This adds in for the staging environment a minimal system for centos
admins to test how items work in staging. Because this server will be
administered by CentOS, we only set up a minimal environment.
ToDo: Put in lines to get and copy the correct root ssh key into
/root/.ssh/authorized_keys for CentOS admins.
Signed-off-by: Stephen Smoogen <ssmoogen@redhat.com>
This gives the web-data-analysis `countme` user a .gitconfig file so the
commits it makes in its local git repo have a proper user name and
email address. (Also it makes git stop complaining..)
The email address might not actually be valid, but this repo doesn't
currently go anywhere public so it shouldn't really matter.
This should automate running the "countme" scripts every day to parse
new log data and publish updated totals.
Here's what I've added to the ansible role:
* install package deps for `mirrors-countme`
* make "countme" user with home /srv/countme
* clone 'prod' branch of https://pagure.io/mirrors-countme to /srv/countme
* if changed: pip install /srv/countme/mirrors-countme
* make web subdir /var/www/html/csv-reports/countme
* make local data dir /var/lib/countme
* install `countme-update.sh` to /usr/local/bin
* install `countme-update.cron` to /etc/cron.d
* runs /usr/local/bin/countme-update.sh daily, as user `countme`
That should make sure `countme-update.sh` runs every day.
That script works like this:
1. Run `countme-update-rawdb.sh`
* parse new mirrors.fp.o logs in /var/log/hosts/proxy*
* write data to /var/lib/countme/raw.db
2. Run `countme-update-totals.sh`
* parse raw data from /var/lib/countme/raw.db
* write updated totals to /var/lib/countme/totals.{db,csv}
3. Track changes in updated totals
* set up /var/lib/countme as git repo (if needed)
* commit new `totals.csv` (if changed)
4. Make updated totals public
* Copy totals.{db,csv} to /var/www/html/csv-reports/countme
For safety's sake, I've tried to set up everything so it runs as the
`countme` user rather than running everything as `root`. This might be
an unnecessary complication but it seemed like the right thing to do.
Similarly, keeping totals.csv in a git repo isn't _required_, but it
seemed like a good idea to keep historical records in case we want/need
to change the counting algorithm or something.
I checked the YAML with ansible-lint and tested that all the scripts
work as expected when run as `wwoods`, so unless I've missed something
this should do the trick.
The download-sync for rdu-cc needs to use the /root versus default
quick-fedora-mirror git repository. This allows for it to only copy
the modules needed.
Signed-off-by: Stephen Smoogen <ssmoogen@redhat.com>
Otherwise there are some mapped files on / volume and keep consuming the storage.
I do not know what is actually consuming the storage, but the restart instead of reload helps.
It looks like the data it retrieves is in bytes and thus needs to be
decoded into a unicode string so we can use it as a regular string
in our code later.
Fixes https://pagure.io/fedora-infrastructure/issue/9372
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
We need an x86_64 worker for lab, apparently there are no more
boxes spare right now, so we have to steal one from prod.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
