add 'countme' stuff to web-data-analysis role

This should automate running the "countme" scripts every day to parse
new log data and publish updated totals.

Here's what I've added to the ansible role:

* install package deps for `mirrors-countme`
* make "countme" user with home /srv/countme
* clone 'prod' branch of https://pagure.io/mirrors-countme to /srv/countme
  * if changed: pip install /srv/countme/mirrors-countme
* make web subdir /var/www/html/csv-reports/countme
* make local data dir /var/lib/countme
* install `countme-update.sh` to /usr/local/bin
* install `countme-update.cron` to /etc/cron.d
  * runs /usr/local/bin/countme-update.sh daily, as user `countme`

That should make sure `countme-update.sh` runs every day.
That script works like this:

1. Run `countme-update-rawdb.sh`
  * parse new mirrors.fp.o logs in /var/log/hosts/proxy*
  * write data to /var/lib/countme/raw.db
2. Run `countme-update-totals.sh`
  * parse raw data from /var/lib/countme/raw.db
  * write updated totals to /var/lib/countme/totals.{db,csv}
3. Track changes in updated totals
  * set up /var/lib/countme as git repo (if needed)
  * commit new `totals.csv` (if changed)
4. Make updated totals public
  * Copy totals.{db,csv} to /var/www/html/csv-reports/countme

For safety's sake, I've tried to set up everything so it runs as the
`countme` user rather than running everything as `root`. This might be
an unnecessary complication but it seemed like the right thing to do.

Similarly, keeping totals.csv in a git repo isn't _required_, but it
seemed like a good idea to keep historical records in case we want/need
to change the counting algorithm or something.

I checked the YAML with ansible-lint and tested that all the scripts
work as expected when run as `wwoods`, so unless I've missed something
this should do the trick.
This commit is contained in:
Will Woods 2020-08-10 17:53:26 -04:00 committed by smooge
parent 5868f77c53
commit f8a5720535
3 changed files with 156 additions and 0 deletions

View file

@ -85,3 +85,57 @@
- web-data
- cron
- name: install package deps for mirrors-countme
package:
# tqdm is optional but it gives nice progress meters for interactive use
name: ['python3-pip', 'python3-setuptools', 'python3-tqdm']
state: present
tags:
- packages
- web-data
- name: make countme user
user:
name: countme
group: countme
shell: /sbin/nologin
home: /srv/countme
comment: "DNF countme counter"
tags:
- web-data
- name: checkout mirrors-countme from git
git:
repo: https://pagure.io/mirrors-countme
dest: /srv/countme/mirrors-countme
version: prod
register: gitcountme
tags:
- web-data
- name: install mirrors-countme from git checkout
command: "pip install --no-index --no-deps /srv/countme/mirrors-countme"
when: "gitcountme is changed"
tags:
- web-data
- name: make countme web subdir
file: path=/var/www/html/csv-reports/countme state=directory mode=0775 owner=countme group=countme
tags:
- web-data
- name: make countme local data dir
file: path=/var/lib/countme state=directory mode=0775 owner=countme group=countme
tags:
- web-data
- name: install countme script to parse new logs & update totals
copy: src=countme-update.sh dest=/usr/local/bin/ mode=0755
tags:
- web-data
- name: install cron file to run countme-update.sh daily
copy: src=countme-update.cron dest=/etc/cron.d/ mode=0644
tags:
- web-data
- cron