add 'countme' stuff to web-data-analysis role
This should automate running the "countme" scripts every day to parse new log data and publish updated totals. Here's what I've added to the ansible role: * install package deps for `mirrors-countme` * make "countme" user with home /srv/countme * clone 'prod' branch of https://pagure.io/mirrors-countme to /srv/countme * if changed: pip install /srv/countme/mirrors-countme * make web subdir /var/www/html/csv-reports/countme * make local data dir /var/lib/countme * install `countme-update.sh` to /usr/local/bin * install `countme-update.cron` to /etc/cron.d * runs /usr/local/bin/countme-update.sh daily, as user `countme` That should make sure `countme-update.sh` runs every day. That script works like this: 1. Run `countme-update-rawdb.sh` * parse new mirrors.fp.o logs in /var/log/hosts/proxy* * write data to /var/lib/countme/raw.db 2. Run `countme-update-totals.sh` * parse raw data from /var/lib/countme/raw.db * write updated totals to /var/lib/countme/totals.{db,csv} 3. Track changes in updated totals * set up /var/lib/countme as git repo (if needed) * commit new `totals.csv` (if changed) 4. Make updated totals public * Copy totals.{db,csv} to /var/www/html/csv-reports/countme For safety's sake, I've tried to set up everything so it runs as the `countme` user rather than running everything as `root`. This might be an unnecessary complication but it seemed like the right thing to do. Similarly, keeping totals.csv in a git repo isn't _required_, but it seemed like a good idea to keep historical records in case we want/need to change the counting algorithm or something. I checked the YAML with ansible-lint and tested that all the scripts work as expected when run as `wwoods`, so unless I've missed something this should do the trick.
This commit is contained in:
parent
5868f77c53
commit
f8a5720535
3 changed files with 156 additions and 0 deletions
|
@ -85,3 +85,57 @@
|
|||
- web-data
|
||||
- cron
|
||||
|
||||
- name: install package deps for mirrors-countme
|
||||
package:
|
||||
# tqdm is optional but it gives nice progress meters for interactive use
|
||||
name: ['python3-pip', 'python3-setuptools', 'python3-tqdm']
|
||||
state: present
|
||||
tags:
|
||||
- packages
|
||||
- web-data
|
||||
|
||||
- name: make countme user
|
||||
user:
|
||||
name: countme
|
||||
group: countme
|
||||
shell: /sbin/nologin
|
||||
home: /srv/countme
|
||||
comment: "DNF countme counter"
|
||||
tags:
|
||||
- web-data
|
||||
|
||||
- name: checkout mirrors-countme from git
|
||||
git:
|
||||
repo: https://pagure.io/mirrors-countme
|
||||
dest: /srv/countme/mirrors-countme
|
||||
version: prod
|
||||
register: gitcountme
|
||||
tags:
|
||||
- web-data
|
||||
|
||||
- name: install mirrors-countme from git checkout
|
||||
command: "pip install --no-index --no-deps /srv/countme/mirrors-countme"
|
||||
when: "gitcountme is changed"
|
||||
tags:
|
||||
- web-data
|
||||
|
||||
- name: make countme web subdir
|
||||
file: path=/var/www/html/csv-reports/countme state=directory mode=0775 owner=countme group=countme
|
||||
tags:
|
||||
- web-data
|
||||
|
||||
- name: make countme local data dir
|
||||
file: path=/var/lib/countme state=directory mode=0775 owner=countme group=countme
|
||||
tags:
|
||||
- web-data
|
||||
|
||||
- name: install countme script to parse new logs & update totals
|
||||
copy: src=countme-update.sh dest=/usr/local/bin/ mode=0755
|
||||
tags:
|
||||
- web-data
|
||||
|
||||
- name: install cron file to run countme-update.sh daily
|
||||
copy: src=countme-update.cron dest=/etc/cron.d/ mode=0644
|
||||
tags:
|
||||
- web-data
|
||||
- cron
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue