Add and use optimized http log syncing script

The previous one synced all hosts serially and ran rsync for each log
file. This reimplements the shell script in Python, with these changes:

- Run rsync on whole directories of log files, with much reduced
  overhead.
- Use a pool of five workers which process hosts in parallel.

Additionally, remove download-rdu01.vpn.fedoraproject.org from the list
of synced hosts.

Signed-off-by: Nils Philippsen <nils@redhat.com>
This commit is contained in:
Nils Philippsen 2021-07-29 14:51:19 +02:00 committed by nphilipp
parent ea1c19d522
commit f703e7a771
4 changed files with 282 additions and 140 deletions

View file

@ -151,9 +151,30 @@
- web-data
- cron
- name: install a sync httpd logs cron script only on log01
copy: src=syncHttpLogs.sh dest=/etc/cron.daily/syncHttpLogs.sh mode=0755
- name: remove old syncHttpLogs.sh cron script only on log01
file:
path: /etc/cron.daily/syncHttpLogs.sh
state: absent
when: inventory_hostname.startswith('log01')
tags:
- web-data
- cron
- name: write configuration file for script to sync httpd logs
template:
src: sync-http-logs.yaml.j2
dest: /etc/sync-http-logs.yaml
when: inventory_hostname.startswith('log01')
tags:
- web-data
- config
- name: install a script to sync httpd logs via cron only on log01
copy:
src: sync-http-logs.py
dest: /etc/cron.daily/sync-http-logs.py
mode: 0755
when: inventory_hostname.startswith('log01')
tags:
- web-data
- cron