Add and use optimized http log syncing script

The previous one synced all hosts serially and ran rsync for each log
file. This reimplements the shell script in Python, with these changes:

- Run rsync on whole directories of log files, with much reduced
  overhead.
- Use a pool of five workers which process hosts in parallel.

Additionally, remove download-rdu01.vpn.fedoraproject.org from the list
of synced hosts.

Signed-off-by: Nils Philippsen <nils@redhat.com>
This commit is contained in:
Nils Philippsen 2021-07-29 14:51:19 +02:00 committed by nphilipp
parent ea1c19d522
commit f703e7a771
4 changed files with 282 additions and 140 deletions

View file

@ -0,0 +1,56 @@
---
synced_hosts:
- proxy01.iad2.fedoraproject.org
- proxy02.vpn.fedoraproject.org
- proxy03.vpn.fedoraproject.org
- proxy04.vpn.fedoraproject.org
- proxy05.vpn.fedoraproject.org
- proxy06.vpn.fedoraproject.org
# - proxy08.vpn.fedoraproject.org
# proxy09 is acting up
- proxy09.vpn.fedoraproject.org
- proxy10.iad2.fedoraproject.org
- proxy11.vpn.fedoraproject.org
- proxy12.vpn.fedoraproject.org
- proxy13.vpn.fedoraproject.org
- proxy14.vpn.fedoraproject.org
- proxy30.vpn.fedoraproject.org
- proxy31.vpn.fedoraproject.org
- proxy32.vpn.fedoraproject.org
- proxy33.vpn.fedoraproject.org
- proxy34.vpn.fedoraproject.org
- proxy35.vpn.fedoraproject.org
- proxy36.vpn.fedoraproject.org
- proxy37.vpn.fedoraproject.org
- proxy38.vpn.fedoraproject.org
- proxy39.vpn.fedoraproject.org
- proxy40.vpn.fedoraproject.org
- proxy101.iad2.fedoraproject.org
- proxy110.iad2.fedoraproject.org
# - proxy01.stg.iad2.fedoraproject.org
- datagrepper01.iad2.fedoraproject.org
# - datagrepper02.iad2.fedoraproject.org
# - datagrepper01.stg.iad2.fedoraproject.org
# - badges-web01.iad2.fedoraproject.org
# - badges-web02.iad2.fedoraproject.org
# - badges-web01.stg.iad2.fedoraproject.org
# - packages03.iad2.fedoraproject.org
# - packages04.iad2.fedoraproject.org
# - packages03.stg.iad2.fedoraproject.org
- blockerbugs01.iad2.fedoraproject.org
# - blockerbugs02.iad2.fedoraproject.org
# - blockerbugs01.stg.iad2.fedoraproject.org
- value01.iad2.fedoraproject.org
- people02.vpn.fedoraproject.org
- noc01.iad2.fedoraproject.org
- dl01.iad2.fedoraproject.org
- dl02.iad2.fedoraproject.org
- dl03.iad2.fedoraproject.org
- dl04.iad2.fedoraproject.org
- dl05.iad2.fedoraproject.org
# - download-rdu01.vpn.fedoraproject.org
- download-ib01.vpn.fedoraproject.org
- download-cc-rdu01.vpn.fedoraproject.org
- sundries01.iad2.fedoraproject.org
# - sundries02.iad2.fedoraproject.org
# - sundries01.stg.iad2.fedoraproject.org