Add and use optimized http log syncing script
The previous one synced all hosts serially and ran rsync for each log file. This reimplements the shell script in Python, with these changes: - Run rsync on whole directories of log files, with much reduced overhead. - Use a pool of five workers which process hosts in parallel. Additionally, remove download-rdu01.vpn.fedoraproject.org from the list of synced hosts. Signed-off-by: Nils Philippsen <nils@redhat.com>
This commit is contained in:
parent
ea1c19d522
commit
f703e7a771
4 changed files with 282 additions and 140 deletions
56
roles/web-data-analysis/templates/sync-http-logs.yaml.j2
Normal file
56
roles/web-data-analysis/templates/sync-http-logs.yaml.j2
Normal file
|
@ -0,0 +1,56 @@
|
|||
---
|
||||
synced_hosts:
|
||||
- proxy01.iad2.fedoraproject.org
|
||||
- proxy02.vpn.fedoraproject.org
|
||||
- proxy03.vpn.fedoraproject.org
|
||||
- proxy04.vpn.fedoraproject.org
|
||||
- proxy05.vpn.fedoraproject.org
|
||||
- proxy06.vpn.fedoraproject.org
|
||||
# - proxy08.vpn.fedoraproject.org
|
||||
# proxy09 is acting up
|
||||
- proxy09.vpn.fedoraproject.org
|
||||
- proxy10.iad2.fedoraproject.org
|
||||
- proxy11.vpn.fedoraproject.org
|
||||
- proxy12.vpn.fedoraproject.org
|
||||
- proxy13.vpn.fedoraproject.org
|
||||
- proxy14.vpn.fedoraproject.org
|
||||
- proxy30.vpn.fedoraproject.org
|
||||
- proxy31.vpn.fedoraproject.org
|
||||
- proxy32.vpn.fedoraproject.org
|
||||
- proxy33.vpn.fedoraproject.org
|
||||
- proxy34.vpn.fedoraproject.org
|
||||
- proxy35.vpn.fedoraproject.org
|
||||
- proxy36.vpn.fedoraproject.org
|
||||
- proxy37.vpn.fedoraproject.org
|
||||
- proxy38.vpn.fedoraproject.org
|
||||
- proxy39.vpn.fedoraproject.org
|
||||
- proxy40.vpn.fedoraproject.org
|
||||
- proxy101.iad2.fedoraproject.org
|
||||
- proxy110.iad2.fedoraproject.org
|
||||
# - proxy01.stg.iad2.fedoraproject.org
|
||||
- datagrepper01.iad2.fedoraproject.org
|
||||
# - datagrepper02.iad2.fedoraproject.org
|
||||
# - datagrepper01.stg.iad2.fedoraproject.org
|
||||
# - badges-web01.iad2.fedoraproject.org
|
||||
# - badges-web02.iad2.fedoraproject.org
|
||||
# - badges-web01.stg.iad2.fedoraproject.org
|
||||
# - packages03.iad2.fedoraproject.org
|
||||
# - packages04.iad2.fedoraproject.org
|
||||
# - packages03.stg.iad2.fedoraproject.org
|
||||
- blockerbugs01.iad2.fedoraproject.org
|
||||
# - blockerbugs02.iad2.fedoraproject.org
|
||||
# - blockerbugs01.stg.iad2.fedoraproject.org
|
||||
- value01.iad2.fedoraproject.org
|
||||
- people02.vpn.fedoraproject.org
|
||||
- noc01.iad2.fedoraproject.org
|
||||
- dl01.iad2.fedoraproject.org
|
||||
- dl02.iad2.fedoraproject.org
|
||||
- dl03.iad2.fedoraproject.org
|
||||
- dl04.iad2.fedoraproject.org
|
||||
- dl05.iad2.fedoraproject.org
|
||||
# - download-rdu01.vpn.fedoraproject.org
|
||||
- download-ib01.vpn.fedoraproject.org
|
||||
- download-cc-rdu01.vpn.fedoraproject.org
|
||||
- sundries01.iad2.fedoraproject.org
|
||||
# - sundries02.iad2.fedoraproject.org
|
||||
# - sundries01.stg.iad2.fedoraproject.org
|
Loading…
Add table
Add a link
Reference in a new issue