Commit graph

4 commits

Author SHA1 Message Date
Seddik Alaoui Ismaili
246cc76260 keeps only errors on logging 2023-11-16 14:08:35 +00:00
Nils Philippsen
6ecdab22c0 Clean up messages issued from sync-http-logs.py
Previously, the script was very talkative by default. Make the default
to be silent for log levels < WARNING and allow logging (at different
level) to syslog. Additionally, configure the cronjob to log everything
of levels >= INFO to syslog.

Signed-off-by: Nils Philippsen <nils@redhat.com>
2021-09-22 10:49:30 +02:00
Nils Philippsen
6e62fcbe69 Don't drop temporary files all over the place
When renaming a file over another which is the same hard link, the
rename is a no-op. This left many temporary files in /var/log/hosts
because a file is attempted to be synced (and thus hard-linked between
dated and undated file names) over a couple of days. The solution to
this is how the `ln` command does it: rename, then unlink the temporary
file.

Signed-off-by: Nils Philippsen <nils@redhat.com>
2021-08-12 09:45:49 +00:00
Nils Philippsen
f703e7a771 Add and use optimized http log syncing script
The previous one synced all hosts serially and ran rsync for each log
file. This reimplements the shell script in Python, with these changes:

- Run rsync on whole directories of log files, with much reduced
  overhead.
- Use a pool of five workers which process hosts in parallel.

Additionally, remove download-rdu01.vpn.fedoraproject.org from the list
of synced hosts.

Signed-off-by: Nils Philippsen <nils@redhat.com>
2021-08-05 16:44:47 +00:00