Commit graph

16 commits

Author SHA1 Message Date
Pavel Raiskup
f4538051df s3-mirror: sync s3-sync-path script with ideas from s3.sh
1. sync everything except for repomd.xml
2. then sync repomd.xml files only, and invalidate caches
3. gently wait a bit to give current downloads a chance
4. delete outdated RPMs and metadata, shouldn't be needed

Also make the sleep/cache configurable.
2020-04-24 21:34:28 +02:00
Pavel Raiskup
506b6a3e22 s3-mirror: try to set 'max-age' cache opt for repomd.xml files 2020-04-24 21:34:27 +02:00
Pavel Raiskup
33f2be7c65 s3-mirror: delay the sync --delete a bit
So the running DNF processes can finish transaction.
2020-04-24 21:34:27 +02:00
Pavel Raiskup
164bd1e7a9 s3-mirror: only skip repomd.xml files in first sync run
.. and include _only_ those in second sync run.
2020-04-24 21:34:27 +02:00
Pavel Raiskup
19f148c842 s3-mirror: avoid potential unexpected shell-expansion, dedup
Doing this has risk that '*' expands to directories:

    $> x='echo *'
    $> $x
    bin boot dev etc home

So it is better to use bash array (we have bash shebang anyways):
    $> x=( echo '*' )
    $> "${x[@]}"
    *

Also de-duplicate few things so it is easier to concentrate on the code.
2020-04-24 21:34:27 +02:00
Kevin Fenzi
96190cec94 Revert "Revert "s3-mirror: Split things into 2 sync runs, one without repodata and delete, the other with both.""
This reverts commit 0798930ebea639871194525789061161b2c19bf9.

Got some +1s, so pushing this now.
2020-04-24 21:34:26 +02:00
Kevin Fenzi
76fb2e0871 Revert "s3-mirror: Split things into 2 sync runs, one without repodata and delete, the other with both."
This reverts commit 626c976f8ce616750a56cd0de7847beeaf3924f7.
2020-04-24 21:34:26 +02:00
Kevin Fenzi
899d1767eb s3-mirror: Split things into 2 sync runs, one without repodata and delete, the other with both.
In order to make sure the s3 mirror always is consistent, split out the commands
to make it sync without repodata and delete, then do another run with those, then finally
invalidate all the repodata/* files.

Some of the cron jobs are adjusted to allow the repodata invalidation.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
2020-04-24 21:34:26 +02:00
Rick Elrod
50b7f1d22c s3-mirror: Stop echoing the command we are running
Signed-off-by: Rick Elrod <relrod@redhat.com>
2020-04-24 21:34:11 +02:00
Rick Elrod
139af32030 s3-mirror: No reason not to sync releases/test
Signed-off-by: Rick Elrod <relrod@redhat.com>
2020-04-24 21:34:10 +02:00
Rick Elrod
97adecb9b4 s3-mirror: stop excluding development/*, just exclude development/rawhide
Signed-off-by: Rick Elrod <relrod@redhat.com>
2020-04-24 21:34:08 +02:00
Rick Elrod
59a4d25a1e s3-mirror: remove a few excludes on the "normal" sync so that we sync images too
Signed-off-by: Rick Elrod <relrod@redhat.com>
2020-04-24 21:34:07 +02:00
Rick Elrod
52535a0fca log start/stop times so we can get an idea of how long each sync is taking
Signed-off-by: Rick Elrod <relrod@redhat.com>
2018-11-29 18:14:52 +00:00
Rick Elrod
5ead8fbd49 invalidate repomd.xml files after every sync
Signed-off-by: Rick Elrod <relrod@redhat.com>
2018-11-29 18:03:14 +00:00
Rick Elrod
759e01310d This is under /srv/ on this box
Signed-off-by: Rick Elrod <relrod@redhat.com>
2018-10-09 17:26:22 +00:00
Rick Elrod
7984e52034 attempt at cron-ify-ing new s3 sync
Signed-off-by: Rick Elrod <relrod@redhat.com>
2018-10-09 17:17:53 +00:00