The two crawlers used to start 25 threads (each) every 12 hours to crawl
the mirrors. The second crawler was already started two hours later to
reduce the database load. This commit increases the delay of the second
crawler to 6 hours, so that every crawler has the MM database for 6
hours on its own. At the same time the number of parallel crawlers is
reduced from 25 to 20 to also reduce the load on the database.
In addition the crawl timeout has been increased from 3 to 4 hours. This
is related to the fact the especially pub/archive has grown and
pub/fedora with the addition of the modular tree. Crawl timeouts can now
be seen more often, which can lead to mirrors being auto-disabled.
The main reason for these changes is that it can be seen in the logs that
the actual crawling of the mirrors does not always require most of time
but updating the state of all directories of each mirror in the database
can take a very long time. By reducing the number of parallel accesses to
the database, in the best case from 50 to 20, the crawling should get
faster (hopefully).
Signed-off-by: Adrian Reber <adrian@lisas.de>
The propagation scripts were still using pkgdb to get a list of active
releases. This switches the scripts to use pdc instead of pkgdb. One
difference between pdc and pkgdb is that pkgdb did not mark a release
as active until the actual release day. In pdc the release (in this case
f27) is already marked as active and it is not clear how to filter it
out until the official release.
As the propagation scripts are looking at the updates-released repository
it will not provide useful data for f27 until content is pushed to the
updates-released repository.
Signed-off-by: Adrian Reber <adrian@lisas.de>
With the stat()-less umdl and adapted wrapper script the umdl check can
now run more often.
Every 30 minutes:
* fedora
* epel
* fedora-secondary
Every 2 hours:
* alt
* archive
Signed-off-by: Adrian Reber <adrian@lisas.de>
Now that umdl can run much faster and often the message bus is no
longer queried. Only the ctime of the fullfiletimelist-* is checked.
Signed-off-by: Adrian Reber <adrian@lisas.de>
The umdl-required script used to query the message bus if there have
been changes on the master mirror. With the new stat()-less umdl
implementation the message bus is no longer queried but only the ctime
of the fullfiletimelist-* is checked to see if it has changed since the
last run.
Signed-off-by: Adrian Reber <adrian@lisas.de>
Disable master rsync server ACL
Fedora does not use it and therefore it is set to False
MASTER_RSYNC_ACL = False
The netblock size parameters define which netblock sizes can be
added by a site administrator. Larger networks can only be added by
mirrormanager admins.
MM_IPV4_NETBLOCK_SIZE = '/16'
MM_IPV6_NETBLOCK_SIZE = '/32'
Signed-off-by: Adrian Reber <adrian@lisas.de>
