Infrastructure/ansible
1
0
Fork 0
Code Issues Pull requests 6 Projects Releases Packages Wiki Activity Actions

Reduce the number of parallel crawled hosts to 29.

Browse source 
There has been a report that the MM database was not updated correctly
and dmesg on mm-crawler01 shows three OOM killed crawlers.

https://fedorahosted.org/fedora-infrastructure/ticket/4845
This commit is contained in:
Adrian Reber 2015-07-29 19:20:19 +00:00
parent c9eac0bb2f
commit 2c9a493271
1 changed files with 2 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

4
roles/mirrormanager/crawler/files/crawler.cron
View file

@ -1,8 +1,8 @@
# run the crawler twice a day
# logs sent to /var/log/mirrormanager/crawler.log and crawl/* by default
# 30 threads with 32GB of RAM seems to work so far
# 29 threads with 32GB of RAM seems to work so far
#
# [ "`hostname -s`" == "mm-crawler02" ] && sleep 2h is used to start the crawl
# later on the second crawler to reduce the number of parallel accesses to
# the database
0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 2h; /usr/bin/mm2_crawler --timeout-minutes 180 --threads 30 `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1
0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 2h; /usr/bin/mm2_crawler --timeout-minutes 180 --threads 29 `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1