Infrastructure/ansible
1
0
Fork 0
Code Issues Pull requests 6 Projects Releases Packages Wiki Activity Actions

Reduce number of parallel crawlers from 38 to 35

Browse source 
To avoid regular nagios memory warnings the number of parallel crawlers
is reduced from 38 to 35 to reduce the required memory.
This commit is contained in:
Adrian Reber 2015-05-31 08:11:04 +00:00
parent 492db6878a
commit af9462caab
1 changed files with 2 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

4
roles/mirrormanager/crawler/files/crawler.cron
View file

@ -1,8 +1,8 @@
# run the crawler twice a day
# logs sent to /var/log/mirrormanager/crawler.log and crawl/* by default
# 32GB of RAM is not enough for 75 threads, 38 seems to work so far
# 32GB of RAM is not enough for 75 threads, 35 seems to work so far
#
# [ "`hostname -s`" == "mm-crawler02" ] && sleep 3h is used to start the crawl
# later on the second crawler to reduce the number of parallel accesses to
# the database
0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 3h; /usr/bin/mm2_crawler --timeout-minutes 180 --threads 38 `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1
0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 3h; /usr/bin/mm2_crawler --timeout-minutes 180 --threads 35 `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1