infrastructure/ansible
1
0
Fork 0
Code Pull requests 7 Activity Actions
ansible/roles/mirrormanager/crawler/files/crawler.cron

15 lines
1.1 KiB
Text
Raw Normal View History

Add a cron job for the crawler of MM2
2014-12-08 16:17:09 +01:00
# run the crawler twice a day
# logs sent to /var/log/mirrormanager/crawler.log and crawl/* by default
Start the crawl later on the second crawler. Even with rsync as crawl method some hosts are taking a very long time to be crawled. The network connection with rsync is only open for a short time, but with both crawlers reading and writing from the database it takes a very long time until the status of all directories is updated. Therefore this patch introduces a 3 hour delay of the crawl on the second crawler. This could also be solved with two different cron.d files; one for each crawler.
2015-05-13 11:23:21 +00:00
#
Reduce number of parallel crawlers from 32 to 31 Only wait 2 hours before starting the crawl on the second crawler.
2015-06-28 10:23:43 +00:00
# [ "`hostname -s`" == "mm-crawler02" ] && sleep 2h is used to start the crawl
Start the crawl later on the second crawler. Even with rsync as crawl method some hosts are taking a very long time to be crawled. The network connection with rsync is only open for a short time, but with both crawlers reading and writing from the database it takes a very long time until the status of all directories is updated. Therefore this patch introduces a 3 hour delay of the crawl on the second crawler. This could also be solved with two different cron.d files; one for each crawler.
2015-05-13 11:23:21 +00:00
# later on the second crawler to reduce the number of parallel accesses to
# the database
mm2_crawler: make sure only one crawler is running To make sure only one cron started crawler is running the previous running (cron) crawlers are being signaled to shut down. The crawler can try to gracefully shutdown if it gets the signal SIGALRM(14). After the signal we wait for 5 minutes to give the crawler a chance to shutdown. After that the crawler is killed. To make sure we only end the cron started crawler we look for the following process "/usr/bin/python /usr/bin/mm2_crawler --threads 25". Signed-off-by: Adrian Reber <adrian@lisas.de>
2016-07-13 10:00:42 +02:00
#
# To make sure only one cron started crawler is running the previous running
# (cron) crawlers are being signaled to shut down. The crawler can try to
# gracefully shutdown if it gets the signal SIGALRM(14). After the signal we
# wait for 5 minutes to give the crawler a chance to shutdown. After that the
# crawler is killed. To make sure we only end the cron started crawler we look
# for the following process "/usr/bin/python /usr/bin/mm2_crawler --threads 25".
mm2_crawler: fix crawler auto kill Before starting the crawler old crawler processes are killed to not have old crawlers running and consuming (lots of) memory. This adapts the pgrep and pkill from ^/usr/bin/python to ^/usr/bin/python2 -s Signed-off-by: Adrian Reber <adrian@lisas.de>
2016-12-31 12:34:18 +01:00
0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 2h; pkill -14 -f "^/usr/bin/python2 -s /usr/bin/mm2_crawler --threads 25"; sleep 5m; pkill -9 -f "^/usr/bin/python2 -s /usr/bin/mm2_crawler --threads 25"; /usr/bin/mm2_crawler --threads 25 --timeout-minutes 180 `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1
Copy permalink