Infrastructure/ansible
1
0
Fork 0
Code Issues Pull requests 6 Projects Releases Packages Wiki Activity Actions

openqa/dispatcher: restart all fm-consumer@ services daily

Browse source 
fedora-messaging has an annoying bug where consumers get stuck
allegedly working fine but actually not handling any messages:

https://github.com/fedora-infra/fedora-messaging/issues/208

to deal with this on our openQA servers, just have the dispatcher
role install a cron job which restarts all the services nightly.
Strictly speaking this shouldn't really be part of a specific
role like this, but this is the easiest way to do it and it's
not really that "wrong".

Signed-off-by: Adam Williamson <awilliam@redhat.com>
This commit is contained in:
Adam Williamson 2019-11-22 14:21:36 -08:00 • committed by Pierre-Yves Chibon
parent 33b5278939
commit be356508ae
2 changed files with 17 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

6
roles/openqa/dispatcher/files/restart-consumers.cron Normal file
View file

@ -0,0 +1,6 @@
#!/bin/sh
# restart all fedora messaging consumer services in case they've got stuck:
# https://github.com/fedora-infra/fedora-messaging/issues/208
/usr/bin/systemctl try-restart fm-consumer@*

11
roles/openqa/dispatcher/tasks/main.yml
View file

@ -362,3 +362,14 @@
copy: src=schedule-live-respins.cron dest=/etc/cron.hourly/schedule-live-respins owner=root group=root mode=0755
tags:
- config
# This is to deal with an annoying bug in fedora-messaging: sometimes
# consumers seem to get stuck allegedly running fine but not parsing
# any messages. So we'll restart all running consumers every day just
# to kick any that are in this state. Strictly speaking this shouldn't
# really be tied to the dispatcher role, but separating it out would
# be a pain, it's easier just to stuff it here
- name: Set up cron job to restart all running fm-consumer services
copy: src=restart-consumers.cron dest=/etc/cron.daily/restart-consumers owner=root group=root mode=0755
tags:
- config