This is causing f31 updates to not be synced. The cron job reports:
Subject: Cron <s3-mirror@mm-backend01> /usr/local/bin/lock-wrapper s3sync-updates-current "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/31/Everything/x86_64/os" 2>&1 |
/usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1
Syntax: /usr/local/bin/s3-sync-path.sh /pub/path/to/sync/
NOTE! Path must end with a trailing /
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
We have been having the cluster fall over for still unknown reasons,
but this patch should at least help prevent them:
first we increase the net_ticktime parameter from it's default of 60 to 120.
rabbitmq sends 4 'ticks' to other cluster members over this time and if 25%
of them are lost it assumes that cluster member is down. All these vm's are
on the same net and in the same datacenter, but perhaps heavy load
from other vm's causes them to sometimes not get a tick in time?
http://www.rabbitmq.com/nettick.html
Also, set our partitioning strategy to autoheal. Currently if some cluster
member gets booted out, it gets paused, and stops processing at all.
With autoheal it will try and figure out a 'winning' partition and restart
all the nodes that are not in that partition.
https://www.rabbitmq.com/partitions.html
Hopefully the first thing will make partitions less likely and the second
will make them repair without causing massive pain to the cluster.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
openqa-aarch64-02.qa is broken in some very mysterious way:
https://pagure.io/fedora-infrastructure/issue/8750
until we can figure that out, this should prevent it picking up
normal jobs, but let us manually target a job at it whenever we
need to for debugging.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Apparently, date +%W gives week number with leading zero and there is no alternative.
Shell math's % cannot deal with that, so we sed the zero out.
Signed-off-by: Miro Hrončok <miro@hroncok.cz>
Consider my patch Public Domain or CC0.
In order to make sure the s3 mirror always is consistent, split out the commands
to make it sync without repodata and delete, then do another run with those, then finally
invalidate all the repodata/* files.
Some of the cron jobs are adjusted to allow the repodata invalidation.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
addressing:
TASK [abrt/faf : run database migrations] *******************************************************************************
Friday 06 March 2020 07:56:21 +0000 (0:00:00.045) 0:04:59.958 **********
fatal: [retrace-stg.aws.fedoraproject.org]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/libexec/platform-python"}, "changed": true, "cmd": ["faf-migrate-db"], "delta": "0:00:01.301859", "end": "2020-03-06 07:56:23.505802", "msg": "non-zero return code", "rc": 1, "start": "2020-03-06 07:56:22.203943", "stderr": "[2020-03-06 07:56:23] ERROR:faf.SaveReports: Required directories can't be created: Required directory '/srv/faf/reports/incoming' does not exist and can't be created: Permission denied\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/pyfaf/common.py\", line 140, in ensure_dirs\n os.makedirs(dirname)\n File \"/usr/lib64/python3.6/os.py\", line 210, in makedirs\n makedirs(head, mode, exist_ok)\n File \"/usr/lib64/python3.6/os.py\", line 210, in makedirs\n makedirs(head, mode, exist_ok)\n File \"/usr/lib64/python3.6/os.py\", line 220, in makedirs\n mkdir(name, mode)\nPermissionError: [Errno 13] Permission denied: '/srv/faf'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/bin/faf-migrate-db\", line 27, in <module>\n from pyfaf.common import get_connect_string\n File \"/usr/lib/python3.6/site-packages/pyfaf/__init__.py\", line 25, in <module>\n from . import cmdline\n File \"/usr/lib/python3.6/site-packages/pyfaf/cmdline.py\", line 22, in <module>\n from pyfaf.actions import actions\n File \"/usr/lib/python3.6/site-packages/pyfaf/actions/__init__.py\", line 106, in <module>\n load_plugins(Action, actions)\n File \"/usr/lib/python3.6/site-packages/pyfaf/common.py\", line 113, in load_plugins\n result[plugin.name] = plugin()\n File \"/usr/lib/python3.6/site-packages/pyfaf/actions/save_reports.py\", line 69, in __init__\n self.dir_attach_saved, self.dir_attach_deferred])\n File \"/usr/lib/python3.6/site-packages/pyfaf/common.py\", line 145, in ensure_dirs\n .format(dirname, ex.strerror))\npyfaf.common.FafError: Required directory '/srv/faf/reports/incoming' does not exist and can't be created: Permission denied", "stderr_lines": ["[2020-03-06 07:56:23] ERROR:faf.SaveReports: Required directories can't be created: Required directory '/srv/faf/reports/incoming' does not exist and can't be created: Permission denied", "Traceback (most recent call last):", " File \"/usr/lib/python3.6/site-packages/pyfaf/common.py\", line 140, in ensure_dirs", " os.makedirs(dirname)", " File \"/usr/lib64/python3.6/os.py\", line 210, in makedirs", " makedirs(head, mode, exist_ok)", " File \"/usr/lib64/python3.6/os.py\", line 210, in makedirs", " makedirs(head, mode, exist_ok)", " File \"/usr/lib64/python3.6/os.py\", line 220, in makedirs", " mkdir(name, mode)", "PermissionError: [Errno 13] Permission denied: '/srv/faf'", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", " File \"/bin/faf-migrate-db\", line 27, in <module>", " from pyfaf.common import get_connect_string", " File \"/usr/lib/python3.6/site-packages/pyfaf/__init__.py\", line 25, in <module>", " from . import cmdline", " File \"/usr/lib/python3.6/site-packages/pyfaf/cmdline.py\", line 22, in <module>", " from pyfaf.actions import actions", " File \"/usr/lib/python3.6/site-packages/pyfaf/actions/__init__.py\", line 106, in <module>", " load_plugins(Action, actions)", " File \"/usr/lib/python3.6/site-packages/pyfaf/common.py\", line 113, in load_plugins", " result[plugin.name] = plugin()", " File \"/usr/lib/python3.6/site-packages/pyfaf/actions/save_reports.py\", line 69, in __init__", " self.dir_attach_saved, self.dir_attach_deferred])", " File \"/usr/lib/python3.6/site-packages/pyfaf/common.py\", line 145, in ensure_dirs", " .format(dirname, ex.strerror))", "pyfaf.common.FafError: Required directory '/srv/faf/reports/incoming' does not exist and can't be created: Permission denied"], "stdout": "", "stdout_lines": []}
