Infrastructure/fedora-infrastructure

Fork 0

Improve monitoring for container registry #9231

New issue

Closed

opened 2020-08-13 10:31:08 +00:00 by mizdebsk · 12 comments

mizdebsk commented

2020-08-13 10:31:08 +00:00

Container registry is listed as having Important SLE, yet one of our registries was down for about 11 hours (see #9230 for details) and we didn't get any Nagios notification about the issue.
Monitoring should be improved so that we are notified about this kind of issues sooner.

[Container registry](https://registry.fedoraproject.org/) is listed as having [Important SLE](https://docs.fedoraproject.org/en-US/cpe/sle_services/), yet one of our registries was down for about 11 hours (see #9230 for details) and we didn't get any Nagios notification about the issue. Monitoring should be improved so that we are notified about this kind of issues sooner.

nasirhm commented

2020-08-13 15:12:06 +00:00

I would like to work on it.

smooge commented

2020-08-13 18:08:56 +00:00

Metadata Update from @smooge:

Issue assigned to nasirhm
Issue priority set to: Waiting on Assignee (was: Needs Review)
Issue tagged with: medium-gain, medium-trouble, ops

**Metadata Update from @smooge**: - Issue assigned to nasirhm - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops

seddik commented

2020-08-22 15:17:00 +00:00

@mizdebsk i think we can get notification when systemd-monitored service enters failed state if we do OnFailure to unit !
for more details about option => https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Specifiers

@mizdebsk i think we can get notification when systemd-monitored service enters failed state if we do ```OnFailure ``` to unit ! for more details about option => https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Specifiers

mizdebsk commented

2020-08-26 07:49:46 +00:00

Author

We have existing Nagios setup that would be trivial to configure to cover OCI registry -- for example adding checks for:

presence of word "fedora" under https://registry.fedoraproject.org/v2/_catalog (that covers v2 API)
presence of word "rawhide" under https://registry.fedoraproject.org/repo/fedora/tags/ (covers web interface)

Hint: the relevant file in ansible.git is roles/nagios_server/templates/nagios/services/websites.cfg.j2

We have existing Nagios setup that would be trivial to configure to cover OCI registry -- for example adding checks for: - presence of word "fedora" under https://registry.fedoraproject.org/v2/_catalog (that covers v2 API) - presence of word "rawhide" under https://registry.fedoraproject.org/repo/fedora/tags/ (covers web interface) Hint: the relevant file in ansible.git is `roles/nagios_server/templates/nagios/services/websites.cfg.j2`

nasirhm commented

2020-08-26 08:13:25 +00:00

Thank You very much @mizdebsk and @seddik for the pointers, I will work on it after work today.

mizdebsk commented

2020-08-26 08:24:00 +00:00

Author

Take your time. We are in beta freeze, so changes to monitoring will need to wait until the freeze ends, or follow FBR SOP

Take your time. We are in [beta freeze](https://fedorapeople.org/groups/schedule/f-33/f-33-infrastructure-tasks.html), so changes to monitoring will need to wait until the freeze ends, or follow [FBR SOP](https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/fedora-releases.html#change-freeze)

seddik commented

2020-10-30 22:46:44 +00:00

Hi,

Could you give any update ?

Hi, Could you give any update ?

darknao commented

2020-11-16 19:42:26 +00:00

I can work on that if needed

mizdebsk commented

2020-12-02 09:16:35 +00:00

Author

Monitoring for container registry is still needed, patches are welcome. Please let me know if you need any help with implementing this.

darknao commented

2020-12-03 00:52:04 +00:00

Related PR fedora-infra/ansible#321 has been merged

mizdebsk commented

2020-12-03 19:45:57 +00:00

Author

The change has been deployed and can be seen eg. here and here.
Thank you for your contribution. This issue is resolved.

The change has been deployed and can be seen eg. [here](https://nagios.fedoraproject.org/nagios/cgi-bin//extinfo.cgi?type=2&host=proxy01.fedoraproject.org&service=http-oci-registry-api) and [here](https://nagios-external.fedoraproject.org/nagios/cgi-bin//extinfo.cgi?type=2&host=proxy01.fedoraproject.org&service=http-oci-registry-api). Thank you for your contribution. This issue is resolved.

mizdebsk commented

2020-12-03 19:45:58 +00:00

Author

Metadata Update from @mizdebsk: