Make nagios check_ssl_cert checks happen from one server instead of from all proxies #8209

New issue

Closed

opened 2019-09-14 15:19:46 +00:00 by codeblock · 7 comments

codeblock commented

2019-09-14 15:19:46 +00:00

In some cases, when a cert is about to expire, we get an alert about it from every single proxy, which is excessive. This notably happens with the checks in nagios_server/files/nagios/services/ssl.cfg which have hostgroup_name proxies.

These should all just be listed under one server, not all the proxies for our sanities.

This is a relatively simple change and would make a good easyfix for someone looking to make one of their first patches.

In some cases, when a cert is about to expire, we get an alert about it from every single proxy, which is excessive. This notably happens with the checks in `nagios_server/files/nagios/services/ssl.cfg` which have `hostgroup_name proxies`. These should all just be listed under one server, not all the proxies for our sanities. This is a relatively simple change and would make a good easyfix for someone looking to make one of their first patches.

kevin commented

2019-09-16 00:33:37 +00:00

Metadata Update from @kevin:

Issue priority set to: Waiting on Assignee (was: Needs Review)

**Metadata Update from @kevin**: - Issue priority set to: Waiting on Assignee (was: Needs Review)

kevin commented

2019-09-23 18:12:53 +00:00

This is easy enough to change, but... one reason we wanted all the proxies checked was in case we rolled out a new cert and somehow some subset of them didn't update and still had the old cert.

ie, we could just check one, but then we might miss if there's others that are effected.

Not sure whats best here...

This is easy enough to change, but... one reason we wanted all the proxies checked was in case we rolled out a new cert and somehow some subset of them didn't update and still had the old cert. ie, we could just check one, but then we might miss if there's others that are effected. Not sure whats best here...

kevin commented

2019-10-30 21:36:25 +00:00

@codeblock you still want to try and do this some different way?

I wonder if we could make them all depend on each other as far as nagios knows and the other ones would all be deps of the first one in the case where they are all bad, but only one would alert in the case of one bad?

@codeblock you still want to try and do this some different way? I wonder if we could make them all depend on each other as far as nagios knows and the other ones would all be deps of the first one in the case where they are all bad, but only one would alert in the case of one bad?

karsten commented

2019-11-14 15:52:19 +00:00

I'll take a look, lets see what I can come up with
@lgriffin

I'll take a look, lets see what I can come up with @lgriffin

karsten commented

2019-11-15 12:29:14 +00:00

diff --git a/roles/nagios_server/files/nagios/services/ssl.cfg b/roles/nagios_server/files/nagios/services/ssl.cfg
index 275571cc9..b857c8fd2 100644
--- a/roles/nagios_server/files/nagios/services/ssl.cfg
+++ b/roles/nagios_server/files/nagios/services/ssl.cfg
@@ -39,3 +39,19 @@ define service {
check_command check_ssl_cert!pagure.io!60
use defaulttemplate
}
+
+define servicedependency {

host_name koji.fedoraproject.org
dependent_hostgroup_name koji
dependent_service_description koji hosts running SSL checks
notification_failure_criteria w,c
execution_failure_criteria w,c
+}

+define servicedependency {

host_name proxy03.fedoraproject.org
dependent_hostgroup_name proxies
dependent_service_description Proxies running SSL checks
notification_failure_criteria w,c
execution_failure_criteria w,c
+}

I wonder if this will work. It checks the certs on koji.fp.o and proxy03.fp.o and suppresses warnings of the koji and proxies hostgroups when this check already returns WARN or CRIT. pagure as a single host can send out notifications whenever necessary. caveat: we won't get notifications about ssl certs from koji and proxies hostgroups until the certs on koji.fp.o and proxie03.fp.o get fixed. diff --git a/roles/nagios_server/files/nagios/services/ssl.cfg b/roles/nagios_server/files/nagios/services/ssl.cfg index 275571cc9..b857c8fd2 100644 --- a/roles/nagios_server/files/nagios/services/ssl.cfg +++ b/roles/nagios_server/files/nagios/services/ssl.cfg @@ -39,3 +39,19 @@ define service { check_command check_ssl_cert!pagure.io!60 use defaulttemplate } + +define servicedependency { + host_name koji.fedoraproject.org + dependent_hostgroup_name koji + dependent_service_description koji hosts running SSL checks + notification_failure_criteria w,c + execution_failure_criteria w,c +} + +define servicedependency { + host_name proxy03.fedoraproject.org + dependent_hostgroup_name proxies + dependent_service_description Proxies running SSL checks + notification_failure_criteria w,c + execution_failure_criteria w,c +}