Infrastructure/fedora-infrastructure

Fork 0

Please issue new fedora-messaging/rabbitmq TLS certs for CentOS Stream infra #12532

New issue

Closed

opened 2025-04-29 15:30:54 +00:00 by arrfab · 8 comments

arrfab commented

2025-04-29 15:30:54 +00:00

Expiring soon (we have Zabbix monitoring at our side for this), and probably to be assigned to @abompard :

        Issuer: CN = RabbitMQ PRODUCTION CA
        Validity
            Not Before: Feb 22 11:11:22 2023 GMT
            Not After : May 27 11:11:22 2025 GMT
        Subject: CN = centos-odcs-private-queue

and

     Issuer: CN = RabbitMQ PRODUCTION CA
        Validity
            Not Before: Feb 21 15:47:34 2023 GMT
            Not After : May 26 15:47:34 2025 GMT
        Subject: CN = centos-odcs

These are both used for ODCS service for CentOS Stream infra

Expiring soon (we have Zabbix monitoring at our side for this), and probably to be assigned to @abompard : ``` Issuer: CN = RabbitMQ PRODUCTION CA Validity Not Before: Feb 22 11:11:22 2023 GMT Not After : May 27 11:11:22 2025 GMT Subject: CN = centos-odcs-private-queue ``` and ``` Issuer: CN = RabbitMQ PRODUCTION CA Validity Not Before: Feb 21 15:47:34 2023 GMT Not After : May 26 15:47:34 2025 GMT Subject: CN = centos-odcs ``` These are both used for ODCS service for CentOS Stream infra

phsmoura commented

2025-04-29 19:04:27 +00:00

Metadata Update from @phsmoura:

Issue priority set to: Waiting on Assignee (was: Needs Review)
Issue tagged with: low-gain, low-trouble, ops

**Metadata Update from @phsmoura**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, low-trouble, ops

t0xic0der commented

2025-04-30 02:56:43 +00:00

This got to desk this morning from previous (Indian) night's Infra & Releng meeting. All I can ask is for the certificate renewal to be held back by a day or two as I expect a Firmitas notification tomorrow morning. The service downloads and validates these certificates on 1st of every month, see this.

Once we verify the functioning, we can actually go ahead with the renewal and discuss whether the current cron schedule makes sense or if it should be lowered down to fortnightly checks (or perhaps, weekly checks)? Should avoid hairy situations like when the certs are issued on 1st of a month itself.

This got to desk this morning from previous (Indian) night's [Infra & Releng meeting](https://meetbot.fedoraproject.org/meeting-3_matrix_fedoraproject-org/2025-04-29/fedora-infrastructure-ops-daily-standup-meeting.2025-04-29-19.01.log.html). All I can ask is for the certificate renewal to be held back by a day or two as I expect a Firmitas notification tomorrow morning. The service downloads and validates these certificates on 1st of every month, see [this](https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/firmitas/templates/cronjob.yml.j2#_8). Once we verify the functioning, we can actually go ahead with the renewal and discuss whether the current cron schedule makes sense or if it should be lowered down to fortnightly checks (or perhaps, weekly checks)? Should avoid hairy situations like when the certs are issued on 1st of a month itself.

t0xic0der commented

2025-04-30 03:02:21 +00:00

Also @arrfab, could you please check if the RabbitMQ certs from CentOS Infra are here in this production services list and this staging services list?

As this list is automatically updated with the public certificates and Firmitas reviews them for notification purposes - this could help you.

@kevin spotted centos-odcs-private-queue.crt and centos-odcs.crt in the directory so we know that at least those two RabbitMQ certs are monitored.

Also @arrfab, could you please check if the RabbitMQ certs from CentOS Infra are here in this [production services list](https://infrastructure.fedoraproject.org/infra/rabbitmq-certs/production/) and this [staging services list](https://infrastructure.fedoraproject.org/infra/rabbitmq-certs/staging/)? As this list is automatically updated with the public certificates and Firmitas reviews them for notification purposes - this could help you. @kevin spotted [centos-odcs-private-queue.crt](https://infrastructure.fedoraproject.org/infra/rabbitmq-certs/production/centos-odcs-private-queue.crt) and [centos-odcs.crt](https://infrastructure.fedoraproject.org/infra/rabbitmq-certs/production/centos-odcs.crt) in the directory so we know that at least those two RabbitMQ certs are monitored.

zlopez commented

2025-04-30 07:28:45 +00:00

Contributor

@t0xic0der I'm for changing the cronjob to run each week. I assume that this check is not taking up much resources.

t0xic0der commented

2025-05-01 02:47:38 +00:00

[t0xic0der@os-control01 ~][PROD-IAD2]$ oc get cronjobs -n firmitas
NAME       SCHEDULE      TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
firmitas   */2 * * * *   Etc/UTC    True      0        99d             99d

I have reenabled this and set the schedule to be once per week but I did that from the production node, so my changes are most likely ephemeral. @zlopez could you make the changes on the ansible repo so that they persist?

[t0xic0der@os-control01 ~][PROD-IAD2]$ oc get cronjobs -n firmitas
NAME       SCHEDULE    TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
firmitas   0 0 * * 0   Etc/UTC    False     0        99d             99d

I have manually triggered the job as well for now and that has resulted in four notifications.

monitor-gating #12538
coreos-ostree-importer #12537
centos-odcs-private-queue #12536
centos-odcs-service #12535

Also, prolly the same for the staging deployment too, thanks!

Get a load of this. The cronjob has been suspended since over the last three months on production. Not just that, it had a weird schedule of running every after a couple of minutes (which is excessive, in my honest opinion). ``` [t0xic0der@os-control01 ~][PROD-IAD2]$ oc get cronjobs -n firmitas NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE firmitas */2 * * * * Etc/UTC True 0 99d 99d ``` I have reenabled this and set the schedule to be once per week but I did that from the production node, so my changes are most likely ephemeral. @zlopez could you make the changes on the ansible repo so that they persist? ``` [t0xic0der@os-control01 ~][PROD-IAD2]$ oc get cronjobs -n firmitas NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE firmitas 0 0 * * 0 Etc/UTC False 0 99d 99d ``` I have manually triggered the job as well for now and that has resulted in four notifications. - monitor-gating #12538 - coreos-ostree-importer #12537 - centos-odcs-private-queue #12536 - centos-odcs-service #12535 Also, prolly the same for the staging deployment too, thanks!

abompard commented

2025-05-02 11:12:38 +00:00

Duplicate of #12535 and #12536

abompard commented

2025-05-02 11:12:39 +00:00

Metadata Update from @abompard:

Issue untagged with: low-gain, low-trouble, ops
Issue close_status updated to: Duplicate
Issue priority set to: Needs Review (was: Waiting on Assignee)
Issue status updated to: Closed (was: Open)

**Metadata Update from @abompard**: - Issue **un**tagged with: low-gain, low-trouble, ops - Issue close_status updated to: Duplicate - Issue priority set to: Needs Review (was: Waiting on Assignee) - Issue status updated to: Closed (was: Open)

zlopez commented

2025-05-05 11:31:53 +00:00

Contributor

@t0xic0der The PR with the change is here https://pagure.io/fedora-infra/ansible/pull-request/2607