Infrastructure/fedora-infrastructure

Fork 0

please don't remove enrolled centos machines from IPA in staging #12514

New issue

Open

opened 2025-04-24 07:11:56 +00:00 by arrfab · 13 comments

arrfab commented

2025-04-24 07:11:56 +00:00

Someone (who ? or a script ?) removed ipsilon.stg.iad2.centos.org from the ipsilon HBAC rule, denying so all auth requests .

Can you identify the root cause and ensure it wouldn't happen again please ?

Thanks a lot

As CentOS and Fedora are using shared IPA backend for authentication, I'd request that nothing touching enrolled centos machines in IPA would be done (manually or through scripts) I had to just waste my time this morning investigating why ipsilon (https://id.stg.centos.org) wasn't allowing anyone to auth (and so no openidc for services using our ipsilon instance) Someone (who ? or a script ?) removed `ipsilon.stg.iad2.centos.org` from the ipsilon HBAC rule, denying so all auth requests . Can you identify the root cause and ensure it wouldn't happen again please ? Thanks *a lot*

zlopez commented

2025-04-24 07:13:06 +00:00

Contributor

Metadata Update from @zlopez:

Issue priority set to: Waiting on Assignee (was: Needs Review)
Issue tagged with: Needs investigation, high-gain, ops

**Metadata Update from @zlopez**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation, high-gain, ops

arrfab commented

2025-04-24 07:17:39 +00:00

Author

arrfab commented

2025-04-24 07:17:40 +00:00

Author

Metadata Update from @arrfab:

Issue untagged with: Needs investigation, high-gain, ops
Issue priority set to: Needs Review (was: Waiting on Assignee)

**Metadata Update from @arrfab**: - Issue **un**tagged with: Needs investigation, high-gain, ops - Issue priority set to: Needs Review (was: Waiting on Assignee)

zlopez commented

2025-04-24 07:19:08 +00:00

Contributor

Metadata Update from @zlopez:

Issue priority set to: Waiting on Assignee (was: Needs Review)
Issue tagged with: Needs investigation, high-gain, ops

**Metadata Update from @zlopez**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation, high-gain, ops

zlopez commented

2025-04-24 07:23:54 +00:00

Contributor

The change you are referring to happened 4 years ago. So I assume that didn't caused the machine to be removed.

The [change](https://pagure.io/fedora-infra/ansible/c/e92d0dda1a54a8576586450578f8873ce920530d) you are referring to happened 4 years ago. So I assume that didn't caused the machine to be removed.

kevin commented

2025-04-24 17:33:19 +00:00

Metadata Update from @kevin:

Issue assigned to kevin

**Metadata Update from @kevin**: - Issue assigned to kevin

kevin commented

2025-04-24 17:33:22 +00:00

It's actually https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_101

It was using the wrong hostname... but that was set in 2021?
b8e6754f97c (Aurélien Bompard 2021-03-22 17:07:45 +0100 101) host: "{{ (env == 'production')|ternary('ipsilon.iad2.centos.org', 'centos-ipa-client02.stg.iad2.fedoraproject.org') }}"

anyhow, I changed it to ipsilon.stg.iad2.centos.org

If you can confirm it's fixed / working?

It's actually https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_101 It was using the wrong hostname... but that was set in 2021? b8e6754f97c (Aurélien Bompard 2021-03-22 17:07:45 +0100 101) host: "{{ (env == 'production')|ternary('ipsilon.iad2.centos.org', 'centos-ipa-client02.stg.iad2.fedoraproject.org') }}" anyhow, I changed it to ipsilon.stg.iad2.centos.org If you can confirm it's fixed / working?

arrfab commented

2025-04-29 15:37:40 +00:00

Author

forgot to give feedback (PTO) but yes, now working again ...
Is there a way to ensure that ansible would not remove hosts not managed by itself ?
Otherwise, just ensure that it's documented somewhere as that means that on my centos ansible infra side I'll never be autonomous to deploy/enroll a machine as then Fedora ansible would remove it :/

forgot to give feedback (PTO) but yes, now working again ... Is there a way to ensure that ansible would not remove hosts not managed by itself ? Otherwise, just ensure that it's documented somewhere as that means that on my centos ansible infra side I'll never be autonomous to deploy/enroll a machine as then Fedora ansible would remove it :/

kevin commented

2025-04-29 16:50:07 +00:00

Well, I mean we could remove that task from our playbooks and you could depend on yourself to create that ?

Otherwise ansible is going to setup the thing thats defined.

Do you want us to remove that setup from our side?

Well, I mean we could remove that task from our playbooks and you could depend on yourself to create that ? Otherwise ansible is going to setup the thing thats defined. Do you want us to remove that setup from our side?

arrfab commented

2025-04-29 17:31:00 +00:00

Author

@kevin : even if you remove that block, I guess that the other one about Fedora host would then remove the existing centos machines again .. so isn't there a parameter for that ipahbacrule to not purge things not declared ? if not then we'll have to still hard-code centos machines there but with dc move and so multiple machines that will need to be enrolled, I'd like that to be documented at least :)

@kevin : even if you remove that block, I guess that the other one about Fedora host would then remove the existing centos machines again .. so isn't there a parameter for that `ipahbacrule` to not purge things not declared ? if not then we'll have to still hard-code centos machines there but with dc move and so multiple machines that will need to be enrolled, I'd like that to be documented at least :)

kevin commented

2025-04-29 17:53:36 +00:00

well, my reading / understanding of it is that:

https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_85 creates the 'ipsilon' hbac rule and

https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_101
has action: member, so it adds those members/hosts.

So, I guess we could change the first one to just confirm/add the fedora ipsilon servers and then it wouldn't recreate the rule...
but then the rule would have to be... manually created or something?

well, my reading / understanding of it is that: https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_85 creates the 'ipsilon' hbac rule and https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_101 has action: member, so it adds those members/hosts. So, I guess we could change the first one to just confirm/add the fedora ipsilon servers and then it wouldn't recreate the rule... but then the rule would have to be... manually created or something?

smilner commented

2025-05-07 13:54:42 +00:00

Would this be high trouble, medium trouble, low trouble or a timeboxed investigation?

kevin commented

2025-05-08 00:33:34 +00:00

What if we do this:

diff --git a/playbooks/groups/ipsilon.yml b/playbooks/groups/ipsilon.yml
index 8c95c4344a..1f4636ece3 100644
--- a/playbooks/groups/ipsilon.yml
+++ b/playbooks/groups/ipsilon.yml
@@ -89,6 +89,14 @@
       hbacsvc:
       - ipsilon
       usercategory: all
+      ipaadmin_password: "{{ ipa_admin_password }}"
+    tags:
+    - ipsilon
+
+  - name: Allow login through ipsilon for the Fedora instance
+    ipahbacrule:
+      name: ipsilon
+      action: member
       host: "{{ (env == 'production')|ternary(groups['ipsilon'], groups['ipsilon_stg']) }}"
       ipaadmin_password: "{{ ipa_admin_password }}"
     tags:

ie, we remove the host call from the one that makes the hbrbac rule, so it should just create it if it doesn't exist.
Then we populate the fedora hosts in a seperate host level task.

What if we do this: ``` diff --git a/playbooks/groups/ipsilon.yml b/playbooks/groups/ipsilon.yml index 8c95c4344a..1f4636ece3 100644 --- a/playbooks/groups/ipsilon.yml +++ b/playbooks/groups/ipsilon.yml @@ -89,6 +89,14 @@ hbacsvc: - ipsilon usercategory: all + ipaadmin_password: "{{ ipa_admin_password }}" + tags: + - ipsilon + + - name: Allow login through ipsilon for the Fedora instance + ipahbacrule: + name: ipsilon + action: member host: "{{ (env == 'production')|ternary(groups['ipsilon'], groups['ipsilon_stg']) }}" ipaadmin_password: "{{ ipa_admin_password }}" tags: ``` ie, we remove the host call from the one that makes the hbrbac rule, so it should just create it if it doesn't exist. Then we populate the fedora hosts in a seperate host level task.