please don't remove enrolled centos machines from IPA in staging #12514

Open
opened 2025-04-24 07:11:56 +00:00 by arrfab · 13 comments

As CentOS and Fedora are using shared IPA backend for authentication, I'd request that nothing touching enrolled centos machines in IPA would be done (manually or through scripts)
I had to just waste my time this morning investigating why ipsilon (https://id.stg.centos.org) wasn't allowing anyone to auth (and so no openidc for services using our ipsilon instance)

Someone (who ? or a script ?) removed ipsilon.stg.iad2.centos.org from the ipsilon HBAC rule, denying so all auth requests .

Can you identify the root cause and ensure it wouldn't happen again please ?

Thanks a lot

As CentOS and Fedora are using shared IPA backend for authentication, I'd request that nothing touching enrolled centos machines in IPA would be done (manually or through scripts) I had to just waste my time this morning investigating why ipsilon (https://id.stg.centos.org) wasn't allowing anyone to auth (and so no openidc for services using our ipsilon instance) Someone (who ? or a script ?) removed `ipsilon.stg.iad2.centos.org` from the ipsilon HBAC rule, denying so all auth requests . Can you identify the root cause and ensure it wouldn't happen again please ? Thanks *a lot*
Contributor

Metadata Update from @zlopez:

  • Issue priority set to: Waiting on Assignee (was: Needs Review)
  • Issue tagged with: Needs investigation, high-gain, ops
**Metadata Update from @zlopez**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation, high-gain, ops
Author
seems related to https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_92 ...
Author

Metadata Update from @arrfab:

  • Issue untagged with: Needs investigation, high-gain, ops
  • Issue priority set to: Needs Review (was: Waiting on Assignee)
**Metadata Update from @arrfab**: - Issue **un**tagged with: Needs investigation, high-gain, ops - Issue priority set to: Needs Review (was: Waiting on Assignee)
Contributor

Metadata Update from @zlopez:

  • Issue priority set to: Waiting on Assignee (was: Needs Review)
  • Issue tagged with: Needs investigation, high-gain, ops
**Metadata Update from @zlopez**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation, high-gain, ops
Contributor

The change you are referring to happened 4 years ago. So I assume that didn't caused the machine to be removed.

The [change](https://pagure.io/fedora-infra/ansible/c/e92d0dda1a54a8576586450578f8873ce920530d) you are referring to happened 4 years ago. So I assume that didn't caused the machine to be removed.

Metadata Update from @kevin:

  • Issue assigned to kevin
**Metadata Update from @kevin**: - Issue assigned to kevin

It's actually https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_101

It was using the wrong hostname... but that was set in 2021?
b8e6754f97c (Aurélien Bompard 2021-03-22 17:07:45 +0100 101) host: "{{ (env == 'production')|ternary('ipsilon.iad2.centos.org', 'centos-ipa-client02.stg.iad2.fedoraproject.org') }}"

anyhow, I changed it to ipsilon.stg.iad2.centos.org

If you can confirm it's fixed / working?

It's actually https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_101 It was using the wrong hostname... but that was set in 2021? b8e6754f97c (Aurélien Bompard 2021-03-22 17:07:45 +0100 101) host: "{{ (env == 'production')|ternary('ipsilon.iad2.centos.org', 'centos-ipa-client02.stg.iad2.fedoraproject.org') }}" anyhow, I changed it to ipsilon.stg.iad2.centos.org If you can confirm it's fixed / working?
Author

forgot to give feedback (PTO) but yes, now working again ...
Is there a way to ensure that ansible would not remove hosts not managed by itself ?
Otherwise, just ensure that it's documented somewhere as that means that on my centos ansible infra side I'll never be autonomous to deploy/enroll a machine as then Fedora ansible would remove it :/

forgot to give feedback (PTO) but yes, now working again ... Is there a way to ensure that ansible would not remove hosts not managed by itself ? Otherwise, just ensure that it's documented somewhere as that means that on my centos ansible infra side I'll never be autonomous to deploy/enroll a machine as then Fedora ansible would remove it :/

Well, I mean we could remove that task from our playbooks and you could depend on yourself to create that ?

Otherwise ansible is going to setup the thing thats defined.

Do you want us to remove that setup from our side?

Well, I mean we could remove that task from our playbooks and you could depend on yourself to create that ? Otherwise ansible is going to setup the thing thats defined. Do you want us to remove that setup from our side?
Author

@kevin : even if you remove that block, I guess that the other one about Fedora host would then remove the existing centos machines again .. so isn't there a parameter for that ipahbacrule to not purge things not declared ? if not then we'll have to still hard-code centos machines there but with dc move and so multiple machines that will need to be enrolled, I'd like that to be documented at least :)

@kevin : even if you remove that block, I guess that the other one about Fedora host would then remove the existing centos machines again .. so isn't there a parameter for that `ipahbacrule` to not purge things not declared ? if not then we'll have to still hard-code centos machines there but with dc move and so multiple machines that will need to be enrolled, I'd like that to be documented at least :)

well, my reading / understanding of it is that:

https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_85 creates the 'ipsilon' hbac rule and

https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_101
has action: member, so it adds those members/hosts.

So, I guess we could change the first one to just confirm/add the fedora ipsilon servers and then it wouldn't recreate the rule...
but then the rule would have to be... manually created or something?

well, my reading / understanding of it is that: https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_85 creates the 'ipsilon' hbac rule and https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_101 has action: member, so it adds those members/hosts. So, I guess we could change the first one to just confirm/add the fedora ipsilon servers and then it wouldn't recreate the rule... but then the rule would have to be... manually created or something?

Would this be high trouble, medium trouble, low trouble or a timeboxed investigation?

Would this be high trouble, medium trouble, low trouble or a timeboxed investigation?

What if we do this:

diff --git a/playbooks/groups/ipsilon.yml b/playbooks/groups/ipsilon.yml
index 8c95c4344a..1f4636ece3 100644
--- a/playbooks/groups/ipsilon.yml
+++ b/playbooks/groups/ipsilon.yml
@@ -89,6 +89,14 @@
       hbacsvc:
       - ipsilon
       usercategory: all
+      ipaadmin_password: "{{ ipa_admin_password }}"
+    tags:
+    - ipsilon
+
+  - name: Allow login through ipsilon for the Fedora instance
+    ipahbacrule:
+      name: ipsilon
+      action: member
       host: "{{ (env == 'production')|ternary(groups['ipsilon'], groups['ipsilon_stg']) }}"
       ipaadmin_password: "{{ ipa_admin_password }}"
     tags:

ie, we remove the host call from the one that makes the hbrbac rule, so it should just create it if it doesn't exist.
Then we populate the fedora hosts in a seperate host level task.

What if we do this: ``` diff --git a/playbooks/groups/ipsilon.yml b/playbooks/groups/ipsilon.yml index 8c95c4344a..1f4636ece3 100644 --- a/playbooks/groups/ipsilon.yml +++ b/playbooks/groups/ipsilon.yml @@ -89,6 +89,14 @@ hbacsvc: - ipsilon usercategory: all + ipaadmin_password: "{{ ipa_admin_password }}" + tags: + - ipsilon + + - name: Allow login through ipsilon for the Fedora instance + ipahbacrule: + name: ipsilon + action: member host: "{{ (env == 'production')|ternary(groups['ipsilon'], groups['ipsilon_stg']) }}" ipaadmin_password: "{{ ipa_admin_password }}" tags: ``` ie, we remove the host call from the one that makes the hbrbac rule, so it should just create it if it doesn't exist. Then we populate the fedora hosts in a seperate host level task.
Sign in to join this conversation.
No milestone
No project
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Infrastructure/fedora-infrastructure#12514
No description provided.