zlopez commented

Contributor

Describe what you would like us to do:

I'm creating this ticket to track what machines were affected by this httpd issue and to be able to close the other tickets related to this issue as those are actually solved.

So for now we have httpd updates disabled for:

people01.iad2.fedoraproject.org
ipsilon01.iad2.fedoraproject.org

When do you need this to be done by? (YYYY/MM/DD)

This is waiting for apache folks to provide a fix.

# Describe what you would like us to do: ---- I'm creating this ticket to track what machines were affected by this [httpd issue](https://bugzilla.redhat.com/show_bug.cgi?id=2298843) and to be able to close the other tickets related to this issue as those are actually solved. So for now we have httpd updates disabled for: * people01.iad2.fedoraproject.org * ipsilon01.iad2.fedoraproject.org # When do you need this to be done by? (YYYY/MM/DD) ---- This is waiting for apache folks to provide a fix.

kevin commented

2024-07-29 17:15:14 +00:00

ipsilon02.iad2.fedoraproject.org
and
ipsilon01.stg.iad2.fedoraproject.org

also. ;)

ipsilon02.iad2.fedoraproject.org and ipsilon01.stg.iad2.fedoraproject.org also. ;)

zlopez commented

2024-08-02 11:29:48 +00:00

Author

Contributor

I tested out https://bodhi.fedoraproject.org/updates/FEDORA-2024-e83af0855e and it's still causing the issue. I downgraded it back to working version on ipsilon01.stg, so people can use it.

@luhliarik is backporting patches for the regressions fixed upstream. So hopefully the issue will be fixed soon.

I tested out https://bodhi.fedoraproject.org/updates/FEDORA-2024-e83af0855e and it's still causing the issue. I downgraded it back to working version on `ipsilon01.stg`, so people can use it. @luhliarik is backporting patches for the regressions fixed upstream. So hopefully the issue will be fixed soon.

kevin commented

2024-09-11 23:32:52 +00:00

Is this solved? I see the errata pushed out?

Or do we need to wait for freeze to end to confirm?

Is this solved? I see the errata pushed out? Or do we need to wait for freeze to end to confirm?

zlopez commented

2024-09-12 08:53:02 +00:00

Author

Contributor

I reached to @luhliarik few days back, but didn't get any response.

The last version I tested still had the issue.

I reached to @luhliarik few days back, but didn't get any response. The last version I tested still had the issue.

kevin commented

2024-10-02 21:22:43 +00:00

Any news here?

The ipsilon hosts are on f39 still... we could try moving to f40 and see if the issue is solved there? (But I suspect not, because it's the same httpd version).

Any news here? The ipsilon hosts are on f39 still... we could try moving to f40 and see if the issue is solved there? (But I suspect not, because it's the same httpd version).

zlopez commented

2024-10-03 09:13:51 +00:00

Author

Contributor

@kevin I will try to ping @luhliarik again as he probably forgot about this.

zlopez commented

2024-10-07 07:52:29 +00:00

Author

Contributor

@luhliarik Is still looking into that. There is no new upstream version, so he needs to patch the current version up.

luhliarik commented

2024-10-11 12:37:40 +00:00

I created new httpd-2.4.62-4.fc42 build including the latest mod_rewrite regression fixes. Feel free to test it.

zlopez commented

2024-10-11 13:21:18 +00:00

Author

Contributor

I installed the new build on ipsilon01.stg and login on https://stg.pagure.io works. I can confirm that this build is solving the issue we have.

Now we need to officially get it to F39 and EPEL9. So we can fix our servers.

I installed the new build on `ipsilon01.stg` and login on https://stg.pagure.io works. I can confirm that this build is solving the issue we have. Now we need to officially get it to F39 and EPEL9. So we can fix our servers.

zlopez commented

2024-10-11 14:47:42 +00:00

Author

Contributor

After another testing on https://src.stg.fedoraproject.org we (me and @luhliarik) found out that the issue is still there. Not sure why it didn't show up on https://stg.pagure.io. But unfortunately the issue is still not solved for us with the update.

zlopez commented

2024-10-11 14:50:43 +00:00

Author

Contributor

@kevin Would it be possible to give @luhliarik access to ipsilon01.stg, so he can test out the changes himself?

kevin commented

2024-10-11 16:18:24 +00:00

Sure. We should look at moving them to f40 (or I suppose we could wait for f41 to come out and move them to f41 then).

zlopez commented

2024-10-14 10:22:13 +00:00

Author

Contributor

@kevin What is needed to give @luhliarik the correct permissions?

Let's wait for F41 release before upgrading the server.

@kevin What is needed to give @luhliarik the correct permissions? Let's wait for F41 release before upgrading the server.

kevin commented

2024-10-14 15:09:19 +00:00

Well, it seems to do not have any ipa groups/shell access/sudo defined for ipsilon, so I think we need to set at least
ipa_client_shell_groups
ipa_client_sudo_groups
ipa_host_group
ipa_host_group_desc

in inventory/groups/ipsilon_stg

(make sure STAGING ONLY)

Then, add them to a group listed there. Or make a new group for it in staging...

I think that should work?

Well, it seems to do not have any ipa groups/shell access/sudo defined for ipsilon, so I think we need to set at least ipa_client_shell_groups ipa_client_sudo_groups ipa_host_group ipa_host_group_desc in inventory/groups/ipsilon_stg (make sure STAGING ONLY) Then, add them to a group listed there. Or make a new group for it in staging... I think that should work?

zlopez commented

2024-10-15 12:47:40 +00:00

Author

Contributor

@kevin I created a PR for it, when it gets merged I will create the group as well.

@kevin I created a [PR](https://pagure.io/fedora-infra/ansible/pull-request/2314) for it, when it gets merged I will create the group as well.

kevin commented

2024-10-15 18:31:04 +00:00

Looks good to me.

zlopez commented

2024-10-16 10:22:29 +00:00

Author

Contributor

I merged the PR, but when creating the sysadmin-ipsilon group on staging I got following error:

Operations error: Allocation of a new value for range cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config failed! Unable to proceed.

@kevin Did you saw this before?

I merged the PR, but when creating the `sysadmin-ipsilon` group on staging I got following error: ``` Operations error: Allocation of a new value for range cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config failed! Unable to proceed. ``` @kevin Did you saw this before?

zlopez commented

2024-10-16 10:26:57 +00:00

Author

Contributor

Found this on ipa01.stg logs:

zlopez@STG.FEDORAPROJECT.ORG: group_add('sysadmin-ipsilon', description='Sysadmin group for ipsilon servers.', fasgroup=True, version='2.253'): DatabaseError

Found this on `ipa01.stg` logs: ``` zlopez@STG.FEDORAPROJECT.ORG: group_add('sysadmin-ipsilon', description='Sysadmin group for ipsilon servers.', fasgroup=True, version='2.253'): DatabaseError ```

kevin commented

2024-10-16 20:57:44 +00:00

I have seen this before... there's some IPA uid/gid range thing... you from time to time need to allocate new ranges. It should be in the ipa docs... I don't recall the details. ;(

zlopez commented

2024-10-17 07:29:59 +00:00

Author

Contributor

@kevin I remember that, I just didn't realized this is the issue. Let me fix that.

zlopez commented

2024-10-17 12:48:11 +00:00

Author

Contributor

I added the DNA ranges, but it seems like the replicas are somewhat broken I can see plenty of errors in /var/log/dirsrv/slapd-STG-FEDORAPROJECT-ORG/errors on ipa01.stg and trying to fix that.

Currently trying to re-initialize replica by running ipa-replica-manage re-initialize, but it doesn't seems to be helping. Not sure what actually happened and why it's not working as it should.

I added the DNA ranges, but it seems like the replicas are somewhat broken I can see plenty of errors in `/var/log/dirsrv/slapd-STG-FEDORAPROJECT-ORG/errors` on `ipa01.stg` and trying to fix that. Currently trying to re-initialize replica by running `ipa-replica-manage re-initialize`, but it doesn't seems to be helping. Not sure what actually happened and why it's not working as it should.

zlopez commented

2024-10-18 14:26:29 +00:00

Author

Contributor

I changed the group to have access in ipsilon_stg to sysadmin-noc and added @luhliarik to the group.

@kevin Do I need to run some playbook to reflect the changes? Or do I need to add permissions to sysadmin-noc group on staging IPA?

I [changed the group](https://pagure.io/fedora-infra/ansible/pull-request/2322) to have access in `ipsilon_stg` to `sysadmin-noc` and added @luhliarik to the group. @kevin Do I need to run some playbook to reflect the changes? Or do I need to add permissions to `sysadmin-noc` group on staging IPA?

kevin commented

2024-10-18 18:41:10 +00:00

Nothing should be needed. sssd on the vm will read the info from ipa, so it should be updated...

zlopez commented

2024-10-21 11:44:56 +00:00

Author

Contributor

I notified @luhliarik that he should now had access to the machine.

kevin commented

2024-12-02 22:41:35 +00:00

ok, so, I moved our ipsilon servers to f41. (because f39 went eol last week).

And the problem still seems to be happening. ;(

We can't keep kicking this can down the road.

I saved off the xml and lv for the old f39 ones, I guess I can roll back to them... but... I really don't want to do that.
I might look and see if I can figure out anything first.

ok, so, I moved our ipsilon servers to f41. (because f39 went eol last week). And the problem still seems to be happening. ;( We can't keep kicking this can down the road. I saved off the xml and lv for the old f39 ones, I guess I can roll back to them... but... I really don't want to do that. I might look and see if I can figure out anything first.

kevin commented

2024-12-02 23:18:39 +00:00

I downgraded the prod ones to httpd-2.4.59-2.fc41 and it resumed working again. ;(

zlopez commented

2024-12-03 09:06:35 +00:00

Author

Contributor

At least the old package is still available on fc41.

zlopez commented

2025-02-20 12:24:54 +00:00

Author

Contributor

I reached to @luhliarik and got info that most of the regressions should be fixed in the latest httpd version.

I tested out ipsilon01.stg with httpd-2.4.63-1.fc41.x86_64 and it seems that the SSO from stg.pagure.io is working without issue. I will try the same with people01 and see the output, but it seems that this issue is solved now.

I reached to @luhliarik and got info that most of the regressions should be fixed in the latest httpd version. I tested out ipsilon01.stg with httpd-2.4.63-1.fc41.x86_64 and it seems that the SSO from stg.pagure.io is working without issue. I will try the same with people01 and see the output, but it seems that this issue is solved now.

zlopez commented

2025-02-20 13:04:34 +00:00

Author

Contributor

Unfortunately on EPEL9 there is still httpd-2.4.62-1.el9_5.2.x86_64, which has the issue. So people01 still has the problem.

Updated httpd on ipsilon01 and ipsilon02 and the issue didn't show up on pagure.io and src.fedoraproject.org, but I'm still getting too many redirects from https://copr.fedorainfracloud.org/. So it's not completely fixing the issue, but at least partially it does.

The problem is that I can't undo the dnf transaction:

Cannot perform Install action, no match for: httpd-2.4.59-2.fc41.x86_64

I will try and find the builds in koji and restore it from that.

Unfortunately on EPEL9 there is still httpd-2.4.62-1.el9_5.2.x86_64, which has the issue. So people01 still has the problem. Updated httpd on ipsilon01 and ipsilon02 and the issue didn't show up on pagure.io and src.fedoraproject.org, but I'm still getting too many redirects from https://copr.fedorainfracloud.org/. So it's not completely fixing the issue, but at least partially it does. The problem is that I can't undo the dnf transaction: ``` Cannot perform Install action, no match for: httpd-2.4.59-2.fc41.x86_64 ``` I will try and find the builds in koji and restore it from that.

zlopez commented

2025-02-20 13:55:16 +00:00

Author

Contributor

I reverted the changes back using old koji build, the packages are in /root/httpd_rpm_backup folder on ipsilon01 and ipsilon02, just in case we need them in future.

I reverted the changes back using old koji build, the packages are in `/root/httpd_rpm_backup` folder on ipsilon01 and ipsilon02, just in case we need them in future.

kevin commented

2025-02-21 17:31:20 +00:00

Just slight side note... we are in freeze now, so we should be careful about making changes to prod without +1s. ;)

Too bad it didn't fix it entirely though. ;(

Just slight side note... we are in freeze now, so we should be careful about making changes to prod without +1s. ;) Too bad it didn't fix it entirely though. ;(

zlopez commented

2025-02-24 12:01:22 +00:00

Author

Contributor

Forgot that this needs a freeze break :-/. Not sure if the reason is that COPR is still using OpenID authentication or there is still something wrong with httpd package (there is definitely a regression, but the migration to OIDC could maybe solve the issue as well).

kevin commented

2025-04-21 18:59:29 +00:00

Any news on a fix here?

zlopez commented

2025-04-23 13:44:11 +00:00

Author

Contributor

Unfortunately not, the last experiment with update didn't work as expected.

I will try to update it again and test it out with COPR as that was the only one which didn't work and it's not migrated to OIDC.

Unfortunately not, the last experiment with update didn't work as expected. I will try to update it again and test it out with COPR as that was the only one which didn't work and it's not migrated to OIDC.

zlopez was assigned by ryanlerch

2025-06-19 01:00:46 +00:00

Rows
Columns

httpd 2.4.61 causing issue in fedora infrastructure #12092

Describe what you would like us to do:

When do you need this to be done by? (YYYY/MM/DD)