Infrastructure/ansible
5
0
Fork 0
Code Issues Pull requests 3 Projects Releases Packages Wiki Activity Actions

roles/bodhi2: Set EPEL to use zstd repodata and disable unusable features #2391

Merged
ryanlerch merged 1 commit from zstd-epelrepodata into main 2025-05-14 23:30:18 +00:00
Conversation 26 Commits 1 Files changed 1 +4 -4
ngompa commented 2024-11-28 01:06:43 +00:00
Contributor
Copy link

Zstd repodata support was introduced in DNF quite some time ago, and
is quite a bit faster and less resource intensive.

Additionally, there is no point in producing sqlmd or DeltaRPMs when
support for both has been removed from RHEL for a fair bit of time.

Regardless, we should leave things mostly as-is for EPEL 9, and thus
we retain older xz compression settings for that.

Signed-off-by: Neal Gompa ngompa@fedoraproject.org

Zstd repodata support was introduced in DNF quite some time ago, and is quite a bit faster and less resource intensive. Additionally, there is no point in producing sqlmd or DeltaRPMs when support for both has been removed from RHEL for a fair bit of time. Regardless, we should leave things mostly as-is for EPEL 9, and thus we retain older xz compression settings for that. Signed-off-by: Neal Gompa <ngompa@fedoraproject.org>
zuul commented 2024-11-28 01:09:03 +00:00
First-time contributor
Copy link

Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/a0b5a3de387742589470c9c9d9c4b4ec

Build succeeded. https://fedora.softwarefactory-project.io/zuul/buildset/a0b5a3de387742589470c9c9d9c4b4ec - [fi-ansible-lint-diff ](https://fedora.softwarefactory-project.io/zuul/build/c6583971e3314e45a5cd4b324343fe21) : SUCCESS in 2m 01s - [fi-yamllint-diff ](https://fedora.softwarefactory-project.io/zuul/build/857c06e73ac44f7d8905b820d3683607) : SUCCESS in 1m 57s
carlwgeorge commented 2024-11-28 01:50:32 +00:00
Contributor
Copy link

I think we should just drop the [prefix.FEDORA-EPEL-NEXT] section. We're not creating any more and the only one still active is EPEL-9N, so we can just set the the options for that one. While we're at it we can drop the [release.EPEL-8N] section since it's already retired.

I think we should just drop the `[prefix.FEDORA-EPEL-NEXT]` section. We're not creating any more and the only one still active is EPEL-9N, so we can just set the the options for that one. While we're at it we can drop the `[release.EPEL-8N]` section since it's already retired.
carlwgeorge commented 2024-11-28 01:53:53 +00:00
Contributor
Copy link

If I'm reading this right, this will be changing EPEL-9 from compatibility = True to compatibility = False. If we want to keep that the same we'll need an override in this section.

If I'm reading this right, this will be changing EPEL-9 from `compatibility = True` to `compatibility = False`. If we want to keep that the same we'll need an override in this section.
mattia commented 2024-12-01 12:08:07 +00:00
Contributor
Copy link

If you want to retain settings for EPEL-9 / EPEL-9N just copy the existing lines of prefix.FEDORA-EPEL section in two release.EPEL-9 / release.EPEL-9N sections, then changes in prefix.FEDORA-EPEL will apply only to EPEL-10.

Note that AFAIK the compatibility=True flag will override the following settings to:

repodata-compress-type = gz
zchunk = False
sqlite_enabled = True

You can only change them if compatibility=False.

If you want to retain settings for EPEL-9 / EPEL-9N just copy the existing lines of prefix.FEDORA-EPEL section in two release.EPEL-9 / release.EPEL-9N sections, then changes in prefix.FEDORA-EPEL will apply only to EPEL-10. Note that AFAIK the `compatibility=True` flag will override the following settings to: ``` repodata-compress-type = gz zchunk = False sqlite_enabled = True ``` You can only change them if `compatibility=False`.
ngompa commented 2024-12-01 12:19:25 +00:00
Author
Contributor
Copy link

I turned compatibility to False because shipping sqlmd is pointless for EPEL 9 since nothing supports it.

I turned compatibility to False because shipping sqlmd is pointless for EPEL 9 since nothing supports it.
ngompa commented 2024-12-01 12:22:04 +00:00
Author
Contributor
Copy link

rebased onto 1bb258138d

rebased onto 1bb258138d0ea80e598158c0f760f3e89cd66b1e
zuul commented 2024-12-01 12:24:28 +00:00
First-time contributor
Copy link

Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/2d332d3106514fa3be2163a90fd5cca8

Build succeeded. https://fedora.softwarefactory-project.io/zuul/buildset/2d332d3106514fa3be2163a90fd5cca8 - [fi-ansible-lint-diff ](https://fedora.softwarefactory-project.io/zuul/build/a44c191dbe4e4530abc26f8370112594) : SUCCESS in 2m 09s - [fi-yamllint-diff ](https://fedora.softwarefactory-project.io/zuul/build/42425149614546dea9ab8136776b27da) : SUCCESS in 1m 53s
kevin commented 2024-12-04 21:29:44 +00:00
Contributor
Copy link

So, to summarize, this:

  • drops sqlite_enabled for epel9 / epel9-next

  • switches epel10 to zstd

I think dropping sqlite might be fine for epel9/epel9-next.

Note that switching epel10 to zstd means rhel8/9 hosts can't mergerepo_c / createrepo_c those repos because it doesn't understand zstd).
( https://issues.redhat.com/browse/RHEL-67689 )
So that might be something to run by the EPEL sig?

So, to summarize, this: * drops sqlite_enabled for epel9 / epel9-next * switches epel10 to zstd I think dropping sqlite might be fine for epel9/epel9-next. Note that switching epel10 to zstd means rhel8/9 hosts can't mergerepo_c / createrepo_c those repos because it doesn't understand zstd). ( https://issues.redhat.com/browse/RHEL-67689 ) So that might be something to run by the EPEL sig?
ngompa commented 2024-12-05 00:00:39 +00:00
Author
Contributor
Copy link

Fedora infrastructure doesn't run repository generation on RHEL, does it?

Fedora infrastructure doesn't run repository generation on RHEL, does it?
kevin commented 2024-12-05 02:10:03 +00:00
Contributor
Copy link

No, but there's external folks who do... possibly centos?

No, but there's external folks who do... possibly centos?
ngompa commented 2025-01-15 14:53:56 +00:00
Author
Contributor
Copy link

CBS runs on RHEL 9 machines I believe. I guess that means we need a patched createrepo for CBS infrastructure?

FYI: @arrfab

CBS runs on RHEL 9 machines I believe. I guess that means we need a patched createrepo for CBS infrastructure? FYI: @arrfab
arrfab commented 2025-01-15 15:05:52 +00:00
First-time contributor
Copy link

I confirm that builders behind cbs.centos.org are all migrated to RHEL9 and so are using createrepo_c-0.20.1-2.el9.x86_64

I confirm that builders behind cbs.centos.org are all migrated to RHEL9 and so are using `createrepo_c-0.20.1-2.el9.x86_64`
kevin commented 2025-01-15 17:41:00 +00:00
Contributor
Copy link

Yeah, rhel8/9 createrepo_c doesn't have zstd support (yet).

https://issues.redhat.com/browse/RHEL-67689

There is rhel/stream createrepo_c with zstd enabled at https://copr.fedorainfracloud.org/coprs/amatej/createrepo_c/

Currently Fedora 41 updates and now rawhide are using zstd.

Ideally we would wait for rhel8/9/stream8/9 to update ctreaterepo_c to support it before moving epel... but if thats going to be a long time or never, perhaps we don't wait.

Yeah, rhel8/9 createrepo_c doesn't have zstd support (yet). https://issues.redhat.com/browse/RHEL-67689 There is rhel/stream createrepo_c with zstd enabled at https://copr.fedorainfracloud.org/coprs/amatej/createrepo_c/ Currently Fedora 41 updates and now rawhide are using zstd. Ideally we would wait for rhel8/9/stream8/9 to update ctreaterepo_c to support it before moving epel... but if thats going to be a long time or never, perhaps we don't wait.
kevin commented 2025-03-19 20:18:40 +00:00
Contributor
Copy link

Some further notes. mdapi breaks without sqlite. ;( @t0xic0der is working on a replacement application that can do xml.

The createrepo_c for rhel9 is still showing in progress: https://issues.redhat.com/browse/RHEL-67689

I don't know the status on stream9

So should we just drop this for now? Or push to get rhel9/stream9 at least with a createrepo_c/mergerepos_c that works before we do?

Some further notes. mdapi breaks without sqlite. ;( @t0xic0der is working on a replacement application that can do xml. The createrepo_c for rhel9 is still showing in progress: https://issues.redhat.com/browse/RHEL-67689 I don't know the status on stream9 So should we just drop this for now? Or push to get rhel9/stream9 at least with a createrepo_c/mergerepos_c that works before we do?
ngompa commented 2025-05-02 19:43:40 +00:00
Author
Contributor
Copy link

We can turn back on the sqlmd for the time being? And that should be okay for everything.

We can turn back on the sqlmd for the time being? And that should be okay for everything.
ngompa commented 2025-05-02 19:50:34 +00:00
Author
Contributor
Copy link

rebased onto 715567cb9f

rebased onto 715567cb9f0c415caa3960296cca4a54813a1f80
ngompa commented 2025-05-02 19:51:00 +00:00
Author
Contributor
Copy link

I've turned back on SQLite metadata generation.

I've turned back on SQLite metadata generation.
zuul commented 2025-05-02 19:53:38 +00:00
First-time contributor
Copy link

Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/acf011060e9448c3a1ceb1b8428f3ced

Build succeeded. https://fedora.softwarefactory-project.io/zuul/buildset/acf011060e9448c3a1ceb1b8428f3ced - [fi-ansible-lint-diff ](https://fedora.softwarefactory-project.io/zuul/build/5980391ab29244e0b2899a645bd31ca1) : SUCCESS in 2m 45s - [fi-yamllint-diff ](https://fedora.softwarefactory-project.io/zuul/build/ba98e353996c49d9bb930fc4d5cff701) : SUCCESS in 2m 11s
carlwgeorge commented 2025-05-02 19:56:40 +00:00
Contributor
Copy link

LGTM

LGTM
kevin commented 2025-05-02 20:08:29 +00:00
Contributor
Copy link

This doesn't help rhel9 mergerepo_c... I guess thats going to be fixed in the next release?

This doesn't help rhel9 mergerepo_c... I guess thats going to be fixed in the next release?
carlwgeorge commented 2025-05-03 00:04:30 +00:00
Contributor
Copy link

Indeed that createrepo_c issue has "Fix Version/s: rhel-9.7", so we likely won't see that fix until the fall. That said, the mergerepo_c problem with mirroring repos using zstd is already a problem with Fedora, so I'm not sure it's worth worrying about for EPEL 10.

Another way we could go about this is set the EPEL default to zstd, and put in stanzas to keep using gz (or switch to xz) for 10.0 and 10.1, and then let 10.2 use the default zstd.

Indeed that createrepo_c issue has "Fix Version/s: rhel-9.7", so we likely won't see that fix until the fall. That said, the mergerepo_c problem with mirroring repos using zstd is already a problem with Fedora, so I'm not sure it's worth worrying about for EPEL 10. Another way we could go about this is set the EPEL default to zstd, and put in stanzas to keep using gz (or switch to xz) for 10.0 and 10.1, and then let 10.2 use the default zstd.
carlwgeorge commented 2025-05-03 00:10:46 +00:00
Contributor
Copy link

Created epel#330 to tag this for the ESC meeting.

Created [epel#330](https://pagure.io/epel/issue/330) to tag this for the ESC meeting.
kevin commented 2025-05-14 23:26:45 +00:00
Contributor
Copy link

rebased onto 5d1e014ccd

rebased onto 5d1e014ccd05aff0f3e59ab83c174364f8696a56
kevin commented 2025-05-14 23:29:31 +00:00
Contributor
Copy link

ESC said go ahead, so...

ESC said go ahead, so...
zuul commented 2025-05-14 23:29:53 +00:00
First-time contributor
Copy link

Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/84a51d21415d4cae98d71aaf4c24ece6

Build succeeded. https://fedora.softwarefactory-project.io/zuul/buildset/84a51d21415d4cae98d71aaf4c24ece6 - [fi-ansible-lint-diff ](https://fedora.softwarefactory-project.io/zuul/build/6ad2782083284c4db18ffa9d1a7af3f1) : SUCCESS in 2m 35s - [fi-yamllint-diff ](https://fedora.softwarefactory-project.io/zuul/build/d1dfb23da5db4dd2a2c2524711cc7eff) : SUCCESS in 2m 20s
kevin commented 2025-05-14 23:30:25 +00:00
Contributor
Copy link

Pull-Request has been merged by kevin

Pull-Request has been merged by kevin
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
freeze-break-request

   
post-freeze
No labels
freeze-break-request
post-freeze
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
6 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Infrastructure/ansible#2391
No description provided.
Delete branch "zstd-epelrepodata"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?