Update Final Release SOP
Signed-off-by: Patrik Polakovič <patrik@alphamail.org>
This commit is contained in:
Patrik Polakovič
parent
b52b32f2d9
commit
b6ad8733c3
1 changed files with 193 additions and 32 deletions
|
|
@ -11,20 +11,31 @@ $ koji edit-tag --lock f{branched}
|
|||
|
||||
== Bodhi Changes
|
||||
|
||||
Set the bodhi release to `current`
|
||||
Set the Bodhi release to `current`.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Ensure your Bodhi configuration is properly set up.
|
||||
`sysadmin-bodhi` FAS group membership is required.
|
||||
====
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ bodhi releases edit --name F{branched} --state current
|
||||
$ bodhi releases edit --name F{branched}F --state current
|
||||
....
|
||||
|
||||
Set the bodhi stable tag to point to updates instead of base repo
|
||||
Set the Bodhi stable tag to point to updates instead of base repo.
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ bodhi releases edit --name F{branched} --stable-tag f{branched}-updates
|
||||
....
|
||||
|
||||
Set EOL of oldest release to correct date: four weeks after the official release date of {branched}.
|
||||
Set EOL of oldest release to correct date: four weeks after the official release
|
||||
date of {branched}.
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ bodhi releases edit --name F{old_release} --eol YYYY-MM-DD
|
||||
|
|
@ -32,18 +43,26 @@ $ bodhi releases edit --name F{old_release}C --eol YYYY-MM-DD
|
|||
$ bodhi releases edit --name F{old_release}F --eol YYYY-MM-DD
|
||||
....
|
||||
|
||||
If this date is different from the one in the https://fedorapeople.org/groups/schedule/f-{old_release}/f-{old_release}-key-tasks.html[Fedora schedule],
|
||||
If this date is different from the one in the
|
||||
https://fedorapeople.org/groups/schedule/f-{old_release}/f-{old_release}-key-tasks.html[Fedora schedule],
|
||||
ask the program manager to update the date in the schedule.
|
||||
If it is different from the one in F{old_release}'s `/etc/os-release`, https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&format&component=fedora-release&version=40[file a bug against fedora-release] asking for it to be updated.
|
||||
If it is different from the one in F{old_release}'s `/etc/os-release`,
|
||||
https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&format&component=fedora-release&version=40[file a bug against fedora-release]
|
||||
asking for it to be updated.
|
||||
|
||||
You may also wish to check the current EOL estimates for other releases on Bodhi, the schedule, and fedora-release,
|
||||
and make sure they are in sync and sensible.
|
||||
You may also wish to check the current EOL estimates for other releases on
|
||||
Bodhi, the schedule, and `fedora-release`, and make sure they are in sync and
|
||||
sensible.
|
||||
|
||||
== Atomic desktops changes
|
||||
|
||||
Run the releng scripts/update_ostree_refs.sh on a compose machine with /mnt/koji/ mounted.
|
||||
Run the `releng/scripts/update_ostree_refs.sh` script on a compose machine with
|
||||
`/mnt/koji/` mounted.
|
||||
|
||||
$ scripts/update_ostree_refs.sh {branched}
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ ./update_ostree_refs.sh {branched}
|
||||
....
|
||||
|
||||
This updates the refs for Atomic desktops to have users follow updates
|
||||
for the new release rather than being stuck on the GA compose forever.
|
||||
|
|
@ -73,40 +92,138 @@ $ sudo rbac-playbook openshift-apps/greenwave.yml
|
|||
$ sudo rbac-playbook groups/openqa.yml -t testcase_stats
|
||||
....
|
||||
|
||||
If you do not have permissions to run a playbook, ask a sysadmin-main member for help.
|
||||
A sysadmin-qa member can run the openQA playbook.
|
||||
If you do not have permissions to run a playbook, ask a `sysadmin-main` member for help.
|
||||
A `sysadmin-qa` member can run the openQA playbook.
|
||||
|
||||
For reference, see the https://pagure.io/fedora-infra/ansible/c/7e501dcb9432885c9ef10e78c0418b0d40c85061[Fedora 41 release PR],
|
||||
though note it is somewhat old and includes some things that are no longer needed, and leaves out some that should be done.
|
||||
For reference, see the https://pagure.io/fedora-infra/ansible/pull-request/2567[Fedora 42 release PR].
|
||||
|
||||
== Stage Final Release for Mirrors
|
||||
== Staging Final Release to Mirrors
|
||||
|
||||
To prepare for running the staging script, make sure you have the following information:
|
||||
Staging the Final Release consists of two steps:
|
||||
|
||||
. Release Version: This is the numerical version number of the release, for example, {branched}.
|
||||
. ComposeID: The ID associated with the Compose, such as "Fedora-{branched}-20160614.0".
|
||||
. Label: The label used for the location in stage, for example, "Compsoe label for the location in stage 39_RC-1.2."
|
||||
. Key: The name of the release key, which can be "fedora-{branched}" or "fedora-{branched}-secondary," as examples.
|
||||
. Prerelease: Set this to 0 or 1 to determine if the release should be placed in a testing environment or not.
|
||||
. Arch (Optional): For secondary architectures, this parameter can be used to make changes to some internal locations.
|
||||
+
|
||||
[source,subs="attributes+"]
|
||||
. Signing the checksums
|
||||
. Staging the release
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
The signing script requires the person running it to have
|
||||
https://docs.fedoraproject.org/en-US/infra/releng_misc_guide/sop_sigul_client_setup/[Sigul]
|
||||
set up.
|
||||
|
||||
The staging script can be run on any compose machine where `/pub` is mounted with `rw` permissions.
|
||||
====
|
||||
|
||||
For example:
|
||||
....
|
||||
$ scripts/stage-release.sh {branched} Fedora-{branched}-20160614.0 {branched}_RC-1.2 fedora-{branched} 0
|
||||
$ ssh bodhi-backend01.iad2.fedoraproject.org
|
||||
....
|
||||
. Sync the release to the Red Hat internal archive following internally
|
||||
documented
|
||||
|
||||
=== Signing the checksums
|
||||
|
||||
Run the `releng/scripts/stage-release-checksum-sign.sh` script.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
This script should be run as the release engineer, not as root.
|
||||
====
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
Make sure to grab the directory size usage numbers which is used to send
|
||||
an email to [.title-ref]#mirror-admin@lists.fedoraproject.org# list.
|
||||
an email to the `mirror-admin@lists.fedoraproject.org` mailing list.
|
||||
====
|
||||
|
||||
== Make sure there are no tagging leftovers from bodhi
|
||||
For example:
|
||||
|
||||
Sometimes builds are not properly tagged during the freeze process.
|
||||
Make sure all builds are properly tagged, for more info about the [issue](https://pagure.io/releng/issue/11775)
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ ./stage-release-checksum-sign.sh {branched} Fedora-{branched}-20250411.n.0 fedora-{branched} 0
|
||||
....
|
||||
|
||||
Where:
|
||||
|
||||
. `{branched}` is the `Release Version`.
|
||||
. `Fedora-{branched}-20250411.n.0` is the `Compose ID`. It can be found https://kojipkgs.fedoraproject.org/compose/{branched}/[here].
|
||||
. `fedora-{branched}` is the `Signing Key`.
|
||||
. `0` is the `Pre-release`. `1` places the release in a testing environment, `0` does not.
|
||||
|
||||
=== Staging the release
|
||||
|
||||
Run the `releng/scripts/stage.sh` script.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
This script should be run as the `ftpsync` user.
|
||||
====
|
||||
|
||||
For example:
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ sudo -u ftpsync scripts/stage-release.sh {branched} Fedora-{branched}-20250411.n.0 {branched}_RC-1.1 fedora-{branched} 0
|
||||
....
|
||||
|
||||
`{branched}_RC-1.1` is the `Label`.
|
||||
|
||||
== Syncing the release to the Red Hat internal archive
|
||||
|
||||
This is documented internally.
|
||||
|
||||
== Making sure there are no tagging issues in Bodhi
|
||||
|
||||
Builds should not be tagged in both `f{branched}` and `f{branched}-updates-testing` at once.
|
||||
Any builds that are in both tags should be untagged from `f{branched}-updates-testing`.
|
||||
|
||||
Get the list of builds that are tagged in `f{branched}`.
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ koji list-tagged --quiet f{branched} > f{branched}.txt
|
||||
....
|
||||
|
||||
Get the list of builds that are tagged in `f{branched}-updates-testing`.
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ koji list-tagged --quiet f{branched}-updates-testing > updates-testing.txt
|
||||
....
|
||||
|
||||
Compare which builds match and put it in a file.
|
||||
|
||||
....
|
||||
$ grep -Fxf <(awk '{print $1}' f42.txt) <(awk '{print $1}' updates-testing.txt) > tagged-in-both.txt
|
||||
....
|
||||
|
||||
We now have a text file (`tagged-in-both.txt`) with only the names of the builds, one word per line.
|
||||
|
||||
Use a simple while loop and the `koji untag-build f{branched}-updates-testing <BUILD>`
|
||||
utility to untag all of the builds.
|
||||
|
||||
This may take several hours.
|
||||
|
||||
If any of the untagging fails because of permission issues the Koji `--force`
|
||||
option can be used.
|
||||
|
||||
After the script finishes repeat the process above (comparing matches in two
|
||||
files) to make sure that no builds tagged in `f{branched}` are also tagged in
|
||||
`f{branched}-updates-testing.`
|
||||
|
||||
== Pushing the updates to the stable repository
|
||||
|
||||
Updates are usually pushed to the stable repository as part of a cronjob that runs during the weekend.
|
||||
Stage Release Day (as opposed to the official Release Day on Tuesdays) usually falls on a Friday.
|
||||
If we want to avoid weekend hiccups we can push the updates to the stable repository manually.
|
||||
|
||||
For example:
|
||||
|
||||
....
|
||||
$ sudo -u apache bodhi-push --username <username> --releases F42 --request stable
|
||||
|
||||
name: f42-updates success: True
|
||||
....
|
||||
|
||||
== Verification
|
||||
|
||||
|
|
@ -115,8 +232,52 @@ the web server due to permissions. Typically we ask somebody from the
|
|||
Infrastructure team to give the tree a second set of eyes.
|
||||
|
||||
Some things to check for:
|
||||
. are the CHECKSUM files signed?
|
||||
. are the spins/labs present?
|
||||
|
||||
. Are the CHECKSUM files signed?
|
||||
. Are the spins/labs present?
|
||||
|
||||
== One day before Release Day (Monday)
|
||||
|
||||
. Check that https://docs.fedoraproject.org/en-US/infra/release_guide/torrentrelease/[torrents] are published and seeding properly.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
We want to have the torrents up in advance so people can download the content to seed it to other users.
|
||||
We do not typically announce this widely, just to those people who normally seed torrents.
|
||||
====
|
||||
|
||||
[start=2]
|
||||
. Make the directories that were hidden as part of the https://pagure.io/releng/blob/main/f/scripts/stage-release.sh[stage-release.sh] script unhidden.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
These directories had their permissions set to `750` and now they should be changed to `755`.
|
||||
The directories in question can be found here:
|
||||
|
||||
`/pub/fedora/linux/releases/`
|
||||
|
||||
`/pub/alt/releases/`
|
||||
|
||||
`/pub/fedora-secondary/releases/`
|
||||
|
||||
This step ensures that the content officially becomes public and everything is accessible after the release GO decision.
|
||||
====
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
This should be done late on Monday night, before the release.
|
||||
This is to allow mirrors that are not tier 1 and do not have access to pre-release content to start syncing up
|
||||
so that more of them are in sync by the time of the actual release announcement.
|
||||
We typically do not announce this widely either.
|
||||
Mirrors will just start syncing it, we don't need to tell users about it.
|
||||
====
|
||||
|
||||
== Release Day (Tuesday @ 14:00 UTC)
|
||||
|
||||
?
|
||||
|
||||
== Post Release
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue