From b6ad8733c32daa2a39b94e607157935fa8a0dc21 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patrik=20Polakovi=C4=8D?= Date: Fri, 11 Apr 2025 14:28:16 +0200 Subject: [PATCH] Update Final Release SOP MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Patrik Polakovič --- .../release_guide/pages/final_release.adoc | 225 +++++++++++++++--- 1 file changed, 193 insertions(+), 32 deletions(-) diff --git a/modules/release_guide/pages/final_release.adoc b/modules/release_guide/pages/final_release.adoc index 1d34b54..3e8ae68 100644 --- a/modules/release_guide/pages/final_release.adoc +++ b/modules/release_guide/pages/final_release.adoc @@ -11,20 +11,31 @@ $ koji edit-tag --lock f{branched} == Bodhi Changes -Set the bodhi release to `current` +Set the Bodhi release to `current`. + +[NOTE] +.Note +==== +Ensure your Bodhi configuration is properly set up. +`sysadmin-bodhi` FAS group membership is required. +==== + [source,subs="attributes+"] .... $ bodhi releases edit --name F{branched} --state current $ bodhi releases edit --name F{branched}F --state current .... -Set the bodhi stable tag to point to updates instead of base repo +Set the Bodhi stable tag to point to updates instead of base repo. + [source,subs="attributes+"] .... $ bodhi releases edit --name F{branched} --stable-tag f{branched}-updates .... -Set EOL of oldest release to correct date: four weeks after the official release date of {branched}. +Set EOL of oldest release to correct date: four weeks after the official release +date of {branched}. + [source,subs="attributes+"] .... $ bodhi releases edit --name F{old_release} --eol YYYY-MM-DD @@ -32,18 +43,26 @@ $ bodhi releases edit --name F{old_release}C --eol YYYY-MM-DD $ bodhi releases edit --name F{old_release}F --eol YYYY-MM-DD .... -If this date is different from the one in the https://fedorapeople.org/groups/schedule/f-{old_release}/f-{old_release}-key-tasks.html[Fedora schedule], +If this date is different from the one in the +https://fedorapeople.org/groups/schedule/f-{old_release}/f-{old_release}-key-tasks.html[Fedora schedule], ask the program manager to update the date in the schedule. -If it is different from the one in F{old_release}'s `/etc/os-release`, https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&format&component=fedora-release&version=40[file a bug against fedora-release] asking for it to be updated. +If it is different from the one in F{old_release}'s `/etc/os-release`, +https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&format&component=fedora-release&version=40[file a bug against fedora-release] +asking for it to be updated. -You may also wish to check the current EOL estimates for other releases on Bodhi, the schedule, and fedora-release, -and make sure they are in sync and sensible. +You may also wish to check the current EOL estimates for other releases on +Bodhi, the schedule, and `fedora-release`, and make sure they are in sync and +sensible. == Atomic desktops changes -Run the releng scripts/update_ostree_refs.sh on a compose machine with /mnt/koji/ mounted. +Run the `releng/scripts/update_ostree_refs.sh` script on a compose machine with +`/mnt/koji/` mounted. -$ scripts/update_ostree_refs.sh {branched} +[source,subs="attributes+"] +.... +$ ./update_ostree_refs.sh {branched} +.... This updates the refs for Atomic desktops to have users follow updates for the new release rather than being stuck on the GA compose forever. @@ -73,40 +92,138 @@ $ sudo rbac-playbook openshift-apps/greenwave.yml $ sudo rbac-playbook groups/openqa.yml -t testcase_stats .... -If you do not have permissions to run a playbook, ask a sysadmin-main member for help. -A sysadmin-qa member can run the openQA playbook. +If you do not have permissions to run a playbook, ask a `sysadmin-main` member for help. +A `sysadmin-qa` member can run the openQA playbook. -For reference, see the https://pagure.io/fedora-infra/ansible/c/7e501dcb9432885c9ef10e78c0418b0d40c85061[Fedora 41 release PR], -though note it is somewhat old and includes some things that are no longer needed, and leaves out some that should be done. +For reference, see the https://pagure.io/fedora-infra/ansible/pull-request/2567[Fedora 42 release PR]. -== Stage Final Release for Mirrors +== Staging Final Release to Mirrors -To prepare for running the staging script, make sure you have the following information: +Staging the Final Release consists of two steps: -. Release Version: This is the numerical version number of the release, for example, {branched}. -. ComposeID: The ID associated with the Compose, such as "Fedora-{branched}-20160614.0". -. Label: The label used for the location in stage, for example, "Compsoe label for the location in stage 39_RC-1.2." -. Key: The name of the release key, which can be "fedora-{branched}" or "fedora-{branched}-secondary," as examples. -. Prerelease: Set this to 0 or 1 to determine if the release should be placed in a testing environment or not. -. Arch (Optional): For secondary architectures, this parameter can be used to make changes to some internal locations. -+ -[source,subs="attributes+"] +. Signing the checksums +. Staging the release + +[NOTE] +.Note +==== +The signing script requires the person running it to have +https://docs.fedoraproject.org/en-US/infra/releng_misc_guide/sop_sigul_client_setup/[Sigul] +set up. + +The staging script can be run on any compose machine where `/pub` is mounted with `rw` permissions. +==== + +For example: .... -$ scripts/stage-release.sh {branched} Fedora-{branched}-20160614.0 {branched}_RC-1.2 fedora-{branched} 0 +$ ssh bodhi-backend01.iad2.fedoraproject.org .... -. Sync the release to the Red Hat internal archive following internally -documented + +=== Signing the checksums + +Run the `releng/scripts/stage-release-checksum-sign.sh` script. + +[NOTE] +.Note +==== +This script should be run as the release engineer, not as root. +==== [NOTE] ==== Make sure to grab the directory size usage numbers which is used to send -an email to [.title-ref]#mirror-admin@lists.fedoraproject.org# list. +an email to the `mirror-admin@lists.fedoraproject.org` mailing list. ==== -== Make sure there are no tagging leftovers from bodhi +For example: -Sometimes builds are not properly tagged during the freeze process. -Make sure all builds are properly tagged, for more info about the [issue](https://pagure.io/releng/issue/11775) +[source,subs="attributes+"] +.... +$ ./stage-release-checksum-sign.sh {branched} Fedora-{branched}-20250411.n.0 fedora-{branched} 0 +.... + +Where: + +. `{branched}` is the `Release Version`. +. `Fedora-{branched}-20250411.n.0` is the `Compose ID`. It can be found https://kojipkgs.fedoraproject.org/compose/{branched}/[here]. +. `fedora-{branched}` is the `Signing Key`. +. `0` is the `Pre-release`. `1` places the release in a testing environment, `0` does not. + +=== Staging the release + +Run the `releng/scripts/stage.sh` script. + +[NOTE] +.Note +==== +This script should be run as the `ftpsync` user. +==== + +For example: + +[source,subs="attributes+"] +.... +$ sudo -u ftpsync scripts/stage-release.sh {branched} Fedora-{branched}-20250411.n.0 {branched}_RC-1.1 fedora-{branched} 0 +.... + +`{branched}_RC-1.1` is the `Label`. + +== Syncing the release to the Red Hat internal archive + +This is documented internally. + +== Making sure there are no tagging issues in Bodhi + +Builds should not be tagged in both `f{branched}` and `f{branched}-updates-testing` at once. +Any builds that are in both tags should be untagged from `f{branched}-updates-testing`. + +Get the list of builds that are tagged in `f{branched}`. + +[source,subs="attributes+"] +.... +$ koji list-tagged --quiet f{branched} > f{branched}.txt +.... + +Get the list of builds that are tagged in `f{branched}-updates-testing`. + +[source,subs="attributes+"] +.... +$ koji list-tagged --quiet f{branched}-updates-testing > updates-testing.txt +.... + +Compare which builds match and put it in a file. + +.... +$ grep -Fxf <(awk '{print $1}' f42.txt) <(awk '{print $1}' updates-testing.txt) > tagged-in-both.txt +.... + +We now have a text file (`tagged-in-both.txt`) with only the names of the builds, one word per line. + +Use a simple while loop and the `koji untag-build f{branched}-updates-testing ` +utility to untag all of the builds. + +This may take several hours. + +If any of the untagging fails because of permission issues the Koji `--force` +option can be used. + +After the script finishes repeat the process above (comparing matches in two +files) to make sure that no builds tagged in `f{branched}` are also tagged in +`f{branched}-updates-testing.` + +== Pushing the updates to the stable repository + +Updates are usually pushed to the stable repository as part of a cronjob that runs during the weekend. +Stage Release Day (as opposed to the official Release Day on Tuesdays) usually falls on a Friday. +If we want to avoid weekend hiccups we can push the updates to the stable repository manually. + +For example: + +.... +$ sudo -u apache bodhi-push --username --releases F42 --request stable + +name: f42-updates success: True +.... == Verification @@ -115,8 +232,52 @@ the web server due to permissions. Typically we ask somebody from the Infrastructure team to give the tree a second set of eyes. Some things to check for: -. are the CHECKSUM files signed? -. are the spins/labs present? + +. Are the CHECKSUM files signed? +. Are the spins/labs present? + +== One day before Release Day (Monday) + +. Check that https://docs.fedoraproject.org/en-US/infra/release_guide/torrentrelease/[torrents] are published and seeding properly. + +[NOTE] +.Note +==== +We want to have the torrents up in advance so people can download the content to seed it to other users. +We do not typically announce this widely, just to those people who normally seed torrents. +==== + +[start=2] +. Make the directories that were hidden as part of the https://pagure.io/releng/blob/main/f/scripts/stage-release.sh[stage-release.sh] script unhidden. + +[NOTE] +.Note +==== +These directories had their permissions set to `750` and now they should be changed to `755`. +The directories in question can be found here: + +`/pub/fedora/linux/releases/` + +`/pub/alt/releases/` + +`/pub/fedora-secondary/releases/` + +This step ensures that the content officially becomes public and everything is accessible after the release GO decision. +==== + +[NOTE] +.Note +==== +This should be done late on Monday night, before the release. +This is to allow mirrors that are not tier 1 and do not have access to pre-release content to start syncing up +so that more of them are in sync by the time of the actual release announcement. +We typically do not announce this widely either. +Mirrors will just start syncing it, we don't need to tell users about it. +==== + +== Release Day (Tuesday @ 14:00 UTC) + +? == Post Release