Update Final Release SOP

Signed-off-by: Patrik Polakovič <patrik@alphamail.org>

modules/release_guide/pages/final_release.adoc

@@ -11,20 +11,31 @@ $ koji edit-tag --lock f{branched}
 
 == Bodhi Changes
 
-Set the bodhi release to `current`
+Set the Bodhi release to `current`.
+
+[NOTE]
+.Note
+====
+Ensure your Bodhi configuration is properly set up.
+`sysadmin-bodhi` FAS group membership is required.
+====
+
 [source,subs="attributes+"]
 ....
 $ bodhi releases edit --name F{branched} --state current
 $ bodhi releases edit --name F{branched}F --state current
 ....
 
-Set the bodhi stable tag to point to updates instead of base repo
+Set the Bodhi stable tag to point to updates instead of base repo.
+
 [source,subs="attributes+"]
 ....
 $ bodhi releases edit --name F{branched} --stable-tag f{branched}-updates
 ....
 
-Set EOL of oldest release to correct date: four weeks after the official release date of {branched}.
+Set EOL of oldest release to correct date: four weeks after the official release
+date of {branched}.
+
 [source,subs="attributes+"]
 ....
 $ bodhi releases edit --name F{old_release} --eol YYYY-MM-DD
@@ -32,18 +43,26 @@ $ bodhi releases edit --name F{old_release}C --eol YYYY-MM-DD
 $ bodhi releases edit --name F{old_release}F --eol YYYY-MM-DD
 ....
 
-If this date is different from the one in the https://fedorapeople.org/groups/schedule/f-{old_release}/f-{old_release}-key-tasks.html[Fedora schedule],
+If this date is different from the one in the
+https://fedorapeople.org/groups/schedule/f-{old_release}/f-{old_release}-key-tasks.html[Fedora schedule],
 ask the program manager to update the date in the schedule.
-If it is different from the one in F{old_release}'s `/etc/os-release`, https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&format&component=fedora-release&version=40[file a bug against fedora-release] asking for it to be updated.
+If it is different from the one in F{old_release}'s `/etc/os-release`,
+https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&format&component=fedora-release&version=40[file a bug against fedora-release]
+asking for it to be updated.
 
-You may also wish to check the current EOL estimates for other releases on Bodhi, the schedule, and fedora-release,
+You may also wish to check the current EOL estimates for other releases on
-and make sure they are in sync and sensible.
+Bodhi, the schedule, and `fedora-release`, and make sure they are in sync and
+sensible.
 
 == Atomic desktops changes
 
-Run the releng scripts/update_ostree_refs.sh on a compose machine with /mnt/koji/ mounted.
+Run the `releng/scripts/update_ostree_refs.sh` script on a compose machine with
+`/mnt/koji/` mounted.
 
-$ scripts/update_ostree_refs.sh {branched}
+[source,subs="attributes+"]
+....
+$ ./update_ostree_refs.sh {branched}
+....
 
 This updates the refs for Atomic desktops to have users follow updates
 for the new release rather than being stuck on the GA compose forever.
@@ -73,40 +92,138 @@ $ sudo rbac-playbook openshift-apps/greenwave.yml
 $ sudo rbac-playbook groups/openqa.yml -t testcase_stats
 ....
 
-If you do not have permissions to run a playbook, ask a sysadmin-main member for help.
+If you do not have permissions to run a playbook, ask a `sysadmin-main` member for help.
-A sysadmin-qa member can run the openQA playbook.
+A `sysadmin-qa` member can run the openQA playbook.
 
-For reference, see the https://pagure.io/fedora-infra/ansible/c/7e501dcb9432885c9ef10e78c0418b0d40c85061[Fedora 41 release PR],
+For reference, see the https://pagure.io/fedora-infra/ansible/pull-request/2567[Fedora 42 release PR].
-though note it is somewhat old and includes some things that are no longer needed, and leaves out some that should be done.
 
-== Stage Final Release for Mirrors
+== Staging Final Release to Mirrors
 
-To prepare for running the staging script, make sure you have the following information:
+Staging the Final Release consists of two steps:
 
-. Release Version: This is the numerical version number of the release, for example, {branched}.
+. Signing the checksums
-. ComposeID: The ID associated with the Compose, such as "Fedora-{branched}-20160614.0".
+. Staging the release
-. Label: The label used for the location in stage, for example, "Compsoe label for the location in stage 39_RC-1.2."
+
-. Key: The name of the release key, which can be "fedora-{branched}" or "fedora-{branched}-secondary," as examples.
+[NOTE]
-. Prerelease: Set this to 0 or 1 to determine if the release should be placed in a testing environment or not.
+.Note
-. Arch (Optional): For secondary architectures, this parameter can be used to make changes to some internal locations.
+====
-+
+The signing script requires the person running it to have
-[source,subs="attributes+"]
+https://docs.fedoraproject.org/en-US/infra/releng_misc_guide/sop_sigul_client_setup/[Sigul]
+set up.
+
+The staging script can be run on any compose machine where `/pub` is mounted with `rw` permissions.
+====
+
+For example:
 ....
-$ scripts/stage-release.sh {branched} Fedora-{branched}-20160614.0 {branched}_RC-1.2 fedora-{branched} 0
+$ ssh bodhi-backend01.iad2.fedoraproject.org
 ....
-. Sync the release to the Red Hat internal archive following internally
+
-documented
+=== Signing the checksums
+
+Run the `releng/scripts/stage-release-checksum-sign.sh` script.
+
+[NOTE]
+.Note
+====
+This script should be run as the release engineer, not as root.
+====
 
 [NOTE]
 ====
 Make sure to grab the directory size usage numbers which is used to send
-an email to [.title-ref]#mirror-admin@lists.fedoraproject.org# list.
+an email to the `mirror-admin@lists.fedoraproject.org` mailing list.
 ====
 
-== Make sure there are no tagging leftovers from bodhi
+For example:
 
-Sometimes builds are not properly tagged during the freeze process.
+[source,subs="attributes+"]
-Make sure all builds are properly tagged, for more info about the [issue](https://pagure.io/releng/issue/11775)
+....
+$ ./stage-release-checksum-sign.sh {branched} Fedora-{branched}-20250411.n.0 fedora-{branched} 0
+....
+
+Where:
+
+. `{branched}` is the `Release Version`.
+. `Fedora-{branched}-20250411.n.0` is the `Compose ID`. It can be found https://kojipkgs.fedoraproject.org/compose/{branched}/[here].
+. `fedora-{branched}` is the `Signing Key`.
+. `0` is the `Pre-release`. `1` places the release in a testing environment, `0` does not.
+
+=== Staging the release
+
+Run the `releng/scripts/stage.sh` script.
+
+[NOTE]
+.Note
+====
+This script should be run as the `ftpsync` user.
+====
+
+For example:
+
+[source,subs="attributes+"]
+....
+$ sudo -u ftpsync scripts/stage-release.sh {branched} Fedora-{branched}-20250411.n.0 {branched}_RC-1.1 fedora-{branched} 0
+....
+
+`{branched}_RC-1.1` is the `Label`.
+
+== Syncing the release to the Red Hat internal archive
+
+This is documented internally.
+
+== Making sure there are no tagging issues in Bodhi
+
+Builds should not be tagged in both `f{branched}` and `f{branched}-updates-testing` at once.
+Any builds that are in both tags should be untagged from `f{branched}-updates-testing`.
+
+Get the list of builds that are tagged in `f{branched}`.
+
+[source,subs="attributes+"]
+....
+$ koji list-tagged --quiet f{branched} > f{branched}.txt
+....
+
+Get the list of builds that are tagged in `f{branched}-updates-testing`.
+
+[source,subs="attributes+"]
+....
+$ koji list-tagged --quiet f{branched}-updates-testing > updates-testing.txt
+....
+
+Compare which builds match and put it in a file.
+
+....
+$ grep -Fxf <(awk '{print $1}' f42.txt) <(awk '{print $1}' updates-testing.txt) > tagged-in-both.txt
+....
+
+We now have a text file (`tagged-in-both.txt`) with only the names of the builds, one word per line.
+
+Use a simple while loop and the `koji untag-build f{branched}-updates-testing <BUILD>`
+utility to untag all of the builds.
+
+This may take several hours.
+
+If any of the untagging fails because of permission issues the Koji `--force`
+option can be used.
+
+After the script finishes repeat the process above (comparing matches in two
+files) to make sure that no builds tagged in `f{branched}` are also tagged in
+`f{branched}-updates-testing.`
+
+== Pushing the updates to the stable repository
+
+Updates are usually pushed to the stable repository as part of a cronjob that runs during the weekend.
+Stage Release Day (as opposed to the official Release Day on Tuesdays) usually falls on a Friday.
+If we want to avoid weekend hiccups we can push the updates to the stable repository manually.
+
+For example:
+
+....
+$ sudo -u apache bodhi-push --username <username> --releases F42 --request stable
+
+name:  f42-updates           success:  True
+....
 
 == Verification
 
@@ -115,8 +232,52 @@ the web server due to permissions. Typically we ask somebody from the
 Infrastructure team to give the tree a second set of eyes.
 
 Some things to check for:
-. are the CHECKSUM files signed?
+
-. are the spins/labs present?
+. Are the CHECKSUM files signed?
+. Are the spins/labs present?
+
+== One day before Release Day (Monday)
+
+. Check that https://docs.fedoraproject.org/en-US/infra/release_guide/torrentrelease/[torrents] are published and seeding properly.
+
+[NOTE]
+.Note
+====
+We want to have the torrents up in advance so people can download the content to seed it to other users.
+We do not typically announce this widely, just to those people who normally seed torrents.
+====
+
+[start=2]
+. Make the directories that were hidden as part of the https://pagure.io/releng/blob/main/f/scripts/stage-release.sh[stage-release.sh] script unhidden.
+
+[NOTE]
+.Note
+====
+These directories had their permissions set to `750` and now they should be changed to `755`.
+The directories in question can be found here:
+
+`/pub/fedora/linux/releases/`
+
+`/pub/alt/releases/`
+
+`/pub/fedora-secondary/releases/`
+
+This step ensures that the content officially becomes public and everything is accessible after the release GO decision.
+====
+
+[NOTE]
+.Note
+====
+This should be done late on Monday night, before the release.
+This is to allow mirrors that are not tier 1 and do not have access to pre-release content to start syncing up
+so that more of them are in sync by the time of the actual release announcement.
+We typically do not announce this widely either.
+Mirrors will just start syncing it, we don't need to tell users about it.
+====
+
+== Release Day (Tuesday @ 14:00 UTC)
+
+?
 
 == Post Release