diff --git a/modules/release_guide/pages/beta_freeze.adoc b/modules/release_guide/pages/beta_freeze.adoc index 240969e..e8ad880 100644 --- a/modules/release_guide/pages/beta_freeze.adoc +++ b/modules/release_guide/pages/beta_freeze.adoc @@ -1,9 +1,155 @@ include::_partials/attributes.adoc[] -== Fedora Beta Freeze +== Beta freeze and Bodhi Activation Point -=== Ansible changes +=== Description -=== Update bodhi release +Bodhi must be activated after two weeks of +https://docs.pagure.org/releng/sop_mass_branching.html[Mass Branching] +at 14:00 UTC. + +=== Action + +==== Making koji changes + +Make the following koji tag changes + +[source,subs="attributes+"] +.... +$ koji remove-tag-inheritance f{branched}-updates-candidate f{branched} +$ koji remove-tag-inheritance f{branched}-updates-testing f{branched} +$ koji remove-tag-inheritance f{branched}-updates-pending f{branched} +$ koji remove-tag-inheritance f{branched}-override f{branched} +$ koji add-tag-inheritance f{branched}-updates-candidate f{branched}-updates +$ koji add-tag-inheritance f{branched}-updates-testing f{branched}-updates +$ koji add-tag-inheritance f{branched}-updates-pending f{branched}-updates +$ koji add-tag-inheritance f{branched}-override f{branched}-updates +$ koji edit-tag --perm=admin f{branched} +.... + +==== Update bodhi rpm release + +Set the bodhi rpm to release to not to automatically create the update +and also bodhi knows to compose the updates + +[source,subs="attributes+"] +.... +$ bodhi releases edit --name "F{branched}" --stable-tag f{branched}-updates --testing-repository updates-testing --package-manager dnf --no-create-automatic-updates --composed-by-bodhi --state frozen +.... + +==== Add the modular release + +Run the following command on your own workstation to add the modular +release + +[source,subs="attributes+"] +.... +$ bodhi releases create --name F{branched}M --long-name "Fedora {branched} Modular" --id-prefix FEDORA-MODULAR --version {branched} --branch f{branched}m --dist-tag f{branched}-modular --stable-tag f{branched}-modular-updates --testing-tag f{branched}-modular-updates-testing --candidate-tag f{branched}-modular-updates-candidate --pending-stable-tag f{branched}-modular-updates-pending --pending-testing-tag f{branched}-modular-updates-testing-pending --pending-signing-tag f{branched}-modular-signing-pending --override-tag f{branched}-modular-override --state pending --user +.... + +Please update fas account username in above command. + +[WARNING] +.Warning +==== +Due to a https://github.com/fedora-infra/bodhi/issues/2177[bug] in +Bodhi, it is critical that Bodhi processes be restarted any time +`bodhi releases create` or `bodhi releases edit` are used. +==== + +[NOTE] +.Note +==== +Add the container and flatpak releases if they weren't already added to +bodhi +==== + +=== Update vars + +Update the _FedoraBranchedBodhi_ and _RelEngFrozen_ vars in infra +ansible + +=== Update all relevant projects in ansible +As in https://pagure.io/fedora-infra/ansible/pull-request/1327[this Ansible Pull request] create changes for the {branched} release +==== Run the playbooks + +.... +$ rbac-playbook openshift-apps/greenwave.yml +$ rbac-playbook openshift-apps/bodhi.yml +$ rbac-playbook groups/bodhi-backend.yml +$ rbac-playbook groups/releng-compose.yml +$ rbac-playbook manual/autosign.yml +.... + +Greenwave runs in OpenShift (as implied by the playbook paths), and so +the change will not be live right away when the playbook finishes. You +can monitor +https://greenwave-web-greenwave.app.os.fedoraproject.org/api/v1.0/policies +to wait for the new policy to appear (it should take a few minutes). + +==== Restart bodhi services + +Restart bodhi services to understand the bodhi new release on +bodhi-backend01 (Look at warning in +https://docs.pagure.org/releng/sop_bodhi_activation.html#action and the +bug is https://github.com/fedora-infra/bodhi/issues/2177) + +.... +$ sudo systemctl restart bodhi-celery +$ sudo systemctl restart fm-consumer@config +$ sudo systemctl restart koji-sync-listener +.... + +==== Send Announcement + +Email *devel-announce* and *test-announce* lists about Bodhi Activation. +Please find the body of the email in templates dir in https://pagure.io/releng/blob/main/f/mail-templates/04-beta-freeze.txt[releng repository] + + +=== Verification + +Compare koji tagging structure with older release + +[source,subs="attributes+"] +.... +$ koji list-tag-inheritance {branched} --reverse +$ koji list-tag-inheritance {current} --reverse +.... + +Compare the bodhi release with older release + +[source,subs="attributes+"] +.... +$ bodhi releases info {branched} +$ bodhi releases info {current} +.... + +Check for other variants like modular, container and flatpaks === Process stable push requests + +During feezes we need to push to stable builds included in the compose. +QA will file a ticket with the nvrs to push. + +[NOTE] +.Note +==== +If you are pushing a bodhi update that contains multiple builds, you +need only pass bodhi-push a single build nvr and all the others in that +update will be detected and pushed along with it. However, if you are +pushing multiple disjoint bodhi updates then each build will need to be +listed individually. +==== + +.... +$ sudo -u apache bodhi-push --builds ',,...' --username +.... + + +=== Consider Before Running + +No considerations at this time. The docs git repository is simply a +static html hosting space and we can just re-render the docs and push to +it again if necessary. + + diff --git a/modules/release_guide/pages/mass_rebuild.adoc b/modules/release_guide/pages/mass_rebuild.adoc index 1f4f984..766c26f 100644 --- a/modules/release_guide/pages/mass_rebuild.adoc +++ b/modules/release_guide/pages/mass_rebuild.adoc @@ -24,3 +24,265 @@ Check with secondary arches, whether they up-to-date enough with primary, create The following steps may be completed in the weeks leading up to the scheduled mass rebuild. +== Mass Rebuild + +=== Description + +Periodically we do mass rebuilds of rawhide during the development +cycle. This SOP will outline the steps necessary to do this. + +=== Assumptions + +This assumes that the mass rebuild has already been approved and +scheduled via release engineering and FESCo. Coordinate with +infrastructure as well for any needed koji updates. + +This also assumes that the mass rebuild does not need to be done in +dependency order, and that the mass rebuild does not involve a ABI +change. + +=== Considerations + +* The most important thing to keep in mind while doing a mass rebuild is +to communicate clearly what actions are being performed and the status +of the rebuild. +* Check in on scripts frequently to avoid a long stalled command from +adding significant delays in completing the rebuild. +* Check with secondary arches, whether they up-to-date enough with +primary, create rebuild tag and target when they are. It will then take +care of rebuilds of the arch specific packages in appropriate kojis. + +=== Actions + +==== Preparatory Steps + +The following steps may be completed in the weeks leading up to the +scheduled mass rebuild. + +. Create the Mass Rebuild Pagure Issue ++ +____ +Create an issue on the https://pagure.io/releng/issues[Release +Engineering issues page] that points at the schedule for the current +release. + +See https://pagure.io/releng/issue/6898[the Fedora 27 mass rebuild issue +example]. +____ +. Set up the Mass Rebuild Wiki Page ++ +____ +The mass rebuild wiki page should answer the following questions for +maintainers: + +* Why the mass rebuild is happening +* How to opt out of the mass rebuild + +[NOTE] +.Note +==== +See https://fedoraproject.org/wiki/Fedora_26_Mass_Rebuild[the Fedora 26 +Wiki example]. +==== +____ +. Send out the Mass Rebuild Notice ++ +____ +Send out the same information posted on the wiki to the +[.title-ref]#devel-announce@lists.fedoraproject.org# mailing list. + +[NOTE] +.Note +==== +See +https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[the +Fedora 26 e-mail example]. +==== +____ +. Create a Tag to Contain the Mass Rebuild ++ +____ +Mass rebuilds require their own tag to contain all related builds. The +example assumes we are doing a rebuild for Fedora 26. + +.... +$ koji add-tag f26-rebuild --parent f26 +.... +____ +. Request Package Auto-Signing for New Mass-Rebuild Tag ++ +____ +File a ticket with https://pagure.io/fedora-infrastructure/issues[Fedora +Infrastructure] requesting the new mass-rebuild tag be enabled for +package auto-signing. +____ +. Create the Koji Target for the Mass Rebuild ++ +____ +Using the same [.title-ref]#f26-rebuild# tag created in the previous +example: + +.... +$ koji add-target f26-rebuild f26-build +.... + +[NOTE] +.Note +==== +*koji add-target* _target-name_ _buildroot-tag_ _destination-tag_ +describes the syntax format above. If the _destination-tag_ is not +specified then it will be the same as the _target-name_. +==== +____ +. Update Scripts ++ +____ +The mass rebuild depends on four main scripts from the +https://pagure.io/releng[releng git repository]. Each one requires some +changes in variables for each new mass rebuild cycle. + +* {blank} ++ +mass-rebuild.py:: + ** buildtag + ** targets + ** epoch + ** comment + ** target +* {blank} ++ +find-failures.py:: + ** buildtag + ** desttag + ** epoch +* mass-tag.py +* {blank} ++ +need-rebuild.py:: + ** buildtag + ** target + ** updates + ** epoch +____ + +Change the following items: + +* the build tag, holding tag, and target tag should be updated to +reflect the Fedora release you're building for +* the `epoch` should be updated to the point at which all features that +the mass rebuild is for have landed in the build system (and a newRepo +task completed with those features) +* the comment which is inserted into spec changelogs + +==== Starting the Mass Rebuild + +The `mass-rebuild.py` script takes care of: + +* Discovering available packages in koji +* Trimming out packages which have already been rebuilt +* Checking out packages from git +* Bumping the spec file +* Committing the change +* git tagging the change +* Submitting the build request to Koji + +. Connect to the mass-rebuild Machine ++ +____ +.... +$ ssh branched-composer.phx2.fedoraproject.org +.... +____ +. Start a terminal multiplexer ++ +____ +.... +$ tmux +.... +____ +. Clone or checkout the latest copy of the +https://pagure.io/releng[releng git repository]. +. Run the mass-rebuild.py script from _releng/scripts_ ++ +____ +.... +$ cd path/to/releng_repo/scripts +$ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out +.... +____ + +==== Monitoring Mass Rebuilds + +The community has a very high interest in the status of rebuilds and +many maintainers will want to know if their build failed right away. The +`find-failures.py` and `need-rebuild.py` scripts are designed to update +publicly available URLs for stakeholders to monitor. + +. Connect to a Compose Machine ++ +____ +.... +$ ssh compose-x86-02.phx2.fedoraproject.org +.... +____ +. Start a terminal multiplexer ++ +____ +.... +$ tmux +.... +____ +. Clone or checkout the latest copy of the +https://pagure.io/releng[releng git repository] +. {blank} ++ +Set Up the Rebuild Failures Notification Web Site:: + The `find_failures.py` script discovers attempted builds that have + failed. It lists those failed builds and sorts them by package owner. + + +.... +$ while true; do ./find_failures.py > f26-failures.html && cp f26-failures.html /mnt/koji/mass-rebuild/f26-failures.html; sleep 600; done +.... +. Start a second pane in the terminal emulator +. {blank} ++ +Set up the Site for Packages that Need Rebuilt:: + The `need-rebuild.py` script discovers packages that have not yet been + rebuilt and generates an html file listing them sorted by package + owner. This gives external stakeholders a rough idea of how much work + is remaining in the mass rebuild. + + +.... +$ while true; do ./need-rebuild.py > f26-need-rebuild.html && cp f26-need-rebuild.html /mnt/koji/mass-rebuild/f26-need-rebuild.html; sleep 600; done +.... + +==== Post Mass Rebuild Tasks + +Once the mass rebuild script completes, and all the pending builds have +finished, the builds will need to be tagged. The `mass-tag.py` script +will accomplish this task. The script will: + +* Discover completed builds +* Trim out builds that are older than the latest build for a given +package +* Tag remaining builds into their final destination (without generating +email) + +. Clone or checkout the latest copy of the +https://pagure.io/releng[releng git repository] +. Run the `mass-tag.py` script (requires koji kerberos authentication) ++ +____ +.... +$ cd path/to/releng_repo/scripts +$ ./mass-tag.py --source f36-rebuild --target f36 +.... +____ +. Send the final notification to the +_devel-announce@lists.fedoraproject.org_ list ++ +____ +The contents should look something like this +https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[example +email]. +____