Update info about mass rebuild and beta freeze
Signed-off-by: Tomas Hrcka <thrcka@redhat.com>
This commit is contained in:
parent
a5a2851d50
commit
9af9868aa1
2 changed files with 411 additions and 3 deletions
|
|
@ -1,9 +1,155 @@
|
||||||
include::_partials/attributes.adoc[]
|
include::_partials/attributes.adoc[]
|
||||||
|
|
||||||
== Fedora Beta Freeze
|
== Beta freeze and Bodhi Activation Point
|
||||||
|
|
||||||
=== Ansible changes
|
=== Description
|
||||||
|
|
||||||
=== Update bodhi release
|
Bodhi must be activated after two weeks of
|
||||||
|
https://docs.pagure.org/releng/sop_mass_branching.html[Mass Branching]
|
||||||
|
at 14:00 UTC.
|
||||||
|
|
||||||
|
=== Action
|
||||||
|
|
||||||
|
==== Making koji changes
|
||||||
|
|
||||||
|
Make the following koji tag changes
|
||||||
|
|
||||||
|
[source,subs="attributes+"]
|
||||||
|
....
|
||||||
|
$ koji remove-tag-inheritance f{branched}-updates-candidate f{branched}
|
||||||
|
$ koji remove-tag-inheritance f{branched}-updates-testing f{branched}
|
||||||
|
$ koji remove-tag-inheritance f{branched}-updates-pending f{branched}
|
||||||
|
$ koji remove-tag-inheritance f{branched}-override f{branched}
|
||||||
|
$ koji add-tag-inheritance f{branched}-updates-candidate f{branched}-updates
|
||||||
|
$ koji add-tag-inheritance f{branched}-updates-testing f{branched}-updates
|
||||||
|
$ koji add-tag-inheritance f{branched}-updates-pending f{branched}-updates
|
||||||
|
$ koji add-tag-inheritance f{branched}-override f{branched}-updates
|
||||||
|
$ koji edit-tag --perm=admin f{branched}
|
||||||
|
....
|
||||||
|
|
||||||
|
==== Update bodhi rpm release
|
||||||
|
|
||||||
|
Set the bodhi rpm to release to not to automatically create the update
|
||||||
|
and also bodhi knows to compose the updates
|
||||||
|
|
||||||
|
[source,subs="attributes+"]
|
||||||
|
....
|
||||||
|
$ bodhi releases edit --name "F{branched}" --stable-tag f{branched}-updates --testing-repository updates-testing --package-manager dnf --no-create-automatic-updates --composed-by-bodhi --state frozen
|
||||||
|
....
|
||||||
|
|
||||||
|
==== Add the modular release
|
||||||
|
|
||||||
|
Run the following command on your own workstation to add the modular
|
||||||
|
release
|
||||||
|
|
||||||
|
[source,subs="attributes+"]
|
||||||
|
....
|
||||||
|
$ bodhi releases create --name F{branched}M --long-name "Fedora {branched} Modular" --id-prefix FEDORA-MODULAR --version {branched} --branch f{branched}m --dist-tag f{branched}-modular --stable-tag f{branched}-modular-updates --testing-tag f{branched}-modular-updates-testing --candidate-tag f{branched}-modular-updates-candidate --pending-stable-tag f{branched}-modular-updates-pending --pending-testing-tag f{branched}-modular-updates-testing-pending --pending-signing-tag f{branched}-modular-signing-pending --override-tag f{branched}-modular-override --state pending --user <fas username>
|
||||||
|
....
|
||||||
|
|
||||||
|
Please update fas account username in above command.
|
||||||
|
|
||||||
|
[WARNING]
|
||||||
|
.Warning
|
||||||
|
====
|
||||||
|
Due to a https://github.com/fedora-infra/bodhi/issues/2177[bug] in
|
||||||
|
Bodhi, it is critical that Bodhi processes be restarted any time
|
||||||
|
`bodhi releases create` or `bodhi releases edit` are used.
|
||||||
|
====
|
||||||
|
|
||||||
|
[NOTE]
|
||||||
|
.Note
|
||||||
|
====
|
||||||
|
Add the container and flatpak releases if they weren't already added to
|
||||||
|
bodhi
|
||||||
|
====
|
||||||
|
|
||||||
|
=== Update vars
|
||||||
|
|
||||||
|
Update the _FedoraBranchedBodhi_ and _RelEngFrozen_ vars in infra
|
||||||
|
ansible
|
||||||
|
|
||||||
|
=== Update all relevant projects in ansible
|
||||||
|
As in https://pagure.io/fedora-infra/ansible/pull-request/1327[this Ansible Pull request] create changes for the {branched} release
|
||||||
|
==== Run the playbooks
|
||||||
|
|
||||||
|
....
|
||||||
|
$ rbac-playbook openshift-apps/greenwave.yml
|
||||||
|
$ rbac-playbook openshift-apps/bodhi.yml
|
||||||
|
$ rbac-playbook groups/bodhi-backend.yml
|
||||||
|
$ rbac-playbook groups/releng-compose.yml
|
||||||
|
$ rbac-playbook manual/autosign.yml
|
||||||
|
....
|
||||||
|
|
||||||
|
Greenwave runs in OpenShift (as implied by the playbook paths), and so
|
||||||
|
the change will not be live right away when the playbook finishes. You
|
||||||
|
can monitor
|
||||||
|
https://greenwave-web-greenwave.app.os.fedoraproject.org/api/v1.0/policies
|
||||||
|
to wait for the new policy to appear (it should take a few minutes).
|
||||||
|
|
||||||
|
==== Restart bodhi services
|
||||||
|
|
||||||
|
Restart bodhi services to understand the bodhi new release on
|
||||||
|
bodhi-backend01 (Look at warning in
|
||||||
|
https://docs.pagure.org/releng/sop_bodhi_activation.html#action and the
|
||||||
|
bug is https://github.com/fedora-infra/bodhi/issues/2177)
|
||||||
|
|
||||||
|
....
|
||||||
|
$ sudo systemctl restart bodhi-celery
|
||||||
|
$ sudo systemctl restart fm-consumer@config
|
||||||
|
$ sudo systemctl restart koji-sync-listener
|
||||||
|
....
|
||||||
|
|
||||||
|
==== Send Announcement
|
||||||
|
|
||||||
|
Email *devel-announce* and *test-announce* lists about Bodhi Activation.
|
||||||
|
Please find the body of the email in templates dir in https://pagure.io/releng/blob/main/f/mail-templates/04-beta-freeze.txt[releng repository]
|
||||||
|
|
||||||
|
|
||||||
|
=== Verification
|
||||||
|
|
||||||
|
Compare koji tagging structure with older release
|
||||||
|
|
||||||
|
[source,subs="attributes+"]
|
||||||
|
....
|
||||||
|
$ koji list-tag-inheritance {branched} --reverse
|
||||||
|
$ koji list-tag-inheritance {current} --reverse
|
||||||
|
....
|
||||||
|
|
||||||
|
Compare the bodhi release with older release
|
||||||
|
|
||||||
|
[source,subs="attributes+"]
|
||||||
|
....
|
||||||
|
$ bodhi releases info {branched}
|
||||||
|
$ bodhi releases info {current}
|
||||||
|
....
|
||||||
|
|
||||||
|
Check for other variants like modular, container and flatpaks
|
||||||
|
|
||||||
=== Process stable push requests
|
=== Process stable push requests
|
||||||
|
|
||||||
|
During feezes we need to push to stable builds included in the compose.
|
||||||
|
QA will file a ticket with the nvrs to push.
|
||||||
|
|
||||||
|
[NOTE]
|
||||||
|
.Note
|
||||||
|
====
|
||||||
|
If you are pushing a bodhi update that contains multiple builds, you
|
||||||
|
need only pass bodhi-push a single build nvr and all the others in that
|
||||||
|
update will be detected and pushed along with it. However, if you are
|
||||||
|
pushing multiple disjoint bodhi updates then each build will need to be
|
||||||
|
listed individually.
|
||||||
|
====
|
||||||
|
|
||||||
|
....
|
||||||
|
$ sudo -u apache bodhi-push --builds '<nvr1>,<nvr2>,...' --username <username>
|
||||||
|
....
|
||||||
|
|
||||||
|
|
||||||
|
=== Consider Before Running
|
||||||
|
|
||||||
|
No considerations at this time. The docs git repository is simply a
|
||||||
|
static html hosting space and we can just re-render the docs and push to
|
||||||
|
it again if necessary.
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -24,3 +24,265 @@ Check with secondary arches, whether they up-to-date enough with primary, create
|
||||||
|
|
||||||
The following steps may be completed in the weeks leading up to the scheduled mass rebuild.
|
The following steps may be completed in the weeks leading up to the scheduled mass rebuild.
|
||||||
|
|
||||||
|
== Mass Rebuild
|
||||||
|
|
||||||
|
=== Description
|
||||||
|
|
||||||
|
Periodically we do mass rebuilds of rawhide during the development
|
||||||
|
cycle. This SOP will outline the steps necessary to do this.
|
||||||
|
|
||||||
|
=== Assumptions
|
||||||
|
|
||||||
|
This assumes that the mass rebuild has already been approved and
|
||||||
|
scheduled via release engineering and FESCo. Coordinate with
|
||||||
|
infrastructure as well for any needed koji updates.
|
||||||
|
|
||||||
|
This also assumes that the mass rebuild does not need to be done in
|
||||||
|
dependency order, and that the mass rebuild does not involve a ABI
|
||||||
|
change.
|
||||||
|
|
||||||
|
=== Considerations
|
||||||
|
|
||||||
|
* The most important thing to keep in mind while doing a mass rebuild is
|
||||||
|
to communicate clearly what actions are being performed and the status
|
||||||
|
of the rebuild.
|
||||||
|
* Check in on scripts frequently to avoid a long stalled command from
|
||||||
|
adding significant delays in completing the rebuild.
|
||||||
|
* Check with secondary arches, whether they up-to-date enough with
|
||||||
|
primary, create rebuild tag and target when they are. It will then take
|
||||||
|
care of rebuilds of the arch specific packages in appropriate kojis.
|
||||||
|
|
||||||
|
=== Actions
|
||||||
|
|
||||||
|
==== Preparatory Steps
|
||||||
|
|
||||||
|
The following steps may be completed in the weeks leading up to the
|
||||||
|
scheduled mass rebuild.
|
||||||
|
|
||||||
|
. Create the Mass Rebuild Pagure Issue
|
||||||
|
+
|
||||||
|
____
|
||||||
|
Create an issue on the https://pagure.io/releng/issues[Release
|
||||||
|
Engineering issues page] that points at the schedule for the current
|
||||||
|
release.
|
||||||
|
|
||||||
|
See https://pagure.io/releng/issue/6898[the Fedora 27 mass rebuild issue
|
||||||
|
example].
|
||||||
|
____
|
||||||
|
. Set up the Mass Rebuild Wiki Page
|
||||||
|
+
|
||||||
|
____
|
||||||
|
The mass rebuild wiki page should answer the following questions for
|
||||||
|
maintainers:
|
||||||
|
|
||||||
|
* Why the mass rebuild is happening
|
||||||
|
* How to opt out of the mass rebuild
|
||||||
|
|
||||||
|
[NOTE]
|
||||||
|
.Note
|
||||||
|
====
|
||||||
|
See https://fedoraproject.org/wiki/Fedora_26_Mass_Rebuild[the Fedora 26
|
||||||
|
Wiki example].
|
||||||
|
====
|
||||||
|
____
|
||||||
|
. Send out the Mass Rebuild Notice
|
||||||
|
+
|
||||||
|
____
|
||||||
|
Send out the same information posted on the wiki to the
|
||||||
|
[.title-ref]#devel-announce@lists.fedoraproject.org# mailing list.
|
||||||
|
|
||||||
|
[NOTE]
|
||||||
|
.Note
|
||||||
|
====
|
||||||
|
See
|
||||||
|
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[the
|
||||||
|
Fedora 26 e-mail example].
|
||||||
|
====
|
||||||
|
____
|
||||||
|
. Create a Tag to Contain the Mass Rebuild
|
||||||
|
+
|
||||||
|
____
|
||||||
|
Mass rebuilds require their own tag to contain all related builds. The
|
||||||
|
example assumes we are doing a rebuild for Fedora 26.
|
||||||
|
|
||||||
|
....
|
||||||
|
$ koji add-tag f26-rebuild --parent f26
|
||||||
|
....
|
||||||
|
____
|
||||||
|
. Request Package Auto-Signing for New Mass-Rebuild Tag
|
||||||
|
+
|
||||||
|
____
|
||||||
|
File a ticket with https://pagure.io/fedora-infrastructure/issues[Fedora
|
||||||
|
Infrastructure] requesting the new mass-rebuild tag be enabled for
|
||||||
|
package auto-signing.
|
||||||
|
____
|
||||||
|
. Create the Koji Target for the Mass Rebuild
|
||||||
|
+
|
||||||
|
____
|
||||||
|
Using the same [.title-ref]#f26-rebuild# tag created in the previous
|
||||||
|
example:
|
||||||
|
|
||||||
|
....
|
||||||
|
$ koji add-target f26-rebuild f26-build
|
||||||
|
....
|
||||||
|
|
||||||
|
[NOTE]
|
||||||
|
.Note
|
||||||
|
====
|
||||||
|
*koji add-target* _target-name_ _buildroot-tag_ _destination-tag_
|
||||||
|
describes the syntax format above. If the _destination-tag_ is not
|
||||||
|
specified then it will be the same as the _target-name_.
|
||||||
|
====
|
||||||
|
____
|
||||||
|
. Update Scripts
|
||||||
|
+
|
||||||
|
____
|
||||||
|
The mass rebuild depends on four main scripts from the
|
||||||
|
https://pagure.io/releng[releng git repository]. Each one requires some
|
||||||
|
changes in variables for each new mass rebuild cycle.
|
||||||
|
|
||||||
|
* {blank}
|
||||||
|
+
|
||||||
|
mass-rebuild.py::
|
||||||
|
** buildtag
|
||||||
|
** targets
|
||||||
|
** epoch
|
||||||
|
** comment
|
||||||
|
** target
|
||||||
|
* {blank}
|
||||||
|
+
|
||||||
|
find-failures.py::
|
||||||
|
** buildtag
|
||||||
|
** desttag
|
||||||
|
** epoch
|
||||||
|
* mass-tag.py
|
||||||
|
* {blank}
|
||||||
|
+
|
||||||
|
need-rebuild.py::
|
||||||
|
** buildtag
|
||||||
|
** target
|
||||||
|
** updates
|
||||||
|
** epoch
|
||||||
|
____
|
||||||
|
|
||||||
|
Change the following items:
|
||||||
|
|
||||||
|
* the build tag, holding tag, and target tag should be updated to
|
||||||
|
reflect the Fedora release you're building for
|
||||||
|
* the `epoch` should be updated to the point at which all features that
|
||||||
|
the mass rebuild is for have landed in the build system (and a newRepo
|
||||||
|
task completed with those features)
|
||||||
|
* the comment which is inserted into spec changelogs
|
||||||
|
|
||||||
|
==== Starting the Mass Rebuild
|
||||||
|
|
||||||
|
The `mass-rebuild.py` script takes care of:
|
||||||
|
|
||||||
|
* Discovering available packages in koji
|
||||||
|
* Trimming out packages which have already been rebuilt
|
||||||
|
* Checking out packages from git
|
||||||
|
* Bumping the spec file
|
||||||
|
* Committing the change
|
||||||
|
* git tagging the change
|
||||||
|
* Submitting the build request to Koji
|
||||||
|
|
||||||
|
. Connect to the mass-rebuild Machine
|
||||||
|
+
|
||||||
|
____
|
||||||
|
....
|
||||||
|
$ ssh branched-composer.phx2.fedoraproject.org
|
||||||
|
....
|
||||||
|
____
|
||||||
|
. Start a terminal multiplexer
|
||||||
|
+
|
||||||
|
____
|
||||||
|
....
|
||||||
|
$ tmux
|
||||||
|
....
|
||||||
|
____
|
||||||
|
. Clone or checkout the latest copy of the
|
||||||
|
https://pagure.io/releng[releng git repository].
|
||||||
|
. Run the mass-rebuild.py script from _releng/scripts_
|
||||||
|
+
|
||||||
|
____
|
||||||
|
....
|
||||||
|
$ cd path/to/releng_repo/scripts
|
||||||
|
$ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out
|
||||||
|
....
|
||||||
|
____
|
||||||
|
|
||||||
|
==== Monitoring Mass Rebuilds
|
||||||
|
|
||||||
|
The community has a very high interest in the status of rebuilds and
|
||||||
|
many maintainers will want to know if their build failed right away. The
|
||||||
|
`find-failures.py` and `need-rebuild.py` scripts are designed to update
|
||||||
|
publicly available URLs for stakeholders to monitor.
|
||||||
|
|
||||||
|
. Connect to a Compose Machine
|
||||||
|
+
|
||||||
|
____
|
||||||
|
....
|
||||||
|
$ ssh compose-x86-02.phx2.fedoraproject.org
|
||||||
|
....
|
||||||
|
____
|
||||||
|
. Start a terminal multiplexer
|
||||||
|
+
|
||||||
|
____
|
||||||
|
....
|
||||||
|
$ tmux
|
||||||
|
....
|
||||||
|
____
|
||||||
|
. Clone or checkout the latest copy of the
|
||||||
|
https://pagure.io/releng[releng git repository]
|
||||||
|
. {blank}
|
||||||
|
+
|
||||||
|
Set Up the Rebuild Failures Notification Web Site::
|
||||||
|
The `find_failures.py` script discovers attempted builds that have
|
||||||
|
failed. It lists those failed builds and sorts them by package owner.
|
||||||
|
+
|
||||||
|
....
|
||||||
|
$ while true; do ./find_failures.py > f26-failures.html && cp f26-failures.html /mnt/koji/mass-rebuild/f26-failures.html; sleep 600; done
|
||||||
|
....
|
||||||
|
. Start a second pane in the terminal emulator
|
||||||
|
. {blank}
|
||||||
|
+
|
||||||
|
Set up the Site for Packages that Need Rebuilt::
|
||||||
|
The `need-rebuild.py` script discovers packages that have not yet been
|
||||||
|
rebuilt and generates an html file listing them sorted by package
|
||||||
|
owner. This gives external stakeholders a rough idea of how much work
|
||||||
|
is remaining in the mass rebuild.
|
||||||
|
+
|
||||||
|
....
|
||||||
|
$ while true; do ./need-rebuild.py > f26-need-rebuild.html && cp f26-need-rebuild.html /mnt/koji/mass-rebuild/f26-need-rebuild.html; sleep 600; done
|
||||||
|
....
|
||||||
|
|
||||||
|
==== Post Mass Rebuild Tasks
|
||||||
|
|
||||||
|
Once the mass rebuild script completes, and all the pending builds have
|
||||||
|
finished, the builds will need to be tagged. The `mass-tag.py` script
|
||||||
|
will accomplish this task. The script will:
|
||||||
|
|
||||||
|
* Discover completed builds
|
||||||
|
* Trim out builds that are older than the latest build for a given
|
||||||
|
package
|
||||||
|
* Tag remaining builds into their final destination (without generating
|
||||||
|
email)
|
||||||
|
|
||||||
|
. Clone or checkout the latest copy of the
|
||||||
|
https://pagure.io/releng[releng git repository]
|
||||||
|
. Run the `mass-tag.py` script (requires koji kerberos authentication)
|
||||||
|
+
|
||||||
|
____
|
||||||
|
....
|
||||||
|
$ cd path/to/releng_repo/scripts
|
||||||
|
$ ./mass-tag.py --source f36-rebuild --target f36
|
||||||
|
....
|
||||||
|
____
|
||||||
|
. Send the final notification to the
|
||||||
|
_devel-announce@lists.fedoraproject.org_ list
|
||||||
|
+
|
||||||
|
____
|
||||||
|
The contents should look something like this
|
||||||
|
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[example
|
||||||
|
email].
|
||||||
|
____
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue