Update info about mass rebuild and beta freeze
Signed-off-by: Tomas Hrcka <thrcka@redhat.com>
This commit is contained in:
parent
a5a2851d50
commit
9af9868aa1
2 changed files with 411 additions and 3 deletions
|
|
@ -1,9 +1,155 @@
|
|||
include::_partials/attributes.adoc[]
|
||||
|
||||
== Fedora Beta Freeze
|
||||
== Beta freeze and Bodhi Activation Point
|
||||
|
||||
=== Ansible changes
|
||||
=== Description
|
||||
|
||||
=== Update bodhi release
|
||||
Bodhi must be activated after two weeks of
|
||||
https://docs.pagure.org/releng/sop_mass_branching.html[Mass Branching]
|
||||
at 14:00 UTC.
|
||||
|
||||
=== Action
|
||||
|
||||
==== Making koji changes
|
||||
|
||||
Make the following koji tag changes
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ koji remove-tag-inheritance f{branched}-updates-candidate f{branched}
|
||||
$ koji remove-tag-inheritance f{branched}-updates-testing f{branched}
|
||||
$ koji remove-tag-inheritance f{branched}-updates-pending f{branched}
|
||||
$ koji remove-tag-inheritance f{branched}-override f{branched}
|
||||
$ koji add-tag-inheritance f{branched}-updates-candidate f{branched}-updates
|
||||
$ koji add-tag-inheritance f{branched}-updates-testing f{branched}-updates
|
||||
$ koji add-tag-inheritance f{branched}-updates-pending f{branched}-updates
|
||||
$ koji add-tag-inheritance f{branched}-override f{branched}-updates
|
||||
$ koji edit-tag --perm=admin f{branched}
|
||||
....
|
||||
|
||||
==== Update bodhi rpm release
|
||||
|
||||
Set the bodhi rpm to release to not to automatically create the update
|
||||
and also bodhi knows to compose the updates
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ bodhi releases edit --name "F{branched}" --stable-tag f{branched}-updates --testing-repository updates-testing --package-manager dnf --no-create-automatic-updates --composed-by-bodhi --state frozen
|
||||
....
|
||||
|
||||
==== Add the modular release
|
||||
|
||||
Run the following command on your own workstation to add the modular
|
||||
release
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ bodhi releases create --name F{branched}M --long-name "Fedora {branched} Modular" --id-prefix FEDORA-MODULAR --version {branched} --branch f{branched}m --dist-tag f{branched}-modular --stable-tag f{branched}-modular-updates --testing-tag f{branched}-modular-updates-testing --candidate-tag f{branched}-modular-updates-candidate --pending-stable-tag f{branched}-modular-updates-pending --pending-testing-tag f{branched}-modular-updates-testing-pending --pending-signing-tag f{branched}-modular-signing-pending --override-tag f{branched}-modular-override --state pending --user <fas username>
|
||||
....
|
||||
|
||||
Please update fas account username in above command.
|
||||
|
||||
[WARNING]
|
||||
.Warning
|
||||
====
|
||||
Due to a https://github.com/fedora-infra/bodhi/issues/2177[bug] in
|
||||
Bodhi, it is critical that Bodhi processes be restarted any time
|
||||
`bodhi releases create` or `bodhi releases edit` are used.
|
||||
====
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Add the container and flatpak releases if they weren't already added to
|
||||
bodhi
|
||||
====
|
||||
|
||||
=== Update vars
|
||||
|
||||
Update the _FedoraBranchedBodhi_ and _RelEngFrozen_ vars in infra
|
||||
ansible
|
||||
|
||||
=== Update all relevant projects in ansible
|
||||
As in https://pagure.io/fedora-infra/ansible/pull-request/1327[this Ansible Pull request] create changes for the {branched} release
|
||||
==== Run the playbooks
|
||||
|
||||
....
|
||||
$ rbac-playbook openshift-apps/greenwave.yml
|
||||
$ rbac-playbook openshift-apps/bodhi.yml
|
||||
$ rbac-playbook groups/bodhi-backend.yml
|
||||
$ rbac-playbook groups/releng-compose.yml
|
||||
$ rbac-playbook manual/autosign.yml
|
||||
....
|
||||
|
||||
Greenwave runs in OpenShift (as implied by the playbook paths), and so
|
||||
the change will not be live right away when the playbook finishes. You
|
||||
can monitor
|
||||
https://greenwave-web-greenwave.app.os.fedoraproject.org/api/v1.0/policies
|
||||
to wait for the new policy to appear (it should take a few minutes).
|
||||
|
||||
==== Restart bodhi services
|
||||
|
||||
Restart bodhi services to understand the bodhi new release on
|
||||
bodhi-backend01 (Look at warning in
|
||||
https://docs.pagure.org/releng/sop_bodhi_activation.html#action and the
|
||||
bug is https://github.com/fedora-infra/bodhi/issues/2177)
|
||||
|
||||
....
|
||||
$ sudo systemctl restart bodhi-celery
|
||||
$ sudo systemctl restart fm-consumer@config
|
||||
$ sudo systemctl restart koji-sync-listener
|
||||
....
|
||||
|
||||
==== Send Announcement
|
||||
|
||||
Email *devel-announce* and *test-announce* lists about Bodhi Activation.
|
||||
Please find the body of the email in templates dir in https://pagure.io/releng/blob/main/f/mail-templates/04-beta-freeze.txt[releng repository]
|
||||
|
||||
|
||||
=== Verification
|
||||
|
||||
Compare koji tagging structure with older release
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ koji list-tag-inheritance {branched} --reverse
|
||||
$ koji list-tag-inheritance {current} --reverse
|
||||
....
|
||||
|
||||
Compare the bodhi release with older release
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ bodhi releases info {branched}
|
||||
$ bodhi releases info {current}
|
||||
....
|
||||
|
||||
Check for other variants like modular, container and flatpaks
|
||||
|
||||
=== Process stable push requests
|
||||
|
||||
During feezes we need to push to stable builds included in the compose.
|
||||
QA will file a ticket with the nvrs to push.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
If you are pushing a bodhi update that contains multiple builds, you
|
||||
need only pass bodhi-push a single build nvr and all the others in that
|
||||
update will be detected and pushed along with it. However, if you are
|
||||
pushing multiple disjoint bodhi updates then each build will need to be
|
||||
listed individually.
|
||||
====
|
||||
|
||||
....
|
||||
$ sudo -u apache bodhi-push --builds '<nvr1>,<nvr2>,...' --username <username>
|
||||
....
|
||||
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
No considerations at this time. The docs git repository is simply a
|
||||
static html hosting space and we can just re-render the docs and push to
|
||||
it again if necessary.
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -24,3 +24,265 @@ Check with secondary arches, whether they up-to-date enough with primary, create
|
|||
|
||||
The following steps may be completed in the weeks leading up to the scheduled mass rebuild.
|
||||
|
||||
== Mass Rebuild
|
||||
|
||||
=== Description
|
||||
|
||||
Periodically we do mass rebuilds of rawhide during the development
|
||||
cycle. This SOP will outline the steps necessary to do this.
|
||||
|
||||
=== Assumptions
|
||||
|
||||
This assumes that the mass rebuild has already been approved and
|
||||
scheduled via release engineering and FESCo. Coordinate with
|
||||
infrastructure as well for any needed koji updates.
|
||||
|
||||
This also assumes that the mass rebuild does not need to be done in
|
||||
dependency order, and that the mass rebuild does not involve a ABI
|
||||
change.
|
||||
|
||||
=== Considerations
|
||||
|
||||
* The most important thing to keep in mind while doing a mass rebuild is
|
||||
to communicate clearly what actions are being performed and the status
|
||||
of the rebuild.
|
||||
* Check in on scripts frequently to avoid a long stalled command from
|
||||
adding significant delays in completing the rebuild.
|
||||
* Check with secondary arches, whether they up-to-date enough with
|
||||
primary, create rebuild tag and target when they are. It will then take
|
||||
care of rebuilds of the arch specific packages in appropriate kojis.
|
||||
|
||||
=== Actions
|
||||
|
||||
==== Preparatory Steps
|
||||
|
||||
The following steps may be completed in the weeks leading up to the
|
||||
scheduled mass rebuild.
|
||||
|
||||
. Create the Mass Rebuild Pagure Issue
|
||||
+
|
||||
____
|
||||
Create an issue on the https://pagure.io/releng/issues[Release
|
||||
Engineering issues page] that points at the schedule for the current
|
||||
release.
|
||||
|
||||
See https://pagure.io/releng/issue/6898[the Fedora 27 mass rebuild issue
|
||||
example].
|
||||
____
|
||||
. Set up the Mass Rebuild Wiki Page
|
||||
+
|
||||
____
|
||||
The mass rebuild wiki page should answer the following questions for
|
||||
maintainers:
|
||||
|
||||
* Why the mass rebuild is happening
|
||||
* How to opt out of the mass rebuild
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
See https://fedoraproject.org/wiki/Fedora_26_Mass_Rebuild[the Fedora 26
|
||||
Wiki example].
|
||||
====
|
||||
____
|
||||
. Send out the Mass Rebuild Notice
|
||||
+
|
||||
____
|
||||
Send out the same information posted on the wiki to the
|
||||
[.title-ref]#devel-announce@lists.fedoraproject.org# mailing list.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
See
|
||||
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[the
|
||||
Fedora 26 e-mail example].
|
||||
====
|
||||
____
|
||||
. Create a Tag to Contain the Mass Rebuild
|
||||
+
|
||||
____
|
||||
Mass rebuilds require their own tag to contain all related builds. The
|
||||
example assumes we are doing a rebuild for Fedora 26.
|
||||
|
||||
....
|
||||
$ koji add-tag f26-rebuild --parent f26
|
||||
....
|
||||
____
|
||||
. Request Package Auto-Signing for New Mass-Rebuild Tag
|
||||
+
|
||||
____
|
||||
File a ticket with https://pagure.io/fedora-infrastructure/issues[Fedora
|
||||
Infrastructure] requesting the new mass-rebuild tag be enabled for
|
||||
package auto-signing.
|
||||
____
|
||||
. Create the Koji Target for the Mass Rebuild
|
||||
+
|
||||
____
|
||||
Using the same [.title-ref]#f26-rebuild# tag created in the previous
|
||||
example:
|
||||
|
||||
....
|
||||
$ koji add-target f26-rebuild f26-build
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
*koji add-target* _target-name_ _buildroot-tag_ _destination-tag_
|
||||
describes the syntax format above. If the _destination-tag_ is not
|
||||
specified then it will be the same as the _target-name_.
|
||||
====
|
||||
____
|
||||
. Update Scripts
|
||||
+
|
||||
____
|
||||
The mass rebuild depends on four main scripts from the
|
||||
https://pagure.io/releng[releng git repository]. Each one requires some
|
||||
changes in variables for each new mass rebuild cycle.
|
||||
|
||||
* {blank}
|
||||
+
|
||||
mass-rebuild.py::
|
||||
** buildtag
|
||||
** targets
|
||||
** epoch
|
||||
** comment
|
||||
** target
|
||||
* {blank}
|
||||
+
|
||||
find-failures.py::
|
||||
** buildtag
|
||||
** desttag
|
||||
** epoch
|
||||
* mass-tag.py
|
||||
* {blank}
|
||||
+
|
||||
need-rebuild.py::
|
||||
** buildtag
|
||||
** target
|
||||
** updates
|
||||
** epoch
|
||||
____
|
||||
|
||||
Change the following items:
|
||||
|
||||
* the build tag, holding tag, and target tag should be updated to
|
||||
reflect the Fedora release you're building for
|
||||
* the `epoch` should be updated to the point at which all features that
|
||||
the mass rebuild is for have landed in the build system (and a newRepo
|
||||
task completed with those features)
|
||||
* the comment which is inserted into spec changelogs
|
||||
|
||||
==== Starting the Mass Rebuild
|
||||
|
||||
The `mass-rebuild.py` script takes care of:
|
||||
|
||||
* Discovering available packages in koji
|
||||
* Trimming out packages which have already been rebuilt
|
||||
* Checking out packages from git
|
||||
* Bumping the spec file
|
||||
* Committing the change
|
||||
* git tagging the change
|
||||
* Submitting the build request to Koji
|
||||
|
||||
. Connect to the mass-rebuild Machine
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ssh branched-composer.phx2.fedoraproject.org
|
||||
....
|
||||
____
|
||||
. Start a terminal multiplexer
|
||||
+
|
||||
____
|
||||
....
|
||||
$ tmux
|
||||
....
|
||||
____
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository].
|
||||
. Run the mass-rebuild.py script from _releng/scripts_
|
||||
+
|
||||
____
|
||||
....
|
||||
$ cd path/to/releng_repo/scripts
|
||||
$ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out
|
||||
....
|
||||
____
|
||||
|
||||
==== Monitoring Mass Rebuilds
|
||||
|
||||
The community has a very high interest in the status of rebuilds and
|
||||
many maintainers will want to know if their build failed right away. The
|
||||
`find-failures.py` and `need-rebuild.py` scripts are designed to update
|
||||
publicly available URLs for stakeholders to monitor.
|
||||
|
||||
. Connect to a Compose Machine
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ssh compose-x86-02.phx2.fedoraproject.org
|
||||
....
|
||||
____
|
||||
. Start a terminal multiplexer
|
||||
+
|
||||
____
|
||||
....
|
||||
$ tmux
|
||||
....
|
||||
____
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository]
|
||||
. {blank}
|
||||
+
|
||||
Set Up the Rebuild Failures Notification Web Site::
|
||||
The `find_failures.py` script discovers attempted builds that have
|
||||
failed. It lists those failed builds and sorts them by package owner.
|
||||
+
|
||||
....
|
||||
$ while true; do ./find_failures.py > f26-failures.html && cp f26-failures.html /mnt/koji/mass-rebuild/f26-failures.html; sleep 600; done
|
||||
....
|
||||
. Start a second pane in the terminal emulator
|
||||
. {blank}
|
||||
+
|
||||
Set up the Site for Packages that Need Rebuilt::
|
||||
The `need-rebuild.py` script discovers packages that have not yet been
|
||||
rebuilt and generates an html file listing them sorted by package
|
||||
owner. This gives external stakeholders a rough idea of how much work
|
||||
is remaining in the mass rebuild.
|
||||
+
|
||||
....
|
||||
$ while true; do ./need-rebuild.py > f26-need-rebuild.html && cp f26-need-rebuild.html /mnt/koji/mass-rebuild/f26-need-rebuild.html; sleep 600; done
|
||||
....
|
||||
|
||||
==== Post Mass Rebuild Tasks
|
||||
|
||||
Once the mass rebuild script completes, and all the pending builds have
|
||||
finished, the builds will need to be tagged. The `mass-tag.py` script
|
||||
will accomplish this task. The script will:
|
||||
|
||||
* Discover completed builds
|
||||
* Trim out builds that are older than the latest build for a given
|
||||
package
|
||||
* Tag remaining builds into their final destination (without generating
|
||||
email)
|
||||
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository]
|
||||
. Run the `mass-tag.py` script (requires koji kerberos authentication)
|
||||
+
|
||||
____
|
||||
....
|
||||
$ cd path/to/releng_repo/scripts
|
||||
$ ./mass-tag.py --source f36-rebuild --target f36
|
||||
....
|
||||
____
|
||||
. Send the final notification to the
|
||||
_devel-announce@lists.fedoraproject.org_ list
|
||||
+
|
||||
____
|
||||
The contents should look something like this
|
||||
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[example
|
||||
email].
|
||||
____
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue