277 lines
No EOL
8.2 KiB
Text
277 lines
No EOL
8.2 KiB
Text
include::_partials/attributes.adoc[]
|
|
|
|
= Mass Rebuild
|
|
|
|
== Description
|
|
|
|
Periodically we do mass rebuilds of rawhide during the development
|
|
cycle. This SOP will outline the steps necessary to do this.
|
|
|
|
== Assumptions
|
|
|
|
This assumes that the mass rebuild has already been approved and
|
|
scheduled via release engineering and FESCo. Coordinate with
|
|
infrastructure as well for any needed koji updates.
|
|
|
|
This also assumes that the mass rebuild does not need to be done in
|
|
dependency order, and that the mass rebuild does not involve a ABI
|
|
change.
|
|
|
|
== Considerations
|
|
|
|
* The most important thing to keep in mind while doing a mass rebuild is
|
|
to communicate clearly what actions are being performed and the status
|
|
of the rebuild.
|
|
* Check in on scripts frequently to avoid a long stalled command from
|
|
adding significant delays in completing the rebuild.
|
|
* Check with secondary arches, whether they up-to-date enough with
|
|
primary, create rebuild tag and target when they are. It will then take
|
|
care of rebuilds of the arch specific packages in appropriate kojis.
|
|
* Please make sure you have right ACLs to run the script mention below, one needs to be atleast in sysadmin-releng/releng-team/cvsadmin/sysadmin in order to get root access of the machine if required.
|
|
|
|
== Actions
|
|
|
|
=== Preparatory Steps
|
|
|
|
The following steps may be completed in the weeks leading up to the
|
|
scheduled mass rebuild.
|
|
|
|
. Create the Mass Rebuild Pagure Issue
|
|
+
|
|
____
|
|
Create an issue on the https://pagure.io/releng/issues[Release
|
|
Engineering issues page] that points at the schedule for the current
|
|
release.
|
|
|
|
See https://pagure.io/releng/issue/6898[the Fedora 27 mass rebuild issue
|
|
example].
|
|
____
|
|
. Set up the Mass Rebuild Wiki Page
|
|
+
|
|
____
|
|
The mass rebuild wiki page should answer the following questions for
|
|
maintainers:
|
|
|
|
* Why the mass rebuild is happening?
|
|
* How to opt out of the mass rebuild?
|
|
* Make sure to inform people either on mail, or in the wiki page about the `PKG_SKIP_LIST` which allows packages to be skipped from rebuilding as well as `noautobuild` file that they can add it to there root of a distgit repository and the mass rebuild script will skip it.
|
|
* What changes from changesets are taken in account for F{branched}?
|
|
|
|
[NOTE,subs="attributes+"]
|
|
====
|
|
See https://fedoraproject.org/wiki/Fedora_{branched}_Mass_Rebuild[the Fedora {branched}
|
|
Wiki example].
|
|
====
|
|
____
|
|
. Send out the Mass Rebuild Notice
|
|
+
|
|
____
|
|
Send out the same information posted on the wiki to the
|
|
[.title-ref]#devel-announce@lists.fedoraproject.org# mailing list. Please check if the email is through, if stuck in moderation please ping respective members.
|
|
|
|
[NOTE]
|
|
====
|
|
See the template in releng repository
|
|
https://pagure.io/releng/blob/main/f/mail-templates/01-mass-rebuild-start.txt[template].
|
|
====
|
|
____
|
|
. Create a Tag to Contain the Mass Rebuild
|
|
+
|
|
____
|
|
Mass rebuilds require their own tag to contain all related builds. The
|
|
example assumes we are doing a rebuild for Fedora {rawhide}.
|
|
|
|
[source,subs="attributes+"]
|
|
....
|
|
$ koji add-tag f{rawhide}-rebuild --parent f{rawhide}
|
|
....
|
|
____
|
|
. Request Package Auto-Signing for New Mass-Rebuild Tag
|
|
+
|
|
____
|
|
File a ticket with https://pagure.io/fedora-infrastructure/issues[Fedora
|
|
Infrastructure] requesting the new mass-rebuild tag be enabled for
|
|
package auto-signing.
|
|
____
|
|
. Create the Koji Target for the Mass Rebuild
|
|
+
|
|
____
|
|
Using the same [.title-ref]#f{rawhide}-rebuild# tag created in the previous
|
|
example:
|
|
|
|
[source,subs="attributes+"]
|
|
....
|
|
$ koji add-target f{rawhide}-rebuild f{rawhide}-build
|
|
....
|
|
|
|
[NOTE]
|
|
====
|
|
*koji add-target* _target-name_ _buildroot-tag_ _destination-tag_
|
|
describes the syntax format above. If the _destination-tag_ is not
|
|
specified then it will be the same as the _target-name_.
|
|
====
|
|
____
|
|
. Update Scripts
|
|
+
|
|
____
|
|
The mass rebuild depends on four main scripts from the
|
|
https://pagure.io/releng[releng git repository]. Each one requires some
|
|
changes in variables for each new mass rebuild cycle.
|
|
|
|
* {blank}
|
|
+
|
|
mass-rebuild.py::
|
|
** buildtag
|
|
** targets
|
|
** epoch
|
|
** comment
|
|
** target
|
|
* {blank}
|
|
+
|
|
find-failures.py::
|
|
** buildtag
|
|
** desttag
|
|
** epoch
|
|
* mass-tag.py
|
|
* {blank}
|
|
+
|
|
need-rebuild.py::
|
|
** buildtag
|
|
** target
|
|
** updates
|
|
** epoch
|
|
____
|
|
|
|
Change the following items:
|
|
|
|
* the build tag, holding tag, and target tag should be updated to
|
|
reflect the Fedora release you're building for
|
|
* the `epoch` should be updated to the point at which all features that
|
|
the mass rebuild is for have landed in the build system (and a newRepo
|
|
task completed with those features)
|
|
* the comment which is inserted into spec changelogs
|
|
* for the references of the above mentioned changes refer to the https://pagure.io/releng/c/f8d89a58efc0b81b9358baadfeb2ba4f52edeeb4?branch=main[F40 mass rebuild PR].
|
|
|
|
=== Starting the Mass Rebuild
|
|
|
|
The `mass-rebuild.py` script takes care of:
|
|
|
|
* Discovering available packages in koji
|
|
* Trimming out packages which have already been rebuilt
|
|
* Checking out packages from git
|
|
* Bumping the spec file
|
|
* Committing the change
|
|
* git tagging the change
|
|
* Submitting the build request to Koji
|
|
|
|
=====
|
|
. Connect to the mass-rebuild Machine
|
|
+
|
|
____
|
|
....
|
|
$ ssh compose-branched01.rdu3.fedoraproject.org
|
|
....
|
|
____
|
|
. Start a terminal multiplexer (this ensures if user gets interrupted due to various reasons, the script can continue in a tmux session)
|
|
+
|
|
____
|
|
....
|
|
$ tmux
|
|
....
|
|
____
|
|
. Clone or checkout the latest copy of the
|
|
https://pagure.io/releng[releng git repository].
|
|
. Run the mass-rebuild.py script from _releng/scripts_
|
|
+
|
|
____
|
|
....
|
|
$ cd path/to/releng_repo/scripts
|
|
$ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out
|
|
....
|
|
____
|
|
=====
|
|
|
|
=== Monitoring Mass Rebuilds
|
|
|
|
The community has a very high interest in the status of rebuilds and
|
|
many maintainers will want to know if their build failed right away. The
|
|
`find-failures.py` and `need-rebuild.py` scripts are designed to update
|
|
publicly available URLs for stakeholders to monitor.
|
|
|
|
=====
|
|
. Connect to a Compose Machine or the monitoring scripts can be run on the same machine where the mass rebuild is happening.
|
|
+
|
|
____
|
|
....
|
|
$ ssh compose-x86-02.rdu3.fedoraproject.org / compose-branched01.rdu3.fedoraproject.org:22
|
|
....
|
|
____
|
|
. Start a terminal multiplexer (this ensures if user gets interrupted due to various reasons, the script can continue in a tmux session)
|
|
+
|
|
____
|
|
....
|
|
$ tmux
|
|
....
|
|
____
|
|
. Clone or checkout the latest copy of the
|
|
https://pagure.io/releng[releng git repository]
|
|
. {blank}
|
|
+
|
|
Set Up the Rebuild Failures Notification Web Site::
|
|
The `find_failures.py` script discovers attempted builds that have
|
|
failed. It lists those failed builds and sorts them by package owner.
|
|
+
|
|
|
|
[source,subs="attributes+"]
|
|
....
|
|
$ while true; do ./find_failures.py > f{rawhide}-failures.html && cp f{rawhide}-failures.html /mnt/koji/mass-rebuild/f{rawhide}-failures.html; sleep 600; done
|
|
....
|
|
. Start a second pane in the terminal emulator
|
|
. {blank}
|
|
+
|
|
Set up the Site for Packages that Need Rebuilt::
|
|
The `need-rebuild.py` script discovers packages that have not yet been
|
|
rebuilt and generates an html file listing them sorted by package
|
|
owner. This gives external stakeholders a rough idea of how much work
|
|
is remaining in the mass rebuild.
|
|
+
|
|
|
|
[source,subs="attributes+"]
|
|
....
|
|
$ while true; do ./need-rebuild.py > f{rawhide}-need-rebuild.html && cp f{rawhide}-need-rebuild.html /mnt/koji/mass-rebuild/f{rawhide}-need-rebuild.html; sleep 600; done
|
|
....
|
|
=====
|
|
|
|
=== Post Mass Rebuild Tasks
|
|
|
|
Once the mass rebuild script completes, and all the pending builds have
|
|
finished, the builds will need to be tagged. The `mass-tag.py` script
|
|
will accomplish this task. The script will:
|
|
|
|
* Discover completed builds
|
|
* Trim out builds that are older than the latest build for a given
|
|
package
|
|
* Tag remaining builds into their final destination (without generating
|
|
email)
|
|
|
|
=====
|
|
. Clone or checkout the latest copy of the
|
|
https://pagure.io/releng[releng git repository]
|
|
. Run the `mass-tag.py` script (requires koji kerberos authentication)
|
|
. Remove the koji target for the rebuild:
|
|
$ koji remove-target f{rawhide}-rebuild
|
|
+
|
|
____
|
|
[source,subs="attributes+"]
|
|
....
|
|
$ cd path/to/releng_repo/scripts
|
|
$ ./mass-tag.py --source f{rawhide}-rebuild --target f{rawhide}
|
|
....
|
|
____
|
|
. Send the final notification to the
|
|
_devel-announce@lists.fedoraproject.org_ list
|
|
+
|
|
____
|
|
For the contents use this https://pagure.io/releng/blob/main/f/mail-templates/02-mass-rebuild-finished.txt[template].
|
|
____
|
|
===== |