include::_partials/attributes.adoc[]

= Mass Rebuild

== Description

Periodically we do mass rebuilds of rawhide during the development
cycle. This SOP will outline the steps necessary to do this.

== Assumptions

This assumes that the mass rebuild has already been approved and
scheduled via release engineering and FESCo. Coordinate with
infrastructure as well for any needed koji updates.

This also assumes that the mass rebuild does not need to be done in
dependency order, and that the mass rebuild does not involve a ABI
change.

== Considerations

* The most important thing to keep in mind while doing a mass rebuild is
to communicate clearly what actions are being performed and the status
of the rebuild.
* Check in on scripts frequently to avoid a long stalled command from
adding significant delays in completing the rebuild.
* Check with secondary arches, whether they up-to-date enough with
primary, create rebuild tag and target when they are. It will then take
care of rebuilds of the arch specific packages in appropriate kojis.
* Please make sure you have right ACLs to run the script mention below, one needs to be atleast in sysadmin-releng/releng-team/cvsadmin/sysadmin in order to get root access of the machine if required.

== Actions

=== Preparatory Steps

The following steps may be completed in the weeks leading up to the
scheduled mass rebuild.

. Create the Mass Rebuild Pagure Issue
+
____
Create an issue on the https://pagure.io/releng/issues[Release
Engineering issues page] that points at the schedule for the current
release.

See https://pagure.io/releng/issue/6898[the Fedora 27 mass rebuild issue
example].
____
. Set up the Mass Rebuild Wiki Page
+
____
The mass rebuild wiki page should answer the following questions for
maintainers:

* Why the mass rebuild is happening?
* How to opt out of the mass rebuild?
  * Make sure to inform people either on mail, or in the wiki page about the `PKG_SKIP_LIST` which allows packages to be skipped from rebuilding as well as `noautobuild` file that they can add it to there root of a distgit repository and the mass rebuild script will skip it.
* What changes from changesets are taken in account for F{branched}?

[NOTE,subs="attributes+"]
====
See https://fedoraproject.org/wiki/Fedora_{branched}_Mass_Rebuild[the Fedora {branched}
Wiki example].
====
____
. Send out the Mass Rebuild Notice
+
____
Send out the same information posted on the wiki to the
[.title-ref]#devel-announce@lists.fedoraproject.org# mailing list. Please check if the email is through, if stuck in moderation please ping respective members.

[NOTE]
====
See the template in releng repository
https://pagure.io/releng/blob/main/f/mail-templates/01-mass-rebuild-start.txt[template].
====
____
. Create a Tag to Contain the Mass Rebuild
+
____
Mass rebuilds require their own tag to contain all related builds. The
example assumes we are doing a rebuild for Fedora {rawhide}.

[source,subs="attributes+"]
....
$ koji add-tag f{rawhide}-rebuild --parent f{rawhide}
....
____
. Request Package Auto-Signing for New Mass-Rebuild Tag
+
____
File a ticket with https://pagure.io/fedora-infrastructure/issues[Fedora
Infrastructure] requesting the new mass-rebuild tag be enabled for
package auto-signing.
____
. Create the Koji Target for the Mass Rebuild
+
____
Using the same [.title-ref]#f{rawhide}-rebuild# tag created in the previous
example:

[source,subs="attributes+"]
....
$ koji add-target f{rawhide}-rebuild f{rawhide}-build
....

[NOTE]
====
*koji add-target* _target-name_ _buildroot-tag_ _destination-tag_
describes the syntax format above. If the _destination-tag_ is not
specified then it will be the same as the _target-name_.
====
____
. Update Scripts
+
____
The mass rebuild depends on four main scripts from the
https://pagure.io/releng[releng git repository]. Each one requires some
changes in variables for each new mass rebuild cycle.

* {blank}
+
mass-rebuild.py::
  ** buildtag
  ** targets
  ** epoch
  ** comment
  ** target
* {blank}
+
find-failures.py::
  ** buildtag
  ** desttag
  ** epoch
* mass-tag.py
* {blank}
+
need-rebuild.py::
  ** buildtag
  ** target
  ** updates
  ** epoch
____

Change the following items:

* the build tag, holding tag, and target tag should be updated to
reflect the Fedora release you're building for
* the `epoch` should be updated to the point at which all features that
the mass rebuild is for have landed in the build system (and a newRepo
task completed with those features)
* the comment which is inserted into spec changelogs
* for the references of the above mentioned changes refer to the https://pagure.io/releng/c/f8d89a58efc0b81b9358baadfeb2ba4f52edeeb4?branch=main[F40 mass rebuild PR].

=== Starting the Mass Rebuild

The `mass-rebuild.py` script takes care of:

* Discovering available packages in koji
* Trimming out packages which have already been rebuilt
* Checking out packages from git
* Bumping the spec file
* Committing the change
* git tagging the change
* Submitting the build request to Koji

=====
. Connect to the mass-rebuild Machine
+
____
....
$ ssh compose-branched01.iad2.fedoraproject.org
....
____
. Start a terminal multiplexer (this ensures if user gets interrupted due to various reasons, the script can continue in a tmux session)
+
____
....
$ tmux
....
____
. Clone or checkout the latest copy of the
https://pagure.io/releng[releng git repository].
. Run the mass-rebuild.py script from _releng/scripts_
+
____
....
$ cd path/to/releng_repo/scripts
$ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out
....
____
=====

=== Monitoring Mass Rebuilds

The community has a very high interest in the status of rebuilds and
many maintainers will want to know if their build failed right away. The
`find-failures.py` and `need-rebuild.py` scripts are designed to update
publicly available URLs for stakeholders to monitor.

=====
. Connect to a Compose Machine or the monitoring scripts can be run on the same machine where the mass rebuild is happening.
+
____
....
$ ssh compose-x86-02.iad2.fedoraproject.org / compose-branched01.iad2.fedoraproject.org:22
....
____
. Start a terminal multiplexer (this ensures if user gets interrupted due to various reasons, the script can continue in a tmux session)
+
____
....
$ tmux
....
____
. Clone or checkout the latest copy of the
https://pagure.io/releng[releng git repository]
. {blank}
+
Set Up the Rebuild Failures Notification Web Site::
  The `find_failures.py` script discovers attempted builds that have
  failed. It lists those failed builds and sorts them by package owner.
  +

[source,subs="attributes+"]
....
$ while true; do ./find_failures.py > f{rawhide}-failures.html && cp f{rawhide}-failures.html /mnt/koji/mass-rebuild/f{rawhide}-failures.html; sleep 600; done
....
. Start a second pane in the terminal emulator
. {blank}
+
Set up the Site for Packages that Need Rebuilt::
  The `need-rebuild.py` script discovers packages that have not yet been
  rebuilt and generates an html file listing them sorted by package
  owner. This gives external stakeholders a rough idea of how much work
  is remaining in the mass rebuild.
  +

[source,subs="attributes+"]
....
$ while true; do ./need-rebuild.py > f{rawhide}-need-rebuild.html && cp f{rawhide}-need-rebuild.html /mnt/koji/mass-rebuild/f{rawhide}-need-rebuild.html; sleep 600; done
....
=====

=== Post Mass Rebuild Tasks

Once the mass rebuild script completes, and all the pending builds have
finished, the builds will need to be tagged. The `mass-tag.py` script
will accomplish this task. The script will:

* Discover completed builds
* Trim out builds that are older than the latest build for a given
package
* Tag remaining builds into their final destination (without generating
email)

=====
. Clone or checkout the latest copy of the
https://pagure.io/releng[releng git repository]
. Run the `mass-tag.py` script (requires koji kerberos authentication)
. Remove the koji target for the rebuild: 
$ koji remove-target f{rawhide}-rebuild
+
____
[source,subs="attributes+"]
....
$ cd path/to/releng_repo/scripts
$ ./mass-tag.py --source f{rawhide}-rebuild --target f{rawhide}
....
____
. Send the final notification to the
_devel-announce@lists.fedoraproject.org_ list
+
____
For the contents use this https://pagure.io/releng/blob/main/f/mail-templates/02-mass-rebuild-finished.txt[template].
____
=====