[releng] Update mass rebuild SOP

Signed-off-by: Tomas Hrcka <thrcka@redhat.com>

modules/release_guide/pages/mass_rebuild.adoc

@@ -1,16 +1,18 @@
-= Mass rebuild information.
+include::_partials/attributes.adoc[]
 
-== Description
+== Mass rebuild information.
+
+=== Description
 
 Periodically we do mass rebuilds of rawhide during the development cycle. This SOP will outline the steps necessary to do this.
 
-== Assumptions
+=== Assumptions
 
 This assumes that the mass rebuild has already been approved and scheduled via release engineering and FESCo. Coordinate with infrastructure as well for any needed koji updates.
 
 This also assumes that the mass rebuild does not need to be done in dependency order, and that the mass rebuild does not involve a ABI change.
 
-== Considerations
+=== Considerations
 
 The most important thing to keep in mind while doing a mass rebuild is to communicate clearly what actions are being performed and the status of the rebuild.
 
@@ -18,20 +20,20 @@ Check in on scripts frequently to avoid a long stalled command from adding signi
 
 Check with secondary arches, whether they up-to-date enough with primary, create rebuild tag and target when they are. It will then take care of rebuilds of the arch specific packages in appropriate kojis.
 
-== Actions
+=== Actions
 
-=== Preparatory Steps
+==== Preparatory Steps
 
 The following steps may be completed in the weeks leading up to the scheduled mass rebuild.
 
-== Mass Rebuild
+=== Mass Rebuild
 
-=== Description
+==== Description
 
 Periodically we do mass rebuilds of rawhide during the development
 cycle. This SOP will outline the steps necessary to do this.
 
-=== Assumptions
+==== Assumptions
 
 This assumes that the mass rebuild has already been approved and
 scheduled via release engineering and FESCo. Coordinate with
@@ -41,7 +43,7 @@ This also assumes that the mass rebuild does not need to be done in
 dependency order, and that the mass rebuild does not involve a ABI
 change.
 
-=== Considerations
+==== Considerations
 
 * The most important thing to keep in mind while doing a mass rebuild is
 to communicate clearly what actions are being performed and the status
@@ -52,9 +54,9 @@ adding significant delays in completing the rebuild.
 primary, create rebuild tag and target when they are. It will then take
 care of rebuilds of the arch specific packages in appropriate kojis.
 
-=== Actions
+==== Actions
 
-==== Preparatory Steps
+===== Preparatory Steps
 
 The following steps may be completed in the weeks leading up to the
 scheduled mass rebuild.
@@ -78,10 +80,10 @@ maintainers:
 * Why the mass rebuild is happening
 * How to opt out of the mass rebuild
 
-[NOTE]
+[NOTE,subs="attributes+"]
 .Note
 ====
-See https://fedoraproject.org/wiki/Fedora_26_Mass_Rebuild[the Fedora 26
+See https://fedoraproject.org/wiki/Fedora_{branched}_Mass_Rebuild[the Fedora {branched}
 Wiki example].
 ====
 ____
@@ -94,19 +96,19 @@ Send out the same information posted on the wiki to the
 [NOTE]
 .Note
 ====
-See
+See the template in releng repository
-https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[the
+https://pagure.io/releng/blob/main/f/mail-templates/01-mass-rebuild-start.txt[template].
-Fedora 26 e-mail example].
 ====
 ____
 . Create a Tag to Contain the Mass Rebuild
 +
 ____
 Mass rebuilds require their own tag to contain all related builds. The
-example assumes we are doing a rebuild for Fedora 26.
+example assumes we are doing a rebuild for Fedora {rawhide}.
 
+[source,subs="attributes+"]
 ....
-$ koji add-tag f26-rebuild --parent f26
+$ koji add-tag f{rawhide}-rebuild --parent f{rawhide}
 ....
 ____
 . Request Package Auto-Signing for New Mass-Rebuild Tag
@@ -119,11 +121,12 @@ ____
 . Create the Koji Target for the Mass Rebuild
 +
 ____
-Using the same [.title-ref]#f26-rebuild# tag created in the previous
+Using the same [.title-ref]#f{rawhide}-rebuild# tag created in the previous
 example:
 
+[source,subs="attributes+"]
 ....
-$ koji add-target f26-rebuild f26-build
+$ koji add-target f{rawhide}-rebuild f{rawhide}-build
 ....
 
 [NOTE]
@@ -174,7 +177,7 @@ the mass rebuild is for have landed in the build system (and a newRepo
 task completed with those features)
 * the comment which is inserted into spec changelogs
 
-==== Starting the Mass Rebuild
+===== Starting the Mass Rebuild
 
 The `mass-rebuild.py` script takes care of:
 
@@ -190,7 +193,7 @@ The `mass-rebuild.py` script takes care of:
 +
 ____
 ....
-$ ssh branched-composer.phx2.fedoraproject.org
+$ ssh branched-composer.iad2.fedoraproject.org
 ....
 ____
 . Start a terminal multiplexer
@@ -211,7 +214,7 @@ $ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out
 ....
 ____
 
-==== Monitoring Mass Rebuilds
+===== Monitoring Mass Rebuilds
 
 The community has a very high interest in the status of rebuilds and
 many maintainers will want to know if their build failed right away. The
@@ -222,7 +225,7 @@ publicly available URLs for stakeholders to monitor.
 +
 ____
 ....
-$ ssh compose-x86-02.phx2.fedoraproject.org
+$ ssh compose-x86-02.iad2.fedoraproject.org
 ....
 ____
 . Start a terminal multiplexer
@@ -240,8 +243,10 @@ Set Up the Rebuild Failures Notification Web Site::
   The `find_failures.py` script discovers attempted builds that have
   failed. It lists those failed builds and sorts them by package owner.
   +
+
+[source,subs="attributes+"]
 ....
-$ while true; do ./find_failures.py > f26-failures.html && cp f26-failures.html /mnt/koji/mass-rebuild/f26-failures.html; sleep 600; done
+$ while true; do ./find_failures.py > f{rawhide}-failures.html && cp f{rawhide}-failures.html /mnt/koji/mass-rebuild/f{rawhide}-failures.html; sleep 600; done
 ....
 . Start a second pane in the terminal emulator
 . {blank}
@@ -252,11 +257,13 @@ Set up the Site for Packages that Need Rebuilt::
   owner. This gives external stakeholders a rough idea of how much work
   is remaining in the mass rebuild.
   +
+
+[source,subs="attributes+"]
 ....
-$ while true; do ./need-rebuild.py > f26-need-rebuild.html && cp f26-need-rebuild.html /mnt/koji/mass-rebuild/f26-need-rebuild.html; sleep 600; done
+$ while true; do ./need-rebuild.py > f{rawhide}-need-rebuild.html && cp f{rawhide}-need-rebuild.html /mnt/koji/mass-rebuild/f{rawhide}-need-rebuild.html; sleep 600; done
 ....
 
-==== Post Mass Rebuild Tasks
+===== Post Mass Rebuild Tasks
 
 Once the mass rebuild script completes, and all the pending builds have
 finished, the builds will need to be tagged. The `mass-tag.py` script
@@ -273,9 +280,10 @@ https://pagure.io/releng[releng git repository]
 . Run the `mass-tag.py` script (requires koji kerberos authentication)
 +
 ____
+[source,subs="attributes+"]
 ....
 $ cd path/to/releng_repo/scripts
-$ ./mass-tag.py --source f36-rebuild --target f36
+$ ./mass-tag.py --source f{rawhide}-rebuild --target f{rawhide}
 ....
 ____
 . Send the final notification to the

modules/release_guide/pages/mass_rebuild_modules.adoc

@@ -1,122 +0,0 @@
-== Mass Rebuild of Modules
-
-=== Description
-
-Periodically we do mass rebuilds of modules in rawhide during the
-development cycle. This SOP will outline the steps necessary to do this.
-
-=== Assumptions
-
-This assumes that the mass rebuild has already been approved and
-scheduled via release engineering and FESCo. Coordinate with
-infrastructure as well for any needed updates.
-
-=== Considerations
-
-* The most important thing to keep in mind while doing a mass rebuild is
-to communicate clearly what actions are being performed and the status
-of the rebuild.
-* Check in on scripts frequently to avoid a long stalled command from
-adding significant delays in completing the rebuild.
-
-=== Actions
-
-==== Preparatory Steps
-
-The following steps should be completed after the
-https://docs.pagure.org/releng/sop_mass_rebuild_packages.html[mass
-rebuild of packages] is done.
-
-. Update Scripts
-
-The mass rebuild depends on two main scripts from the
-https://pagure.io/releng[releng git repository]. Each one requires some
-changes in variables for each new mass rebuild cycle.
-
-____
-* {blank}
-+
-_mass-rebuild-modules.py_::
-  ** rebuildid
-* {blank}
-+
-_massrebuildsinfo.py_::
-  ** module_mass_rebuild_epoch
-  ** module_mass_rebuild_platform
-____
-
-Change the following items:
-
-* the `rebuildid` to match the release for which you are mass rebuilding
-modules as per in massrebuildsinfo.py
-* `module_mass_rebuild_epoch` mostly will be the epoch of mass rebuild
-of packages
-* `module_mass_rebuild_platform` should be the rawhide module platform
-
-==== Starting the Mass Rebuild of Modules
-
-The `mass-rebuild-modules.py` script takes care of:
-
-* Discovering available available modules from PDC
-* Find the module info from mbs and check if a module build is submitted
-after the epoch date
-* Checking out modules from dist-git
-* Switching to appropriate stream
-* Find modulemd file
-* Use libmodulemd to determine if this module stream applies to this
-platform version
-* If needs rebuilding, committing the change
-* Push the commit
-* Submitting the build request through mbs
-
-. Connect to the mass-rebuild Machine
-+
-____
-....
-$ ssh compose-branched01.iad2.fedoraproject.org
-....
-____
-. Start a terminal multiplexer
-+
-____
-....
-$ tmux
-....
-____
-. Clone or checkout the latest copy of the
-https://pagure.io/releng[releng git repository].
-. Run the [.title-ref]#mass-rebuild-modules.py# script from
-_releng/scripts_
-+
-____
-....
-$ cd path/to/releng_repo/scripts
-$ ./mass-rebuild-modules.py <path_to_token_file> build --wait 2>&1 | tee ~/massbuildmodules.out
-....
-____
-
-[NOTE]
-.Note
-====
-The token file should be located in infra's private ansible repo, or ask
-infra to get it to you using this
-https://pagure.io/fedora-infrastructure/issue/8048#comment-587789[process].
-====
-
-[NOTE]
-.Note
-====
-The [.title-ref]#build# option is really important to pay attention,
-since the mass branching of modules will also use the same script, just
-changing the option to [.title-ref]#branch# and
-[.title-ref]#module_mass_branching_platform# in
-[.title-ref]#massrebuildsinfo.py#
-====
-
-==== Post Mass Rebuild Tasks
-
-Once the module mass rebuild is done, send an email to the
-devel-announce@ list
-
-. Send the final notification to the
-_devel-announce@lists.fedoraproject.org_ list

modules/release_guide/pages/mass_rebuild_packages.adoc

@@ -1,266 +0,0 @@
-include::_partials/attributes.adoc[]
-
-== Mass Rebuild
-
-=== Description
-
-Periodically we do mass rebuilds of rawhide during the development
-cycle. This SOP will outline the steps necessary to do this.
-
-=== Assumptions
-
-This assumes that the mass rebuild has already been approved and
-scheduled via release engineering and FESCo. Coordinate with
-infrastructure as well for any needed koji updates.
-
-This also assumes that the mass rebuild does not need to be done in
-dependency order, and that the mass rebuild does not involve a ABI
-change.
-
-=== Considerations
-
-* The most important thing to keep in mind while doing a mass rebuild is
-to communicate clearly what actions are being performed and the status
-of the rebuild.
-* Check in on scripts frequently to avoid a long stalled command from
-adding significant delays in completing the rebuild.
-* Check with secondary arches, whether they up-to-date enough with
-primary, create rebuild tag and target when they are. It will then take
-care of rebuilds of the arch specific packages in appropriate kojis.
-
-=== Actions
-
-==== Preparatory Steps
-
-The following steps may be completed in the weeks leading up to the
-scheduled mass rebuild.
-
-. Create the Mass Rebuild Pagure Issue
-+
-____
-Create an issue on the https://pagure.io/releng/issues[Release
-Engineering issues page] that points at the schedule for the current
-release.
-
-See https://pagure.io/releng/issue/6898[the Fedora 27 mass rebuild issue
-example].
-____
-. Set up the Mass Rebuild Wiki Page
-+
-____
-The mass rebuild wiki page should answer the following questions for
-maintainers:
-
-* Why the mass rebuild is happening
-* How to opt out of the mass rebuild
-
-[NOTE]
-.Note
-====
-See https://fedoraproject.org/wiki/Fedora_26_Mass_Rebuild[the Fedora 26
-Wiki example].
-====
-____
-. Send out the Mass Rebuild Notice
-+
-____
-Send out the same information posted on the wiki to the
-[.title-ref]#devel-announce@lists.fedoraproject.org# mailing list.
-
-[NOTE]
-.Note
-====
-See
-https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[the
-Fedora 26 e-mail example].
-====
-____
-. Create a Tag to Contain the Mass Rebuild
-+
-____
-Mass rebuilds require their own tag to contain all related builds. The
-example assumes we are doing a rebuild for Fedora 26.
-
-....
-$ koji add-tag f26-rebuild --parent f26
-....
-____
-. Request Package Auto-Signing for New Mass-Rebuild Tag
-+
-____
-File a ticket with https://pagure.io/fedora-infrastructure/issues[Fedora
-Infrastructure] requesting the new mass-rebuild tag be enabled for
-package auto-signing.
-____
-. Create the Koji Target for the Mass Rebuild
-+
-____
-Using the same [.title-ref]#f26-rebuild# tag created in the previous
-example:
-
-....
-$ koji add-target f26-rebuild f26-build
-....
-
-[NOTE]
-.Note
-====
-*koji add-target* _target-name_ _buildroot-tag_ _destination-tag_
-describes the syntax format above. If the _destination-tag_ is not
-specified then it will be the same as the _target-name_.
-====
-____
-. Update Scripts
-+
-____
-The mass rebuild depends on four main scripts from the
-https://pagure.io/releng[releng git repository]. Each one requires some
-changes in variables for each new mass rebuild cycle.
-
-* {blank}
-+
-mass-rebuild.py::
-  ** buildtag
-  ** targets
-  ** epoch
-  ** comment
-  ** target
-* {blank}
-+
-find-failures.py::
-  ** buildtag
-  ** desttag
-  ** epoch
-* mass-tag.py
-* {blank}
-+
-need-rebuild.py::
-  ** buildtag
-  ** target
-  ** updates
-  ** epoch
-____
-
-Change the following items:
-
-* the build tag, holding tag, and target tag should be updated to
-reflect the Fedora release you're building for
-* the `epoch` should be updated to the point at which all features that
-the mass rebuild is for have landed in the build system (and a newRepo
-task completed with those features)
-* the comment which is inserted into spec changelogs
-
-==== Starting the Mass Rebuild
-
-The `mass-rebuild.py` script takes care of:
-
-* Discovering available packages in koji
-* Trimming out packages which have already been rebuilt
-* Checking out packages from git
-* Bumping the spec file
-* Committing the change
-* git tagging the change
-* Submitting the build request to Koji
-
-. Connect to the mass-rebuild Machine
-+
-____
-....
-$ ssh branched-composer.phx2.fedoraproject.org
-....
-____
-. Start a terminal multiplexer
-+
-____
-....
-$ tmux
-....
-____
-. Clone or checkout the latest copy of the
-https://pagure.io/releng[releng git repository].
-. Run the mass-rebuild.py script from _releng/scripts_
-+
-____
-....
-$ cd path/to/releng_repo/scripts
-$ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out
-....
-____
-
-==== Monitoring Mass Rebuilds
-
-The community has a very high interest in the status of rebuilds and
-many maintainers will want to know if their build failed right away. The
-`find-failures.py` and `need-rebuild.py` scripts are designed to update
-publicly available URLs for stakeholders to monitor.
-
-. Connect to a Compose Machine
-+
-____
-....
-$ ssh compose-x86-02.phx2.fedoraproject.org
-....
-____
-. Start a terminal multiplexer
-+
-____
-....
-$ tmux
-....
-____
-. Clone or checkout the latest copy of the
-https://pagure.io/releng[releng git repository]
-. {blank}
-+
-Set Up the Rebuild Failures Notification Web Site::
-  The `find_failures.py` script discovers attempted builds that have
-  failed. It lists those failed builds and sorts them by package owner.
-  +
-....
-$ while true; do ./find_failures.py > f26-failures.html && cp f26-failures.html /mnt/koji/mass-rebuild/f26-failures.html; sleep 600; done
-....
-. Start a second pane in the terminal emulator
-. {blank}
-+
-Set up the Site for Packages that Need Rebuilt::
-  The `need-rebuild.py` script discovers packages that have not yet been
-  rebuilt and generates an html file listing them sorted by package
-  owner. This gives external stakeholders a rough idea of how much work
-  is remaining in the mass rebuild.
-  +
-....
-$ while true; do ./need-rebuild.py > f26-need-rebuild.html && cp f26-need-rebuild.html /mnt/koji/mass-rebuild/f26-need-rebuild.html; sleep 600; done
-....
-
-==== Post Mass Rebuild Tasks
-
-Once the mass rebuild script completes, and all the pending builds have
-finished, the builds will need to be tagged. The `mass-tag.py` script
-will accomplish this task. The script will:
-
-* Discover completed builds
-* Trim out builds that are older than the latest build for a given
-package
-* Tag remaining builds into their final destination (without generating
-email)
-
-. Clone or checkout the latest copy of the
-https://pagure.io/releng[releng git repository]
-. Run the `mass-tag.py` script (requires koji kerberos authentication)
-+
-____
-
-[source,subs="attributes+"]
-....
-$ cd path/to/releng_repo/scripts
-$ ./mass-tag.py --source f{branched}-rebuild --target f{branched}
-....
-____
-. Send the final notification to the
-_devel-announce@lists.fedoraproject.org_ list
-+
-____
-The contents should look something like this
-https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[example
-email].
-____