[releng] Update mass rebuild SOP
Signed-off-by: Tomas Hrcka <thrcka@redhat.com>
This commit is contained in:
parent
783f0f00ef
commit
ae4028711b
3 changed files with 37 additions and 417 deletions
|
|
@ -1,16 +1,18 @@
|
|||
= Mass rebuild information.
|
||||
include::_partials/attributes.adoc[]
|
||||
|
||||
== Description
|
||||
== Mass rebuild information.
|
||||
|
||||
=== Description
|
||||
|
||||
Periodically we do mass rebuilds of rawhide during the development cycle. This SOP will outline the steps necessary to do this.
|
||||
|
||||
== Assumptions
|
||||
=== Assumptions
|
||||
|
||||
This assumes that the mass rebuild has already been approved and scheduled via release engineering and FESCo. Coordinate with infrastructure as well for any needed koji updates.
|
||||
|
||||
This also assumes that the mass rebuild does not need to be done in dependency order, and that the mass rebuild does not involve a ABI change.
|
||||
|
||||
== Considerations
|
||||
=== Considerations
|
||||
|
||||
The most important thing to keep in mind while doing a mass rebuild is to communicate clearly what actions are being performed and the status of the rebuild.
|
||||
|
||||
|
|
@ -18,20 +20,20 @@ Check in on scripts frequently to avoid a long stalled command from adding signi
|
|||
|
||||
Check with secondary arches, whether they up-to-date enough with primary, create rebuild tag and target when they are. It will then take care of rebuilds of the arch specific packages in appropriate kojis.
|
||||
|
||||
== Actions
|
||||
=== Actions
|
||||
|
||||
=== Preparatory Steps
|
||||
==== Preparatory Steps
|
||||
|
||||
The following steps may be completed in the weeks leading up to the scheduled mass rebuild.
|
||||
|
||||
== Mass Rebuild
|
||||
=== Mass Rebuild
|
||||
|
||||
=== Description
|
||||
==== Description
|
||||
|
||||
Periodically we do mass rebuilds of rawhide during the development
|
||||
cycle. This SOP will outline the steps necessary to do this.
|
||||
|
||||
=== Assumptions
|
||||
==== Assumptions
|
||||
|
||||
This assumes that the mass rebuild has already been approved and
|
||||
scheduled via release engineering and FESCo. Coordinate with
|
||||
|
|
@ -41,7 +43,7 @@ This also assumes that the mass rebuild does not need to be done in
|
|||
dependency order, and that the mass rebuild does not involve a ABI
|
||||
change.
|
||||
|
||||
=== Considerations
|
||||
==== Considerations
|
||||
|
||||
* The most important thing to keep in mind while doing a mass rebuild is
|
||||
to communicate clearly what actions are being performed and the status
|
||||
|
|
@ -52,9 +54,9 @@ adding significant delays in completing the rebuild.
|
|||
primary, create rebuild tag and target when they are. It will then take
|
||||
care of rebuilds of the arch specific packages in appropriate kojis.
|
||||
|
||||
=== Actions
|
||||
==== Actions
|
||||
|
||||
==== Preparatory Steps
|
||||
===== Preparatory Steps
|
||||
|
||||
The following steps may be completed in the weeks leading up to the
|
||||
scheduled mass rebuild.
|
||||
|
|
@ -78,10 +80,10 @@ maintainers:
|
|||
* Why the mass rebuild is happening
|
||||
* How to opt out of the mass rebuild
|
||||
|
||||
[NOTE]
|
||||
[NOTE,subs="attributes+"]
|
||||
.Note
|
||||
====
|
||||
See https://fedoraproject.org/wiki/Fedora_26_Mass_Rebuild[the Fedora 26
|
||||
See https://fedoraproject.org/wiki/Fedora_{branched}_Mass_Rebuild[the Fedora {branched}
|
||||
Wiki example].
|
||||
====
|
||||
____
|
||||
|
|
@ -94,19 +96,19 @@ Send out the same information posted on the wiki to the
|
|||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
See
|
||||
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[the
|
||||
Fedora 26 e-mail example].
|
||||
See the template in releng repository
|
||||
https://pagure.io/releng/blob/main/f/mail-templates/01-mass-rebuild-start.txt[template].
|
||||
====
|
||||
____
|
||||
. Create a Tag to Contain the Mass Rebuild
|
||||
+
|
||||
____
|
||||
Mass rebuilds require their own tag to contain all related builds. The
|
||||
example assumes we are doing a rebuild for Fedora 26.
|
||||
example assumes we are doing a rebuild for Fedora {rawhide}.
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ koji add-tag f26-rebuild --parent f26
|
||||
$ koji add-tag f{rawhide}-rebuild --parent f{rawhide}
|
||||
....
|
||||
____
|
||||
. Request Package Auto-Signing for New Mass-Rebuild Tag
|
||||
|
|
@ -119,11 +121,12 @@ ____
|
|||
. Create the Koji Target for the Mass Rebuild
|
||||
+
|
||||
____
|
||||
Using the same [.title-ref]#f26-rebuild# tag created in the previous
|
||||
Using the same [.title-ref]#f{rawhide}-rebuild# tag created in the previous
|
||||
example:
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ koji add-target f26-rebuild f26-build
|
||||
$ koji add-target f{rawhide}-rebuild f{rawhide}-build
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
|
|
@ -174,7 +177,7 @@ the mass rebuild is for have landed in the build system (and a newRepo
|
|||
task completed with those features)
|
||||
* the comment which is inserted into spec changelogs
|
||||
|
||||
==== Starting the Mass Rebuild
|
||||
===== Starting the Mass Rebuild
|
||||
|
||||
The `mass-rebuild.py` script takes care of:
|
||||
|
||||
|
|
@ -190,7 +193,7 @@ The `mass-rebuild.py` script takes care of:
|
|||
+
|
||||
____
|
||||
....
|
||||
$ ssh branched-composer.phx2.fedoraproject.org
|
||||
$ ssh branched-composer.iad2.fedoraproject.org
|
||||
....
|
||||
____
|
||||
. Start a terminal multiplexer
|
||||
|
|
@ -211,7 +214,7 @@ $ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out
|
|||
....
|
||||
____
|
||||
|
||||
==== Monitoring Mass Rebuilds
|
||||
===== Monitoring Mass Rebuilds
|
||||
|
||||
The community has a very high interest in the status of rebuilds and
|
||||
many maintainers will want to know if their build failed right away. The
|
||||
|
|
@ -222,7 +225,7 @@ publicly available URLs for stakeholders to monitor.
|
|||
+
|
||||
____
|
||||
....
|
||||
$ ssh compose-x86-02.phx2.fedoraproject.org
|
||||
$ ssh compose-x86-02.iad2.fedoraproject.org
|
||||
....
|
||||
____
|
||||
. Start a terminal multiplexer
|
||||
|
|
@ -240,8 +243,10 @@ Set Up the Rebuild Failures Notification Web Site::
|
|||
The `find_failures.py` script discovers attempted builds that have
|
||||
failed. It lists those failed builds and sorts them by package owner.
|
||||
+
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ while true; do ./find_failures.py > f26-failures.html && cp f26-failures.html /mnt/koji/mass-rebuild/f26-failures.html; sleep 600; done
|
||||
$ while true; do ./find_failures.py > f{rawhide}-failures.html && cp f{rawhide}-failures.html /mnt/koji/mass-rebuild/f{rawhide}-failures.html; sleep 600; done
|
||||
....
|
||||
. Start a second pane in the terminal emulator
|
||||
. {blank}
|
||||
|
|
@ -252,11 +257,13 @@ Set up the Site for Packages that Need Rebuilt::
|
|||
owner. This gives external stakeholders a rough idea of how much work
|
||||
is remaining in the mass rebuild.
|
||||
+
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ while true; do ./need-rebuild.py > f26-need-rebuild.html && cp f26-need-rebuild.html /mnt/koji/mass-rebuild/f26-need-rebuild.html; sleep 600; done
|
||||
$ while true; do ./need-rebuild.py > f{rawhide}-need-rebuild.html && cp f{rawhide}-need-rebuild.html /mnt/koji/mass-rebuild/f{rawhide}-need-rebuild.html; sleep 600; done
|
||||
....
|
||||
|
||||
==== Post Mass Rebuild Tasks
|
||||
===== Post Mass Rebuild Tasks
|
||||
|
||||
Once the mass rebuild script completes, and all the pending builds have
|
||||
finished, the builds will need to be tagged. The `mass-tag.py` script
|
||||
|
|
@ -273,9 +280,10 @@ https://pagure.io/releng[releng git repository]
|
|||
. Run the `mass-tag.py` script (requires koji kerberos authentication)
|
||||
+
|
||||
____
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ cd path/to/releng_repo/scripts
|
||||
$ ./mass-tag.py --source f36-rebuild --target f36
|
||||
$ ./mass-tag.py --source f{rawhide}-rebuild --target f{rawhide}
|
||||
....
|
||||
____
|
||||
. Send the final notification to the
|
||||
|
|
|
|||
|
|
@ -1,122 +0,0 @@
|
|||
== Mass Rebuild of Modules
|
||||
|
||||
=== Description
|
||||
|
||||
Periodically we do mass rebuilds of modules in rawhide during the
|
||||
development cycle. This SOP will outline the steps necessary to do this.
|
||||
|
||||
=== Assumptions
|
||||
|
||||
This assumes that the mass rebuild has already been approved and
|
||||
scheduled via release engineering and FESCo. Coordinate with
|
||||
infrastructure as well for any needed updates.
|
||||
|
||||
=== Considerations
|
||||
|
||||
* The most important thing to keep in mind while doing a mass rebuild is
|
||||
to communicate clearly what actions are being performed and the status
|
||||
of the rebuild.
|
||||
* Check in on scripts frequently to avoid a long stalled command from
|
||||
adding significant delays in completing the rebuild.
|
||||
|
||||
=== Actions
|
||||
|
||||
==== Preparatory Steps
|
||||
|
||||
The following steps should be completed after the
|
||||
https://docs.pagure.org/releng/sop_mass_rebuild_packages.html[mass
|
||||
rebuild of packages] is done.
|
||||
|
||||
. Update Scripts
|
||||
|
||||
The mass rebuild depends on two main scripts from the
|
||||
https://pagure.io/releng[releng git repository]. Each one requires some
|
||||
changes in variables for each new mass rebuild cycle.
|
||||
|
||||
____
|
||||
* {blank}
|
||||
+
|
||||
_mass-rebuild-modules.py_::
|
||||
** rebuildid
|
||||
* {blank}
|
||||
+
|
||||
_massrebuildsinfo.py_::
|
||||
** module_mass_rebuild_epoch
|
||||
** module_mass_rebuild_platform
|
||||
____
|
||||
|
||||
Change the following items:
|
||||
|
||||
* the `rebuildid` to match the release for which you are mass rebuilding
|
||||
modules as per in massrebuildsinfo.py
|
||||
* `module_mass_rebuild_epoch` mostly will be the epoch of mass rebuild
|
||||
of packages
|
||||
* `module_mass_rebuild_platform` should be the rawhide module platform
|
||||
|
||||
==== Starting the Mass Rebuild of Modules
|
||||
|
||||
The `mass-rebuild-modules.py` script takes care of:
|
||||
|
||||
* Discovering available available modules from PDC
|
||||
* Find the module info from mbs and check if a module build is submitted
|
||||
after the epoch date
|
||||
* Checking out modules from dist-git
|
||||
* Switching to appropriate stream
|
||||
* Find modulemd file
|
||||
* Use libmodulemd to determine if this module stream applies to this
|
||||
platform version
|
||||
* If needs rebuilding, committing the change
|
||||
* Push the commit
|
||||
* Submitting the build request through mbs
|
||||
|
||||
. Connect to the mass-rebuild Machine
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ssh compose-branched01.iad2.fedoraproject.org
|
||||
....
|
||||
____
|
||||
. Start a terminal multiplexer
|
||||
+
|
||||
____
|
||||
....
|
||||
$ tmux
|
||||
....
|
||||
____
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository].
|
||||
. Run the [.title-ref]#mass-rebuild-modules.py# script from
|
||||
_releng/scripts_
|
||||
+
|
||||
____
|
||||
....
|
||||
$ cd path/to/releng_repo/scripts
|
||||
$ ./mass-rebuild-modules.py <path_to_token_file> build --wait 2>&1 | tee ~/massbuildmodules.out
|
||||
....
|
||||
____
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
The token file should be located in infra's private ansible repo, or ask
|
||||
infra to get it to you using this
|
||||
https://pagure.io/fedora-infrastructure/issue/8048#comment-587789[process].
|
||||
====
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
The [.title-ref]#build# option is really important to pay attention,
|
||||
since the mass branching of modules will also use the same script, just
|
||||
changing the option to [.title-ref]#branch# and
|
||||
[.title-ref]#module_mass_branching_platform# in
|
||||
[.title-ref]#massrebuildsinfo.py#
|
||||
====
|
||||
|
||||
==== Post Mass Rebuild Tasks
|
||||
|
||||
Once the module mass rebuild is done, send an email to the
|
||||
devel-announce@ list
|
||||
|
||||
. Send the final notification to the
|
||||
_devel-announce@lists.fedoraproject.org_ list
|
||||
|
|
@ -1,266 +0,0 @@
|
|||
include::_partials/attributes.adoc[]
|
||||
|
||||
== Mass Rebuild
|
||||
|
||||
=== Description
|
||||
|
||||
Periodically we do mass rebuilds of rawhide during the development
|
||||
cycle. This SOP will outline the steps necessary to do this.
|
||||
|
||||
=== Assumptions
|
||||
|
||||
This assumes that the mass rebuild has already been approved and
|
||||
scheduled via release engineering and FESCo. Coordinate with
|
||||
infrastructure as well for any needed koji updates.
|
||||
|
||||
This also assumes that the mass rebuild does not need to be done in
|
||||
dependency order, and that the mass rebuild does not involve a ABI
|
||||
change.
|
||||
|
||||
=== Considerations
|
||||
|
||||
* The most important thing to keep in mind while doing a mass rebuild is
|
||||
to communicate clearly what actions are being performed and the status
|
||||
of the rebuild.
|
||||
* Check in on scripts frequently to avoid a long stalled command from
|
||||
adding significant delays in completing the rebuild.
|
||||
* Check with secondary arches, whether they up-to-date enough with
|
||||
primary, create rebuild tag and target when they are. It will then take
|
||||
care of rebuilds of the arch specific packages in appropriate kojis.
|
||||
|
||||
=== Actions
|
||||
|
||||
==== Preparatory Steps
|
||||
|
||||
The following steps may be completed in the weeks leading up to the
|
||||
scheduled mass rebuild.
|
||||
|
||||
. Create the Mass Rebuild Pagure Issue
|
||||
+
|
||||
____
|
||||
Create an issue on the https://pagure.io/releng/issues[Release
|
||||
Engineering issues page] that points at the schedule for the current
|
||||
release.
|
||||
|
||||
See https://pagure.io/releng/issue/6898[the Fedora 27 mass rebuild issue
|
||||
example].
|
||||
____
|
||||
. Set up the Mass Rebuild Wiki Page
|
||||
+
|
||||
____
|
||||
The mass rebuild wiki page should answer the following questions for
|
||||
maintainers:
|
||||
|
||||
* Why the mass rebuild is happening
|
||||
* How to opt out of the mass rebuild
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
See https://fedoraproject.org/wiki/Fedora_26_Mass_Rebuild[the Fedora 26
|
||||
Wiki example].
|
||||
====
|
||||
____
|
||||
. Send out the Mass Rebuild Notice
|
||||
+
|
||||
____
|
||||
Send out the same information posted on the wiki to the
|
||||
[.title-ref]#devel-announce@lists.fedoraproject.org# mailing list.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
See
|
||||
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[the
|
||||
Fedora 26 e-mail example].
|
||||
====
|
||||
____
|
||||
. Create a Tag to Contain the Mass Rebuild
|
||||
+
|
||||
____
|
||||
Mass rebuilds require their own tag to contain all related builds. The
|
||||
example assumes we are doing a rebuild for Fedora 26.
|
||||
|
||||
....
|
||||
$ koji add-tag f26-rebuild --parent f26
|
||||
....
|
||||
____
|
||||
. Request Package Auto-Signing for New Mass-Rebuild Tag
|
||||
+
|
||||
____
|
||||
File a ticket with https://pagure.io/fedora-infrastructure/issues[Fedora
|
||||
Infrastructure] requesting the new mass-rebuild tag be enabled for
|
||||
package auto-signing.
|
||||
____
|
||||
. Create the Koji Target for the Mass Rebuild
|
||||
+
|
||||
____
|
||||
Using the same [.title-ref]#f26-rebuild# tag created in the previous
|
||||
example:
|
||||
|
||||
....
|
||||
$ koji add-target f26-rebuild f26-build
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
*koji add-target* _target-name_ _buildroot-tag_ _destination-tag_
|
||||
describes the syntax format above. If the _destination-tag_ is not
|
||||
specified then it will be the same as the _target-name_.
|
||||
====
|
||||
____
|
||||
. Update Scripts
|
||||
+
|
||||
____
|
||||
The mass rebuild depends on four main scripts from the
|
||||
https://pagure.io/releng[releng git repository]. Each one requires some
|
||||
changes in variables for each new mass rebuild cycle.
|
||||
|
||||
* {blank}
|
||||
+
|
||||
mass-rebuild.py::
|
||||
** buildtag
|
||||
** targets
|
||||
** epoch
|
||||
** comment
|
||||
** target
|
||||
* {blank}
|
||||
+
|
||||
find-failures.py::
|
||||
** buildtag
|
||||
** desttag
|
||||
** epoch
|
||||
* mass-tag.py
|
||||
* {blank}
|
||||
+
|
||||
need-rebuild.py::
|
||||
** buildtag
|
||||
** target
|
||||
** updates
|
||||
** epoch
|
||||
____
|
||||
|
||||
Change the following items:
|
||||
|
||||
* the build tag, holding tag, and target tag should be updated to
|
||||
reflect the Fedora release you're building for
|
||||
* the `epoch` should be updated to the point at which all features that
|
||||
the mass rebuild is for have landed in the build system (and a newRepo
|
||||
task completed with those features)
|
||||
* the comment which is inserted into spec changelogs
|
||||
|
||||
==== Starting the Mass Rebuild
|
||||
|
||||
The `mass-rebuild.py` script takes care of:
|
||||
|
||||
* Discovering available packages in koji
|
||||
* Trimming out packages which have already been rebuilt
|
||||
* Checking out packages from git
|
||||
* Bumping the spec file
|
||||
* Committing the change
|
||||
* git tagging the change
|
||||
* Submitting the build request to Koji
|
||||
|
||||
. Connect to the mass-rebuild Machine
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ssh branched-composer.phx2.fedoraproject.org
|
||||
....
|
||||
____
|
||||
. Start a terminal multiplexer
|
||||
+
|
||||
____
|
||||
....
|
||||
$ tmux
|
||||
....
|
||||
____
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository].
|
||||
. Run the mass-rebuild.py script from _releng/scripts_
|
||||
+
|
||||
____
|
||||
....
|
||||
$ cd path/to/releng_repo/scripts
|
||||
$ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out
|
||||
....
|
||||
____
|
||||
|
||||
==== Monitoring Mass Rebuilds
|
||||
|
||||
The community has a very high interest in the status of rebuilds and
|
||||
many maintainers will want to know if their build failed right away. The
|
||||
`find-failures.py` and `need-rebuild.py` scripts are designed to update
|
||||
publicly available URLs for stakeholders to monitor.
|
||||
|
||||
. Connect to a Compose Machine
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ssh compose-x86-02.phx2.fedoraproject.org
|
||||
....
|
||||
____
|
||||
. Start a terminal multiplexer
|
||||
+
|
||||
____
|
||||
....
|
||||
$ tmux
|
||||
....
|
||||
____
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository]
|
||||
. {blank}
|
||||
+
|
||||
Set Up the Rebuild Failures Notification Web Site::
|
||||
The `find_failures.py` script discovers attempted builds that have
|
||||
failed. It lists those failed builds and sorts them by package owner.
|
||||
+
|
||||
....
|
||||
$ while true; do ./find_failures.py > f26-failures.html && cp f26-failures.html /mnt/koji/mass-rebuild/f26-failures.html; sleep 600; done
|
||||
....
|
||||
. Start a second pane in the terminal emulator
|
||||
. {blank}
|
||||
+
|
||||
Set up the Site for Packages that Need Rebuilt::
|
||||
The `need-rebuild.py` script discovers packages that have not yet been
|
||||
rebuilt and generates an html file listing them sorted by package
|
||||
owner. This gives external stakeholders a rough idea of how much work
|
||||
is remaining in the mass rebuild.
|
||||
+
|
||||
....
|
||||
$ while true; do ./need-rebuild.py > f26-need-rebuild.html && cp f26-need-rebuild.html /mnt/koji/mass-rebuild/f26-need-rebuild.html; sleep 600; done
|
||||
....
|
||||
|
||||
==== Post Mass Rebuild Tasks
|
||||
|
||||
Once the mass rebuild script completes, and all the pending builds have
|
||||
finished, the builds will need to be tagged. The `mass-tag.py` script
|
||||
will accomplish this task. The script will:
|
||||
|
||||
* Discover completed builds
|
||||
* Trim out builds that are older than the latest build for a given
|
||||
package
|
||||
* Tag remaining builds into their final destination (without generating
|
||||
email)
|
||||
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository]
|
||||
. Run the `mass-tag.py` script (requires koji kerberos authentication)
|
||||
+
|
||||
____
|
||||
|
||||
[source,subs="attributes+"]
|
||||
....
|
||||
$ cd path/to/releng_repo/scripts
|
||||
$ ./mass-tag.py --source f{branched}-rebuild --target f{branched}
|
||||
....
|
||||
____
|
||||
. Send the final notification to the
|
||||
_devel-announce@lists.fedoraproject.org_ list
|
||||
+
|
||||
____
|
||||
The contents should look something like this
|
||||
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[example
|
||||
email].
|
||||
____
|
||||
Loading…
Add table
Add a link
Reference in a new issue