Initial import
Add missing pages Fix the nav for release guide Cleanup branching sop Add release version substitutions Signed-off-by: Tomas Hrcka <thrcka@redhat.com>
This commit is contained in:
parent
cd836eed90
commit
14c92455cb
62 changed files with 7546 additions and 0 deletions
|
|
@ -16,3 +16,4 @@ nav:
|
|||
- modules/ROOT/nav.adoc
|
||||
- modules/developer_guide/nav.adoc
|
||||
- modules/sysadmin_guide/nav.adoc
|
||||
- modules/release_guide/nav.adoc
|
||||
|
|
|
|||
9
modules/release_guide/nav.adoc
Normal file
9
modules/release_guide/nav.adoc
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
* xref:release_process.adoc[Release process]
|
||||
* xref:index.adoc[Release guide]
|
||||
** xref:mass_rebuild.adoc[Mass Rebuild]
|
||||
** xref:sop_file_ftbfs.adoc[File FTBFS bugs]
|
||||
** xref:sop_mass_branching.adoc[Mass Branching]
|
||||
** xref:sop_bodhi_activation.adoc[Updates testing activation]
|
||||
** xref:beta_freeze.adoc[Beta freeze]
|
||||
** xref:beta_release.adoc[Beta release]
|
||||
** xref:final_freeze.adoc[Final freeze]
|
||||
5
modules/release_guide/pages/_partials/attributes.adoc
Normal file
5
modules/release_guide/pages/_partials/attributes.adoc
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
:year: 2023
|
||||
:rawhide: 39
|
||||
:branched: 38
|
||||
:current: 37
|
||||
:old_release: 36
|
||||
12
modules/release_guide/pages/architecture.adoc
Normal file
12
modules/release_guide/pages/architecture.adoc
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
[[architecture]]
|
||||
== Fedora Release Engineering Architecture
|
||||
|
||||
Here you will find an index of documentation that describes the Fedora
|
||||
Release Engineering Architecture. The goal of this is to provide an
|
||||
overview of the environment and various systems that are used to build
|
||||
it as a reference to members of the Fedora Community as well as
|
||||
materials to help prospective Release Engineering Community members.
|
||||
|
||||
=== Architecture Documents
|
||||
|
||||
layered_image_build_service workflow_automation
|
||||
1
modules/release_guide/pages/beta_freeze.adoc
Normal file
1
modules/release_guide/pages/beta_freeze.adoc
Normal file
|
|
@ -0,0 +1 @@
|
|||
== Fedora Beta Freeze
|
||||
1
modules/release_guide/pages/beta_release.adoc
Normal file
1
modules/release_guide/pages/beta_release.adoc
Normal file
|
|
@ -0,0 +1 @@
|
|||
== Fedora Beta Release
|
||||
248
modules/release_guide/pages/contributing.adoc
Normal file
248
modules/release_guide/pages/contributing.adoc
Normal file
|
|
@ -0,0 +1,248 @@
|
|||
== Fedora Release Engineering Contributing Guide
|
||||
|
||||
Fedora Release Engineering works with many different utilities that are
|
||||
maintained in respective upstream locations. Fedora Release Engineering
|
||||
maintains an "Upstream First" policy such that if there is a bug in an
|
||||
utility or a feature needed, we pursue that upstream before entertaining
|
||||
the idea of carrying a Fedora specific patch.
|
||||
|
||||
Fedora Release Engineering also has a number of scripts and utilities
|
||||
that are strictly Fedora centric in order to automate tasks and
|
||||
processes as they relate to Fedora itself which is what is contained in
|
||||
the https://pagure.io/releng[releng git repository] hosted on
|
||||
https://pagure.io/pagure[Pagure]. If you would like to contribute to
|
||||
something in this repository, please reference the
|
||||
`contributing-to-releng` section.
|
||||
|
||||
=== Contributing to releng
|
||||
|
||||
In order to contribute to the releng http://www.git-scm.com[git]
|
||||
repository (where the source reStructured Text version of these docs
|
||||
live), you should first have a
|
||||
https://admin.fedoraproject.org/accounts[Fedora Account System] (FAS)
|
||||
account, log into https://pagure.io[pagure.io] and the fork the
|
||||
https://pagure.io/releng[releng git repository].
|
||||
|
||||
Once you've forked the https://pagure.io/releng[releng git repository],
|
||||
you will need to set the remote upstream git clone of your fork in order
|
||||
to track the official releng repository. While not mandatory, it's
|
||||
conventional to call the remote upstream the name `upstream` which can
|
||||
be done with the following command while within the directory of your
|
||||
local git clone of your fork.
|
||||
|
||||
[source,bash]
|
||||
----
|
||||
$ git remote add upstream https://pagure.io/releng.git
|
||||
----
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
If you are not currently familiar with git, it is highly recommended to
|
||||
visit git's upstream and familiarize yourself.
|
||||
|
||||
http://www.git-scm.com/
|
||||
====
|
||||
|
||||
==== RelEng Developer Workflow
|
||||
|
||||
There are many options for developer workflow, but the recommended
|
||||
workflow for Fedora releng repository is known as a
|
||||
"http://www.git-scm.com/book/en/v2/Git-Branching-Branching-Workflows#Topic-Branches[Topic
|
||||
Branch]" based workflow in git nomenclature. This is how Fedora Release
|
||||
Engineering contributors can submit changes to the
|
||||
https://pagure.io/releng[releng git repository] for both code and
|
||||
documentation.
|
||||
|
||||
The general workflow is as follows:
|
||||
|
||||
* Checkout `main` branch of the local git clone of your releng
|
||||
repository clone.
|
||||
+
|
||||
....
|
||||
$ git checkout main
|
||||
....
|
||||
* Pull upstream and merge into local main to make sure that your main
|
||||
branch is in line with the latest changes from upstream. Then push it to
|
||||
your clone so that origin knows about the changes.
|
||||
+
|
||||
....
|
||||
$ git pull --rebase upstream main
|
||||
$ git push origin main
|
||||
....
|
||||
* Create a topic branch from main.
|
||||
+
|
||||
....
|
||||
$ git checkout -b my-topic-branch
|
||||
....
|
||||
* Make changes in your topic branch and commit them to your topic
|
||||
branch.
|
||||
+
|
||||
....
|
||||
$ vim somefile.py
|
||||
|
||||
.... make some change ...
|
||||
|
||||
$ git add somefile.py
|
||||
$ git commit -s -m "awesome patch to somefile.py"
|
||||
....
|
||||
* This step is optional but recommended in order to avoid collisions
|
||||
when submitting upstream. Here we will checkout main again and merge
|
||||
`upstream/main` so that we can resolve any conflicts locally.
|
||||
+
|
||||
....
|
||||
$ git checkout main
|
||||
$ git pull --rebase upstream main
|
||||
$ git push origin main
|
||||
....
|
||||
* Rebase on main before submitting a pull request
|
||||
+
|
||||
....
|
||||
$ git rebase main
|
||||
|
||||
..... Resolve any conflicts if needed ......
|
||||
....
|
||||
* Push your topic branch to your fork's origin in pagure.
|
||||
+
|
||||
....
|
||||
$ git push origin my-topic-branch
|
||||
....
|
||||
* Open a pull request in Rel Eng Pagure.
|
||||
https://pagure.io/releng/pull-requests
|
||||
|
||||
==== Developer Workflow Tips and Tricks
|
||||
|
||||
Below are some Fedora Release Engineering Developer Workflow Tips and
|
||||
Tricks used by current members of the team in order to help assist with
|
||||
development.
|
||||
|
||||
===== pullupstream
|
||||
|
||||
The following is an useful shell function to place in your `~/.bashrc`
|
||||
to help automate certain aspects of the developer workflow. It will
|
||||
allow you to merge in the upstream main or devel branch into your forked
|
||||
repository for easily keeping in line with the upstream repository.
|
||||
|
||||
The following is the bash function to be added to your `~/.bashrc` and
|
||||
make sure to `source ~/.bashrc` after adding it in order to "enable" the
|
||||
function.
|
||||
|
||||
....
|
||||
pullupstream () {
|
||||
if [[ -z "$1" ]]; then
|
||||
printf "Error: must specify a branch name (e.g. - main, devel)\n"
|
||||
else
|
||||
pullup_startbranch=$(git describe --contains --all HEAD)
|
||||
git checkout $1
|
||||
git pull --rebase upstream $1
|
||||
git push origin $1
|
||||
git checkout ${pullup_startbranch}
|
||||
fi
|
||||
}
|
||||
....
|
||||
|
||||
With the function in place you can easily pull and merge in the releng
|
||||
main branch even while using a topic branch as follows:
|
||||
|
||||
....
|
||||
$ git status
|
||||
On branch docs
|
||||
nothing to commit, working directory clean
|
||||
|
||||
$ pullupstream main
|
||||
Switched to branch 'main'
|
||||
Your branch is up-to-date with 'origin/main'.
|
||||
Already up-to-date.
|
||||
Everything up-to-date
|
||||
Switched to branch 'docs'
|
||||
|
||||
$ git status
|
||||
On branch docs
|
||||
nothing to commit, working directory clean
|
||||
....
|
||||
|
||||
Now that you're back on your topic branch you can easily rebase on your
|
||||
local main branch in order to resolve any merge conflicts that may come
|
||||
up for clean pull request submission.
|
||||
|
||||
....
|
||||
$ git rebase main
|
||||
Current branch docs is up to date.
|
||||
....
|
||||
|
||||
=== RelEng Upstream Tools
|
||||
|
||||
Fedora Release Engineering uses many tools that exist in their own
|
||||
upstream project space. These are tools that every Fedora Release
|
||||
Engineer should be familiar with and in the event there is a bug or a
|
||||
feature needed, we should participate in the respective upstream to
|
||||
resolve the issue first before considering carrying a Fedora specific
|
||||
patch.
|
||||
|
||||
==== Tools List
|
||||
|
||||
===== Tools Release Engineering is actively involved with upstream
|
||||
|
||||
Below are a set of tools that are centric to the Release Engineering
|
||||
team and our processes. We actively engage with upstreams of these
|
||||
projects. For these tools, we recommend the same git contribution
|
||||
workflow that is outlined above for this git repository.
|
||||
|
||||
* https://pagure.io/koji[koji] -Build System used by Fedora
|
||||
* https://pagure.io/mash[mash] -Tool that creates repositories from koji
|
||||
tags, and solves them for multilib dependencies.
|
||||
* https://pagure.io/pungi[pungi] -Fedora Compose tool
|
||||
* https://github.com/release-engineering/product-definition-center[Product
|
||||
Defintion Center (PDC)] -Repository and API for storing and querying
|
||||
product metadata
|
||||
* https://github.com/release-engineering/koji-containerbuild[koji-containerbuild]
|
||||
-Koji plugin to integrate OSBS with koji
|
||||
|
||||
===== Tools Release Engineering is actively mostly consumers of
|
||||
|
||||
Below are the set of tools that the Release Engineering team either
|
||||
consumes directly or as the side effect of other tools in the Release
|
||||
Engineering Infrastructure. Tools here should always be engaged upstream
|
||||
in the event of a bug or enhancement needed but are not tools that the
|
||||
Release Engineering team is extremely active in their continued upstream
|
||||
development and will defer to each upstream for recommended
|
||||
contributions workflow.
|
||||
|
||||
* https://pagure.io/fedpkg[fedpkg] -command line utility for Fedora (and
|
||||
EPEL) developers. It interacts with dist-git, koji, rpmbuild, git, etc.
|
||||
* https://pagure.io/rpkg[rpkg] -library for dealing with rpm packaging
|
||||
in a git source control (used by fedpkg)
|
||||
* https://github.com/release-engineering/dist-git[dist-git] -remote Git
|
||||
repository specificaly designed to hold RPM package sources.
|
||||
* http://createrepo.baseurl.org/[creatrepo] -A python program which
|
||||
generate repodata from a set of rpm files.
|
||||
* https://github.com/rpm-software-management/createrepo_c[createrepo_c]
|
||||
-C implementation of createrepo
|
||||
* https://github.com/clalancette/oz[oz] -set of programs and classes to
|
||||
do automated installations of operating systems.
|
||||
* http://imgfac.org/[imagefactory] -imagefactory builds images for a
|
||||
variety of operating system/cloud combinations.
|
||||
* https://pagure.io/sigul[sigul] -An automated gpg signing system
|
||||
* https://github.com/rpm-software-management/mock/wiki[mock] -a tool for
|
||||
building packages in prestine buildroots
|
||||
* http://www.fedmsg.com/en/latest/[fedmsg] -Fedora Infrastructure
|
||||
Message Bus
|
||||
* https://github.com/rhinstaller/lorax[lorax] -tool to build install
|
||||
trees and images
|
||||
* http://www.openshift.org/[OpenShift] -Open Source Platform as a
|
||||
Service by Red Hat
|
||||
* https://github.com/projectatomic/osbs-client[OSBS] -set of utilities
|
||||
that turn OpenShift into a layered image build system
|
||||
* https://fedoraproject.org/wiki/Taskotron[taskotron] -a framework for
|
||||
automated task execution.
|
||||
* http://www.pulpproject.org/[pulp] -a platform for managing
|
||||
repositories of content, such as software packages, and pushing that
|
||||
content out to large numbers of consumer
|
||||
* https://github.com/pulp/crane[crane] -Crane is a small read-only web
|
||||
application that provides enough of the docker registry API to support
|
||||
"docker pull"
|
||||
* https://pagure.io/pagure[pagure] A git centered forge
|
||||
* https://github.com/projectatomic/rpm-ostree[rpm-ostree] -Store RPMs in
|
||||
OSTree repository, and atomically upgrade from commits
|
||||
* https://wiki.gnome.org/Projects/OSTree[ostree] -a tool for managing
|
||||
bootable, immutable, versioned filesystem trees.
|
||||
1
modules/release_guide/pages/final_freeze.adoc
Normal file
1
modules/release_guide/pages/final_freeze.adoc
Normal file
|
|
@ -0,0 +1 @@
|
|||
== Fedora Final Freeze
|
||||
150
modules/release_guide/pages/index.adoc
Normal file
150
modules/release_guide/pages/index.adoc
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
== Fedora Release Engineering
|
||||
|
||||
Contents:
|
||||
|
||||
overview philosophy contributing troubleshooting architecture sop
|
||||
|
||||
This page contains information about the Fedora Release Engineering
|
||||
team.
|
||||
|
||||
[[releng-contact-info]]
|
||||
=== Contact Information
|
||||
|
||||
* IRC: `#fedora-releng` on irc.libera.chat
|
||||
* Mailing List: https://admin.fedoraproject.org/mailman/listinfo/rel-eng[rel-eng@lists.fedoraproject.org]
|
||||
* Issue tracker: https://pagure.io/releng/new_issue[Fedora Releng Pagure Tickets]
|
||||
|
||||
If you want the to get something done (e.g. moving packages to
|
||||
buildroots or into frozen compositions) by the ReleaseEngineering Team,
|
||||
please create a ticket in the issue tracker mentioned above. Please
|
||||
enter your FAS-username or e-mail address in the respective textbox, to
|
||||
make sure the team can contact you.
|
||||
|
||||
[[index-team-composition]]
|
||||
=== Team Composition
|
||||
* https://fedoraproject.org/wiki/User:kevin[Kevin Fenzi (nirik)]
|
||||
* https://fedoraproject.org/wiki/User:sharkcz[Dan Horák (sharkcz)](secondary arches)
|
||||
* https://fedoraproject.org/wiki/User:pbrobinson[Peter Robinson(pbrobinson)]
|
||||
* https://fedoraproject.org/wiki/User:maxamillion[Adam Miller(maxamillion)]
|
||||
* https://fedoraproject.org/wiki/User:humaton[Tomas Hrcka(humaton)]
|
||||
|
||||
Release Team members are approved by FESCo. However, FESCo has delegated
|
||||
this power to the Release Team itself. If you want to join the team,
|
||||
please read `join-releng`.
|
||||
|
||||
=== What is Fedora Release Engineering?
|
||||
|
||||
For a Broad Overview, see `overview <overview>`.
|
||||
|
||||
=== Why we do things the way we do them
|
||||
|
||||
For information on the Fedora Release Engineering Philosophy, see
|
||||
`philosophy <philosophy>`.
|
||||
|
||||
=== Fedora Release Engineering Leadership
|
||||
|
||||
Mohan Boddu (mboddu on IRC, FAS username mohanboddu)
|
||||
|
||||
Leadership is currently appointed by FESCo with input from the current
|
||||
release team.
|
||||
|
||||
=== Things we Do
|
||||
|
||||
* {blank}
|
||||
+
|
||||
Create official Fedora releases.::
|
||||
** {blank}
|
||||
+
|
||||
Fedora Products;;
|
||||
*** Cloud
|
||||
*** Server
|
||||
*** Workstation
|
||||
** Fedora Spins
|
||||
* Report progress towards release from
|
||||
https://fedoraproject.org/wiki/Releases/Branched[branched] creation on.
|
||||
* Give reports to FESCo on changes to processes.
|
||||
* If something is known to be controversial, we let FESCo know before
|
||||
implementing otherwise implementation generally happens concurrently to
|
||||
reporting.
|
||||
* Set policy on freeze management
|
||||
* Administrate the build system(s)
|
||||
* Remove unmaintained packages from Fedora
|
||||
* Push updated packages
|
||||
* write and maintain tools to compose and push Fedora
|
||||
|
||||
[[join-releng]]
|
||||
=== Joining Release Engineering
|
||||
|
||||
Much of rel-eng's communication is via IRC. One of the best ways to
|
||||
initially get involved is to attend one of the meetings and say that
|
||||
you're interested in doing some work during the open floor at the end of
|
||||
the meeting. If you can't make the meeting times, you can also ping one
|
||||
of us on IRC or sign up for the
|
||||
https://admin.fedoraproject.org/mailman/listinfo/rel-eng[mailing list].
|
||||
|
||||
Since release engineering needs special access to systems essential to
|
||||
Fedora people new to rel-eng will usually get access a little bit at a
|
||||
time. Typically people won't immediately be granted the ability to sign
|
||||
packages and push updates for example. A couple of tasks you could start
|
||||
out with are troubleshooting why builds are failing (and if rel-eng
|
||||
could take actions to fix it) as the requests are submitted to pagure or
|
||||
help with scripts for various rel-eng tasks.
|
||||
|
||||
There are also a number of tools that Fedora Release Engineering uses
|
||||
and relies upon, working on improving these upstream to fascilitate with
|
||||
new things that the Fedora Project is aiming to deliver is also a great
|
||||
way to get involved with Fedora Rel-Eng.
|
||||
|
||||
=== How we do it
|
||||
|
||||
See our `Standard Operating Procedures <sop>` for details on how we do
|
||||
the things we do.
|
||||
|
||||
Most discussions regarding release engineering will happen either in
|
||||
[.title-ref]##fedora-releng# or on the releng mailing list. For
|
||||
requests, please consult the `releng-contact-info`
|
||||
|
||||
=== Meetings
|
||||
|
||||
rel-eng holds regular meetings every Monday at 14:30 UTC in
|
||||
[.title-ref]##fedora-meeting-3# on the Libera IRC network.
|
||||
|
||||
* https://pagure.io/releng/issues?status=Open&tags=meeting[Meeting
|
||||
agendas] are created from open tickets in pagure that contain the
|
||||
meeting keyword.
|
||||
|
||||
==== Meeting Minutes
|
||||
|
||||
Minutes are posted to the rel-eng mailing list. They are also available
|
||||
at the
|
||||
https://meetbot.fedoraproject.org/sresults/?group_id=releng&type=team[Meetbot
|
||||
team page for releng]
|
||||
|
||||
There are also
|
||||
https://fedoraproject.org/wiki/ReleaseEngineering/Meetings[historical
|
||||
Meeting Minutes for 2007-04-16 to 2009-05-04].
|
||||
|
||||
=== Current activities
|
||||
|
||||
See our https://pagure.io/releng/issues[ticket queue] for the things we
|
||||
are currently working.
|
||||
|
||||
See https://fedoraproject.org/wiki/Releases[Releases] for information
|
||||
about Fedora releases, including schedules.
|
||||
|
||||
=== Freeze Policies
|
||||
|
||||
* https://fedoraproject.org/wiki/Milestone_freezes[Milestone (Alpha,
|
||||
Beta, Final) freezes]
|
||||
* https://fedoraproject.org/wiki/Software_String_Freeze_Policy[String
|
||||
Freeze Policy] (Same time as Alpha Freeze)
|
||||
* https://fedoraproject.org/wiki/Changes/Policy[Change freeze policy]
|
||||
(that's 'Change' as in 'feature')
|
||||
* https://fedoraproject.org/wiki/Updates_Policy[Updates Policy] (not
|
||||
technically a freeze, but of interest)
|
||||
|
||||
=== Indices and tables
|
||||
|
||||
* `genindex`
|
||||
* `modindex`
|
||||
* `search`
|
||||
263
modules/release_guide/pages/layered_image_build_service.adoc
Normal file
263
modules/release_guide/pages/layered_image_build_service.adoc
Normal file
|
|
@ -0,0 +1,263 @@
|
|||
== Fedora Layered Image Build System
|
||||
|
||||
The Fedora Layered Image Build System aims to provide an new type of
|
||||
official binary artifact produced by Fedora. Currently, we produce two
|
||||
main types of artifacts: RPMs, and images. The RPMs are created in
|
||||
https://fedoraproject.org/wiki/Koji[Koji] from specfiles in dist-git.
|
||||
The images come in different formats, but have in common creation in
|
||||
Koji from kickstart files — this includes the official Fedora Docker
|
||||
Base Image. This change introduces a new type of image, a
|
||||
https://github.com/docker/docker/[Docker] Layered Image, which is
|
||||
created from a Dockerfile and builds on top of that base image.
|
||||
|
||||
=== Layered Image Build Service Architecture
|
||||
|
||||
....
|
||||
+------------------------------+
|
||||
| Future Items to Integrate |
|
||||
+------------------------------+
|
||||
| +--------------------------+ |
|
||||
| |PDC Integration | |
|
||||
| +--------------------------+ |
|
||||
| |New Hotness | |
|
||||
| +--------------------------+ |
|
||||
| |Other??? | |
|
||||
| +--------------------------+ |
|
||||
| Magical Future |
|
||||
| |
|
||||
+------------------------------+
|
||||
|
||||
|
||||
|
||||
+------------------+
|
||||
|Fedora |
|
||||
|Users/Contributors|
|
||||
+--------+---------+
|
||||
^
|
||||
| +----------------+
|
||||
| | Fedora |
|
||||
+-------+-----------+ | Layered Image |
|
||||
| Fedora Production | | Maintainers |
|
||||
| Docker Registry | | |
|
||||
+-------------------+ +-------+--------+
|
||||
^ |
|
||||
| V
|
||||
| +-------------------+--+
|
||||
+-----------+------------+ | |
|
||||
| RelEng Automation | | +------------------+ |
|
||||
| (Release) | | |Dockerfile | |
|
||||
+-----------+------------+ | +------------------+ |
|
||||
^ | |App "init" scripts| |
|
||||
| | +------------------+ |
|
||||
| | |Docs | |
|
||||
+---------+------------+ | +------------------+ |
|
||||
| Taskotron | | DistGit |
|
||||
| Automated QA | | |
|
||||
+---------+-------+----+ +-----------+----------+
|
||||
^ ^ |
|
||||
| | |
|
||||
| | |
|
||||
| | |
|
||||
| +-------------------+ |
|
||||
| | |
|
||||
[docker images] | |
|
||||
| | |
|
||||
| [fedmsg] |
|
||||
+---------------+-----------+ | |
|
||||
| | | +---------------+
|
||||
| +----------------------+ | | |
|
||||
| |Intermediate Docker | | +------+-------------------+ |
|
||||
| |Images for OSBS Builds| | | | |
|
||||
| +----------------------+ | | +----------------------+ | |
|
||||
| |Layered Images | | | |containerbuild plugin | | |
|
||||
| +----------------------+ | | +----------------------+ | |
|
||||
| |docker|distribution | | | |rpmbuilds | | |
|
||||
| +----------------------+ | | +----------------------+ | |
|
||||
| Registry | | koji | |
|
||||
| | | | |
|
||||
+-----+----+----------------+ +-----+---+----+-----------+ |
|
||||
| ^ | ^ ^ |
|
||||
| | | | | |
|
||||
| | +---------------------------+ | | |
|
||||
| | | [osbs-client api] | | |
|
||||
| | | +---------------------------+ | |
|
||||
| | | | | |
|
||||
| | | | | |
|
||||
V | V | | |
|
||||
+--+----+---+---+---+ | V
|
||||
| | +-------------------+---+
|
||||
| +--------------+ | |fedpkg container-build +
|
||||
| |OpenShift v3 | | +-----------------------+
|
||||
| +--------------+ |
|
||||
| |Atomic Reactor| |
|
||||
| +--------------+ |
|
||||
| OSBS |
|
||||
| |
|
||||
+-------------------+
|
||||
|
||||
|
||||
[--------------------- Robosignatory ------------------------------------]
|
||||
[ Every time an image is tagged or changes names, Robosignatory signs it ]
|
||||
[ ]
|
||||
[ NOTE: It's injection points in the diagram are ommitted for brevity ]
|
||||
[------------------------------------------------------------------------]
|
||||
....
|
||||
|
||||
=== Layered Image Build System Components
|
||||
|
||||
The main aspects of the Layered Image Build System are:
|
||||
|
||||
* Koji
|
||||
** koji-containerbuild plugin
|
||||
* OpenShift Origin v3
|
||||
* Atomic Reactor
|
||||
* osbs-client tools
|
||||
* A docker registry
|
||||
** docker-distribution
|
||||
* Taskotron
|
||||
* fedmsg
|
||||
* RelEng Automation
|
||||
|
||||
The build system is setup such that Fedora Layered Image maintainers
|
||||
will submit a build to Koji via the `fedpkg container-build` command a
|
||||
`containers` namespace within
|
||||
https://fedoraproject.org/wiki/Infrastructure/VersionControl/dist-git[DistGit].
|
||||
This will trigger the build to be scheduled in
|
||||
https://www.openshift.org/[OpenShift] via
|
||||
https://github.com/projectatomic/osbs-client[osbs-client] tooling, this
|
||||
will create a custom
|
||||
https://docs.openshift.org/latest/dev_guide/builds.html[OpenShift Build]
|
||||
which will use the pre-made buildroot Docker image that we have created.
|
||||
The https://github.com/projectatomic/atomic-reactor[Atomic Reactor]
|
||||
(`atomic-reactor`) utility will run within the buildroot and prep the
|
||||
build container where the actual build action will execute, it will also
|
||||
maintain uploading the
|
||||
https://fedoraproject.org/wiki/Koji/ContentGenerators[Content Generator]
|
||||
metadata back to https://fedoraproject.org/wiki/Koji[Koji] and upload
|
||||
the built image to the candidate docker registry. This will run on a
|
||||
host with iptables rules restricting access to the docker bridge, this
|
||||
is how we will further limit the access of the buildroot to the outside
|
||||
world verifying that all sources of information come from Fedora.
|
||||
|
||||
Completed layered image builds are hosted in a candidate docker registry
|
||||
which is then used to pull the image and perform tests with
|
||||
https://taskotron.fedoraproject.org/[Taskotron]. The taskotron tests are
|
||||
triggered by a http://www.fedmsg.com/en/latest/[fedmsg] message that is
|
||||
emitted from https://fedoraproject.org/wiki/Koji[Koji] once the build is
|
||||
complete. Once the test is complete, taskotron will send fedmsg which is
|
||||
then caught by the [.title-ref]#RelEng Automation# Engine that will run
|
||||
the Automatic Release tasks in order to push the layered image into a
|
||||
stable docker registry in the production space for end users to consume.
|
||||
|
||||
Note that every time the layered image tagged to a new registry,
|
||||
ultimately changing it's name,
|
||||
https://pagure.io/robosignatory[Robosignatory] will automatically sign
|
||||
the new image. This will also occur as part of the Automated Release
|
||||
process as the image will be tagged from the candidate docker registry
|
||||
into the production docker registry in order to "graduate" the image to
|
||||
stable.
|
||||
|
||||
==== Koji
|
||||
|
||||
https://fedoraproject.org/wiki/Koji[Koji] is the Fedora Build System.
|
||||
|
||||
==== koji-containerbuild plugin
|
||||
|
||||
The
|
||||
https://github.com/release-engineering/koji-containerbuild[koji-containerbuild]
|
||||
plugin integrates Koji and OSBS so that builds can be scheduled by koji
|
||||
and integrated into the build system with imports of metadata, logs,
|
||||
build data, and build artifacts.
|
||||
|
||||
==== OpenShift Origin v3
|
||||
|
||||
https://www.openshift.org/[OpenShift Origin v3] is an open source
|
||||
Container Platform, built on top of http://kubernetes.io/[kubernetes]
|
||||
and https://github.com/docker/docker/[Docker]. This provides many
|
||||
aspects of the system needed including a build pipeline for Docker
|
||||
images with custom build profiles, image streams, and triggers based on
|
||||
events within the system.
|
||||
|
||||
==== Atomic Reactor
|
||||
|
||||
https://github.com/projectatomic/atomic-reactor[Atomic Reactor] is an
|
||||
utility which allows for the building of containers from within other
|
||||
containers providing hooks to trigger automation of builds as well as
|
||||
plugins providing automatic integration many other utilities and
|
||||
services.
|
||||
|
||||
==== osbs-client tools
|
||||
|
||||
https://github.com/projectatomic/osbs-client[osbs-client] tools allow
|
||||
for users to access the build functionality of
|
||||
https://www.openshift.org/[OpenShift Origin v3] using a simple set of
|
||||
command line utilities.
|
||||
|
||||
==== docker-registry
|
||||
|
||||
A https://docs.docker.com/registry/[docker-registry] is a stateless,
|
||||
highly scalable server side application that stores and lets you
|
||||
distribute Docker images.
|
||||
|
||||
There are many different implementations of docker-registries, two main
|
||||
ones are supported by the Fedora Layered Image Build System.
|
||||
|
||||
===== docker-distribution
|
||||
|
||||
The https://github.com/docker/distribution/[docker-distribution]
|
||||
registry is considered the Docker upstream "v2 registry" such that it
|
||||
was used by upstream to implement the new version 2 specification of the
|
||||
docker-registry.
|
||||
|
||||
===== Fedora Production Registry
|
||||
|
||||
Implementation details of this are still unknown at the time of this
|
||||
writing and will be updated at a later date. For the current status and
|
||||
implementation notes please visit the
|
||||
https://fedoraproject.org/wiki/Changes/FedoraDockerRegistry[FedoraDockerRegistry]
|
||||
page.
|
||||
|
||||
==== Taskotron
|
||||
|
||||
https://taskotron.fedoraproject.org/[Taskotron] is an automated task
|
||||
execution framework, written on top of http://buildbot.net/[buildbot]
|
||||
that currently executes many Fedora automated QA tasks and we will be
|
||||
adding the Layered Image automated QA tasks. The tests themselves will
|
||||
be held in DistGit and maintained by the Layered Image maintainers.
|
||||
|
||||
==== RelEng Automation
|
||||
|
||||
https://pagure.io/releng-automation[RelEng Automation] is an ongoing
|
||||
effort to automate as much of the RelEng process as possible by using
|
||||
http://ansible.com/[Ansible] and being driven by
|
||||
http://www.fedmsg.com/en/latest/[fedmsg] via
|
||||
https://github.com/maxamillion/loopabull[Loopabull] to execute Ansible
|
||||
Playbooks based on fedmsg events.
|
||||
|
||||
==== Robosignatory
|
||||
|
||||
https://pagure.io/robosignatory[Robosignatory] is a fedmsg consumer that
|
||||
automatically signs artifacts and will be used to automatically sign
|
||||
docker layered images for verification by client tools as well as end
|
||||
users.
|
||||
|
||||
==== Future Integrations
|
||||
|
||||
In the future various other components of the
|
||||
https://fedoraproject.org/wiki/Infrastructure[Fedora Infrastructure]
|
||||
will likely be incorporated.
|
||||
|
||||
===== PDC
|
||||
|
||||
https://pdc.fedoraproject.org/[PDC] is Fedora's implementation of
|
||||
https://github.com/product-definition-center/product-definition-center[Product
|
||||
Definition Center] which allows Fedora to maintain a database of each
|
||||
Compose and all of it's contents in a way that can be queried and used
|
||||
to make decisions in a programatic way.
|
||||
|
||||
===== The New Hotness
|
||||
|
||||
https://github.com/fedora-infra/the-new-hotness[The New Hotness] is a
|
||||
http://www.fedmsg.com/en/latest/[fedmsg] consumer that listens to
|
||||
release-monitoring.org and files bugzilla bugs in response (to notify
|
||||
packagers that they can update their packages).
|
||||
26
modules/release_guide/pages/mass_rebuild.adoc
Normal file
26
modules/release_guide/pages/mass_rebuild.adoc
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
= Mass rebuild information.
|
||||
|
||||
== Description
|
||||
|
||||
Periodically we do mass rebuilds of rawhide during the development cycle. This SOP will outline the steps necessary to do this.
|
||||
|
||||
== Assumptions
|
||||
|
||||
This assumes that the mass rebuild has already been approved and scheduled via release engineering and FESCo. Coordinate with infrastructure as well for any needed koji updates.
|
||||
|
||||
This also assumes that the mass rebuild does not need to be done in dependency order, and that the mass rebuild does not involve a ABI change.
|
||||
|
||||
== Considerations
|
||||
|
||||
The most important thing to keep in mind while doing a mass rebuild is to communicate clearly what actions are being performed and the status of the rebuild.
|
||||
|
||||
Check in on scripts frequently to avoid a long stalled command from adding significant delays in completing the rebuild.
|
||||
|
||||
Check with secondary arches, whether they up-to-date enough with primary, create rebuild tag and target when they are. It will then take care of rebuilds of the arch specific packages in appropriate kojis.
|
||||
|
||||
== Actions
|
||||
|
||||
=== Preparatory Steps
|
||||
|
||||
The following steps may be completed in the weeks leading up to the scheduled mass rebuild.
|
||||
|
||||
122
modules/release_guide/pages/mass_rebuild_modules.adoc
Normal file
122
modules/release_guide/pages/mass_rebuild_modules.adoc
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
== Mass Rebuild of Modules
|
||||
|
||||
=== Description
|
||||
|
||||
Periodically we do mass rebuilds of modules in rawhide during the
|
||||
development cycle. This SOP will outline the steps necessary to do this.
|
||||
|
||||
=== Assumptions
|
||||
|
||||
This assumes that the mass rebuild has already been approved and
|
||||
scheduled via release engineering and FESCo. Coordinate with
|
||||
infrastructure as well for any needed updates.
|
||||
|
||||
=== Considerations
|
||||
|
||||
* The most important thing to keep in mind while doing a mass rebuild is
|
||||
to communicate clearly what actions are being performed and the status
|
||||
of the rebuild.
|
||||
* Check in on scripts frequently to avoid a long stalled command from
|
||||
adding significant delays in completing the rebuild.
|
||||
|
||||
=== Actions
|
||||
|
||||
==== Preparatory Steps
|
||||
|
||||
The following steps should be completed after the
|
||||
https://docs.pagure.org/releng/sop_mass_rebuild_packages.html[mass
|
||||
rebuild of packages] is done.
|
||||
|
||||
. Update Scripts
|
||||
|
||||
The mass rebuild depends on two main scripts from the
|
||||
https://pagure.io/releng[releng git repository]. Each one requires some
|
||||
changes in variables for each new mass rebuild cycle.
|
||||
|
||||
____
|
||||
* {blank}
|
||||
+
|
||||
_mass-rebuild-modules.py_::
|
||||
** rebuildid
|
||||
* {blank}
|
||||
+
|
||||
_massrebuildsinfo.py_::
|
||||
** module_mass_rebuild_epoch
|
||||
** module_mass_rebuild_platform
|
||||
____
|
||||
|
||||
Change the following items:
|
||||
|
||||
* the `rebuildid` to match the release for which you are mass rebuilding
|
||||
modules as per in massrebuildsinfo.py
|
||||
* `module_mass_rebuild_epoch` mostly will be the epoch of mass rebuild
|
||||
of packages
|
||||
* `module_mass_rebuild_platform` should be the rawhide module platform
|
||||
|
||||
==== Starting the Mass Rebuild of Modules
|
||||
|
||||
The `mass-rebuild-modules.py` script takes care of:
|
||||
|
||||
* Discovering available available modules from PDC
|
||||
* Find the module info from mbs and check if a module build is submitted
|
||||
after the epoch date
|
||||
* Checking out modules from dist-git
|
||||
* Switching to appropriate stream
|
||||
* Find modulemd file
|
||||
* Use libmodulemd to determine if this module stream applies to this
|
||||
platform version
|
||||
* If needs rebuilding, committing the change
|
||||
* Push the commit
|
||||
* Submitting the build request through mbs
|
||||
|
||||
. Connect to the mass-rebuild Machine
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ssh compose-branched01.iad2.fedoraproject.org
|
||||
....
|
||||
____
|
||||
. Start a terminal multiplexer
|
||||
+
|
||||
____
|
||||
....
|
||||
$ tmux
|
||||
....
|
||||
____
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository].
|
||||
. Run the [.title-ref]#mass-rebuild-modules.py# script from
|
||||
_releng/scripts_
|
||||
+
|
||||
____
|
||||
....
|
||||
$ cd path/to/releng_repo/scripts
|
||||
$ ./mass-rebuild-modules.py <path_to_token_file> build --wait 2>&1 | tee ~/massbuildmodules.out
|
||||
....
|
||||
____
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
The token file should be located in infra's private ansible repo, or ask
|
||||
infra to get it to you using this
|
||||
https://pagure.io/fedora-infrastructure/issue/8048#comment-587789[process].
|
||||
====
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
The [.title-ref]#build# option is really important to pay attention,
|
||||
since the mass branching of modules will also use the same script, just
|
||||
changing the option to [.title-ref]#branch# and
|
||||
[.title-ref]#module_mass_branching_platform# in
|
||||
[.title-ref]#massrebuildsinfo.py#
|
||||
====
|
||||
|
||||
==== Post Mass Rebuild Tasks
|
||||
|
||||
Once the module mass rebuild is done, send an email to the
|
||||
devel-announce@ list
|
||||
|
||||
. Send the final notification to the
|
||||
_devel-announce@lists.fedoraproject.org_ list
|
||||
262
modules/release_guide/pages/mass_rebuild_packages.adoc
Normal file
262
modules/release_guide/pages/mass_rebuild_packages.adoc
Normal file
|
|
@ -0,0 +1,262 @@
|
|||
== Mass Rebuild
|
||||
|
||||
=== Description
|
||||
|
||||
Periodically we do mass rebuilds of rawhide during the development
|
||||
cycle. This SOP will outline the steps necessary to do this.
|
||||
|
||||
=== Assumptions
|
||||
|
||||
This assumes that the mass rebuild has already been approved and
|
||||
scheduled via release engineering and FESCo. Coordinate with
|
||||
infrastructure as well for any needed koji updates.
|
||||
|
||||
This also assumes that the mass rebuild does not need to be done in
|
||||
dependency order, and that the mass rebuild does not involve a ABI
|
||||
change.
|
||||
|
||||
=== Considerations
|
||||
|
||||
* The most important thing to keep in mind while doing a mass rebuild is
|
||||
to communicate clearly what actions are being performed and the status
|
||||
of the rebuild.
|
||||
* Check in on scripts frequently to avoid a long stalled command from
|
||||
adding significant delays in completing the rebuild.
|
||||
* Check with secondary arches, whether they up-to-date enough with
|
||||
primary, create rebuild tag and target when they are. It will then take
|
||||
care of rebuilds of the arch specific packages in appropriate kojis.
|
||||
|
||||
=== Actions
|
||||
|
||||
==== Preparatory Steps
|
||||
|
||||
The following steps may be completed in the weeks leading up to the
|
||||
scheduled mass rebuild.
|
||||
|
||||
. Create the Mass Rebuild Pagure Issue
|
||||
+
|
||||
____
|
||||
Create an issue on the https://pagure.io/releng/issues[Release
|
||||
Engineering issues page] that points at the schedule for the current
|
||||
release.
|
||||
|
||||
See https://pagure.io/releng/issue/6898[the Fedora 27 mass rebuild issue
|
||||
example].
|
||||
____
|
||||
. Set up the Mass Rebuild Wiki Page
|
||||
+
|
||||
____
|
||||
The mass rebuild wiki page should answer the following questions for
|
||||
maintainers:
|
||||
|
||||
* Why the mass rebuild is happening
|
||||
* How to opt out of the mass rebuild
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
See https://fedoraproject.org/wiki/Fedora_26_Mass_Rebuild[the Fedora 26
|
||||
Wiki example].
|
||||
====
|
||||
____
|
||||
. Send out the Mass Rebuild Notice
|
||||
+
|
||||
____
|
||||
Send out the same information posted on the wiki to the
|
||||
[.title-ref]#devel-announce@lists.fedoraproject.org# mailing list.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
See
|
||||
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[the
|
||||
Fedora 26 e-mail example].
|
||||
====
|
||||
____
|
||||
. Create a Tag to Contain the Mass Rebuild
|
||||
+
|
||||
____
|
||||
Mass rebuilds require their own tag to contain all related builds. The
|
||||
example assumes we are doing a rebuild for Fedora 26.
|
||||
|
||||
....
|
||||
$ koji add-tag f26-rebuild --parent f26
|
||||
....
|
||||
____
|
||||
. Request Package Auto-Signing for New Mass-Rebuild Tag
|
||||
+
|
||||
____
|
||||
File a ticket with https://pagure.io/fedora-infrastructure/issues[Fedora
|
||||
Infrastructure] requesting the new mass-rebuild tag be enabled for
|
||||
package auto-signing.
|
||||
____
|
||||
. Create the Koji Target for the Mass Rebuild
|
||||
+
|
||||
____
|
||||
Using the same [.title-ref]#f26-rebuild# tag created in the previous
|
||||
example:
|
||||
|
||||
....
|
||||
$ koji add-target f26-rebuild f26-build
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
*koji add-target* _target-name_ _buildroot-tag_ _destination-tag_
|
||||
describes the syntax format above. If the _destination-tag_ is not
|
||||
specified then it will be the same as the _target-name_.
|
||||
====
|
||||
____
|
||||
. Update Scripts
|
||||
+
|
||||
____
|
||||
The mass rebuild depends on four main scripts from the
|
||||
https://pagure.io/releng[releng git repository]. Each one requires some
|
||||
changes in variables for each new mass rebuild cycle.
|
||||
|
||||
* {blank}
|
||||
+
|
||||
mass-rebuild.py::
|
||||
** buildtag
|
||||
** targets
|
||||
** epoch
|
||||
** comment
|
||||
** target
|
||||
* {blank}
|
||||
+
|
||||
find-failures.py::
|
||||
** buildtag
|
||||
** desttag
|
||||
** epoch
|
||||
* mass-tag.py
|
||||
* {blank}
|
||||
+
|
||||
need-rebuild.py::
|
||||
** buildtag
|
||||
** target
|
||||
** updates
|
||||
** epoch
|
||||
____
|
||||
|
||||
Change the following items:
|
||||
|
||||
* the build tag, holding tag, and target tag should be updated to
|
||||
reflect the Fedora release you're building for
|
||||
* the `epoch` should be updated to the point at which all features that
|
||||
the mass rebuild is for have landed in the build system (and a newRepo
|
||||
task completed with those features)
|
||||
* the comment which is inserted into spec changelogs
|
||||
|
||||
==== Starting the Mass Rebuild
|
||||
|
||||
The `mass-rebuild.py` script takes care of:
|
||||
|
||||
* Discovering available packages in koji
|
||||
* Trimming out packages which have already been rebuilt
|
||||
* Checking out packages from git
|
||||
* Bumping the spec file
|
||||
* Committing the change
|
||||
* git tagging the change
|
||||
* Submitting the build request to Koji
|
||||
|
||||
. Connect to the mass-rebuild Machine
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ssh branched-composer.phx2.fedoraproject.org
|
||||
....
|
||||
____
|
||||
. Start a terminal multiplexer
|
||||
+
|
||||
____
|
||||
....
|
||||
$ tmux
|
||||
....
|
||||
____
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository].
|
||||
. Run the mass-rebuild.py script from _releng/scripts_
|
||||
+
|
||||
____
|
||||
....
|
||||
$ cd path/to/releng_repo/scripts
|
||||
$ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out
|
||||
....
|
||||
____
|
||||
|
||||
==== Monitoring Mass Rebuilds
|
||||
|
||||
The community has a very high interest in the status of rebuilds and
|
||||
many maintainers will want to know if their build failed right away. The
|
||||
`find-failures.py` and `need-rebuild.py` scripts are designed to update
|
||||
publicly available URLs for stakeholders to monitor.
|
||||
|
||||
. Connect to a Compose Machine
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ssh compose-x86-02.phx2.fedoraproject.org
|
||||
....
|
||||
____
|
||||
. Start a terminal multiplexer
|
||||
+
|
||||
____
|
||||
....
|
||||
$ tmux
|
||||
....
|
||||
____
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository]
|
||||
. {blank}
|
||||
+
|
||||
Set Up the Rebuild Failures Notification Web Site::
|
||||
The `find_failures.py` script discovers attempted builds that have
|
||||
failed. It lists those failed builds and sorts them by package owner.
|
||||
+
|
||||
....
|
||||
$ while true; do ./find_failures.py > f26-failures.html && cp f26-failures.html /mnt/koji/mass-rebuild/f26-failures.html; sleep 600; done
|
||||
....
|
||||
. Start a second pane in the terminal emulator
|
||||
. {blank}
|
||||
+
|
||||
Set up the Site for Packages that Need Rebuilt::
|
||||
The `need-rebuild.py` script discovers packages that have not yet been
|
||||
rebuilt and generates an html file listing them sorted by package
|
||||
owner. This gives external stakeholders a rough idea of how much work
|
||||
is remaining in the mass rebuild.
|
||||
+
|
||||
....
|
||||
$ while true; do ./need-rebuild.py > f26-need-rebuild.html && cp f26-need-rebuild.html /mnt/koji/mass-rebuild/f26-need-rebuild.html; sleep 600; done
|
||||
....
|
||||
|
||||
==== Post Mass Rebuild Tasks
|
||||
|
||||
Once the mass rebuild script completes, and all the pending builds have
|
||||
finished, the builds will need to be tagged. The `mass-tag.py` script
|
||||
will accomplish this task. The script will:
|
||||
|
||||
* Discover completed builds
|
||||
* Trim out builds that are older than the latest build for a given
|
||||
package
|
||||
* Tag remaining builds into their final destination (without generating
|
||||
email)
|
||||
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository]
|
||||
. Run the `mass-tag.py` script (requires koji kerberos authentication)
|
||||
+
|
||||
____
|
||||
....
|
||||
$ cd path/to/releng_repo/scripts
|
||||
$ ./mass-tag.py --source f36-rebuild --target f36
|
||||
....
|
||||
____
|
||||
. Send the final notification to the
|
||||
_devel-announce@lists.fedoraproject.org_ list
|
||||
+
|
||||
____
|
||||
The contents should look something like this
|
||||
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[example
|
||||
email].
|
||||
____
|
||||
141
modules/release_guide/pages/overview.adoc
Normal file
141
modules/release_guide/pages/overview.adoc
Normal file
|
|
@ -0,0 +1,141 @@
|
|||
[[overview]]
|
||||
== Fedora Release Engineering Overview
|
||||
|
||||
[[overview-intro]]
|
||||
=== Introduction
|
||||
|
||||
The development of Fedora is a very open process, involving over a
|
||||
thousand package maintainers (along with testers, translators,
|
||||
documentation writers and so forth). These maintainers are responsible
|
||||
for the bulk of Fedora distribution development. An elected
|
||||
https://fedoraproject.org/wiki/Fedora_Engineering_Steering_Committee[committee
|
||||
of people] provides some level of direction over the engineering aspects
|
||||
of the project.
|
||||
|
||||
The rapid pace of Fedora development leaves little time for polishing
|
||||
the development system into a quality release. To solve this dilemma,
|
||||
the Fedora project makes use of regular freezes and milestone (Alpha,
|
||||
Beta, Final) releases of the distribution, as well as "branching" of our
|
||||
trees to maintain different strains of development.
|
||||
|
||||
Stable branches of the Fedora tree and associated
|
||||
https://fedoraproject.org/wiki/Repositories[Repositories] are maintained
|
||||
for each Fedora release. The
|
||||
https://fedoraproject.org/wiki/Releases/Rawhide[Rawhide] rolling
|
||||
development tree is the initial entry point for all Fedora development,
|
||||
and the trunk from which all branches diverge. Releases are
|
||||
https://fedoraproject.org/wiki/Releases/Branched[Branched] from Rawhide
|
||||
some time before they are sent out as stable releases, and the milestone
|
||||
releases (Alpha, Beta and Final) are all built from this Branched tree.
|
||||
|
||||
Nightly snapshot images of various kinds are built from Rawhide and
|
||||
Branched (when it exists) and made available for download from within
|
||||
the trees on the https://mirrors.fedoraproject.org/[mirrors] or from the
|
||||
https://fedoraproject.org/wiki/Koji[Koji] build system.
|
||||
|
||||
The https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle[Fedora
|
||||
Release Life Cycle] page is a good entry point for more details on these
|
||||
processes. Some other useful references regarding the Fedora release
|
||||
process include:
|
||||
|
||||
* The https://fedoraproject.org/wiki/Changes/Policy[Release planning
|
||||
process]
|
||||
* The
|
||||
https://fedoraproject.org/wiki/QA:Release_validation_test_plan[release
|
||||
validation test plan]
|
||||
* The https://fedoraproject.org/wiki/QA:Updates_Testing[updates-testing
|
||||
process], via https://fedoraproject.org/wiki/Bodhi[Bodhi] and the
|
||||
https://fedoraproject.org/wiki/Updates_Policy[Updates Policy]
|
||||
* The https://fedoraproject.org/wiki/QA:SOP_compose_request[test compose
|
||||
and release candidate system]
|
||||
* The https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process[blocker
|
||||
bug process] and
|
||||
https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process[freeze
|
||||
exception bug process]
|
||||
* The [.title-ref]#Repositories#
|
||||
* The https://fedoraproject.org/wiki/Bugs_and_feature_requests[Bugzilla
|
||||
system]
|
||||
|
||||
=== Final Release Checklist
|
||||
|
||||
Various tasks need to be accomplished prior to a final Fedora release.
|
||||
Release Engineering is responsible for many of them, as outlined here.
|
||||
|
||||
==== Release Announcement
|
||||
|
||||
The https://fedoraproject.org/wiki/Docs_Project[Fedora Documentation
|
||||
Project] prepares release announcements for the final releases. A
|
||||
https://bugzilla.redhat.com/bugzilla/enter_bug.cgi?product=Fedora%20Documentation&op_sys=Linux&target_milestone=---&bug_status=NEW&version=devel&component=release-notes&rep_platform=All&priority=normal&bug_severity=normal&assigned_to=relnotes%40fedoraproject.org&cc=&estimated_time_presets=0.0&estimated_time=0.0&bug_file_loc=http%3A%2F%2F&short_desc=RELNOTES%20-%20Summarize%20the%20release%20note%20suggestion%2Fcontent&comment=Provide%20details%20here.%20%20Do%20not%20change%20the%20blocking%20bug.&status_whiteboard=&keywords=&issuetrackers=&dependson=&blocked=151189&ext_bz_id=0&ext_bz_bug_id=&data=&description=&contenttypemethod=list&contenttypeselection=text%2Fplain&contenttypeentry=&maketemplate=Remember%20values%20as%20bookmarkable%20template&form_name=enter_bug[bug
|
||||
needs to be filed] for this two weeks before the final release date.
|
||||
|
||||
==== Mirror List Files
|
||||
|
||||
A new set of mirror list files need to be created for the new release.
|
||||
Email mailto:mirror-admin@fedoraproject.org[Fedora Mirror Admins] to
|
||||
have these files created. These should be created at the Final Freeze
|
||||
point but may redirect to Rawhide until final bits have been staged.
|
||||
|
||||
=== Release Composing
|
||||
|
||||
Fedora “releases” can be built by anyone with a fast machine of proper
|
||||
arch and access to a package repository. All of the tools necessary to
|
||||
build a release are available from the package repository. These tools
|
||||
aim to provide a consistent way to build Fedora releases. A complete
|
||||
release can actually be built with only a couple commands, including the
|
||||
creation of network install images, offline install images ('DVDs'),
|
||||
live images, disk images, install repositories, [[FedUp]] upgrade
|
||||
images, and other bits. These commands are named pungi and
|
||||
livecd-creator.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
There is work currently being done to replace livecd-creator with
|
||||
https://github.com/rhinstaller/lorax/blob/master/src/sbin/livemedia-creator[livemedia-creator].
|
||||
====
|
||||
|
||||
==== Pungi
|
||||
|
||||
https://pagure.io/pungi[Pungi] is the tool used to compose Fedora
|
||||
releases. It requires being ran in a chroot of the package set that it
|
||||
is composing. This ensures that the correct userland tools are used to
|
||||
match up with the kernel that will be used to perform the installation.
|
||||
It uses a comps file + yum repos to gather packages for the compose. The
|
||||
https://fedoraproject.org/wiki/Koji[Koji] build system provides a way to
|
||||
run tasks within chroots on the various arches, as well as the ability
|
||||
to produce yum repos of packages from specific collections.
|
||||
|
||||
==== Livecd-creator
|
||||
|
||||
Livecd-creator is part of the
|
||||
https://fedoraproject.org/wiki/FedoraLiveCD[livecd-tools] package in
|
||||
Fedora and it is used to compose the live images as part of the Fedora
|
||||
release. It is also used to compose many of the custom
|
||||
https://spins.fedoraproject.org[Spins] or variants of Fedora.
|
||||
|
||||
=== Distribution
|
||||
|
||||
Once a compose has been completed, the composed tree of release media,
|
||||
installation trees, and frozen
|
||||
https://fedoraproject.org/wiki/Repositories[Repositories] needs to be
|
||||
synchronized with the Fedora mirror system. [[MirrorManager]] has some
|
||||
more details on the mirror system. Many of the images are also offered
|
||||
via BitTorrent as an alternative method of downloading.
|
||||
|
||||
==== Download Mirrors
|
||||
|
||||
Depends on the Fedora Mirror System and infrastructure to populate them
|
||||
privately.
|
||||
|
||||
==== BitTorrent
|
||||
|
||||
BitTorrent is currently served by http://torrent.fedoraproject.org.
|
||||
Images are added to the system via this
|
||||
https://infrastructure.fedoraproject.org/infra/docs/docs/sysadmin-guide/sops/torrentrelease.rst[Standard
|
||||
Operating Procedure].
|
||||
|
||||
=== Acknowledgements
|
||||
|
||||
This document was influenced by
|
||||
http://www.freebsd.org/doc/en_US.ISO8859-1/articles/releng/article.html[release
|
||||
engineering documents] from http://freebsd.org[FreeBSD].
|
||||
130
modules/release_guide/pages/philosophy.adoc
Normal file
130
modules/release_guide/pages/philosophy.adoc
Normal file
|
|
@ -0,0 +1,130 @@
|
|||
[[philosophy]]
|
||||
== Fedora Release Engineering Philosophy
|
||||
|
||||
Being an official part of Fedora means that it is composed and supported
|
||||
by Fedora Release Engineering (releng). Releng bestows this status on
|
||||
items that successfully makes its way through our Manufacturing Train
|
||||
and satisfies all related policies. With this status the item is Open,
|
||||
Integrated, Reproducible, Auditable, Definable, and Deliverable. (in no
|
||||
particular order) This document provides definitions for each of those
|
||||
terms, why they're important, and how we guarantee them. All parts of
|
||||
Fedora should strive to be meet all parts of being official at all
|
||||
times.
|
||||
|
||||
=== Open
|
||||
|
||||
It goes without saying that Fedora is built on
|
||||
https://fedoraproject.org/wiki/Foundations[the four F's]. In releng we
|
||||
are no different, we require everything be open, that is open source,
|
||||
developed in the open, available for all to look at, use and contribute
|
||||
to. All downstreams should be able to take our tooling and make their
|
||||
own derivative of Fedora, either by rebuilding everything, perhaps with
|
||||
different otions or just adding thier own marks and packages and
|
||||
recomposing. At any time anyone should be able to see how Fedora is put
|
||||
together and put together their own version of Fedora.
|
||||
|
||||
=== Integrated
|
||||
|
||||
Fedora is a huge project with a massive number of ever growing
|
||||
deliverables. This means when we add new deliverables we need to have
|
||||
the composing of them tightly integrated into the process. We ship
|
||||
Fedora a whole unit, so we need to make it as a whole unit. Any new
|
||||
tooling we use needs to be consistent with the existing tooling in how
|
||||
it works. The tooling has to ensure that the output is Reproducible,
|
||||
Auditable, Definable so that it can be Deliverable.
|
||||
|
||||
=== Reproducible
|
||||
|
||||
A reproducible component is one we can rebuild from scratch at any time
|
||||
with less than a day's worth of effort. It implies we can look up all of
|
||||
the source code, and know exactly what revisions to use from source
|
||||
control. We know exactly what tools are used in the build environment to
|
||||
transform the source code into product content (binaries). We also know
|
||||
how to reproduce the same build environment to ensure the tools behave
|
||||
like we expect. This aspect is why releng is in the business of
|
||||
standardizing on build tools.
|
||||
|
||||
Reproducible components are important because they make them a lot
|
||||
easier to maintain. The Security Team takes advantage of this aspect of
|
||||
an Product. Not knowing how to rebuild a subsystem in a product to apply
|
||||
security fixes, or bug fixes, makes their job much more difficult. It
|
||||
would be a significant risk to provide a product to users that we are
|
||||
incapable (or merely unsure) of how to build again. Not knowing the
|
||||
origin of source code is also a significant risk, which is why many of
|
||||
our build environments are configured to prevent tools from dynamically
|
||||
downloading content from the internet.
|
||||
|
||||
The combination of Koji and fedpkg is what enables releng to rebuild a
|
||||
component. fedpkg manages the source code in our dist-git system, and
|
||||
Koji archives details about the build environment, the tools used, logs,
|
||||
and of course the binaries themselves. The reproducibility aspect of a
|
||||
product is the primary reason we require all products be built in Koji
|
||||
if they seek to be an officially supported part of Fedora.
|
||||
|
||||
=== Auditable
|
||||
|
||||
Fedora and Red Hat expect auditable output too, which means releng knows
|
||||
who built what, when, and where (and how, but that's reproducibility).
|
||||
Being able to authoritatively say that something was built within Fedora
|
||||
by people who have signed the FPCA is important for several reasons. One
|
||||
big reason is it promotes and fosters accountability within the
|
||||
comunity. It promotes ownership. Another one is more defensive: in the
|
||||
event of a security breach, we have a lot of evidence and data prepared
|
||||
to help us identify what content (if any) was compromised. If a kernel
|
||||
RPM randomly shows up, and we have no records of building it and/or
|
||||
shipping it, that should raise a lot of alarms pretty quickly!
|
||||
|
||||
Red Hat's Infosec team and Fedora Security care about this aspect
|
||||
deeply. We should never be in a position where we cannot definitively
|
||||
answer why a piece of content is available to users. This aspect is also
|
||||
a key part of the verification that is done when Fedora becomes RHEL.
|
||||
All downstream consumers of Fedora expect that they can verify the code
|
||||
and binaries that they consume from Fedora.
|
||||
|
||||
Releng tracks this data in 2 systems, 1 of which we own: Koji and Bodhi.
|
||||
Koji uses ssl certs tied to FAS and bodhi uses FAS for authentication to
|
||||
provide a strong relationship between a user and the content. Koji
|
||||
builds the content of course, the Bodhi tracks the bugs, documentation,
|
||||
and enhancements associated with the content and actually does the
|
||||
delivery. Bodhi maintains records of what was shipped when and where,
|
||||
and who pushed it.
|
||||
|
||||
=== Definable
|
||||
|
||||
The ability to define and predict content is necessary as well. It is
|
||||
important to know exactly what was included in a release. It helps
|
||||
protect against shipping content that unnecessarily causes a support
|
||||
burden. This aspect of a Fedora component helps support other aspects
|
||||
like reproducibility. No need to reproduce software we do not have to
|
||||
ship, right? Ensuring the product content is lean and trim may sound
|
||||
obvious, but in the world of sprawling RPM dependencies, Maven
|
||||
artifacts, and Ruby gems, it is actually rather easy to include content
|
||||
during the course of a multi-month or multi-year development cycle.
|
||||
|
||||
Furthermore, a definable component has the changes made to it vetted and
|
||||
understood by multiple teams. They are not made in an ad-hoc manor or
|
||||
without consent from FESCo, QA, releng, and the Product Working Groups
|
||||
that contribute at the program level. This reduces change risk to the
|
||||
user, which our users and downstreams like to hear.
|
||||
|
||||
Many systems help make components definable. Releng uses Bugzilla, trac,
|
||||
blocker bugs and bodhi to track additions and changes.
|
||||
|
||||
=== Deliverable
|
||||
|
||||
Official parts of Fedora are eligible to be delivered to `/pub/fedora/`
|
||||
or `/pub/alt/releases` on https://dl.fedoraproject.org/pub/[Fedora
|
||||
Download] and to get
|
||||
https://admin.fedoraproject.org/mirrormanager[mirrorlists] in
|
||||
https://github.com/fedora-infra/mirrormanager2[mirrormanager]. These
|
||||
Distribution Platforms are maintained by Fedora Infrastructure and
|
||||
releng. This is not a feature of the product content itself or how it
|
||||
was built, but rather how it was delivered to users through releng's
|
||||
processes. Those platforms are geographically replicated by the
|
||||
volunteer mirror network. They provide a reliable and durable service
|
||||
that ensures users can always reach Fedora for updates, even in the
|
||||
event of a disaster affecting our data center in phx2.
|
||||
|
||||
User support (user mailing lists and [.title-ref]##fedora#) and Fedora
|
||||
Security team depend on these services. It is vital to that critical
|
||||
security fixes and updates are always available to users.
|
||||
40
modules/release_guide/pages/release_process.adoc
Normal file
40
modules/release_guide/pages/release_process.adoc
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
include::_partials/attributes.adoc[]
|
||||
|
||||
= Release process.
|
||||
|
||||
== Rawhide starts Fedora Linux {branched} development
|
||||
== Mass Rebuild: RPMs first, then modules
|
||||
== Branch Fedora Linux {branched} from Rawhide
|
||||
== Rawhide starts Fedora Linux {rawhide} development
|
||||
== Update Greenwave policy product_versions to add branched release
|
||||
== Post-branch Freeze (end date approximate)
|
||||
== Bodhi updates-testing activation point
|
||||
== Beta Freeze (starts at 1400 UTC)
|
||||
== Beta Release Public Availability
|
||||
== Final Freeze (starts at 1400 UTC)
|
||||
== Final Release Public Availability (GA)
|
||||
== Fedora Linux {old_release} EOL auto closure
|
||||
== Beta release Go/No-Go Meeting
|
||||
== Final release Go/No-Go Meeting
|
||||
== Update Greenwave policy product_versions to remove EOL release
|
||||
== Notifications for retirement of packages with security issues
|
||||
== Retire packages with security issues
|
||||
== fedora-repos update with the new keys is available in updates-testing
|
||||
== Begin reminders for packages that haven't rebuilt in last 2 releases
|
||||
== Re-sign F{branched} content with F{rawhide} key
|
||||
== Retire Orphaned and Long-Time FTBFS Rawhide Packages
|
||||
== Retire FTI packages in NEW state
|
||||
== Stage Beta on master minor
|
||||
== Bit-flip Beta to mirrors
|
||||
== Notify Mirrors of Beta Release
|
||||
== Set current release to active in Package database
|
||||
== Retire all packages with broken deps
|
||||
== Retire FTI packages in NEW state
|
||||
== Enable Updates in Bodhi for the current release
|
||||
== Stage Final on master minor
|
||||
== Bit-flip Final to mirrors
|
||||
== Update PDC with F{old_release} EOL date
|
||||
== REMINDER: EOL release final update push in 1 week
|
||||
== REMINDER: EOL release final update push in 1 day
|
||||
== Notify Mirrors of Final Release
|
||||
== Remove Beta images
|
||||
48
modules/release_guide/pages/sop.adoc
Normal file
48
modules/release_guide/pages/sop.adoc
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
[[sop]]
|
||||
== Fedora Release Engineering Standard Operating Procedures
|
||||
|
||||
This page documents the Standard Operating Procedures for Fedora Release
|
||||
Engineering.
|
||||
|
||||
=== Purpose
|
||||
|
||||
The SOP section contains a list of tasks Fedora Release Engineering team
|
||||
provides for the project. Current Fedora Release Engineering team
|
||||
members would can add tasks they know about and list steps to complete
|
||||
the task as well as issues to consider. This is a great way to make sure
|
||||
that individuals aren't the only ones that can fix a problem.
|
||||
|
||||
=== The Community
|
||||
|
||||
The SOP section is left in the public because it is our hope that others
|
||||
in the community will add common problems, fix our steps and just in
|
||||
general look over what we're doing and help us when we're doing
|
||||
something stupid. We encourage anyone interested to go over our
|
||||
processes with a fine tooth comb. It'll make us better and we'll
|
||||
probably learn something in the process. Therefore please open a
|
||||
https://pagure.io/releng/pull-requests[pull request] to suggest
|
||||
improvements.
|
||||
|
||||
=== Procedures
|
||||
|
||||
Needed: #. Composing an official release #. Making a composed release
|
||||
publicly available
|
||||
|
||||
=== Standard Operating Procedures
|
||||
|
||||
sop_adding_build_targets sop_adding_content_generator
|
||||
sop_adding_new_release_engineer sop_adding_new_branch_sla
|
||||
sop_adding_packages_compose_artifact sop_adding_side_build_targets
|
||||
sop_branch_freeze sop_branching sop_breaking_development_freeze
|
||||
sop_composing_fedora sop_clean_amis sop_create_release_signing_key
|
||||
sop_deprecate_ftbfs_packages sop_end_of_life sop_eol_change
|
||||
sop_mass_branching sop_bodhi_activation sop_mass_rebuild_packages
|
||||
sop_mass_rebuild_modules sop_file_ftbfs sop_generating_openh264_composes
|
||||
sop_package_blocking sop_package_unblocking
|
||||
sop_process_dist_git_requests sop_promoting_container_content
|
||||
sop_signing_builds sop_pushing_updates sop_release_package_signing
|
||||
sop_remote_dist_git_branches sop_requesting_task_automation_users
|
||||
sop_retire_orphaned_packages sop_sigul_client_setup
|
||||
sop_stage_final_release_for_mirrors sop_unretire sop_update_critpath
|
||||
sop_update_releng_docs sop_updating_comps sop_fedora_media_writer
|
||||
sop_find_module_info sop_release_container_base_image
|
||||
180
modules/release_guide/pages/sop_adding_build_targets.adoc
Normal file
180
modules/release_guide/pages/sop_adding_build_targets.adoc
Normal file
|
|
@ -0,0 +1,180 @@
|
|||
== Adding Build Targets
|
||||
|
||||
=== Description
|
||||
|
||||
Each new release we create a build target for the next release. This SOP
|
||||
will describe the steps necessary to prepare the new build target.
|
||||
|
||||
=== Action
|
||||
|
||||
Adding a build target is a complicated task. It involves updating koji,
|
||||
git, and fedora-release package.
|
||||
|
||||
[[adding_build_targets_koji]]
|
||||
==== Koji
|
||||
|
||||
In koji a couple collection tags need to be made, and a target created
|
||||
to tie them together. We create a base collection tag named after the
|
||||
release, and a build tag to hold a few things we use in the buildroots
|
||||
that are not part of the distribution (glibc32/glibc64). Inheritance to
|
||||
the previous release is used for ownership and package data, as well as
|
||||
buildroot content data.
|
||||
|
||||
The `add-tag`, `add-tag-inheritance`, `edit-tag`, and `add-target`
|
||||
commands are used.
|
||||
|
||||
....
|
||||
$ koji add-tag --help
|
||||
Usage: koji add-tag [options] name
|
||||
(Specify the --help global option for a list of other help options)
|
||||
|
||||
Options:
|
||||
-h, --help show this help message and exit
|
||||
--parent=PARENT Specify parent
|
||||
--arches=ARCHES Specify arches
|
||||
|
||||
|
||||
$ koji add-tag-inheritance --help
|
||||
Usage: koji add-tag-inheritance [options] tag parent-tag
|
||||
(Specify the --help global option for a list of other help options)
|
||||
|
||||
Options:
|
||||
-h, --help show this help message and exit
|
||||
--priority=PRIORITY Specify priority
|
||||
--maxdepth=MAXDEPTH Specify max depth
|
||||
--intransitive Set intransitive
|
||||
--noconfig Set to packages only
|
||||
--pkg-filter=PKG_FILTER
|
||||
Specify the package filter
|
||||
--force=FORCE Force adding a parent to a tag that already has that
|
||||
parent tag
|
||||
|
||||
$ koji edit-tag --help
|
||||
Usage: koji edit-tag [options] name
|
||||
(Specify the --help global option for a list of other help options)
|
||||
|
||||
Options:
|
||||
-h, --help show this help message and exit
|
||||
--arches=ARCHES Specify arches
|
||||
--perm=PERM Specify permission requirement
|
||||
--no-perm Remove permission requirement
|
||||
--lock Lock the tag
|
||||
--unlock Unlock the tag
|
||||
--rename=RENAME Rename the tag
|
||||
|
||||
$ koji add-target --help
|
||||
Usage: koji add-target name build-tag <dest-tag>
|
||||
(Specify the --help global option for a list of other help options)
|
||||
|
||||
Options:
|
||||
-h, --help show this help message and exit
|
||||
....
|
||||
|
||||
For example if we wanted to create the Fedora 17 tags, we would do the
|
||||
following:
|
||||
|
||||
....
|
||||
koji add-tag --parent f16-updates f17
|
||||
koji add-tag --parent f17 f17-updates
|
||||
koji add-tag --parent f17-updates f17-candidate
|
||||
koji add-tag --parent f17-updates f17-updates-testing
|
||||
koji add-tag --parent f17-updates-testing f17-updates-testing-pending
|
||||
koji add-tag --parent f17-updates f17-updates-pending
|
||||
koji add-tag --parent f17-updates f17-override
|
||||
koji add-tag --parent f17-override --arches=i686,x86_64 f17-build
|
||||
koji add-tag-inheritance --priority 1 f17-build f16-build
|
||||
koji edit-tag --perm=fedora-override f17-override
|
||||
koji edit-tag --lock f17-updates
|
||||
koji add-target f17 f17-build
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
The `-pending` tags are used by
|
||||
https://fedoraproject.org/wiki/Bodhi[Bodhi] and
|
||||
https://fedoraproject.org/wiki/Taskotron[Taskotron] to track and test
|
||||
proposed updates. These tags are not build targets and they don't get
|
||||
made into repos, so proper inheritance isn't vital.
|
||||
====
|
||||
|
||||
==== Git
|
||||
|
||||
The pkgdb_sync_git_branches.py file which is hosted in Fedora
|
||||
Infrastructure ansible
|
||||
(roles/distgit/templates/pkgdb_sync_git_branches.py) needs to be updated
|
||||
for the new target for making branches.
|
||||
|
||||
Update `BRANCHES` with the new branch information. The branch name maps
|
||||
to the branch that is its parent.
|
||||
|
||||
....
|
||||
BRANCHES = {'el4': 'rawhide', 'el5': 'rawhide', 'el6': 'f12',
|
||||
'OLPC-2': 'f7',
|
||||
'rawhide': None,
|
||||
'fc6': 'rawhide',
|
||||
'f7': 'rawhide',
|
||||
'f8': 'rawhide',
|
||||
'f9': 'rawhide',
|
||||
'f10': 'rawhide',
|
||||
'f11': 'rawhide',
|
||||
'f12': 'rawhide',
|
||||
'f13': 'rawhide', 'f14': 'rawhide'}
|
||||
....
|
||||
|
||||
and update `GITBRANCHES` with the translation from pkgdb branch string
|
||||
to git branch string:
|
||||
|
||||
....
|
||||
GITBRANCHES = {'EL-4': 'el4', 'EL-5': 'el5', 'EL-6': 'el6', 'OLPC-2': 'olpc2',
|
||||
'FC-6': 'fc6', 'F-7': 'f7', 'F-8': 'f8', 'F-9': 'f9', 'F-10': 'f10',
|
||||
'F-11': 'f11', 'F-12': 'f12', 'F-13': 'f13', 'F-14': 'f14', 'devel': 'rawhide'}
|
||||
....
|
||||
|
||||
The genacls.pkgdb file also needs to be updated for active branches to
|
||||
generate ACLs for. Update the `ACTIVE` variable. genacls.pkgdb lives in
|
||||
puppet (modules/gitolite/files/distgit/genacls.pkgdb). The format is
|
||||
pkgdb branch string to git branch string (until pkgdb uses git branch
|
||||
strings):
|
||||
|
||||
....
|
||||
ACTIVE = {'OLPC-2': 'olpc2/', 'EL-4': 'el4/', 'EL-5': 'el5/',
|
||||
'EL-6': 'el6/', 'F-11': 'f11/', 'F-12': 'f12/', 'F-13': 'f13/',
|
||||
'F-14': 'f14/', 'devel': 'rawhide'}
|
||||
....
|
||||
|
||||
==== fedora-release
|
||||
|
||||
Currently the fedora-release package provides the `%{?dist}` definitions
|
||||
used during building of packages. When a new target is created,
|
||||
fedora-release must be built for the collection with new dist
|
||||
definitions.
|
||||
|
||||
==== Comps
|
||||
|
||||
* In the comps module in Fedora Hosted git
|
||||
(ssh://git.fedorarhosted.org/git/comps.git), create and add a new comps
|
||||
file based on the previous release. (Just copying it should be fine.)
|
||||
Add the new file to `po/POTFILES.in`.
|
||||
* When rawhide is retargeted in koji to point to the new release, update
|
||||
the `Makefile` to target comps-rawhide.xml at the new version.
|
||||
* Don't forget to `git push` your changes after committing.
|
||||
|
||||
=== Verification
|
||||
|
||||
Given the complexity of all the changes necessary to create a new build
|
||||
target, the best way to verify is to attempt a build. Given that
|
||||
fedora-release has to be built before anything else so that dist tags
|
||||
translate correctly it is a good test case. For example, this was used
|
||||
to test the new Fedora 15 target:
|
||||
|
||||
* Use pkgdb to request an F-15 branch of fedora-release
|
||||
* Use pkgdb2branch.py to actually make the branch
|
||||
* Update fedora-release clone
|
||||
* Adjust .spec file in rawhide for new dist defines
|
||||
* commit/build
|
||||
* Track build in koji to ensure proper tagging is used
|
||||
|
||||
What this won't test is translations of dist at tag time given that
|
||||
`fedora-release` doesn't use dist in it's Release. Verifying with a
|
||||
second package that uses dist is a good idea.
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
== Adding new koji content generator
|
||||
|
||||
=== Description
|
||||
|
||||
Koji added support for content generators some time ago. Basic premise
|
||||
of content generators is that it lets us create build systems for new
|
||||
types of content without affecting or changing core Koji code and in
|
||||
some way simplify integration with rest of the release toolchain. More
|
||||
information about content generators, background, requirements and more
|
||||
can be found in Koji
|
||||
https://docs.pagure.org/koji/content_generators/[content generator
|
||||
documentation]
|
||||
|
||||
For content generator to be able to create/import builds into Koji
|
||||
following prerequisites have to be met:
|
||||
|
||||
* Koji has to recognize the content generator type
|
||||
* User doing the content generator import has to have permissions for
|
||||
this action
|
||||
* Any new content types have to be defined in Koji
|
||||
|
||||
=== Questions to ask
|
||||
|
||||
There are some questions that should be answered before the content
|
||||
generator is enabled/added to Koji
|
||||
|
||||
* Where is the content generator service running, what is its support
|
||||
status etc?
|
||||
* Is new type of content being added or is the content generator
|
||||
providing different way to build content Koji already knows about?
|
||||
* What is the expected size of content that will be imported into Koji?
|
||||
* Does the content generator follow each of the requirements for writing
|
||||
it from Koji documentation referenced above?
|
||||
|
||||
=== Adding a new content generator into koji
|
||||
|
||||
First we create the content generator and give a user permission to do
|
||||
imports for it:
|
||||
|
||||
....
|
||||
koji grant-cg-access <username> <content_generator_name> --new
|
||||
....
|
||||
|
||||
In many cases the content generator will be adding content with new
|
||||
content type. This can be achieved simply by running:
|
||||
|
||||
....
|
||||
koji call addBType <content type>
|
||||
....
|
||||
|
||||
==== Explanation
|
||||
|
||||
* username - is a name of user which will be doing the imports. In most
|
||||
cases this will be a service-level account
|
||||
* content generator name - this name has to be provided by the content
|
||||
generator development team
|
||||
* --new - this switch ensures we create the content generator if it
|
||||
doesn't exist
|
||||
28
modules/release_guide/pages/sop_adding_new_branch_sla.adoc
Normal file
28
modules/release_guide/pages/sop_adding_new_branch_sla.adoc
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
== Adding New Branch SLAs
|
||||
|
||||
=== Description
|
||||
|
||||
In the ArbitraryBranching model, packagers can choose whatever SLAs they
|
||||
want for the branches of their packages, but they must choose from a
|
||||
subset of pre-defined SLAs stored in PDC, maintained by releng.
|
||||
|
||||
This SOP describes the steps necessary for a release engineer to create
|
||||
a new SLA.
|
||||
|
||||
=== Action
|
||||
|
||||
Adding a new SLA is simple. It involves running a script in the releng
|
||||
repo, with an authorized token. There is a token available on
|
||||
[.title-ref]#pdc-backend01# in the [.title-ref]#/etc/pdc.d/# directory.
|
||||
|
||||
....
|
||||
$ ./scripts/pdc/insert-sla.py
|
||||
Name of the SLA: wild_and_cavalier
|
||||
Description of the SLA: Anything goes! This branch may rebase at any time. No stability guarantees provided.
|
||||
....
|
||||
|
||||
=== Verification
|
||||
|
||||
Verifying that the SLA is present is simple: visit the
|
||||
https://pdc.fedoraproject.org/rest_api/v1/component-branch-slas/[appropriate
|
||||
PDC endpoint] and verify that your newly-added SLA is present.
|
||||
160
modules/release_guide/pages/sop_adding_new_release_engineer.adoc
Normal file
160
modules/release_guide/pages/sop_adding_new_release_engineer.adoc
Normal file
|
|
@ -0,0 +1,160 @@
|
|||
== Adding a New Release Engineer
|
||||
|
||||
=== Description
|
||||
|
||||
People volunteer (or get assigned) to doing Fedora release engineering
|
||||
from time to time. This SOP seeks to describe the process to add a new
|
||||
release engineer so that they have the rights to accomplish their tasks,
|
||||
know where to find the tasks, and are introduced to the existing
|
||||
members. There are several groups that manage access to the respective
|
||||
systems:
|
||||
|
||||
* `cvsadmin`: Admin group for pkgdb2 (allows to retire/orphan all
|
||||
packages etc), allows git write access via SSH to all packages in
|
||||
dist-git
|
||||
* `gitreleng`: Allows write access to release engineering git repo
|
||||
* `signers`: Membership is necessary to use
|
||||
https://fedoraproject.org/wiki/Sigul_Client_Setup_SOP[sigul].
|
||||
* `sysadmin`: Allows SSH access to bastion, the SSH gateway to the PHX2
|
||||
data center. SSH access to several other internal systems is only
|
||||
possible from here.
|
||||
* `sysadmin-cvs`: Allows shell access to pkgs01 (pkgs.fedoraproject.org)
|
||||
* `sysadmin-releng`: Allows SSH access to autosign01, koji03, koji04,
|
||||
releng04, relepel01 from bastion
|
||||
|
||||
=== Action
|
||||
|
||||
A new release engineer will access rights in a variety of systems, as
|
||||
well as be introduced to the releng group.
|
||||
|
||||
==== Git
|
||||
|
||||
Fedora Release Engineering maintains a git repo of scripts. This can be
|
||||
found in https://pagure.io/pagure[Pagure] at
|
||||
ssh://git@pagure.io/releng.git. Write access to this group is controlled
|
||||
by access to the 'gitreleng' FAS group. The new member's FAS username
|
||||
will need to be added to this group.
|
||||
|
||||
https://pagure.io/releng
|
||||
|
||||
`FIXME: walkthrough group addition`
|
||||
|
||||
==== FAS
|
||||
|
||||
There is a releng group in FAS that release engineers are added to in
|
||||
order to grant them various rights within the Fedora infrastructure. The
|
||||
new member's FAS username will need to be added to this group.
|
||||
|
||||
`FIXME: walkthrough group addition`
|
||||
|
||||
==== Koji
|
||||
|
||||
In order to perform certain (un)tagging actions a release engineer must
|
||||
be an admin in koji. To grant a user admin rights one uses the
|
||||
`grant-permission` command in koji.
|
||||
|
||||
....
|
||||
$ koji grant-permission --help
|
||||
Usage: koji grant-permission <permission> <user> [<user> ...]
|
||||
(Specify the --help global option for a list of other help options)
|
||||
|
||||
Options:
|
||||
-h, --help show this help message and exit
|
||||
....
|
||||
|
||||
For example if we wanted to grant npetrov admin rights we would issue:
|
||||
|
||||
....
|
||||
$ koji grant-permission admin npetrov
|
||||
....
|
||||
|
||||
==== Sigul
|
||||
|
||||
Sigul is our signing server system. They need to bet setup as a signer
|
||||
if they are going to be signing packages for a release.
|
||||
|
||||
For information on how to setup Sigul, please see:
|
||||
https://fedoraproject.org/wiki/Sigul_Client_Setup_SOP[sigul]
|
||||
|
||||
==== RelEng Docs Page
|
||||
|
||||
The new release engineer should be added to the
|
||||
ref:link:index-team-composition[Release Engineering membership list]
|
||||
|
||||
==== rel-eng email list
|
||||
|
||||
Release engineering ticket spam and discussion happens on our
|
||||
https://admin.fedoraproject.org/mailman/listinfo/rel-eng[Mailing List].
|
||||
New releng people need to subscribe.
|
||||
|
||||
==== IRC
|
||||
|
||||
We ask that release engineers idle in [.title-ref]##fedora-releng# on
|
||||
Libera to be available for queries by other infrastructure admins.
|
||||
Idling on [.title-ref]##fedora-admin# on Libera is optional. It is noisy
|
||||
little bit but people sometimes ask releng stuff.
|
||||
|
||||
==== New member announcement
|
||||
|
||||
When a new releng member starts, we announce it to the email list. This
|
||||
lets the other admins know to expect a new name to show up in tickets
|
||||
and on IRC.
|
||||
|
||||
=== Verification
|
||||
|
||||
==== Git
|
||||
|
||||
You can verify group membership by sshing to a system that is setup with
|
||||
FAS and using `getent` to verify membership in the gitreleng group:
|
||||
|
||||
....
|
||||
$ ssh fedorapeople.org getent group gitreleng
|
||||
gitreleng:x:101647:ausil,dwa,jwboyer,kevin,notting,pbabinca,sharkcz,skvidal,spot
|
||||
....
|
||||
|
||||
You can verify that the new user is in the above list.
|
||||
|
||||
==== FAS
|
||||
|
||||
You can verify group membership by sshing to a system that is setup with
|
||||
FAS and using `getent` to verify membership in the releng group:
|
||||
|
||||
....
|
||||
$ ssh fedorapeople.org getent group releng
|
||||
releng:x:101737:atowns,ausil,dwa,jwboyer,kevin,lmacken,notting,pbabinca,spot
|
||||
....
|
||||
|
||||
You can verify that the new user is in the above list.
|
||||
|
||||
==== Koji
|
||||
|
||||
To verify that the releng member is an admin koji use the
|
||||
`list-permissions` koji command:
|
||||
|
||||
....
|
||||
$ koji list-permissions --user npetrov
|
||||
admin
|
||||
....
|
||||
|
||||
This shows that npetrov is an admin.
|
||||
|
||||
==== Sigul
|
||||
|
||||
* `FIXME`
|
||||
|
||||
==== Wiki Page
|
||||
|
||||
Verification is easy. Just look at the page.
|
||||
|
||||
==== releng mailing list
|
||||
|
||||
Verify by asking the user if they got the announcement email
|
||||
|
||||
==== Announcement email
|
||||
|
||||
See above
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
* Make sure the releng person has a solid grasp of the tasks we do and
|
||||
where to get help if stuck
|
||||
|
|
@ -0,0 +1,81 @@
|
|||
== Adding a package to a Release Artifact
|
||||
|
||||
=== Description
|
||||
|
||||
In the event that a Fedora contributor would like to have a package
|
||||
added to an Artifact of a Compose (such as an installer ISO image, a
|
||||
liveCD, Cloud Image, Vagrant, Docker, etc.) that is slated for Release,
|
||||
the following procedures must be followed due to the interdependence of
|
||||
different components within the distro layout.
|
||||
|
||||
==== Background
|
||||
|
||||
First, some information on where this all comes from and how things fit
|
||||
together.
|
||||
|
||||
There is the concept of the "Install Tree" which is the collection of
|
||||
packages available at install time. This is a vast sub-set of the whole
|
||||
of the Fedora Package Collection and it is the pool of possible packages
|
||||
that is available to end users who choose to customize their install
|
||||
from the https://fedoraproject.org/wiki/Anaconda[Anaconda] installer. It
|
||||
is also the pool of possible packages that is available to the
|
||||
https://pagure.io/fedora-kickstarts[fedora-kickstarts] kickstart files
|
||||
that are used to generate various components of the compose via
|
||||
https://pagure.io/pungi[pungi] which then produces the Release
|
||||
Artifacts.
|
||||
|
||||
The Install Tree itself is defined by the
|
||||
https://pagure.io/fedora-comps[comps] groups so in order to add a net
|
||||
new package to one of the Release Artifacts, the package must be placed
|
||||
in an appropriate https://pagure.io/fedora-comps[comps] xml file. For
|
||||
more information on what specifically defines the "appropriate
|
||||
[.title-ref]#comps_# xml file" and what kinds of approvals or review
|
||||
might be needed for adding new packages, please see
|
||||
https://fedoraproject.org/wiki/How_to_use_and_edit_comps.xml_for_package_groups[this
|
||||
HowTo].
|
||||
|
||||
=== Action
|
||||
|
||||
We will need to edit the comps file specific to the Fedora release we
|
||||
would like to target. For example, if we were to target Fedora 25 we
|
||||
would edit `comps-f25.xml.in` found within the
|
||||
https://pagure.io/fedora-comps[comps] git repository and this should be
|
||||
modified based on the
|
||||
https://fedoraproject.org/wiki/How_to_use_and_edit_comps.xml_for_package_groups#How_to_edit_comps[How
|
||||
to edit comps] procedure.
|
||||
|
||||
If the package that was added is a part of a pre-existing comps group
|
||||
that is already used in the target Release Artifact's
|
||||
https://pagure.io/fedora-kickstarts[fedora-kickstarts] kickstart file
|
||||
then we are done.
|
||||
|
||||
However, if there is a new comps group added then we need to include
|
||||
that new comps group in the respective
|
||||
https://pagure.io/fedora-kickstarts[fedora-kickstarts] kickstart file
|
||||
similar to the following.
|
||||
|
||||
....
|
||||
%packages
|
||||
@mynewcompsgroup
|
||||
....
|
||||
|
||||
Next we will need to tell https://pagure.io/pungi[pungi] Variants data
|
||||
about the new group and it's relationship to the corresponding
|
||||
https://sgallagh.wordpress.com/2016/03/18/sausage-factory-multiple-edition-handling-in-fedora/[Variant].
|
||||
This information is held in the https://pagure.io/pungi-fedora[Fedora
|
||||
Pungi Configs] https://pagure.io/[pagure] git forge repository. The file
|
||||
needed to be edited is `variants-fedora.xml` and can be viewed from a
|
||||
web browser
|
||||
https://pagure.io/pungi-fedora/blob/master/f/variants-fedora.xml[here].
|
||||
|
||||
Once this has been completed, we're all done.
|
||||
|
||||
=== Verification
|
||||
|
||||
Verify that the next compose is successful and that the change made
|
||||
didn't cause any issues. This can be done from the
|
||||
https://pdc.fedoraproject.org/compose/[Fedora Product Definition Center]
|
||||
which is a central store of information about Composes and their
|
||||
resulting artifacts.
|
||||
|
||||
=== Consider Before Running
|
||||
208
modules/release_guide/pages/sop_adding_side_build_targets.adoc
Normal file
208
modules/release_guide/pages/sop_adding_side_build_targets.adoc
Normal file
|
|
@ -0,0 +1,208 @@
|
|||
== Adding Side Build Tags
|
||||
|
||||
=== Description
|
||||
|
||||
Bigger Features can take a while to stabilize and land or need a large
|
||||
number of packages to be built against each other, this is easiest
|
||||
served by having a separate build tag for the development work. This SOP
|
||||
will describe the steps necessary to prepare the new build target.
|
||||
|
||||
=== Action
|
||||
|
||||
Engineers should be aware that adding side build targets incurs extra
|
||||
newRepo tasks in the koji.
|
||||
|
||||
==== Research Tag
|
||||
|
||||
. Verify whether a tag already exists.
|
||||
+
|
||||
The typical tag format is _PRODUCT_-_DESCRIPTOR_. The _DESCRIPTOR_
|
||||
should be something brief that clearly shows why the tag exists.
|
||||
+
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Don't think too hard about what makes a good descriptor. The descriptor
|
||||
for the XFCE version 4.8 side-build target was _xfce48_. KDE often
|
||||
simply uses _kde_ as its descriptor. Use best judgement and if in doubt
|
||||
ask in IRC on `#fedora-releng`.
|
||||
====
|
||||
+
|
||||
EPEL6
|
||||
|
||||
....
|
||||
$ koji taginfo epel6-kde
|
||||
....
|
||||
+
|
||||
EPEL7
|
||||
|
||||
....
|
||||
$ koji taginfo epel7-kde
|
||||
....
|
||||
+
|
||||
Fedora
|
||||
|
||||
....
|
||||
$ koji taginfo f28-kde
|
||||
....
|
||||
+
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
If the tag already exists, continue searching for an available tag by
|
||||
appending `-2` and incrementing the number by one until an available tag
|
||||
is found. For example, if `f28-kde` already exists then search for
|
||||
`f28-kde-2`, `f28-kde-3`, and so on until a suitable tag is found.
|
||||
====
|
||||
. Determine the appropriate architectures.
|
||||
+
|
||||
EPEL6
|
||||
|
||||
....
|
||||
$ koji taginfo dist-6E-epel-build
|
||||
....
|
||||
+
|
||||
EPEL7
|
||||
|
||||
....
|
||||
$ koji taginfo epel7-build
|
||||
....
|
||||
+
|
||||
Fedora
|
||||
|
||||
....
|
||||
$ koji taginfo f28-build
|
||||
....
|
||||
|
||||
==== Create Side Build Target
|
||||
|
||||
. Create the proper tag.
|
||||
+
|
||||
Note the slightly different syntax depending on which product needs the
|
||||
side-build target and the comma delimited list of architectures based on
|
||||
the information from a previous step.
|
||||
+
|
||||
EPEL6
|
||||
|
||||
....
|
||||
$ koji add-tag epel6-kde --parent=dist-6E-epel-build --arches=i686,x86_64,ppc64
|
||||
....
|
||||
+
|
||||
EPEL7
|
||||
|
||||
....
|
||||
$ koji add-tag epel7-kde --parent=epel7-build --arches=aarch64,x86_64,ppc64,ppc64le
|
||||
....
|
||||
+
|
||||
Fedora
|
||||
|
||||
....
|
||||
$ koji add-tag f28-kde --parent=f28-build --arches=armv7hl,i686,x86_64,aarch64,ppc64,ppc64le,s390x
|
||||
....
|
||||
. Create the target.
|
||||
+
|
||||
EPEL6
|
||||
|
||||
....
|
||||
$ koji add-target epel6-kde epel6-kde
|
||||
....
|
||||
+
|
||||
EPEL7
|
||||
|
||||
....
|
||||
$ koji add-target epel7-kde epel7-kde
|
||||
....
|
||||
+
|
||||
Fedora
|
||||
|
||||
....
|
||||
$ koji add-target f28-kde f28-kde
|
||||
....
|
||||
. Find the taskID for the newRepo task associated with the newly created
|
||||
target.
|
||||
+
|
||||
....
|
||||
$ koji list-tasks --method=newRepo
|
||||
ID Pri Owner State Arch Name
|
||||
25101143 15 kojira OPEN noarch newRepo (f28-kde)
|
||||
....
|
||||
. Ensure the newRepo task is being run across all architectures.
|
||||
+
|
||||
....
|
||||
$ koji watch-task 25101143
|
||||
Watching tasks (this may be safely interrupted)...
|
||||
25101143 newRepo (f28-kde): open (buildvm-14.phx2.fedoraproject.org)
|
||||
25101154 createrepo (i386): closed
|
||||
25101150 createrepo (ppc64le): closed
|
||||
25101152 createrepo (ppc64): closed
|
||||
25101151 createrepo (aarch64): closed
|
||||
25101149 createrepo (armhfp): closed
|
||||
25101153 createrepo (s390x): open (buildvm-ppc64le-04.ppc.fedoraproject.org)
|
||||
25101148 createrepo (x86_64): open (buildvm-aarch64-21.arm.fedoraproject.org)
|
||||
....
|
||||
. Request Package Auto-Signing for New Tag
|
||||
+
|
||||
File a ticket in https://pagure.io/fedora-infrastructure/issues[pagure
|
||||
infrastructure] requesting the new tag be enabled for package
|
||||
auto-signing.
|
||||
. Update the Pagure Issue
|
||||
+
|
||||
Update the issue according to the following template which assumes a
|
||||
side target was made for KDE under Fedora 28. _TAG_NAME_ has been
|
||||
created:
|
||||
+
|
||||
____
|
||||
$ koji add-tag f28-kde --parent=f28-build
|
||||
--arches=armv7hl,i686,x86_64,aarch64,ppc64,ppc64le,s390x
|
||||
|
||||
$ koji add-target f28-kde f28-kde
|
||||
|
||||
You can do builds with:
|
||||
|
||||
$ fedpkg build --target=f28-kde
|
||||
|
||||
Let us know when you are done and we will move all the builds into f28.
|
||||
____
|
||||
|
||||
=== Cleanup
|
||||
|
||||
Fedora Release Engineering is responsible for merging the packages from
|
||||
the side-build target and tag back into the main tag. The requestor will
|
||||
update the original ticket when ready for the procedure outlined below.
|
||||
|
||||
. Remove the target
|
||||
+
|
||||
....
|
||||
$ koji remove-target <SIDE_TAG_NAME>
|
||||
....
|
||||
. Merge side build back to main target.
|
||||
+
|
||||
Get the latest checkout from https://pagure.io/releng/[Fedora Release
|
||||
Engineering Repository] and run the [.title-ref]#mass-tag.py# from the
|
||||
scripts directory.
|
||||
+
|
||||
....
|
||||
$ ./mass-tag.py --source <SIDE_TAG_NAME> --target <MAIN_TAG_NAME> > mass_tag.txt
|
||||
....
|
||||
+
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
The _MAIN_TAG_NAME_ for Fedora is typically the pending subtag, e.g.
|
||||
`f28-pending` when bodhi is not managing updates. After bodhi is enabled
|
||||
and managing updates then merge into `f28-updates-candidate`.
|
||||
====
|
||||
. Paste Output to the Original Ticket
|
||||
+
|
||||
Paste the output from mass-tag.py into the pagure/releng ticket to show
|
||||
what packages were merged and what packages need rebuilding for those
|
||||
who work on the buildroot.
|
||||
|
||||
Tags are *never* removed.
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
* Is the amount of work to be done worth the cost of newRepo tasks.
|
||||
* If there is only a small number of packages overrides may be better.
|
||||
* Is there a mass-rebuild going on? no side tags are allowed while a
|
||||
mass rebuild is underway
|
||||
238
modules/release_guide/pages/sop_bodhi_activation.adoc
Normal file
238
modules/release_guide/pages/sop_bodhi_activation.adoc
Normal file
|
|
@ -0,0 +1,238 @@
|
|||
== Bodhi Activation Point
|
||||
|
||||
=== Description
|
||||
|
||||
Bodhi must be activated after two weeks of
|
||||
https://docs.pagure.org/releng/sop_mass_branching.html[Mass Branching]
|
||||
at 14:00 UTC.
|
||||
|
||||
=== Action
|
||||
|
||||
==== Making koji changes
|
||||
|
||||
Make the following koji tag changes
|
||||
|
||||
....
|
||||
$ koji remove-tag-inheritance f33-updates-candidate f33
|
||||
$ koji remove-tag-inheritance f33-updates-testing f33
|
||||
$ koji remove-tag-inheritance f33-updates-pending f33
|
||||
$ koji remove-tag-inheritance f33-override f33
|
||||
$ koji add-tag-inheritance f33-updates-candidate f33-updates
|
||||
$ koji add-tag-inheritance f33-updates-testing f33-updates
|
||||
$ koji add-tag-inheritance f33-updates-pending f33-updates
|
||||
$ koji add-tag-inheritance f33-override f33-updates
|
||||
$ koji edit-tag --perm=admin f33
|
||||
....
|
||||
|
||||
==== Update bodhi rpm release
|
||||
|
||||
Set the bodhi rpm to release to not to automatically create the update
|
||||
and also bodhi knows to compose the updates
|
||||
|
||||
....
|
||||
$ bodhi releases edit --name "F33" --stable-tag f33-updates --testing-repository updates-testing --package-manager dnf --no-create-automatic-updates --composed-by-bodhi
|
||||
....
|
||||
|
||||
==== Add the modular release
|
||||
|
||||
Run the following command on your own workstation to add the modular
|
||||
release
|
||||
|
||||
....
|
||||
$ bodhi releases create --name F33M --long-name "Fedora 33 Modular" --id-prefix FEDORA-MODULAR --version 33 --branch f33m --dist-tag f33-modular --stable-tag f33-modular-updates --testing-tag f33-modular-updates-testing --candidate-tag f33-modular-updates-candidate --pending-stable-tag f33-modular-updates-pending --pending-testing-tag f33-modular-updates-testing-pending --pending-signing-tag f33-modular-signing-pending --override-tag f33-modular-override --state pending --user mohanboddu
|
||||
....
|
||||
|
||||
[WARNING]
|
||||
.Warning
|
||||
====
|
||||
Due to a https://github.com/fedora-infra/bodhi/issues/2177[bug] in
|
||||
Bodhi, it is critical that Bodhi processes be restarted any time
|
||||
`bodhi releases create` or `bodhi releases edit` are used.
|
||||
====
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Add the container and flatpak releases if they weren't already added to
|
||||
bodhi
|
||||
====
|
||||
|
||||
=== Ansible Changes
|
||||
|
||||
==== Update vars
|
||||
|
||||
Update the _FedoraBranchedBodhi_ and _RelEngFrozen_ vars in infra
|
||||
ansible
|
||||
|
||||
....
|
||||
diff --git a/vars/all/FedoraBranchedBodhi.yaml b/vars/all/FedoraBranchedBodhi.yaml
|
||||
index aba8be2..606eb2e 100644
|
||||
--- a/vars/all/FedoraBranchedBodhi.yaml
|
||||
+++ b/vars/all/FedoraBranchedBodhi.yaml
|
||||
@@ -3,4 +3,4 @@
|
||||
# prebeta: After bodhi enablement/beta freeze and before beta release
|
||||
# postbeta: After beta release and before final release
|
||||
# current: After final release
|
||||
-FedoraBranchedBodhi: preenable
|
||||
+FedoraBranchedBodhi: prebeta
|
||||
diff --git a/vars/all/RelEngFrozen.yaml b/vars/all/RelEngFrozen.yaml
|
||||
index 5836689..87d85f3 100644
|
||||
--- a/vars/all/RelEngFrozen.yaml
|
||||
+++ b/vars/all/RelEngFrozen.yaml
|
||||
@@ -1 +1 @@
|
||||
-RelEngFrozen: False
|
||||
+RelEngFrozen: True
|
||||
....
|
||||
|
||||
==== Update Greenwave Policy
|
||||
|
||||
Now edit the Greenwave policy to configure a policy for the new release
|
||||
by editing `roles/openshift-apps/greenwave/templates/configmap.yml` in
|
||||
the Infrastructure Ansible repository.
|
||||
|
||||
....
|
||||
diff --git a/roles/openshift-apps/greenwave/templates/fedora.yaml b/roles/openshift-apps/greenwave/templates/fedora.yaml
|
||||
index 7a76f61..d15e154 100644
|
||||
--- a/roles/openshift-apps/greenwave/templates/fedora.yaml
|
||||
+++ b/roles/openshift-apps/greenwave/templates/fedora.yaml
|
||||
@@ -84,6 +84,9 @@ rules:
|
||||
--- !Policy
|
||||
id: "no_requirements_testing"
|
||||
product_versions:
|
||||
+ - fedora-33-modular
|
||||
+ - fedora-33-containers
|
||||
+ - fedora-33-flatpaks
|
||||
- fedora-32-modular
|
||||
- fedora-32-containers
|
||||
- fedora-32-flatpaks
|
||||
@@ -107,6 +110,9 @@ rules: []
|
||||
--- !Policy
|
||||
id: "no_requirements_for_stable"
|
||||
product_versions:
|
||||
+ - fedora-33-modular
|
||||
+ - fedora-33-containers
|
||||
+ - fedora-33-flatpaks
|
||||
- fedora-32-modular
|
||||
- fedora-32-containers
|
||||
- fedora-32-flatpaks
|
||||
@@ -133,6 +139,7 @@ id: "openqa_release_critical_tasks_for_testing"
|
||||
product_versions:
|
||||
- fedora-rawhide
|
||||
- fedora-eln
|
||||
+ - fedora-33
|
||||
- fedora-32
|
||||
- fedora-31
|
||||
- fedora-30
|
||||
@@ -147,6 +154,7 @@ id: "openqa_release_critical_tasks_for_stable"
|
||||
product_versions:
|
||||
- fedora-rawhide
|
||||
- fedora-eln
|
||||
+ - fedora-33
|
||||
- fedora-32
|
||||
- fedora-31
|
||||
- fedora-30
|
||||
....
|
||||
|
||||
==== Update Robosignatory Config
|
||||
|
||||
Update the robosignatory config in the infra ansible repo as following
|
||||
|
||||
....
|
||||
diff --git a/roles/robosignatory/templates/robosignatory.toml.j2 b/roles/robosignatory/templates/robosignatory.toml.j2
|
||||
index 16a6708..68f4251 100644
|
||||
--- a/roles/robosignatory/templates/robosignatory.toml.j2
|
||||
+++ b/roles/robosignatory/templates/robosignatory.toml.j2
|
||||
@@ -259,8 +259,8 @@ handlers = ["console"]
|
||||
type = "modular"
|
||||
|
||||
[[consumer_config.koji_instances.primary.tags]]
|
||||
- from = "f33-modular-updates-candidate"
|
||||
- to = "f33-modular"
|
||||
+ from = "f33-modular-signing-pending"
|
||||
+ to = "f33-modular-updates-testing-pending"
|
||||
key = "{{ (env == 'production')|ternary('fedora-33', 'testkey') }}"
|
||||
keyid = "{{ (env == 'production')|ternary('9570ff31', 'd300e724') }}"
|
||||
type = "modular"
|
||||
....
|
||||
|
||||
==== Run the playbooks
|
||||
|
||||
....
|
||||
$ rbac-playbook openshift-apps/greenwave.yml
|
||||
$ rbac-playbook openshift-apps/bodhi.yml
|
||||
$ rbac-playbook groups/bodhi-backend.yml
|
||||
$ rbac-playbook groups/releng-compose.yml
|
||||
$ rbac-playbook manual/autosign.yml
|
||||
....
|
||||
|
||||
Greenwave runs in OpenShift (as implied by the playbook paths), and so
|
||||
the change will not be live right away when the playbook finishes. You
|
||||
can monitor
|
||||
https://greenwave-web-greenwave.app.os.fedoraproject.org/api/v1.0/policies
|
||||
to wait for the new policy to appear (it should take a few minutes).
|
||||
|
||||
==== Restart bodhi services
|
||||
|
||||
Restart bodhi services to understand the bodhi new release on
|
||||
bodhi-backend01 (Look at warning in
|
||||
https://docs.pagure.org/releng/sop_bodhi_activation.html#action and the
|
||||
bug is https://github.com/fedora-infra/bodhi/issues/2177)
|
||||
|
||||
....
|
||||
$ sudo systemctl restart bodhi-celery
|
||||
$ sudo systemctl restart fm-consumer@config
|
||||
$ sudo systemctl restart koji-sync-listener
|
||||
....
|
||||
|
||||
==== Send Announcement
|
||||
|
||||
Email *devel-announce* and *test-announce* lists about Bodhi Activation.
|
||||
Please find the body of the email below:
|
||||
|
||||
....
|
||||
Hi all,
|
||||
|
||||
Today's an important day on the Fedora 25 schedule[1], with several significant cut-offs. First of all today is the Bodhi activation point [2]. That means that from now all Fedora 25 packages must be submitted to updates-testing and pass the relevant requirements[3] before they will be marked as 'stable' and moved to the fedora repository.
|
||||
|
||||
Today is also the Alpha freeze[4]. This means that only packages which fix accepted blocker or freeze exception bugs[5][6] will be marked as 'stable' and included in the Alpha composes. Other builds will remain in updates-testing until the Alpha release is approved, at which point the Alpha freeze is lifted and packages can move to 'stable' as usual until the Beta freeze.
|
||||
|
||||
Today is also the Software String freeze[7], which means that strings marked for translation in Fedora-translated projects should not now be changed for Fedora 25.
|
||||
|
||||
Finally, today is the 'completion deadline' Change Checkpoint[8], meaning that Fedora 25 Changes must now be 'feature complete or close enough to completion that a majority of its functionality can be tested'.
|
||||
|
||||
Regards
|
||||
<your_name>
|
||||
|
||||
[1] https://fedorapeople.org/groups/schedule/f-34/f-34-key-tasks.html
|
||||
[2] https://fedoraproject.org/wiki/Updates_Policy#Bodhi_enabling
|
||||
[3] https://fedoraproject.org/wiki/Updates_Policy#Branched_release
|
||||
[4] https://fedoraproject.org/wiki/Milestone_freezes
|
||||
[5] https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process
|
||||
[6] https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process
|
||||
[7] https://fedoraproject.org/wiki/ReleaseEngineering/StringFreezePolicy
|
||||
[8] https://fedoraproject.org/wiki/Changes/Policy
|
||||
....
|
||||
|
||||
=== Verification
|
||||
|
||||
Compare koji tagging structure with older release
|
||||
|
||||
....
|
||||
$ koji list-tag-inheritance <branched_release> --reverse
|
||||
$ koji list-tag-inheritance <latest_stable_release> --reverse
|
||||
....
|
||||
|
||||
Compare the bodhi release with older release
|
||||
|
||||
....
|
||||
$ bodhi releases info <branched_release>
|
||||
$ bodhi releases info <latest_stable_release>
|
||||
....
|
||||
|
||||
Check for other variants like modular, container and flatpaks
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
No considerations at this time. The docs git repository is simply a
|
||||
static html hosting space and we can just re-render the docs and push to
|
||||
it again if necessary.
|
||||
33
modules/release_guide/pages/sop_branch_freeze.adoc
Normal file
33
modules/release_guide/pages/sop_branch_freeze.adoc
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
== Branching Freeze
|
||||
|
||||
=== Introduction/Background
|
||||
|
||||
When the next release is branched from rawhide, it initially composes
|
||||
much like rawhide with nightly composes and no updates process.
|
||||
|
||||
Once the Bodhi is activated, we will push updates to the branched and
|
||||
the nightly composes will start to differ. But two weeks before the
|
||||
scheduled release of either Beta or GA, we will start the freeze for
|
||||
that release and stop pushing updates.
|
||||
|
||||
* Send announcement to devel-announce mailing list noting that the alpha
|
||||
change freeze is going to happen at least one day in advance.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
For updates pushers: In Change freeze only updates that fix accepted
|
||||
blockers or Freeze break bugs are allowed into the main tree. Please
|
||||
coordinate with QA for any stable updates pushes. Otherwise ONLY push
|
||||
updates-testing.
|
||||
====
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
For Final/GA release: During Final freeze, we dont want to block any
|
||||
packages in koji as it will effect the RC composes. So, please update
|
||||
the
|
||||
https://pagure.io/releng/blob/master/f/scripts/block_retired.py[block_retired.py]
|
||||
script and remove the branched release reference.
|
||||
====
|
||||
33
modules/release_guide/pages/sop_branching.adoc
Normal file
33
modules/release_guide/pages/sop_branching.adoc
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
== Branching
|
||||
|
||||
=== Description
|
||||
|
||||
This SOP covers how to make git and pkgdb branches for packages, either
|
||||
for new packages that have passed review, or for existing packages that
|
||||
need a new branch (e.g. EPEL). Release Engineering has written a script
|
||||
to automate this process.
|
||||
|
||||
=== Normal Action (automated)
|
||||
|
||||
. On your local system (not on an infrastructure hosted system), be sure
|
||||
you have the following packages installed:
|
||||
* python-bugzilla
|
||||
* python-configobj
|
||||
* python-fedora
|
||||
. Run "bugzilla login" and successfully receive an Authorization cookie.
|
||||
. {blank}
|
||||
+
|
||||
Clone the fedora-infrastructure tools repository:::
|
||||
....
|
||||
git clone https://pagure.io/releng.git
|
||||
....
|
||||
. In scripts/process-git-requests, run "process-git-requests". Answer
|
||||
the prompts.
|
||||
|
||||
=== Manual Action
|
||||
|
||||
==== Creating a new branch for an existing package
|
||||
|
||||
. ssh into `pkgs.fedoraproject.org`
|
||||
. `pkgdb-client edit -u $YOURUSERNAME -b $NEWBRANCH --master=devel $NAMEOFPACKAGE`
|
||||
. `pkgdb2branch.py $NAMEOFPACKAGE`
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
== Breaking Development Freeze
|
||||
|
||||
`FIXME: NEED TO FIGURE OUT HOW TO FEDORA-VERSION-NEXT`
|
||||
|
||||
=== Description
|
||||
|
||||
Packages which require an exception to freeze policies must be run
|
||||
through this SOP.
|
||||
|
||||
The following freeze policies are set for the following significant
|
||||
release milestones:
|
||||
|
||||
* https://fedoraproject.org/wiki/Software_String_Freeze_Policy[String
|
||||
Freeze]
|
||||
* https://fedoraproject.org/wiki/Milestone_freezes[Beta Freeze]
|
||||
* https://fedoraproject.org/wiki/Milestone_freezes[Final Freeze]
|
||||
|
||||
See https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle[Fedora
|
||||
Release Life Cycle] for a summary of all the freezes, dates, and
|
||||
exception handling, or the release engineering
|
||||
[https://fedorapeople.org/groups/schedule/f-\{\{FedoraVersionNumbernext}}-releng-tasks.html
|
||||
calendar for the current release].
|
||||
|
||||
=== Action
|
||||
|
||||
The commands to tag a package properly once it has been accepted:
|
||||
|
||||
....
|
||||
$ koji move-pkg --force dist-f{{FedoraVersionNumber|next}}-updates-candidate dist-f{{FedoraVersionNumber|next}} <PKGNAME>
|
||||
$ koji tag-pkg --force f{{FedoraVersionNumber|next}}-<RELEASE> <PKGNAME>
|
||||
....
|
||||
|
||||
Where <PKGNAME> is the package name, and <RELEASE> is the first release
|
||||
in which the package should land (e.g. alpha, beta, final).
|
||||
|
||||
=== Verification
|
||||
|
||||
The `koji` client reports success or failure. For secondary
|
||||
verification, run these commands:
|
||||
|
||||
....
|
||||
$ koji latest-pkg dist-f{{FedoraVersionNumber|next}} <PKGNAME>
|
||||
$ koji latest-pkg dist-f{{FedoraVersionNumber|next}}-updates-candidate <PKGNAME>
|
||||
....
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
* Change agrees with stated policies (see links above)
|
||||
* Approval for change has been granted under
|
||||
https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process[Blocker Bug
|
||||
Process] or [.title-ref]#Freeze Exception Bug Process#
|
||||
59
modules/release_guide/pages/sop_clean_amis.adoc
Normal file
59
modules/release_guide/pages/sop_clean_amis.adoc
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
== Clean AMIs Process
|
||||
|
||||
=== Description
|
||||
|
||||
The Fedora AMIs are uploaded on a daily basis to Amazon Web Services.
|
||||
Over time the number of AMIs piles up and have to be removed manually.
|
||||
Manual removal comes with it's own set of issues where missing to delete
|
||||
the AMIs is a viable issue.
|
||||
|
||||
The goal of the script is to automate the process and continue regular
|
||||
removal of the AMIs. The report of the script is pushed to a
|
||||
https://pagure.io/ami-purge-report[Pagure repo]
|
||||
|
||||
=== Action
|
||||
|
||||
There is a script in the https://pagure.io/releng[Fedora RelEng repo]
|
||||
named `clean-amis.py` under the `scripts` directory.
|
||||
|
||||
The script runs as a cron job within the Fedora Infrastructure to delete
|
||||
the old AMIs. The permission of the selected AMIs are changed to
|
||||
private. This is to make sure that if someone from the community raises
|
||||
an issue we have the option to get the AMI back to public. After 10
|
||||
days, if no complaints are raised the AMIs are deleted permanently.
|
||||
|
||||
The complete process can be divided in couple of parts:
|
||||
|
||||
* Fetching the data from datagrepper. Based on the [.title-ref]#--days#
|
||||
param, the script starts fetching the fedmsg messages from datagrepper
|
||||
for the specified timeframe i.e. for lasts [.title-ref]#n# days, where
|
||||
[.title-ref]#n# is the value of [.title-ref]#--days# param. The queried
|
||||
fedmsg topic [.title-ref]#fedimg.image.upload#.
|
||||
* Selection of the AMIs: After the AMIs are parsed from datagrepper. The
|
||||
AMIs are filtered to remove Beta, Two-week Atomic Host and GA released
|
||||
AMIs. Composes with [.title-ref]#compose_type# set to
|
||||
[.title-ref]#nightly# are picked up for deletion. Composes which contain
|
||||
date in the [.title-ref]#compose label# are also picked up for deletion.
|
||||
GA composes also have the compose_type set to production. So to
|
||||
distinguish then we filter them if the compose_label have date in them.
|
||||
The GA composes dont have date whereas they have the version in format
|
||||
of X.Y
|
||||
* Updated permissions of AMIs The permissions of the selected AMIs are
|
||||
changed to private.
|
||||
* Deletion of AMIs After 10 days, the private AMIs are deleted.
|
||||
|
||||
In order to change the permissions of the AMIs use the command given
|
||||
below, add [.title-ref]#--dry-run# argument test the command works.
|
||||
Adding [.title-ref]#--dry-run# argument will print the AMIs to console.
|
||||
|
||||
....
|
||||
AWS_ACCESS_KEY={{ ec2_image_delete_access_key_id }} AWS_SECRET_ACCESS_KEY={{ ec2_image_delete_access_key }} PAGURE_ACCESS_TOKEN={{ ami_purge_report_api_key }} ./clean-amis.py --change-perms --days 7 --permswaitperiod 5
|
||||
....
|
||||
|
||||
In order to delete the AMIs whose launch permissions have been removed,
|
||||
add [.title-ref]#--dry-run# argument test the command works. Adding
|
||||
[.title-ref]#--dry-run# argument will print the AMIs to console.
|
||||
|
||||
....
|
||||
AWS_ACCESS_KEY={{ ec2_image_delete_access_key_id }} AWS_SECRET_ACCESS_KEY={{ ec2_image_delete_access_key }} PAGURE_ACCESS_TOKEN={{ ami_purge_report_api_key }} ./clean-amis.py --delete --days 17 --deletewaitperiod 10
|
||||
....
|
||||
622
modules/release_guide/pages/sop_composing_fedora.adoc
Normal file
622
modules/release_guide/pages/sop_composing_fedora.adoc
Normal file
|
|
@ -0,0 +1,622 @@
|
|||
== Composing Fedora
|
||||
|
||||
=== Description
|
||||
|
||||
All composes are defined by configuration files kept in the
|
||||
https://pagure.io/pungi-fedora[pungi-fedora repository].
|
||||
|
||||
Composes fall into two categories. They may be release candidates
|
||||
created on demand or nightly composes set to run at a scheduled time
|
||||
each day.
|
||||
|
||||
[cols=",,",options="header",]
|
||||
|===
|
||||
|Compose Name |Configuration File |Compose Script
|
||||
|Docker |fedora-docker.conf |docker-nightly.sh
|
||||
|Cloud |fedora-cloud.conf |cloud-nightly.sh
|
||||
|Modular |fedora-modular.conf |modular-nightly.sh
|
||||
|Nightly |fedora.conf |nightly.sh
|
||||
|===
|
||||
|
||||
When Quality Engineering (QE) requests a Release Candidate (RC) they do
|
||||
so by opening an issue in the releng repository on pagure. Release
|
||||
candidate composes are not currently automated.
|
||||
|
||||
[cols=",,",options="header",]
|
||||
|===
|
||||
|Compose Name |Configuration File |Compose Script
|
||||
|Beta |fedora-beta.conf |release-candidate.sh
|
||||
|GA |fedora-final.conf |release-candidate.sh
|
||||
|===
|
||||
|
||||
=== Action
|
||||
|
||||
The following procedures are for release candidates only. They do not
|
||||
apply to the scheduled nightly composes.
|
||||
|
||||
==== Review Compose Tags
|
||||
|
||||
. List any pre-existing builds in the current compose tag
|
||||
+
|
||||
....
|
||||
$ koji list-tagged f[release_version]-compose
|
||||
....
|
||||
. Verify pre-existing builds are in compose tags
|
||||
+
|
||||
The tagged builds from the previous composes should all be present in
|
||||
the output from the previous step. Consult the request ticket for the
|
||||
list of builds expected in this output.
|
||||
+
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
The very first run of an Beta, or GA compose should have no builds
|
||||
listed under the compose tag. It is important to clear pre-existing
|
||||
builds from the compose tag when moving between the Beta and RC
|
||||
composes. Verify that these builds were removed.
|
||||
|
||||
....
|
||||
$ koji list-tagged f[release_version]-compose
|
||||
$ koji untag-build --all f[release_version]-compose [build1 build2 ...]
|
||||
....
|
||||
====
|
||||
+
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
The order in which packages are added into the
|
||||
f[release_version]-compose tag matter. If the builds are untagged
|
||||
erroneously then special attention should be given to adding them back
|
||||
correctly.
|
||||
====
|
||||
. Add builds specified by QE to the current compose tag
|
||||
+
|
||||
....
|
||||
$ koji tag-build f[release_version]-compose [build1 build2 ...]
|
||||
....
|
||||
+
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
These steps may be completed on a local machine as long as the user has
|
||||
appropriate permissions in the koji tool.
|
||||
====
|
||||
|
||||
==== Package Signing before the Compose
|
||||
|
||||
. Check for unsigned packages
|
||||
+
|
||||
....
|
||||
$ koji list-tagged f[release_version]-signing-pending
|
||||
....
|
||||
+
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
If there are unsigned builds then wait for the automated queue to pick
|
||||
them up and sign them. Contact a member of the Fedora infrastructure
|
||||
team if the package signing has taken more than thirty minutes.
|
||||
====
|
||||
|
||||
==== Running the Compose
|
||||
|
||||
. Update the pungi-fedora config file Composes use a configuration file
|
||||
to construct the compose. Each compose uses its own configuration. The
|
||||
`global_release` variable should start from 1.1 and the second number
|
||||
should increment each time a new compose is created.
|
||||
* Beta - `fedora-beta.conf`
|
||||
* RC - `fedora-final.conf`
|
||||
. Log into the compose backend
|
||||
+
|
||||
....
|
||||
$ ssh compose-x86-01.phx2.fedoraproject.org
|
||||
....
|
||||
. Open a screen session
|
||||
+
|
||||
....
|
||||
$ screen
|
||||
....
|
||||
. Obtain the pungi-fedora branch for the current compose
|
||||
+
|
||||
The first time any user account executes a compose the pungi-fedora git
|
||||
repository must be cloned. The compose candidate script that invokes
|
||||
pungi should be run from `compose-x86-01.iad2.fedoraproject.org`.
|
||||
+
|
||||
....
|
||||
$ git clone ssh://git@pagure.io/pungi-fedora.git
|
||||
....
|
||||
+
|
||||
Enter the pungi-fedora directory.
|
||||
+
|
||||
....
|
||||
$ cd pungi-fedora
|
||||
....
|
||||
+
|
||||
If the clone step above was not required then fully update the existing
|
||||
repository checkout from pagure.
|
||||
+
|
||||
....
|
||||
$ git fetch origin
|
||||
$ git checkout f[release_version]
|
||||
$ git pull origin f[release_version]
|
||||
....
|
||||
. Run the compose
|
||||
+
|
||||
....
|
||||
$ sudo ./release-candidate.sh [Beta|RC]-#.#
|
||||
....
|
||||
+
|
||||
The numbering scheme begins with 1.1 and the second number is
|
||||
incremented after each compose.
|
||||
+
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Pungi requires numbers in the format #.# as an argument. It is because
|
||||
of this that composes always start with the number 1 and the second
|
||||
number is incremented with each compose.
|
||||
====
|
||||
+
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
If the compose fails with a directory missing error, then create the
|
||||
compose directory with `mkdir /mnt/koji/compose/[release_version]`
|
||||
====
|
||||
|
||||
==== Syncing the Compose
|
||||
|
||||
We sync the compose to `/pub/alt/stage` to enable faster access to new
|
||||
content for QA and the larger Fedora community.
|
||||
|
||||
. Log into the compose backend
|
||||
+
|
||||
....
|
||||
$ ssh compose-x86-01.iad2.fedoraproject.org
|
||||
....
|
||||
. Open a screen session
|
||||
+
|
||||
....
|
||||
$ screen
|
||||
....
|
||||
. Check the status of the compose
|
||||
+
|
||||
....
|
||||
$ cat /mnt/koji/compose/[release_version]/[compose_id]/STATUS
|
||||
....
|
||||
+
|
||||
Do not continue with any further steps if the output above is `DOOMED`.
|
||||
. Create the directory targeted for the copy :
|
||||
+
|
||||
....
|
||||
$ sudo -u ftpsync mkdir -m 750 -p /pub/alt/stage/[release_version]_[release_label]-[#.#]
|
||||
....
|
||||
. Locate the compose directory that will be the copy source :
|
||||
+
|
||||
....
|
||||
$ ls /mnt/koji/compose/[release_version]/[compose_id]
|
||||
....
|
||||
+
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Take care executing the synchronization if the next compose is already
|
||||
running. Be sure to grab the correct directory.
|
||||
|
||||
If in doubt, check
|
||||
/mnt/koji/compose/[release_version]/[compose_id]/STATUS to be sure it is
|
||||
finished.
|
||||
====
|
||||
. Run the synchronization one-liner
|
||||
+
|
||||
The synchronization of the completed compose to the public domain is
|
||||
currently a one-liner shell script. Pay close attention to what needs
|
||||
replaced in the example below.
|
||||
+
|
||||
....
|
||||
$ sudo -u ftpsync sh -c 'for dir in Everything Cloud Container Kinoite Labs Modular Server Silverblue Spins Workstation metadata; do rsync -avhH /mnt/koji/compose/31/Fedora-31-20190911.0/compose/$dir/ /pub/alt/stage/31_Beta-1.1/$dir/ --link-dest=/pub/fedora/linux/development/31/Everything/ --link-dest=/pub/alt/stage/31_Beta-1.1/Everything/; done'
|
||||
....
|
||||
+
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
If multiple composes are run like 1.2, 1.3, add multiple --link-dest
|
||||
arguments above with multiple composes
|
||||
====
|
||||
. Set the permissions of the synced compose :
|
||||
+
|
||||
....
|
||||
$ sudo -u ftpsync chmod 755 /pub/alt/stage/[release_version]_[release_label]-[#.#]
|
||||
....
|
||||
. Update the issue in the releng pagure repository
|
||||
+
|
||||
Once the compose and sync is complete the issue in pagure should be
|
||||
updated and closed.
|
||||
+
|
||||
Standard Ticket Verbage
|
||||
|
||||
Compose is done and available from
|
||||
https://kojipkgs.fedoraproject.org/compose/26/Fedora-26-20170328.0/compose/
|
||||
it has been synced to
|
||||
http://dl.fedoraproject.org/pub/alt/stage/26_Alpha-1.4/ rpms have all be
|
||||
hardlinked to /pub/fedora/linux/development/26/
|
||||
|
||||
===== Verification
|
||||
|
||||
The method for verifying a compose has completed is checking
|
||||
`/mnt/koji/compose/[release_version]/[compose_dir]/STATUS`. Any status
|
||||
other than DOOMED is OK.
|
||||
|
||||
=== Pre-Release Work
|
||||
|
||||
==== Pushing Updates to Stable
|
||||
|
||||
When the release is signed off on Thursday after the Go/No-Go meeting,
|
||||
push the freeze and blocker to stable updates
|
||||
|
||||
Generally the updates are requested stable by QA. If they are not
|
||||
available, you can request them by following
|
||||
|
||||
....
|
||||
$ bodhi updates request <updateid> stable
|
||||
....
|
||||
|
||||
Once the updates are requested stable, please push them to stable by
|
||||
following the
|
||||
https://docs.pagure.org/releng/sop_pushing_updates.html#pushing-stable-updates-during-freeze[bodhi
|
||||
push to stable sop]
|
||||
|
||||
==== koji tag changes
|
||||
|
||||
Once the updates are pushed stable, we need to clone the koji tag for
|
||||
beta release or lock the koji tag for final release.
|
||||
|
||||
===== For Beta Release
|
||||
|
||||
....
|
||||
$ koji clone-tag --all --latest-only f31 f31-Beta
|
||||
$ koji clone-tag --all --latest-only f31-modular f31-Beta-modular
|
||||
....
|
||||
|
||||
===== For Final Release
|
||||
|
||||
....
|
||||
$ koji edit-tag --lock f31
|
||||
$ koji edit-tag --lock f31-modular
|
||||
....
|
||||
|
||||
==== Bodhi Changes
|
||||
|
||||
Set the bodhi release to `current`
|
||||
|
||||
....
|
||||
$ bodhi releases edit --name F31 --state current
|
||||
....
|
||||
|
||||
=== Changes for Final Release
|
||||
|
||||
Once Final is GO, we need to perform different changes as that of Beta
|
||||
release.
|
||||
|
||||
==== Last Branched Compose
|
||||
|
||||
Manually run a branched compose so that the GOLD content is same as the
|
||||
nightly compose. This also helps in updating the silverblue refs as that
|
||||
of the GOLD content.
|
||||
|
||||
==== Update silverblue refs
|
||||
|
||||
Please update the refs as per the following commands on
|
||||
[.title-ref]#bodhi-backend01.phx2.fedoraproject.org#
|
||||
|
||||
Run the following commands from
|
||||
[.title-ref]#/mnt/koji/compose/ostree/repo# and
|
||||
[.title-ref]#/mnt/koji/ostree/repo/#
|
||||
|
||||
....
|
||||
$ sudo -u ftpsync ostree refs --create=fedora/31/x86_64/updates/silverblue fedora/31/x86_64/silverblue
|
||||
$ sudo -u ftpsync ostree refs --create=fedora/31/aarch64/updates/silverblue fedora/31/aarch64/silverblue
|
||||
$ sudo -u ftpsync ostree refs --create=fedora/31/ppc64le/updates/silverblue fedora/31/ppc64le/silverblue
|
||||
|
||||
$ sudo ostree refs --delete fedora/31/x86_64/silverblue
|
||||
$ sudo ostree refs --delete fedora/31/aarch64/silverblue
|
||||
$ sudo ostree refs --delete fedora/31/ppc64le/silverblue
|
||||
|
||||
$ sudo -u ftpsync ostree refs --alias --create=fedora/31/x86_64/silverblue fedora/31/x86_64/updates/silverblue
|
||||
$ sudo -u ftpsync ostree refs --alias --create=fedora/31/aarch64/silverblue fedora/31/aarch64/updates/silverblue
|
||||
$ sudo -u ftpsync ostree refs --alias --create=fedora/31/ppc64le/silverblue fedora/31/ppc64le/updates/silverblue
|
||||
....
|
||||
|
||||
Run the following command only from [.title-ref]#/mnt/koji/ostree/repo/#
|
||||
|
||||
....
|
||||
$ sudo ostree summary -u
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Before pushing the updates to fxx-updates, run the last branched compose
|
||||
so that both branched and rc composes have the same content. Once the
|
||||
branched compose is done, then update the silverblue refs as mentioned
|
||||
above. If the order is changed, that will screw up the refs
|
||||
====
|
||||
|
||||
==== Disable Branched Compose
|
||||
|
||||
Now that we have a final GOLD compose, we dont need nightly branched
|
||||
composes anymore. This is disabled in
|
||||
https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/releng[releng
|
||||
role] in infra ansible repo and then running the playbook.
|
||||
|
||||
....
|
||||
$ sudo rbac-playbook groups/releng-compose.yml
|
||||
....
|
||||
|
||||
==== Lift RelEng freeze
|
||||
|
||||
Lift the RelEng Freeze so that the updates will be pushed to stable.
|
||||
This is done by editing
|
||||
https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/vars/all/RelEngFrozen.yaml[RelEngFrozen
|
||||
variable] in infra ansible repo and then run the bodhi playbook.
|
||||
|
||||
....
|
||||
$ sudo rbac-playbook groups/bodhi-backend.yml
|
||||
....
|
||||
|
||||
==== Other Changes
|
||||
|
||||
These changes include enabling nightly container and cloud composes,
|
||||
other variable changes in infra ansible repo, bodhi pungi config
|
||||
changes, updates sync changes and others.
|
||||
|
||||
Run the appropriate playbooks after the following changes
|
||||
|
||||
....
|
||||
diff --git a/roles/releng/files/branched b/roles/releng/files/branched
|
||||
index 966f5c3..1c0454f 100644
|
||||
--- a/roles/releng/files/branched
|
||||
+++ b/roles/releng/files/branched
|
||||
@@ -1,3 +1,3 @@
|
||||
# branched compose
|
||||
-MAILTO=releng-cron@lists.fedoraproject.org
|
||||
-15 7 * * * root TMPDIR=`mktemp -d /tmp/branched.XXXXXX` && cd $TMPDIR && git clone https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && /usr/local/bin/lock-wrapper branched-compose "PYTHONMALLOC=debug LANG=en_US.UTF-8 ./nightly.sh" && sudo -u ftpsync /usr/local/bin/update-fullfiletimelist -l /pub/fedora-secondary/update-fullfiletimelist.lock -t /pub fedora fedora-secondary
|
||||
+#MAILTO=releng-cron@lists.fedoraproject.org
|
||||
+#15 7 * * * root TMPDIR=`mktemp -d /tmp/branched.XXXXXX` && cd $TMPDIR && git clone https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && /usr/local/bin/lock-wrapper branched-compose "PYTHONMALLOC=debug LANG=en_US.UTF-8 ./nightly.sh" && sudo -u ftpsync /usr/local/bin/update-fullfiletimelist -l /pub/fedora-secondary/update-fullfiletimelist.lock -t /pub fedora fedora-secondary
|
||||
diff --git a/roles/releng/files/cloud-updates b/roles/releng/files/cloud-updates
|
||||
index a0ffbe8..287d57d 100644
|
||||
--- a/roles/releng/files/cloud-updates
|
||||
+++ b/roles/releng/files/cloud-updates
|
||||
@@ -6,6 +6,6 @@ MAILTO=releng-cron@lists.fedoraproject.org
|
||||
MAILTO=releng-cron@lists.fedoraproject.org
|
||||
15 7 * * * root TMPDIR=`mktemp -d /tmp/CloudF29.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f29 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
|
||||
|
||||
-#Fedora 28 Cloud nightly compose
|
||||
-#MAILTO=releng-cron@lists.fedoraproject.org
|
||||
-#15 8 * * * root TMPDIR=`mktemp -d /tmp/CloudF28.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f28 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
|
||||
+#Fedora 31 Cloud nightly compose
|
||||
+MAILTO=releng-cron@lists.fedoraproject.org
|
||||
+15 8 * * * root TMPDIR=`mktemp -d /tmp/CloudF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
|
||||
diff --git a/roles/releng/files/container-updates b/roles/releng/files/container-updates
|
||||
index d763149..5446840 100644
|
||||
--- a/roles/releng/files/container-updates
|
||||
+++ b/roles/releng/files/container-updates
|
||||
@@ -1,6 +1,6 @@
|
||||
-#Fedora 28 Container Updates nightly compose
|
||||
-#MAILTO=releng-cron@lists.fedoraproject.org
|
||||
-#45 5 * * * root TMPDIR=`mktemp -d /tmp/containerF28.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f28 && LANG=en_US.UTF-8 ./container-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
|
||||
+#Fedora 31 Container Updates nightly compose
|
||||
+MAILTO=releng-cron@lists.fedoraproject.org
|
||||
+45 5 * * * root TMPDIR=`mktemp -d /tmp/containerF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./container-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
|
||||
|
||||
# Fedora 30 Container Updates nightly compose
|
||||
MAILTO=releng-cron@lists.fedoraproject.org
|
||||
diff --git a/vars/all/00-FedoraCycleNumber.yaml b/vars/all/00-FedoraCycleNumber.yaml
|
||||
index 22476b0..4bd0d46 100644
|
||||
--- a/vars/all/00-FedoraCycleNumber.yaml
|
||||
+++ b/vars/all/00-FedoraCycleNumber.yaml
|
||||
@@ -1 +1 @@
|
||||
-FedoraCycleNumber: 30
|
||||
+FedoraCycleNumber: 31
|
||||
|
||||
diff --git a/vars/all/FedoraBranched.yaml b/vars/all/FedoraBranched.yaml
|
||||
index 42ac534..0bbcc1d 100644
|
||||
--- a/vars/all/FedoraBranched.yaml
|
||||
+++ b/vars/all/FedoraBranched.yaml
|
||||
@@ -1 +1 @@
|
||||
-FedoraBranched: True
|
||||
+FedoraBranched: False
|
||||
|
||||
diff --git a/vars/all/FedoraPreviousPrevious.yaml b/vars/all/FedoraPreviousPrevious.yaml
|
||||
index a8e3d3b..a061e04 100644
|
||||
--- a/vars/all/FedoraPreviousPrevious.yaml
|
||||
+++ b/vars/all/FedoraPreviousPrevious.yaml
|
||||
@@ -1 +1 @@
|
||||
-FedoraPreviousPrevious: False
|
||||
+FedoraPreviousPrevious: True
|
||||
diff --git a/vars/all/Frozen.yaml b/vars/all/Frozen.yaml
|
||||
index 97d3bc3..7578a88 100644
|
||||
--- a/vars/all/Frozen.yaml
|
||||
+++ b/vars/all/Frozen.yaml
|
||||
@@ -1 +1 @@
|
||||
-Frozen: True
|
||||
+Frozen: False
|
||||
|
||||
|
||||
diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
|
||||
index 688bade..28b524a 100644
|
||||
--- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
|
||||
+++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
|
||||
@@ -179,8 +179,8 @@ ostree = {
|
||||
# In the case of testing, also inject the last stable updates
|
||||
"https://kojipkgs{{ env_suffix }}.fedoraproject.org/compose/updates/f[[ release.version_int ]]-updates/compose/Everything/$basearch/os/",
|
||||
[% endif %]
|
||||
- # For f31 the compose location is going to be under /compose/branched/
|
||||
- [% if release.version_int == 31 %]
|
||||
+ # For F32 the compose location is going to be under /compose/branched/
|
||||
+ [% if release.version_int == 32 %]
|
||||
"https://kojipkgs{{ env_suffix }}.fedoraproject.org/compose/branched/latest-Fedora-[[ release.version_int ]]/compose/Everything/$basearch/os/"
|
||||
[% else %]
|
||||
"https://kojipkgs{{ env_suffix }}.fedoraproject.org/compose/[[ release.version_int ]]/latest-Fedora-[[ release.version_int ]]/compose/Everything/$basearch/os/"
|
||||
|
||||
diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
|
||||
index 28b524a..640ddf0 100644
|
||||
--- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
|
||||
+++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
|
||||
@@ -193,8 +193,8 @@ ostree = {
|
||||
"ostree_ref": "fedora/[[ release.version_int ]]/${basearch}/testing/silverblue",
|
||||
[% endif %]
|
||||
"tag_ref": False,
|
||||
- "arches": ["x86_64"],
|
||||
- "failable": ["x86_64"]
|
||||
+ "arches": ["x86_64", "ppc64le", "aarch64" ],
|
||||
+ "failable": ["x86_64", "ppc64le", "aarch64" ]
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
diff --git a/roles/bodhi2/backend/files/new-updates-sync b/roles/bodhi2/backend/files/new-updates-sync
|
||||
index d08c893..2d0fb4d 100755
|
||||
--- a/roles/bodhi2/backend/files/new-updates-sync
|
||||
+++ b/roles/bodhi2/backend/files/new-updates-sync
|
||||
@@ -25,8 +25,9 @@ RELEASES = {'f31': {'topic': 'fedora',
|
||||
'modules': ['fedora', 'fedora-secondary'],
|
||||
'repos': {'updates': {
|
||||
'from': 'f31-updates',
|
||||
- 'ostrees': [{'ref': 'fedora/31/x86_64/updates/silverblue',
|
||||
- 'dest': OSTREEDEST}],
|
||||
+ 'ostrees': [{'ref': 'fedora/31/%(arch)s/updates/silverblue',
|
||||
+ 'dest': OSTREEDEST,
|
||||
+ 'arches': ['x86_64', 'ppc64le', 'aarch64']}],
|
||||
'to': [{'arches': ['x86_64', 'armhfp', 'aarch64', 'source'],
|
||||
'dest': os.path.join(FEDORADEST, '31', 'Everything')},
|
||||
{'arches': ['ppc64le', 's390x'],
|
||||
@@ -34,8 +35,9 @@ RELEASES = {'f31': {'topic': 'fedora',
|
||||
]},
|
||||
'updates-testing': {
|
||||
'from': 'f31-updates-testing',
|
||||
- 'ostrees': [{'ref': 'fedora/31/x86_64/testing/silverblue',
|
||||
- 'dest': OSTREEDEST}],
|
||||
+ 'ostrees': [{'ref': 'fedora/31/%(arch)s/testing/silverblue',
|
||||
+ 'dest': OSTREEDEST,
|
||||
+ 'arches': ['x86_64', 'ppc64le', 'aarch64']}],
|
||||
'to': [{'arches': ['x86_64', 'aarch64', 'armhfp', 'source'],
|
||||
'dest': os.path.join(FEDORADEST, 'testing', '31', 'Everything')},
|
||||
{'arches': ['ppc64le', 's390x'],
|
||||
|
||||
|
||||
diff --git a/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json b/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json
|
||||
index aac977e..9e0cbf2 100644
|
||||
--- a/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json
|
||||
+++ b/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json
|
||||
@@ -12,14 +12,14 @@
|
||||
"version": "devel"
|
||||
},
|
||||
{
|
||||
- "allow_retire": true,
|
||||
+ "allow_retire": false,
|
||||
"branchname": "f31",
|
||||
"date_created": "2014-05-14 12:36:15",
|
||||
"date_updated": "2018-08-14 17:07:23",
|
||||
"dist_tag": ".fc31",
|
||||
"koji_name": "f31",
|
||||
"name": "Fedora",
|
||||
- "status": "Under Development",
|
||||
+ "status": "Active",
|
||||
"version": "31"
|
||||
},
|
||||
{
|
||||
....
|
||||
|
||||
==== Bodhi config
|
||||
|
||||
==== After Beta
|
||||
|
||||
....
|
||||
diff --git a/vars/all/FedoraBranchedBodhi.yaml b/vars/all/FedoraBranchedBodhi.yaml
|
||||
index 606eb2e..ca2ba61 100644
|
||||
--- a/vars/all/FedoraBranchedBodhi.yaml
|
||||
+++ b/vars/all/FedoraBranchedBodhi.yaml
|
||||
@@ -3,4 +3,4 @@
|
||||
#prebeta: After bodhi enablement/beta freeze and before beta release
|
||||
#postbeta: After beta release and before final release
|
||||
#current: After final release
|
||||
-FedoraBranchedBodhi: prebeta
|
||||
+FedoraBranchedBodhi: postbeta
|
||||
....
|
||||
|
||||
==== After Final
|
||||
|
||||
....
|
||||
diff --git a/vars/all/FedoraBranchedBodhi.yaml b/vars/all/FedoraBranchedBodhi.yaml
|
||||
index 380f61d..76ba14d 100644
|
||||
--- a/vars/all/FedoraBranchedBodhi.yaml
|
||||
+++ b/vars/all/FedoraBranchedBodhi.yaml
|
||||
@@ -1,2 +1,2 @@
|
||||
#options are: prebeta, postbeta, current
|
||||
-FedoraBranchedBodhi: postbeta
|
||||
+FedoraBranchedBodhi: current
|
||||
....
|
||||
|
||||
==== Mirroring
|
||||
|
||||
Run [.title-ref]#stage-release.sh# script from
|
||||
https://pagure.io/releng[releng repo] in pagure on
|
||||
[.title-ref]#bodhi-backend01.phx2.fedoraproject.org#, this will sign the
|
||||
checksums and will put the content on mirrors.
|
||||
|
||||
===== For Beta
|
||||
|
||||
....
|
||||
$ sh scripts/stage-release.sh 32_Beta Fedora-32-20200312.0 32_Beta-1.2 fedora-32 1
|
||||
....
|
||||
|
||||
===== For Final
|
||||
|
||||
....
|
||||
$ sh scripts/stage-release.sh 32 Fedora-32-20200415.0 32_RC-1.1 fedora-32 0
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Make sure to grab the directory size usage numbers which is used to send
|
||||
an email to [.title-ref]#mirror-admin@lists.fedoraproject.org# list.
|
||||
====
|
||||
|
||||
==== Sync the signed checksums to stage
|
||||
|
||||
We need to sync the signed checksums to /pub/alt/stage/ by running the
|
||||
following command
|
||||
|
||||
....
|
||||
$ for dir in Everything Cloud Container Labs Server Spins Workstation Silverblue Kinoite metadata; do sudo -u ftpsync rsync -avhH /mnt/koji/compose/37/Fedora-37-20221105.0/compose/$dir/ /pub/alt/stage/37_RC-1.7/$dir/ --link-dest=/pub/fedora/linux/releases/37/Everything/ --link-dest=/pub/alt/stage/37_RC-1.2/Everything/ --link-dest=/pub/alt/stage/37_RC-1.3/Everything --link-dest=/pub/alt/stage/37_RC-1.4/Everything --link-dest=/pub/alt/stage/37_RC-1.5/Everything --link-dest=/pub/alt/stage/37_RC-1.6/Everything --link-dest=/pub/alt/stage/37_RC-1.7/Everything; done
|
||||
....
|
||||
|
||||
=== Move development to release folder with mirrormanager
|
||||
|
||||
Two weeks after the release move bits from development to release
|
||||
directory
|
||||
|
||||
. {blank}
|
||||
+
|
||||
ssh to the mm-backend01.iad2.fedoraproject.org::
|
||||
::;;
|
||||
$ ssh mm-backend01.iad2.fedoraproject.org
|
||||
. {blank}
|
||||
+
|
||||
get root::
|
||||
::;;
|
||||
$ sudo su
|
||||
. {blank}
|
||||
+
|
||||
run the mm2_move-devel-to-release::
|
||||
::;;
|
||||
$ mm2_move-devel-to-release --version=35 --category='Fedora Linux'
|
||||
|
||||
=== Consider before Running
|
||||
|
||||
Composes and file synchronizations should be run in a screen session on
|
||||
a remote machine. This enables the operator to withstand network
|
||||
connection issues.
|
||||
536
modules/release_guide/pages/sop_create_release_signing_key.adoc
Normal file
536
modules/release_guide/pages/sop_create_release_signing_key.adoc
Normal file
|
|
@ -0,0 +1,536 @@
|
|||
== Create Release Signing Key
|
||||
|
||||
=== Description
|
||||
|
||||
At the beginning of each release under development a new package signing
|
||||
key is created for it. This key is used to prove the authenticity of
|
||||
packages built by Fedora and distributed by Fedora. This key will be
|
||||
used to sign all packages for the public test and final releases.
|
||||
|
||||
=== Action
|
||||
|
||||
==== Sigul
|
||||
|
||||
Sigul is the signing server which holds our keys. In order to make use
|
||||
of a new key, the key will have to be created and access to the key will
|
||||
have to be granted. The `new-key`, `grant-key-access`, and
|
||||
`change-passphrase` commands are used.
|
||||
|
||||
....
|
||||
$ sigul new-key --help
|
||||
usage: client.py new-key [options] key
|
||||
|
||||
Add a key
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
--key-admin=USER Initial key administrator
|
||||
--name-real=NAME_REAL
|
||||
Real name of key subject
|
||||
--name-comment=NAME_COMMENT
|
||||
A comment about of key subject
|
||||
--name-email=NAME_EMAIL
|
||||
E-mail of key subject
|
||||
--expire-date=YYYY-MM-DD
|
||||
Key expiration date
|
||||
|
||||
$ sigul grant-key-access --help
|
||||
usage: client.py grant-key-access key user
|
||||
|
||||
Grant key access to a user
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
|
||||
$ sigul change-passphrase --help
|
||||
usage: client.py change-passphrase key
|
||||
|
||||
Change key passphrase
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
....
|
||||
|
||||
For example if we wanted to create the Fedora 23 signing key, we would
|
||||
do the following:
|
||||
|
||||
. Log into a system configured to run sigul client.
|
||||
. Create the key using a strong passphrase when prompted
|
||||
+
|
||||
....
|
||||
$ sigul new-key --key-admin ausil --key-type gnupg \
|
||||
--gnupg-name-real Fedora \
|
||||
--gnupg-name-comment 23 \
|
||||
--gnupg-name-email fedora-23-primary@fedoraproject.org fedora-23
|
||||
....
|
||||
+
|
||||
For EPEL
|
||||
+
|
||||
....
|
||||
$ sigul new-key --key-admin ausil --key-type gnupg \
|
||||
--gnupg-name-real "Fedora EPEL" \
|
||||
--gnupg-name-comment 7 \
|
||||
--gnupg-name-email epel@fedoraproject.org epel-7
|
||||
....
|
||||
. Wait a while for entropy. This can take several minutes.
|
||||
. For Fedora, also create the IMA signing key
|
||||
+
|
||||
....
|
||||
$ sigul new-key --key-admin ausil --key-type ECC fedora-23-ima
|
||||
....
|
||||
. Grant key access to Fedora Account holders who will be signing
|
||||
packages and protect it with a temporary a passphrase. For example,
|
||||
`CHANGEME.`. Do the same with the -ima key for Fedora.
|
||||
+
|
||||
....
|
||||
$ sigul grant-key-access fedora-23 kevin
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
*IMPORTANT:* Grant the access to autopen user as it's required for
|
||||
robosignatory autosigning and then restart robosignatory service
|
||||
====
|
||||
|
||||
. Provide the key name and temporary passphrase to signers. If they
|
||||
don't respond, revoke access until they are ready to change their
|
||||
passphrase. Signers can change their passphrase using the
|
||||
`change-passphrase` command:
|
||||
+
|
||||
....
|
||||
$ sigul change-passphrase fedora-23
|
||||
....
|
||||
. When your sigul cert expires, you will need to run:
|
||||
+
|
||||
....
|
||||
certutil -d ~/.sigul -D -n sigul-client-cert
|
||||
....
|
||||
+
|
||||
to remove the old cert, then
|
||||
+
|
||||
....
|
||||
sigul_setup_client
|
||||
....
|
||||
+
|
||||
to add a new one.
|
||||
|
||||
==== fedora-repos
|
||||
|
||||
The fedora-repos package houses a copy of the public key information.
|
||||
This is used by rpm to verify the signature on files encountered.
|
||||
Currently the fedora-repos package has a single key file named after the
|
||||
version of the key and the arch the key is for. To continue our example,
|
||||
the file would be named `RPM-GPG-KEY-fedora-27-primary` which is the
|
||||
primary arch key for Fedora 27. To create this file, use the
|
||||
`get-public-key` command from sigul:
|
||||
|
||||
....
|
||||
$ sigul get-public-key fedora-27 > RPM-GPG-KEY-fedora-27-primary
|
||||
....
|
||||
|
||||
Add this file to the repo and update the archmap file for the new
|
||||
release.
|
||||
|
||||
....
|
||||
$ git add RPM-GPG-KEY-fedora-27-primary
|
||||
....
|
||||
|
||||
Then make a new fedora-repos build for rawhide
|
||||
(`FIXME: this should be its own SOP`)
|
||||
|
||||
==== getfedora.org
|
||||
|
||||
getfedora.org/keys lists information about all of our keys. We need to
|
||||
let the websites team know we have created a new key so that they can
|
||||
add it to the list.
|
||||
|
||||
We do this by filing an issues in their pagure instance
|
||||
https://pagure.io/fedora-websites/ we should point them at this SOP
|
||||
|
||||
===== Web team SOP
|
||||
|
||||
....
|
||||
# from git repo root
|
||||
cd fedoraproject.org/
|
||||
curl $KEYURL > /tmp/newkey
|
||||
$EDITOR update-gpg-keys # Add key ID of recently EOL'd version to obsolete_keys
|
||||
./update-gpg-key /tmp/newkey
|
||||
gpg static/fedora.gpg # used to verify the new keyring
|
||||
# it should look something like this:
|
||||
# pub 4096R/57BBCCBA 2009-07-29 Fedora (12) <fedora@fedoraproject.org>
|
||||
# pub 4096R/E8E40FDE 2010-01-19 Fedora (13) <fedora@fedoraproject.org>
|
||||
# pub 4096R/97A1071F 2010-07-23 Fedora (14) <fedora@fedoraproject.org>
|
||||
# pub 1024D/217521F6 2007-03-02 Fedora EPEL <epel@fedoraproject.org>
|
||||
# sub 2048g/B6610DAF 2007-03-02 [expires: 2017-02-27]
|
||||
# it must only have the two supported versions of fedora, rawhide and EPEL
|
||||
# also verify that static/$NEWKEY.txt exists
|
||||
$EDITOR data/content/{keys,verify}.html # see git diff 1840f96~ 1840f96
|
||||
....
|
||||
|
||||
==== sigulsign_unsigned
|
||||
|
||||
`sigulsign_unsigned.py` is the script Release Engineers use to sign
|
||||
content in koji. This script has a hardcoded list of keys and aliases to
|
||||
the keys that needs to be updated when we create new keys.
|
||||
|
||||
Add the key details to the `KEYS` dictionary near the top of the
|
||||
`sigulsign_unsigned.py` script. It lives in Release Engineering's git
|
||||
repo at `ssh://git@pagure.io/releng.git` in the `scripts` directory. You
|
||||
will need to know the key ID to insert the correct information:
|
||||
|
||||
....
|
||||
$ gpg <key block from sigul get-public-key>
|
||||
....
|
||||
|
||||
==== Public Keyservers
|
||||
|
||||
We upload the key to the public key servers when we create the keys. To
|
||||
do this, we need to get the ascii key block from sigul, determine the
|
||||
key ID, import they key into our local keyring, and then upload it to
|
||||
the key servers.
|
||||
|
||||
....
|
||||
$ sigul get-public-key fedora-13 > fedora-13
|
||||
$ gpg fedora-13 (The ID is the "E8E40FDE" part of 4096R/E8E40FDE)
|
||||
$ gpg --import fedora-13
|
||||
$ gpg --send-keys E8E40FDE
|
||||
....
|
||||
|
||||
==== pungi-fedora
|
||||
|
||||
The nightly compose configs come from the pungi-fedora project on
|
||||
https://pagure.io We need to create a pull request to pull in the new
|
||||
key.
|
||||
|
||||
....
|
||||
$ git clone ssh://git@pagure.io/<your fork path>/pungi-fedora.git
|
||||
$ cd pungi-fedora
|
||||
$ vim *conf
|
||||
<set key value in sigkeys = line >
|
||||
$ git commit -m 'Add new key'
|
||||
$ git push
|
||||
$ file a Pull Request
|
||||
....
|
||||
|
||||
==== Koji
|
||||
|
||||
Koji has a garbage collection utility that will find builds that meet
|
||||
criteria to be removed to save space. Part of that criteria has to do
|
||||
with whether or not the build has been signed with a key. If the
|
||||
collection utility doesn't know about a key it will ignore the build.
|
||||
Thus as we create new keys we need to inform the utility of these keys
|
||||
or else builds can pile up. The configuration for the garbage collection
|
||||
lives within ansible.
|
||||
|
||||
On a clone of the infrastructure ansible git repo edit the
|
||||
roles/koji_hub/templates/koji-gc.conf.j2 file:
|
||||
|
||||
....
|
||||
diff --git a/roles/koji_hub/templates/koji-gc.conf.j2 b/roles/koji_hub/templates/koji-gc.conf.j2
|
||||
index 9ecb750..9c48a8e 100644
|
||||
--- a/roles/koji_hub/templates/koji-gc.conf.j2
|
||||
+++ b/roles/koji_hub/templates/koji-gc.conf.j2
|
||||
@@ -35,6 +35,7 @@ key_aliases =
|
||||
81B46521 fedora-24
|
||||
FDB19C98 fedora-25
|
||||
64DAB85D fedora-26
|
||||
+ F5282EE4 fedora-27
|
||||
217521F6 fedora-epel
|
||||
0608B895 fedora-epel-6
|
||||
352C64E5 fedora-epel-7
|
||||
@@ -52,6 +53,7 @@ unprotected_keys =
|
||||
fedora-24
|
||||
fedora-25
|
||||
fedora-26
|
||||
+ fedora-27
|
||||
fedora-extras
|
||||
redhat-beta
|
||||
fedora-epel
|
||||
@@ -91,6 +93,7 @@ policy =
|
||||
sig fedora-24 && age < 12 weeks :: keep
|
||||
sig fedora-25 && age < 12 weeks :: keep
|
||||
sig fedora-26 && age < 12 weeks :: keep
|
||||
+ sig fedora-27 && age < 12 weeks :: keep
|
||||
sig fedora-epel && age < 12 weeks :: keep
|
||||
sig fedora-epel-6 && age < 12 weeks :: keep
|
||||
sig fedora-epel-7 && age < 12 weeks :: keep
|
||||
....
|
||||
|
||||
In this case the fedora-epel key was added to the list of key aliases,
|
||||
then referenced in the list of unprotected_keys, and finally a policy
|
||||
was created for how long to keep builds signed with this key.
|
||||
|
||||
Once you've made your change commit and push. The buildsystem will pick
|
||||
up this change the next time puppet refreshes.
|
||||
|
||||
=== Verification
|
||||
|
||||
We can verify that the key was created in sigul, the correct users have
|
||||
access to the key, the key was added to the fedora-release package, that
|
||||
the website was updated with the right key, that sigulsign_unsigned was
|
||||
properly updated, and that the key was successfully updated to the
|
||||
public key servers.
|
||||
|
||||
==== sigul
|
||||
|
||||
Use the `list-keys` command to verify that the key was indeed added to
|
||||
sigul:
|
||||
|
||||
....
|
||||
$ sigul list-keys
|
||||
Administrator's password:
|
||||
fedora-10
|
||||
fedora-10-testing
|
||||
fedora-11
|
||||
fedora-12
|
||||
fedora-13
|
||||
....
|
||||
|
||||
Our new key should be on the list. This command expects *your*
|
||||
administrative password.
|
||||
|
||||
Use the `list-key-users` command to verify all the signers have access:
|
||||
|
||||
....
|
||||
$ sigul list-key-users fedora-13
|
||||
Key passphrase:
|
||||
jkeating
|
||||
jwboyer
|
||||
....
|
||||
|
||||
This command expects *your* key passphrase for the key in question.
|
||||
|
||||
==== fedora-release
|
||||
|
||||
To verify that the key was added to this package correctly, download the
|
||||
latest build from koji and run rpm2cpio on it, then run gpg on the key
|
||||
file:
|
||||
|
||||
....
|
||||
$ koji download-build --arch noarch --latest f27 fedora-repos
|
||||
fedora-repos-rawhide-27-0.1.noarch.rpm | 7.3 kB 00:00:00
|
||||
fedora-repos-27-0.1.noarch.rpm | 87 kB 00:00:00
|
||||
$ rpmdev-extract fedora-repos-27-0.1.noarch.rpm
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-27-fedora
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-ppc
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-ppc
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-arm
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-armhfp
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-mips
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-secondary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-14-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-14-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-14-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-arm
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-armhfp
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-ppc
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-s390
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-s390x
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-secondary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-arm
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-armhfp
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-ppc
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-s390
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-s390x
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-secondary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-arm
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-armhfp
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-ppc
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-s390
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-s390x
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-secondary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-arm
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-armhfp
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-ppc
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-s390
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-s390x
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-secondary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-armhfp
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-ppc
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-s390
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-s390x
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-secondary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-armhfp
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-ppc
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-s390
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-s390x
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-secondary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-aarch64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-armhfp
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-ppc64le
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-s390
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-s390x
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-secondary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-aarch64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-armhfp
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-ppc64le
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-s390
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-s390x
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-secondary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-aarch64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-armhfp
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-ppc64le
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-s390
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-s390x
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-secondary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-aarch64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-armhfp
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-ppc64le
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-s390x
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-secondary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-aarch64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-armhfp
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-ppc64le
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-s390x
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-secondary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-aarch64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-armhfp
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-ppc64le
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-s390x
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-secondary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-aarch64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-armhfp
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-ppc64le
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-s390x
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-7-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-ppc
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-primary-original
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-i386
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-ia64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-ppc
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-ppc64
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-primary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-primary-original
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-secondary
|
||||
fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-x86_64
|
||||
fedora-repos-27-0.1.noarch/etc/yum.repos.d
|
||||
fedora-repos-27-0.1.noarch/etc/yum.repos.d/fedora-cisco-openh264.repo
|
||||
fedora-repos-27-0.1.noarch/etc/yum.repos.d/fedora-updates-testing.repo
|
||||
fedora-repos-27-0.1.noarch/etc/yum.repos.d/fedora-updates.repo
|
||||
fedora-repos-27-0.1.noarch/etc/yum.repos.d/fedora.repo
|
||||
|
||||
$ gpg2 fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-primary
|
||||
pub rsa4096 2017-02-21 [SCE]
|
||||
860E19B0AFA800A1751881A6F55E7430F5282EE4
|
||||
uid Fedora 27 (27) <fedora-27@fedoraproject.org>
|
||||
pub 4096R/E8E40FDE 2010-01-19 Fedora (13) <fedora@fedoraproject.org>
|
||||
....
|
||||
|
||||
You may wish to do this in a tempoary directory to make cleaning it up
|
||||
easy.
|
||||
|
||||
==== getfedora.org
|
||||
|
||||
One can simply browse to https://getfedora.org/keys to verify that the
|
||||
key has been uploaded.
|
||||
|
||||
==== sigulsign_unsigned
|
||||
|
||||
The best way to test whether or not the key has been added correctly is
|
||||
to sign a package using the key, like our newly built fedora-repos
|
||||
package.
|
||||
|
||||
....
|
||||
$ ./sigulsign_unsigned.py fedora-13 fedora-release-13-0.3
|
||||
Passphrase for fedora-13:
|
||||
....
|
||||
|
||||
The command should exit cleanly.
|
||||
|
||||
==== Public key servers
|
||||
|
||||
One can use the <code>search-keys</code> command from gpg to locate the
|
||||
key on the public server:
|
||||
|
||||
....
|
||||
$ gpg2 --search-keys "Fedora (13)"
|
||||
gpg: searching for "Fedora (13)" from hkp server subkeys.pgp.net
|
||||
(1) Fedora (13) <fedora@fedoraproject.org>
|
||||
4096 bit RSA key E8E40FDE, created: 2010-01-19
|
||||
...
|
||||
....
|
||||
|
||||
==== Koji
|
||||
|
||||
Log into koji02.phx2.fedoraproject.org by way of
|
||||
bastion.fedoraproject.org.
|
||||
|
||||
Verify that `/etc/koji-gc/koji-gc.conf` has the new key in it.
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
Nothing at this time.
|
||||
123
modules/release_guide/pages/sop_deprecate_ftbfs_packages.adoc
Normal file
123
modules/release_guide/pages/sop_deprecate_ftbfs_packages.adoc
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
== Deprecate FTBFS Packages
|
||||
|
||||
=== Description
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
FTBFS = "Fails To Build From Source"
|
||||
====
|
||||
|
||||
Every release prior to the Feature Freeze we deprecate all packages that
|
||||
https://fedoraproject.org/wiki/Fails_to_build_from_source[FTBFS]. This
|
||||
keeps out software that no longer builds from source, and prevents
|
||||
future problems down the road.
|
||||
|
||||
=== Action
|
||||
|
||||
The FTBFS process takes place in stages:
|
||||
|
||||
. Detecting a list of FTBFS packages and the dependencies that will be
|
||||
broken if they are removed.
|
||||
. Sending the list of potential deprecated FTBFS packages to
|
||||
devel@lists.fedoraproject.org for community review and removal from the
|
||||
FTBFS list by fixing the package.
|
||||
. Removing packages confirmed as FTBFS from the Fedora package
|
||||
repositories.
|
||||
|
||||
==== Detecting FTBFS
|
||||
|
||||
We will remove packages that have failed to build for at least two
|
||||
release cycles. For example, in preparation for Fedora 21 branching,
|
||||
packages which FTBFS since the Fedora 19 cycle (i.e. packages that have
|
||||
a dist tag of fc18 or earlier) will be considered candidates for
|
||||
removal. Adjust
|
||||
https://pagure.io/releng/blob/main/f/scripts/find_FTBFS.py[find_FTBFS.py]
|
||||
and run it to get a list of candidate packages.
|
||||
|
||||
Given a candidate list from above, rel-eng should attempt to build each
|
||||
of the candidate packages using koji. Should package building now
|
||||
succeed, the package may be removed from the candidate list.
|
||||
|
||||
==== Announcing Packages to be Deprecated
|
||||
|
||||
Email the output to the development list
|
||||
(`devel@lists.fedodraproject.org`) at least a week before the feature
|
||||
freeze. This gives maintainers an opportunity to fix packages that are
|
||||
important to them. Follow-up on the list where necessary.
|
||||
|
||||
==== Retiring FTBFS packages
|
||||
|
||||
Once maintainers have been given an opportunity to pick up and fix FTBFS
|
||||
packages, the remaining packages are `retired` by blocking them, and
|
||||
creating the `dead.package` file in git.
|
||||
|
||||
===== GIT and Package DB
|
||||
|
||||
Required permissions: provenpackage for GIT, cvsadmin for Package DB.
|
||||
|
||||
We just have to remove the existing files from the `rawhide` branch and
|
||||
replace them with a `dead.package` file whose contents describe why the
|
||||
package is dead. Also the package needs to be marked as retired in
|
||||
PackageDB. Fedpkg takes care of this:
|
||||
|
||||
For example, if we wished to clean up git for the roxterm package we
|
||||
would:
|
||||
|
||||
....
|
||||
$ fedpkg clone roxterm
|
||||
$ cd roxterm
|
||||
$ fedpkg retire "Retired on $(date -I), because it failed to build for two releases (FTBFS Cleanup)."
|
||||
....
|
||||
|
||||
===== Koji
|
||||
|
||||
Required permissions: admin in koji if the automatic blocking fails.
|
||||
|
||||
Blocking should happen automatically a few minutes after the packags was
|
||||
retired in PackageDB. If it does not, use the `block-pkg` `koji` command
|
||||
is used to do the blocking.
|
||||
|
||||
Koji accepts multiple package names as input and thus we can use the
|
||||
FTBFS package list as input. Deprecated packages are only blocked from
|
||||
the latest `f##` tag. For example, if we wanted to `deprecate` (block)
|
||||
`sbackup, roxterm,` and `uisp` from rawhide during the development of
|
||||
Fedora 21 we would run the following command:
|
||||
|
||||
....
|
||||
$ koji block-pkg f21 sbackup roxterm uisp
|
||||
....
|
||||
|
||||
===== Bugs
|
||||
|
||||
This procedure probably leaves open bugs for the deprecated packages
|
||||
behind. It is not within the scope of releng to take care of these. If
|
||||
bugs are closed, only bugs targeted at Rawhide should be affected, since
|
||||
other branches might still be maintained.
|
||||
|
||||
=== Verification
|
||||
|
||||
To verify that the packages were blocked correctly we can use the
|
||||
`latest-pkg` `koji` action.
|
||||
|
||||
....
|
||||
$ koji latest-pkg dist-f16 wdm
|
||||
....
|
||||
|
||||
This should return nothing, as the `wdm` package is blocked.
|
||||
|
||||
Also check that package DB shows that the package is retired and that
|
||||
the rawhide branch contains only a dead.package file.
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
Generally we block anything that doesn't leave broken dependencies. If
|
||||
there are packages whose removal would result in broken dependencies a
|
||||
second warning should be sent to devel@lists.fedoraproject.org and to
|
||||
<package>-owner@fedoraproject.org for each dependent package.
|
||||
|
||||
Allow another couple of days for maintainers to take notice and fix
|
||||
these package so the package repository can be maintained without broken
|
||||
dependencies or needing to deprecate the package. It is not good to have
|
||||
broken package dependencies in our package repositories so every effort
|
||||
should be made to find owners or to fix the broken dependencies.
|
||||
537
modules/release_guide/pages/sop_end_of_life.adoc
Normal file
537
modules/release_guide/pages/sop_end_of_life.adoc
Normal file
|
|
@ -0,0 +1,537 @@
|
|||
== End Of Life
|
||||
|
||||
=== Description
|
||||
|
||||
Each release of Fedora is maintained as laid out in the
|
||||
https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Maintenance_Schedule[maintenance
|
||||
schedule]. At the conclusion of the maintenance period, a Fedora release
|
||||
enters `end of life` status. This procedure describes the tasks
|
||||
necessary to move a release to that status.
|
||||
|
||||
=== Actions
|
||||
|
||||
==== Set date
|
||||
|
||||
* {blank}
|
||||
+
|
||||
Releng responsibilities:::
|
||||
** Follow guidelines of
|
||||
https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Maintenance_Schedule[maintenance
|
||||
schedule]
|
||||
** Take into account any infrastructure or other supporting project
|
||||
resource contention
|
||||
** Announce the closure of the release to the package maintainers.
|
||||
|
||||
==== Reminder announcement
|
||||
|
||||
Send an email to devel@, devel-announce@, test-announce@, announce@
|
||||
lists as remainder about the release EOL.
|
||||
|
||||
===== Announcement Content
|
||||
|
||||
....
|
||||
Hello all,
|
||||
|
||||
Fedora 31 will go end of life for updates and support on 24th of
|
||||
November 2020. No further updates, including security updates, will be
|
||||
available for Fedora 31 after the said date. All the updates of Fedora
|
||||
31 being pushed to stable will be stopped as well.
|
||||
|
||||
Fedora 32 will continue to receive updates until approximately one
|
||||
month after the release of Fedora 34. The maintenance schedule of
|
||||
Fedora releases is documented on the Fedora Project wiki [0]. The
|
||||
fedora Project wiki also contains instructions [1] on how to upgrade
|
||||
from a previous release of Fedora to a version receiving updates.
|
||||
|
||||
Regards,
|
||||
Mohan Boddu.
|
||||
|
||||
[0]https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Maintenance_Schedule
|
||||
[1]https://fedoraproject.org/wiki/Upgrading?rd=DistributionUpgrades
|
||||
....
|
||||
|
||||
==== Koji tasks
|
||||
|
||||
* Disable builds by removing targets
|
||||
|
||||
....
|
||||
$ koji remove-target f31
|
||||
$ koji remove-target f31-candidate
|
||||
$ koji remove-target f31-container-candidate
|
||||
$ koji remove-target f31-flatpak-candidate
|
||||
$ koji remove-target f31-infra
|
||||
$ koji remove-target f31-coreos-continuous
|
||||
$ koji remove-target f31-rebuild
|
||||
$ koji remote-target <side-targets> #any side targets that are still around
|
||||
....
|
||||
|
||||
* Purge from disk the signed copies of rpms that are signed with the
|
||||
EOL'd release key. To acheive this, add the release key to
|
||||
*koji_cleanup_signed.py* script in https://pagure.io/releng[releng] repo
|
||||
and the script on compose-branched01.iad2.fedoraproject.org
|
||||
|
||||
....
|
||||
./scripts/koji_cleanup_signed.py
|
||||
....
|
||||
|
||||
==== PDC tasks
|
||||
|
||||
* Set PDC *active* value for the release to *False*
|
||||
|
||||
....
|
||||
curl -u: -H 'Authorization: Token <token>' -H 'Accept: application/json' -H 'Content-Type:application/json' -X PATCH -d '{"active":"false"}' https://pdc.fedoraproject.org/rest_api/v1/releases/fedora-31/
|
||||
....
|
||||
|
||||
* Set the EOL dates in PDC for all the components to the release EOL
|
||||
date if they are not already set. Run the following script from
|
||||
https://pagure.io/releng[releng] repo
|
||||
|
||||
....
|
||||
python scripts/pdc/adjust-eol-all.py <token> f31 2020-11-24
|
||||
....
|
||||
|
||||
==== Bodhi tasks
|
||||
|
||||
* Run the following bodhi commands to set the releases state to
|
||||
*archived*
|
||||
|
||||
....
|
||||
$ bodhi releases edit --name "F31" --state archived
|
||||
$ bodhi releases edit --name "F31M" --state archived
|
||||
$ bodhi releases edit --name "F31C" --state archived
|
||||
$ bodhi releases edit --name "F31F" --state archived
|
||||
....
|
||||
|
||||
[WARNING]
|
||||
.Warning
|
||||
====
|
||||
Due to a https://github.com/fedora-infra/bodhi/issues/2177[bug] in
|
||||
Bodhi, it is critical that Bodhi processes be restarted any time
|
||||
`bodhi releases create` or `bodhi releases edit` are used.
|
||||
====
|
||||
|
||||
* On bodhi-backend01.iad2.fedoraproject.org, run the following commands
|
||||
|
||||
....
|
||||
$ sudo systemctl restart fm-consumer@config.service
|
||||
$ sudo systemctl restart bodhi-celery.service
|
||||
....
|
||||
|
||||
==== Fedora Infra Ansible Changes
|
||||
|
||||
* We need to make changes to bodhi, koji, mbs, releng, autosign roles in
|
||||
ansible repo.
|
||||
|
||||
[source,diff]
|
||||
----
|
||||
From 73dc8a1042a190f1b88bf78e110d44753cfa7962 Mon Sep 17 00:00:00 2001
|
||||
From: Mohan Boddu <mboddu@bhujji.com>
|
||||
Date: Nov 24 2020 17:19:23 +0000
|
||||
Subject: F31 EOL
|
||||
|
||||
|
||||
Signed-off-by: Mohan Boddu <mboddu@bhujji.com>
|
||||
|
||||
---
|
||||
|
||||
diff --git a/roles/bodhi2/backend/files/new-updates-sync b/roles/bodhi2/backend/files/new-updates-sync
|
||||
index a143047..d8c8a73 100755
|
||||
--- a/roles/bodhi2/backend/files/new-updates-sync
|
||||
+++ b/roles/bodhi2/backend/files/new-updates-sync
|
||||
@@ -113,50 +113,6 @@ RELEASES = {'f33': {'topic': 'fedora',
|
||||
'dest': os.path.join(FEDORAALTDEST, 'testing', '32', 'Modular')}
|
||||
]}}
|
||||
},
|
||||
- 'f31': {'topic': 'fedora',
|
||||
- 'version': '31',
|
||||
- 'modules': ['fedora', 'fedora-secondary'],
|
||||
- 'repos': {'updates': {
|
||||
- 'from': 'f31-updates',
|
||||
- 'ostrees': [{'ref': 'fedora/31/%(arch)s/updates/silverblue',
|
||||
- 'dest': OSTREEDEST,
|
||||
- 'arches': ['x86_64', 'ppc64le', 'aarch64']}],
|
||||
- 'to': [{'arches': ['x86_64', 'armhfp', 'aarch64', 'source'],
|
||||
- 'dest': os.path.join(FEDORADEST, '31', 'Everything')},
|
||||
- {'arches': ['ppc64le', 's390x'],
|
||||
- 'dest': os.path.join(FEDORAALTDEST, '31', 'Everything')}
|
||||
- ]},
|
||||
- 'updates-testing': {
|
||||
- 'from': 'f31-updates-testing',
|
||||
- 'ostrees': [{'ref': 'fedora/31/%(arch)s/testing/silverblue',
|
||||
- 'dest': OSTREEDEST,
|
||||
- 'arches': ['x86_64', 'ppc64le', 'aarch64']}],
|
||||
- 'to': [{'arches': ['x86_64', 'aarch64', 'armhfp', 'source'],
|
||||
- 'dest': os.path.join(FEDORADEST, 'testing', '31', 'Everything')},
|
||||
- {'arches': ['ppc64le', 's390x'],
|
||||
- 'dest': os.path.join(FEDORAALTDEST, 'testing', '31', 'Everything')}
|
||||
- ]}}
|
||||
- },
|
||||
- 'f31m': {'topic': 'fedora',
|
||||
- 'version': '31m',
|
||||
- 'modules': ['fedora', 'fedora-secondary'],
|
||||
- 'repos': {'updates': {
|
||||
- 'from': 'f31-modular-updates',
|
||||
- 'ostrees': [],
|
||||
- 'to': [{'arches': ['x86_64', 'aarch64', 'armhfp', 'source'],
|
||||
- 'dest': os.path.join(FEDORADEST, '31', 'Modular')},
|
||||
- {'arches': ['ppc64le', 's390x'],
|
||||
- 'dest': os.path.join(FEDORAALTDEST, '31', 'Modular')}
|
||||
- ]},
|
||||
- 'updates-testing': {
|
||||
- 'from': 'f31-modular-updates-testing',
|
||||
- 'ostrees': [],
|
||||
- 'to': [{'arches': ['x86_64', 'aarch64', 'armhfp', 'source'],
|
||||
- 'dest': os.path.join(FEDORADEST, 'testing', '31', 'Modular')},
|
||||
- {'arches': ['ppc64le', 's390x'],
|
||||
- 'dest': os.path.join(FEDORAALTDEST, 'testing', '31', 'Modular')}
|
||||
- ]}}
|
||||
- },
|
||||
'epel8': {'topic': 'epel',
|
||||
'version': '8',
|
||||
'modules': ['epel'],
|
||||
diff --git a/roles/bodhi2/backend/tasks/main.yml b/roles/bodhi2/backend/tasks/main.yml
|
||||
index a4b2a2b..d84f86a 100644
|
||||
--- a/roles/bodhi2/backend/tasks/main.yml
|
||||
+++ b/roles/bodhi2/backend/tasks/main.yml
|
||||
@@ -76,7 +76,7 @@
|
||||
# bodhi2/backend/files/koji_sync_listener.py
|
||||
# This cronjob runs only once a day. The listener script runs reactively.
|
||||
cron: name="owner-sync" minute="15" hour="4" user="root"
|
||||
- job="/usr/local/bin/lock-wrapper owner-sync '/usr/local/bin/owner-sync-pagure f34 f34-container f34-modular f33 f33-container f33-modular f33-flatpak f32 f32-container f32-modular f32-flatpak f31 f31-container f31-flatpak f31-modular epel8 epel8-playground epel8-modular epel7 dist-6E-epel module-package-list modular'"
|
||||
+ job="/usr/local/bin/lock-wrapper owner-sync '/usr/local/bin/owner-sync-pagure f34 f34-container f34-modular f33 f33-container f33-modular f33-flatpak f32 f32-container f32-modular f32-flatpak epel8 epel8-playground epel8-modular epel7 dist-6E-epel module-package-list modular'"
|
||||
cron_file=update-koji-owner
|
||||
when: env == "production"
|
||||
tags:
|
||||
diff --git a/roles/bodhi2/backend/templates/koji_sync_listener.toml b/roles/bodhi2/backend/templates/koji_sync_listener.toml
|
||||
index 753adc0..41954ca 100644
|
||||
--- a/roles/bodhi2/backend/templates/koji_sync_listener.toml
|
||||
+++ b/roles/bodhi2/backend/templates/koji_sync_listener.toml
|
||||
@@ -48,10 +48,6 @@ taglist = [
|
||||
"f32-container",
|
||||
"f32-modular",
|
||||
"f32-flatpak",
|
||||
- "f31",
|
||||
- "f31-container",
|
||||
- "f31-flatpak",
|
||||
- "f31-modular",
|
||||
"epel8",
|
||||
"epel8-playground",
|
||||
"epel8-modular",
|
||||
diff --git a/roles/koji_hub/templates/hub.conf.j2 b/roles/koji_hub/templates/hub.conf.j2
|
||||
index 2f8b716..4816dba 100644
|
||||
--- a/roles/koji_hub/templates/hub.conf.j2
|
||||
+++ b/roles/koji_hub/templates/hub.conf.j2
|
||||
@@ -187,6 +187,5 @@ sidetag =
|
||||
tag f34-build :: allow
|
||||
tag f33-build :: allow
|
||||
tag f32-build :: allow
|
||||
- tag f31-build :: allow
|
||||
tag eln-build :: allow
|
||||
all :: deny
|
||||
diff --git a/roles/mbs/common/files/default-modules.production/platform-f31.yaml b/roles/mbs/common/files/default-modules.production/platform-f31.yaml
|
||||
deleted file mode 100644
|
||||
index 0608f93..0000000
|
||||
--- a/roles/mbs/common/files/default-modules.production/platform-f31.yaml
|
||||
+++ /dev/null
|
||||
@@ -1,28 +0,0 @@
|
||||
-data:
|
||||
- description: Fedora 31 traditional base
|
||||
- license:
|
||||
- module: [MIT]
|
||||
- name: platform
|
||||
- profiles:
|
||||
- buildroot:
|
||||
- rpms: [bash, bzip2, coreutils, cpio, diffutils, fedora-release, findutils, gawk,
|
||||
- glibc-minimal-langpack, grep, gzip, info, make, patch, redhat-rpm-config,
|
||||
- rpm-build, sed, shadow-utils, tar, unzip, util-linux, which, xz]
|
||||
- srpm-buildroot:
|
||||
- rpms: [bash, fedora-release, fedpkg-minimal, glibc-minimal-langpack, gnupg2,
|
||||
- redhat-rpm-config, rpm-build, shadow-utils]
|
||||
- stream: f31
|
||||
- summary: Fedora 31 traditional base
|
||||
- context: 00000000
|
||||
- version: 1
|
||||
- xmd:
|
||||
- mbs:
|
||||
- buildrequires: {}
|
||||
- commit: f31
|
||||
- requires: {}
|
||||
- koji_tag: module-f31-build
|
||||
- mse: TRUE
|
||||
- virtual_streams: [fedora]
|
||||
-document: modulemd
|
||||
-version: 1
|
||||
-
|
||||
diff --git a/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json b/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json
|
||||
index d2f9a89..0eae499 100644
|
||||
--- a/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json
|
||||
+++ b/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json
|
||||
@@ -41,7 +41,7 @@
|
||||
"dist_tag": ".fc31",
|
||||
"koji_name": "f31",
|
||||
"name": "Fedora",
|
||||
- "status": "Active",
|
||||
+ "status": "EOL",
|
||||
"version": "31"
|
||||
},
|
||||
{
|
||||
diff --git a/roles/releng/files/cloud-updates b/roles/releng/files/cloud-updates
|
||||
index 0a37b49..ebb807c 100644
|
||||
--- a/roles/releng/files/cloud-updates
|
||||
+++ b/roles/releng/files/cloud-updates
|
||||
@@ -7,5 +7,5 @@ MAILTO=releng-cron@lists.fedoraproject.org
|
||||
15 7 * * * root TMPDIR=`mktemp -d /tmp/CloudF32.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f32 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
|
||||
|
||||
# Fedora 31 Cloud nightly compose
|
||||
-MAILTO=releng-cron@lists.fedoraproject.org
|
||||
-15 8 * * * root TMPDIR=`mktemp -d /tmp/CloudF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
|
||||
+#MAILTO=releng-cron@lists.fedoraproject.org
|
||||
+#15 8 * * * root TMPDIR=`mktemp -d /tmp/CloudF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
|
||||
diff --git a/roles/releng/files/container-updates b/roles/releng/files/container-updates
|
||||
index d3a0e14..6932bec 100644
|
||||
--- a/roles/releng/files/container-updates
|
||||
+++ b/roles/releng/files/container-updates
|
||||
@@ -1,6 +1,6 @@
|
||||
#Fedora 31 Container Updates nightly compose
|
||||
-MAILTO=releng-cron@lists.fedoraproject.org
|
||||
-45 5 * * * root TMPDIR=`mktemp -d /tmp/containerF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./container-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
|
||||
+#MAILTO=releng-cron@lists.fedoraproject.org
|
||||
+#45 5 * * * root TMPDIR=`mktemp -d /tmp/containerF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./container-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
|
||||
|
||||
# Fedora 33 Container Updates nightly compose
|
||||
MAILTO=releng-cron@lists.fedoraproject.org
|
||||
diff --git a/roles/robosignatory/templates/robosignatory.toml.j2 b/roles/robosignatory/templates/robosignatory.toml.j2
|
||||
index 41ab24c..60295c1 100644
|
||||
--- a/roles/robosignatory/templates/robosignatory.toml.j2
|
||||
+++ b/roles/robosignatory/templates/robosignatory.toml.j2
|
||||
@@ -92,30 +92,6 @@ handlers = ["console"]
|
||||
|
||||
# Temporary tags
|
||||
|
||||
- [[consumer_config.koji_instances.primary.tags]]
|
||||
- from = "f31-kde"
|
||||
- to = "f31-kde"
|
||||
- key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}"
|
||||
- keyid = "{{ (env == 'production')|ternary('3c3359c4', 'd300e724') }}"
|
||||
-
|
||||
- [[consumer_config.koji_instances.primary.tags]]
|
||||
- from = "f31-gnome"
|
||||
- to = "f31-gnome"
|
||||
- key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}"
|
||||
- keyid = "{{ (env == 'production')|ternary('3c3359c4', 'd300e724') }}"
|
||||
-
|
||||
- [[consumer_config.koji_instances.primary.tags]]
|
||||
- from = "f31-python"
|
||||
- to = "f31-python"
|
||||
- key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}"
|
||||
- keyid = "{{ (env == 'production')|ternary('3c3359c4', 'd300e724') }}"
|
||||
-
|
||||
- [[consumer_config.koji_instances.primary.tags]]
|
||||
- from = "f30-kde"
|
||||
- to = "f30-kde"
|
||||
- key = "{{ (env == 'production')|ternary('fedora-30', 'testkey') }}"
|
||||
- keyid = "{{ (env == 'production')|ternary('cfc659b9', 'd300e724') }}"
|
||||
-
|
||||
# Infra tags
|
||||
|
||||
[[consumer_config.koji_instances.primary.tags]]
|
||||
@@ -143,12 +119,6 @@ handlers = ["console"]
|
||||
keyid = "{{ (env == 'production')|ternary('47dd8ef9', 'd300e724') }}"
|
||||
|
||||
[[consumer_config.koji_instances.primary.tags]]
|
||||
- from = "f31-infra-candidate"
|
||||
- to = "f31-infra-stg"
|
||||
- key = "{{ (env == 'production')|ternary('fedora-infra', 'testkey') }}"
|
||||
- keyid = "{{ (env == 'production')|ternary('47dd8ef9', 'd300e724') }}"
|
||||
-
|
||||
- [[consumer_config.koji_instances.primary.tags]]
|
||||
from = "f32-infra-candidate"
|
||||
to = "f32-infra-stg"
|
||||
key = "{{ (env == 'production')|ternary('fedora-infra', 'testkey') }}"
|
||||
@@ -170,18 +140,6 @@ handlers = ["console"]
|
||||
# Gated coreos-pool tag
|
||||
|
||||
[[consumer_config.koji_instances.primary.tags]]
|
||||
- from = "f30-coreos-signing-pending"
|
||||
- to = "coreos-pool"
|
||||
- key = "{{ (env == 'production')|ternary('fedora-30', 'testkey') }}"
|
||||
- keyid = "{{ (env == 'production')|ternary('cfc659b9', 'd300e724') }}"
|
||||
-
|
||||
- [[consumer_config.koji_instances.primary.tags]]
|
||||
- from = "f31-coreos-signing-pending"
|
||||
- to = "coreos-pool"
|
||||
- key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}"
|
||||
- keyid = "{{ (env == 'production')|ternary('3c3359c4', 'd300e724') }}"
|
||||
-
|
||||
- [[consumer_config.koji_instances.primary.tags]]
|
||||
from = "f32-coreos-signing-pending"
|
||||
to = "coreos-pool"
|
||||
key = "{{ (env == 'production')|ternary('fedora-32', 'testkey') }}"
|
||||
@@ -297,19 +255,6 @@ handlers = ["console"]
|
||||
keyid = "{{ (env == 'production')|ternary('12c944d0', 'd300e724') }}"
|
||||
type = "modular"
|
||||
|
||||
- [[consumer_config.koji_instances.primary.tags]]
|
||||
- from = "f31-signing-pending"
|
||||
- to = "f31-updates-testing-pending"
|
||||
- key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}"
|
||||
- keyid = "{{ (env == 'production')|ternary('3c3359c4', 'd300e724') }}"
|
||||
-
|
||||
- [[consumer_config.koji_instances.primary.tags]]
|
||||
- from = "f31-modular-signing-pending"
|
||||
- to = "f31-modular-updates-testing-pending"
|
||||
- key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}"
|
||||
- keyid = "{{ (env == 'production')|ternary('3c3359c4', 'd300e724') }}"
|
||||
- type = "modular"
|
||||
-
|
||||
#epel8 modular tags
|
||||
[[consumer_config.koji_instances.primary.tags]]
|
||||
from = "epel8-modular-signing-pending"
|
||||
@@ -417,12 +362,6 @@ handlers = ["console"]
|
||||
key = "{{ (env == 'production')|ternary('fedora-32', 'testkey') }}"
|
||||
keyid = "{{ (env == 'production')|ternary('12c944d0', 'd300e724') }}"
|
||||
|
||||
- [[consumer_config.koji_instances.primary.tags]]
|
||||
- from = "f31-openh264"
|
||||
- to = "f31-openh264"
|
||||
- key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}"
|
||||
- keyid = "{{ (env == 'production')|ternary('3c3359c4', 'd300e724') }}"
|
||||
-
|
||||
# f33-rebuild
|
||||
[[consumer_config.koji_instances.primary.tags]]
|
||||
from = "f33-rebuild"
|
||||
----
|
||||
|
||||
* Run the associated playbooks on _batcave_
|
||||
|
||||
....
|
||||
$ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/bodhi-backend.yml
|
||||
$ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/koji-hub.yml
|
||||
$ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/mbs.yml
|
||||
$ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/releng-compose.yml
|
||||
$ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/proxies -t pkgdb2
|
||||
$ sudo ansible-playbook /srv/web/infra/ansible/playbooks/manual/autosign.yml
|
||||
$ sudo ansible-playbook /srv/web/infra/ansible/playbooks/openshift-apps/bodhi.yml
|
||||
....
|
||||
|
||||
==== MBS Platform Retirement
|
||||
|
||||
* To retire the platform in mbs, run the following command on
|
||||
mbs-backend01.iad2.fedoraproject.org
|
||||
|
||||
....
|
||||
$ sudo mbs-manager retire platform:f31
|
||||
....
|
||||
|
||||
==== Final announcement
|
||||
|
||||
* Send the final announcement to devel@, devel-announce@,
|
||||
test-announce@, announce@ lists
|
||||
|
||||
===== Announcement content
|
||||
|
||||
....
|
||||
Hello all,
|
||||
|
||||
As of the 24th of November 2020, Fedora 31 has reached its end of life
|
||||
for updates and support. No further updates, including security
|
||||
updates, will be available for Fedora 31. All the updates that are
|
||||
currently in testing won't get pushed to stable. Fedora 32 will
|
||||
continue to receive updates until approximately one month after the
|
||||
release of Fedora 34. The maintenance schedule of Fedora releases is
|
||||
documented on the Fedora Project wiki [0]. The Fedora Project wiki
|
||||
also contains instructions [1] on how to upgrade from a previous
|
||||
release of Fedora to a version receiving updates.
|
||||
|
||||
Mohan Boddu.
|
||||
|
||||
[0] https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Maintenance_Schedule
|
||||
[1] https://docs.fedoraproject.org/en-US/quick-docs/dnf-system-upgrade/
|
||||
....
|
||||
|
||||
===== Update eol wiki page
|
||||
|
||||
https://fedoraproject.org/wiki/End_of_life update with release and
|
||||
number of days.
|
||||
|
||||
===== Move the EOL release to archive
|
||||
|
||||
. log into to bodhi-backend01 and become root
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ssh bodhi-backend01.iad2.fedoraproject.org
|
||||
$ sudo su
|
||||
$ su - ftpsync
|
||||
....
|
||||
____
|
||||
. then change into the releases directory.
|
||||
+
|
||||
____
|
||||
....
|
||||
$ cd /pub/fedora/linux/releases
|
||||
....
|
||||
____
|
||||
. check to see that the target directory doesnt already exist.
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ls /pub/archive/fedora/linux/releases/
|
||||
....
|
||||
____
|
||||
. do a recursive rsync to update any changes in the trees since the
|
||||
previous copy.
|
||||
+
|
||||
____
|
||||
....
|
||||
$ rsync -avAXSHP ./35/ /pub/archive/fedora/linux/releases/35/
|
||||
....
|
||||
____
|
||||
. we now do the updates and updates/testing in similar ways.
|
||||
+
|
||||
____
|
||||
....
|
||||
$ cd ../updates/
|
||||
$ rsync -avAXSHP 35/ /pub/archive/fedora/linux/updates/35/
|
||||
$ cd testing
|
||||
$ rsync -avAXSHP 35/ /pub/archive/fedora/linux/updates/testing/35/
|
||||
....
|
||||
____
|
||||
. do the same with fedora-secondary.
|
||||
. announce to the mirror list this has been done and that in 2 weeks you
|
||||
will move the old trees to archives.
|
||||
. in two weeks, log into mm-backend01 and run the archive script
|
||||
+
|
||||
____
|
||||
....
|
||||
$ sudo -u mirrormanager mm2_move-to-archive --originalCategory="Fedora Linux" --archiveCategory="Fedora Archive" --directoryRe='/35/Everything'
|
||||
....
|
||||
____
|
||||
. if there are problems, the postgres DB may have issues and so you need
|
||||
to get a DBA to update the backend to fix items.
|
||||
. wait an hour or so then you can remove the files from the main tree.
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ssh bodhi-backend01
|
||||
$ cd /pub/fedora/linux
|
||||
$ cd releases/35
|
||||
$ ls # make sure you have stuff here
|
||||
$ rm -rf *
|
||||
$ ln ../20/README .
|
||||
$ cd ../../updates/35
|
||||
$ ls #make sure you have stuff here
|
||||
$ rm -rf *
|
||||
$ ln ../20/README .
|
||||
$ cd ../testing/35
|
||||
$ ls # make sure you have stuff here
|
||||
$ rm -rf *
|
||||
$ ln ../20/README .
|
||||
....
|
||||
____
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
* Resource contention in infrastructure, such as outages
|
||||
* Extenuating circumstances for specific planned updates, if any
|
||||
* Send the reminder announcement, if it isn't sent already
|
||||
98
modules/release_guide/pages/sop_eol_change.adoc
Normal file
98
modules/release_guide/pages/sop_eol_change.adoc
Normal file
|
|
@ -0,0 +1,98 @@
|
|||
== Adjust EOLs and SLs on branches
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
This SOP is about adjust EOLs and SLs on so-called "arbitrary" branches.
|
||||
Unretiring a package _also_ involves changing the EOL on a branch, but
|
||||
you won't find information on that here. See the unretirement SOP for
|
||||
more info there.
|
||||
====
|
||||
|
||||
=== Description
|
||||
|
||||
With "arbitrary branching", modules can include streams for an RPM that
|
||||
are not directly associated with a Fedora release. Modules _themselves_
|
||||
can have branches not directly associated with a Fedora release. For
|
||||
instance, our [.title-ref]#python3# module has a [.title-ref]#3.6#
|
||||
branch. The SLs on that module branch all go EOL on 2018-06-01.
|
||||
*@torsava*, one of the
|
||||
https://src.fedoraproject.org/modules/python3k[python3 module
|
||||
maintainers], may request that this branch have its EOL extended until
|
||||
2018-12-01.
|
||||
|
||||
When a maintainer wants to change EOL on a rpm, module, or container
|
||||
branch, they need to file a https://pagure.io/releng/issues[releng
|
||||
ticket] requesting it. They have no way to do it on their own. Releng
|
||||
must review the request, and then process it.
|
||||
|
||||
=== Policy
|
||||
|
||||
Here are some _policy_ guidelines to help you (releng) make decisions
|
||||
about these tickets
|
||||
|
||||
* Clarify. Does the maintainer want the EOL lengthed for an rpm? Or for
|
||||
a module? Or for a container? If they just say "increase the EOL for
|
||||
[.title-ref]#httpd#, please", it is not clear which thing they're really
|
||||
talking about.
|
||||
* Expect that maintainers generally know _when_ their EOL should go
|
||||
until. You don't need to go and research upstream mailing lists to
|
||||
figure out what makes sense. Politely asking the maintainer for some
|
||||
background information on _why_ the EOL should be changed is good to
|
||||
record in the ticket for posterity. Bonus points if they can provide
|
||||
references to upstream discussions that make the request make sense.
|
||||
* EOLs should _almost always_ only be extended into the future.
|
||||
Shortening an EOL should only be done with care. There might be modules
|
||||
out there that depend on an rpm branch with a conflicting EOL of their
|
||||
own. If a _shorter_ EOL is requested for an rpm branch, you should
|
||||
verify that no modules that depend on it have a conflicting EOL. If a
|
||||
_shorter_ EOL is requested for a module branch, you should verify that
|
||||
no other modules require it that have a conflicting EOL.
|
||||
* EOLs should not be arbitrary dates. At Flock 2017, we
|
||||
https://pagure.io/fedrepo_req/issue/100[decided on using two standard
|
||||
dates] for EOLs to help make things less crazy. You should use December
|
||||
1st or June 1st of any given year for all EOL dates.
|
||||
* Many branches will _often_ have multiple SLs all with the _same_ EOL.
|
||||
I.e., the branch is fully supported until such time as it is totally
|
||||
retired. There is no gray area. However, it is _possible_ to have
|
||||
branches with piecemeal SLs and EOLs. A branch may support
|
||||
[.title-ref]#bug_fixes# until time X but will further support
|
||||
[.title-ref]#security_fixes# until time Y. This is nicely flexible for
|
||||
the maintainers, but also introduces complexity. If a maintainer
|
||||
requests piecemeal SL EOLs, ask to make sure they really want this kind
|
||||
of complexity.
|
||||
|
||||
=== Action
|
||||
|
||||
We have a script in the releng repo:
|
||||
|
||||
....
|
||||
$ PYTHONPATH=scripts/pdc python3 scripts/pdc/adjust-eol.py -h
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Run it with [.title-ref]#python3#. It imports [.title-ref]#fedrepo_req#
|
||||
which is python3 by default. Installing [.title-ref]#python2#
|
||||
dependencies should be possible when needed.
|
||||
====
|
||||
|
||||
Here is an example of using it to increase the SL of the
|
||||
[.title-ref]#3.6# branch of the [.title-ref]#python3# module (not the
|
||||
rpm branch):
|
||||
|
||||
....
|
||||
$ PYTHONPATH=scripts/pdc python3 scripts/pdc/adjust-eol.py \
|
||||
fedora \
|
||||
SOME_TOKEN \
|
||||
python3 \
|
||||
module \
|
||||
3.6 \
|
||||
2018-12-01
|
||||
Connecting to PDC args.server 'fedora' with token 'a9a1e4cbca122c21580d1fe4646e603a770c5280'
|
||||
Found 2 existing SLs in PDC.
|
||||
Adjust eol of (module)python3#3.6 security_fixes:2018-06-01 to 2018-12-01? [y/N]: y
|
||||
Adjust eol of (module)python3#3.6 bug_fixes:2018-06-01 to 2018-12-01? [y/N]: y
|
||||
Set eol to 2018-12-01 on ['(module)python3#3.6 security_fixes:2018-06-01', '(module)python3#3.6 bug_fixes:2018-06-01']
|
||||
....
|
||||
156
modules/release_guide/pages/sop_fedora_media_writer.adoc
Normal file
156
modules/release_guide/pages/sop_fedora_media_writer.adoc
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
== Fedora Media Writer Building and Signing
|
||||
|
||||
=== Description
|
||||
|
||||
Whenever a new version of Fedora Media Writer is available, it is
|
||||
required to build and code sign it.
|
||||
|
||||
=== Action
|
||||
|
||||
==== Windows
|
||||
|
||||
{empty}#. Get a windows code signing key from private ansible
|
||||
repository. It is in the ansible-private/files/code-signing/ directory
|
||||
|
||||
. Convert the .cert formatted certificate to .pxf format:
|
||||
+
|
||||
....
|
||||
$ openssl pkcs12 -export -in code-signing.crt -inkey code-signing.key -out authenticode.pfx
|
||||
....
|
||||
. Clone the Fedora Media Writer git repo:
|
||||
+
|
||||
....
|
||||
$ git clone https://github.com/MartinBriza/MediaWriter.git
|
||||
....
|
||||
. Checkout the release tag for which the executable to be created:
|
||||
+
|
||||
....
|
||||
$ git checkout tags/<release_number>
|
||||
....
|
||||
. The script to build and sign the executable is available at
|
||||
dist/win/build.sh
|
||||
. Get the mingw-mediawriter from koji. Make sure the version is the one
|
||||
that needs building and signing.
|
||||
. Install the remaining packages that are listed under PACKAGES variable
|
||||
in build.sh script.
|
||||
. Export CERTPATH to the location where the .pfx file is located and
|
||||
make sure its named as authenticode.pfx and export CERTPASS to the file
|
||||
that contains the password used in creating .pvk file.
|
||||
. Run the build.sh script:
|
||||
+
|
||||
....
|
||||
$ ./build.sh
|
||||
....
|
||||
|
||||
=== Verification
|
||||
|
||||
The FedoraMediaWriter-win32-<release_number>.exe is located under
|
||||
dist/win/ directory.
|
||||
|
||||
==== OS X:
|
||||
|
||||
==== Build:
|
||||
|
||||
. install xcode 8.1 from apple store.
|
||||
. {blank}
|
||||
+
|
||||
install qt for mac from:::
|
||||
http://download.qt.io/official_releases/qt/5.7/5.7.0/qt-opensource-mac-x64-clang-5.7.0.dmg
|
||||
. Open a terminal and run the following commands
|
||||
+
|
||||
::::
|
||||
$ git clone https://github.com/MartinBriza/MediaWriter $ cd
|
||||
MediaWriter $ mkdir build $ cd build $
|
||||
$QT_PREFIX/$QT_VERSION/clang64/bin/qmake .. $ make $ cp
|
||||
build/helper/mac/helper.app/Contents/MacOS/helper
|
||||
build/app/FedoraMediaWriter.app/Contents/MacOS/ $ cd build/app $
|
||||
$QT_PREFIX/$QT_VERSION/clang_64/bin/macdeployqt "Fedora Media
|
||||
Writer.app" -executable="Fedora Media
|
||||
Writer.app/Contents/MacOS/helper" -qmldir="../../app"
|
||||
|
||||
==== Prepare certificates
|
||||
|
||||
This only needs to happen once per build machine, and prepares the
|
||||
certificates by requesting them from Apple.
|
||||
|
||||
. Install Xcode from the Mac App store
|
||||
. Start Xcode
|
||||
. Press Command-, (or in the menu bar click Xcode -> Preferences)
|
||||
. Go to Accounts tab
|
||||
. Click the plus button and sign in
|
||||
. Select the new account
|
||||
. Select the correct team
|
||||
. Click View Details
|
||||
. Under "Signing Identities", find "Developer ID Application"
|
||||
. Click Create
|
||||
. Wait until the button disappears
|
||||
. Done
|
||||
|
||||
==== Sign and DMG
|
||||
|
||||
. Open a terminal
|
||||
. cd to the root directory of the FMW project
|
||||
. Run the following bash script:
|
||||
+
|
||||
____
|
||||
....
|
||||
#/bin/bash
|
||||
|
||||
security -v unlock-keychain login.keychain
|
||||
|
||||
# First sign all dynamic libraries (dylib's)
|
||||
cd "build/app/Fedora Media Writer.app"
|
||||
for dylib in $(find . -name "*dylib")
|
||||
do
|
||||
codesign -s "Developer ID Application: Fedora Gilmore" -v $dylib
|
||||
done
|
||||
# Now sign framework bundles
|
||||
for framework in $(find . -name "*framework")
|
||||
do
|
||||
codesign -s "Developer ID Application: Fedora Gilmore" -v $framework
|
||||
done
|
||||
|
||||
# Sign the two binaries
|
||||
codesign -s "Developer ID Application: Fedora Gilmore" -v Contents/MacOS/helper
|
||||
codesign -s "Developer ID Application: Fedora Gilmore" -v "Contents/MacOS/Fedora Media Writer"
|
||||
|
||||
# Sign the app bundle
|
||||
codesign -s "Developer ID Application: Fedora Gilmore" -v .
|
||||
|
||||
# Create the dmg
|
||||
cd ..
|
||||
rm -f FedoraMediaWriter-osx-*.dmg
|
||||
|
||||
hdiutil create -srcfolder "Fedora Media Writer.app" -format UDCO -imagekey zlib-level=9 -scrub \
|
||||
-volname FedoraMediaWriter-osx FedoraMediaWriter-osx-$(git describe --tags).dmg
|
||||
....
|
||||
____
|
||||
|
||||
==== Account Email(OS X)
|
||||
|
||||
____
|
||||
::::
|
||||
releng@fedoraproject.org
|
||||
____
|
||||
|
||||
==== Account Holders(OS X)
|
||||
|
||||
. Primary: Dennis Gilmore(ausil)
|
||||
. Backup: Kevin Fenzi(kevin)
|
||||
. Manager/bill-payer: Paul Frields(pfrields)
|
||||
|
||||
=== Sync binaries to the web
|
||||
|
||||
copy both files to /srv/web/fmw on sundries01 create symlinks to the
|
||||
FedoraMediaWriter-win32-latest.exe and FedoraMediaWriter-osx-latest.dmg
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
Nothing yet.
|
||||
|
||||
=== Issue with signing
|
||||
|
||||
If the build is done but it is not signed then try editing the
|
||||
`build.sh` and add -askpass argument for all the osslsigncode commands
|
||||
and run the script, when it asks for the password you can enter the
|
||||
password that was used in creating .pvk file.
|
||||
76
modules/release_guide/pages/sop_file_ftbfs.adoc
Normal file
76
modules/release_guide/pages/sop_file_ftbfs.adoc
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
== File FTBFS
|
||||
|
||||
=== Description
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
FTBFS = "Fails To Build From Source"
|
||||
====
|
||||
|
||||
After every mass rebuild, we file FTBFS bugs for the packages that
|
||||
failed to build during mass rebuild.
|
||||
|
||||
This should be run after the
|
||||
https://docs.pagure.org/releng/sop_mass_rebuild_packages.html#post-mass-rebuild-tasks[mass
|
||||
rebuild builds are merged into main tag].
|
||||
|
||||
=== Action
|
||||
|
||||
The FTBFS bugs are filed in bugzilla.
|
||||
|
||||
. {blank}
|
||||
+
|
||||
Create a bugzilla bug for FTBFS::
|
||||
* use the https://bugzilla.redhat.com/show_bug.cgi?id=1750908[previous
|
||||
FTBFS bugzilla bug example] if its not created
|
||||
. {blank}
|
||||
+
|
||||
Set alias for RAWHIDEFTBFS::
|
||||
* remove RAWHIDEFTBFS alias from the previous FTBFS bugzilla
|
||||
* set RAWHIDEFTBFS alias on the new rawhide version FTBFS bugzilla
|
||||
* set the alias on RAWHIDEFailsToInstall bugzilla in same fashion
|
||||
. {blank}
|
||||
+
|
||||
Install [.title-ref]#python-bugzilla-cli# on your local machine if its
|
||||
not installed::
|
||||
....
|
||||
$ sudo dnf install python-bugzilla-cli
|
||||
....
|
||||
. {blank}
|
||||
+
|
||||
Update the [.title-ref]#massrebuildsinfo.py#::
|
||||
* epoch
|
||||
* buildtag
|
||||
* destag
|
||||
* tracking_bug
|
||||
+
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Most of these values are already updated during mass rebuild, only one
|
||||
that might need updating is [.title-ref]#tracking_bug#
|
||||
====
|
||||
. {blank}
|
||||
+
|
||||
Update the [.title-ref]#mass_rebuild_file_bugs.py#::
|
||||
* rebuildid
|
||||
. {blank}
|
||||
+
|
||||
Login into bugzilla in the terminal using [.title-ref]#bugzilla login#
|
||||
command::
|
||||
....
|
||||
$ bugzilla login
|
||||
....
|
||||
+
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Login as [.title-ref]#releng@fedoraproject.org#
|
||||
====
|
||||
. {blank}
|
||||
+
|
||||
Run [.title-ref]#mass_rebuild_file_bugs.py# locally::
|
||||
....
|
||||
$ python mass_rebuild_file_bugs.py
|
||||
....
|
||||
28
modules/release_guide/pages/sop_find_module_info.adoc
Normal file
28
modules/release_guide/pages/sop_find_module_info.adoc
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
== Finding Module Information
|
||||
|
||||
=== Description
|
||||
|
||||
When users submit builds to the Module Build Service (MBS), it in turn
|
||||
submits builds to Koji. Sometimes, you are looking at a koji build, and
|
||||
you want to know what module-build it is a part of.
|
||||
|
||||
=== Caveat
|
||||
|
||||
It requires that the build has been completed and has been tagged, until
|
||||
https://pagure.io/fm-orchestrator/issue/375 is complete.
|
||||
|
||||
=== Setup
|
||||
|
||||
Run the following:
|
||||
|
||||
....
|
||||
$ sudo dnf install python-arrow python-requests koji
|
||||
....
|
||||
|
||||
=== Action
|
||||
|
||||
Run the following:
|
||||
|
||||
....
|
||||
$ scripts/mbs/koji-module-info.py $BUILD_ID
|
||||
....
|
||||
|
|
@ -0,0 +1,185 @@
|
|||
== Generating Openh264 Composes
|
||||
|
||||
=== Description
|
||||
|
||||
Openh264 repos are a special case and we need to generate the composes
|
||||
for it in a different way. We use ODCS to generate the private compose
|
||||
and send the rpms to Cisco to publish them on their CDN. We publish the
|
||||
repodata on our side.
|
||||
|
||||
[WARNING]
|
||||
.Warning
|
||||
====
|
||||
We do not have all the appropriate legal rights to distribute these
|
||||
packages, so we need to be extra carefull to make sure they are never
|
||||
distributed via our build system or websites
|
||||
====
|
||||
|
||||
=== Action
|
||||
|
||||
==== Permissions needed
|
||||
|
||||
You will need some ODCS permissions in order to request private composes
|
||||
and composes from tags. You can set this in infra/ansible in
|
||||
inventory/group_vars/odcs in the odcs_allowed_clients_users variable.
|
||||
See other releng users entries for format.
|
||||
|
||||
==== Get the odcs token
|
||||
|
||||
In order to generate an odcs compose, you need a openidc token.
|
||||
|
||||
Run the odcs-token.py under `scripts/odcs/` from pagure releng
|
||||
repository to generate the token.
|
||||
|
||||
....
|
||||
$ ./odcs-token.py
|
||||
....
|
||||
|
||||
==== Make sure rpms are written out with the right signature
|
||||
|
||||
....
|
||||
$ koji write-signed-rpm eb10b464 openh264-2.2.0-1.fc38
|
||||
....
|
||||
|
||||
Where the key for that branch is listed, then the open264 package and
|
||||
version.
|
||||
|
||||
==== Generate a private odcs compose
|
||||
|
||||
With the token generated above, generate the odcs private compose
|
||||
|
||||
....
|
||||
$ python odcs-private-compose.py <token> <koji_tag> <signingkeyid>
|
||||
....
|
||||
|
||||
`koji_tag`: fxx-openh264 (Openh264 builds are tagged to fxx-openh264
|
||||
tags where [.title-ref]#xx# represents the fedora release)
|
||||
|
||||
`signingkeyid`: The short hash of the key for this Fedora branch.
|
||||
|
||||
The composes are stored under `/srv/odcs/private/` dir on
|
||||
`odcs-backend-releng01.iad2.fedoraproject.org`
|
||||
|
||||
==== Pull the compose to your local machine
|
||||
|
||||
We need to extract the rpms and tar them to send them to Cisco. In order
|
||||
to that, first of all we need to pull the compose to our local machine.
|
||||
|
||||
===== Move the compose to your home dir on odcs-backend-releng01.iad2.fedoraproject.org
|
||||
|
||||
Since the compose is owned by [.title-ref]#odcs-server# pull it into
|
||||
your home dir
|
||||
|
||||
....
|
||||
$ mkdir ~/32-openh264
|
||||
$ sudo rsync -avhHP /srv/odcs/private/odcs-3835/ ~/32-openh264/
|
||||
$ sudo chown -R mohanboddu:mohanboddu ~/32-openh264/
|
||||
....
|
||||
|
||||
===== Sync the compose to your local machine
|
||||
|
||||
Pull in the compose from your home dir on odcs releng backend to your
|
||||
local machine into a temp working dir
|
||||
|
||||
....
|
||||
$ mkdir openh264-20200813
|
||||
$ scp -rv odcs-backend-releng01.iad2.fedoraproject.org:/home/fedora/mohanboddu/32-openh264/ openh264-20200813/
|
||||
....
|
||||
|
||||
===== Make the changes needed
|
||||
|
||||
Please follow the following commands to make the necessary tar files to
|
||||
send to Cisco
|
||||
|
||||
....
|
||||
$ cd openh264-20200813
|
||||
$ mkdir 32-rpms
|
||||
# Copy rpms including devel rpms
|
||||
$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*rpm 32-rpms/
|
||||
# Copy debuginfo rpms
|
||||
$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*/*rpm 32-rpms/
|
||||
# copy the src.rpm
|
||||
$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*src.rpm 32-rpms/
|
||||
$ cd 32-rpms
|
||||
# Create the tar file with the rpms
|
||||
$ tar -cJvf ../fedora-32-openh264-rpms.tar.xz *rpm
|
||||
....
|
||||
|
||||
We need to send this tar file to Cisco along with the list of rpms in
|
||||
each tarball.
|
||||
|
||||
===== Syncing the compose to sundries01
|
||||
|
||||
Once we get a confirmation from Cisco that the rpms are updated on their
|
||||
CDN, verify them by using curl. For example:
|
||||
|
||||
....
|
||||
$ curl -I http://ciscobinary.openh264.org/openh264-2.1.1-1.fc32.x86_64.rpm
|
||||
....
|
||||
|
||||
Now push these composes to *sundries01.iad2.fedoraproject.org* and
|
||||
*mm-backend01.iad2.fedoraproject.org*
|
||||
|
||||
On sundries01 we need to sync to a directory that is owned by _apache_,
|
||||
so first we sync to the home directory on sundries01. Same with
|
||||
mm-backend01 as the directory is owned by _root_.
|
||||
|
||||
Create a temp working directory on sundries01
|
||||
|
||||
....
|
||||
$ ssh sundries01.iad2.fedoraproject.org
|
||||
$ mkdir openh264-20200825
|
||||
....
|
||||
|
||||
Create a temp working directory on mm-backend01
|
||||
|
||||
....
|
||||
$ ssh mm-backend01.iad2.fedoraproject.org
|
||||
$ mkdir openh264-20200825
|
||||
....
|
||||
|
||||
Then from your local machine, sync the compose
|
||||
|
||||
....
|
||||
$ cd openh264-20200825
|
||||
$ rsync -avhHP 32-openh264 sundries01.iad2.fedoraproject.org:/home/fedora/mohanboddu/openh264-20200825
|
||||
$ rsync -avhHP 32-openh264 mm-backend01.iad2.fedoraproject.org:/home/fedora/mohanboddu/openh264-20200825
|
||||
....
|
||||
|
||||
On sundries01
|
||||
|
||||
....
|
||||
$ cd openh264-20200825
|
||||
$ sudo rsync -avhHP 32-openh264/compose/Temporary/ /srv/web/codecs.fedoraproject.org/openh264/32/
|
||||
....
|
||||
|
||||
On mm-backend01
|
||||
|
||||
....
|
||||
$ cd openh264-20200825
|
||||
$ sudo rsync -avhHP 32-openh264/compose/Temporary/ /srv/codecs.fedoraproject.org/openh264/32/
|
||||
....
|
||||
|
||||
===== Extra info
|
||||
|
||||
Normally that should be it, but in some cases you may want to push
|
||||
things out faster than normal, and here's a few things you can do to do
|
||||
that:
|
||||
|
||||
On mm-backend01.iad2.fedoraproject.org you can run:
|
||||
|
||||
....
|
||||
# sudo -u mirrormanager /usr/local/bin/umdl-required codecs /var/log/mirrormanager/umdl-required.log
|
||||
....
|
||||
|
||||
This will have mirrormanager scan the codecs dir and update it if it's
|
||||
changed.
|
||||
|
||||
On batcave01.iad2.fedoraproject.org you can use ansible to force all the
|
||||
proxies to sync the codec content from sundries01:
|
||||
|
||||
....
|
||||
# nsible -a '/usr/bin/rsync --delete -a --no-owner --no-group sundries01::codecs.fedoraproject.org/ /srv/web/codecs.fedoraproject.org/' proxies
|
||||
....
|
||||
|
||||
Mirrorlist servers should update every 15min.
|
||||
520
modules/release_guide/pages/sop_mass_branching.adoc
Normal file
520
modules/release_guide/pages/sop_mass_branching.adoc
Normal file
|
|
@ -0,0 +1,520 @@
|
|||
== Mass Branching
|
||||
|
||||
=== Description
|
||||
|
||||
At each alpha freeze we branch the pending release away from `devel/`
|
||||
which allows rawhide to move on while the pending release goes into
|
||||
bugfix and polish mode.
|
||||
|
||||
=== Action
|
||||
|
||||
==== Update fedscm-admin
|
||||
|
||||
Add the new release to fedscm-admin and create an update and send it to
|
||||
limb.
|
||||
|
||||
Please take a look at the
|
||||
https://pagure.io/fedscm-admin/c/7862d58b5982803dbe4c47e0262c6ce78bc903db?branch=main[fedscm-admin
|
||||
commit].
|
||||
|
||||
==== Repos to branch
|
||||
|
||||
All the following listed repos needs updating, including adding a new
|
||||
branch for branched release and updating rawhide branch with new release
|
||||
values.
|
||||
|
||||
[arabic]
|
||||
. https://pagure.io/pungi-fedora
|
||||
. https://pagure.io/fedora-kickstarts
|
||||
. https://pagure.io/fedora-comps
|
||||
. https://pagure.io/fedora-lorax-templates/
|
||||
. https://pagure.io/releng/fedora-module-defaults/
|
||||
. https://pagure.io/workstation-ostree-config/
|
||||
. https://src.fedoraproject.org/rpms/fedora-release
|
||||
. https://src.fedoraproject.org/rpms/fedora-repos
|
||||
|
||||
==== PDC
|
||||
|
||||
The "product-release" needs to be created in PDC.
|
||||
|
||||
In the `scripts/pdc/` directory, run:
|
||||
|
||||
....
|
||||
$ python create-product-release.py fedora $TOKEN Fedora $NEW_VERSION
|
||||
....
|
||||
|
||||
On `pdc-backend01.stg` (for testing) or `pdc-backend01` (for production)
|
||||
clone (or update an existing one) the releng repo:
|
||||
|
||||
....
|
||||
git clone https://pagure.io/releng.git
|
||||
....
|
||||
|
||||
In the `scripts/pdc/` directory, run (see the `--help` of the script for
|
||||
information on how to run it):
|
||||
|
||||
....
|
||||
$ python create-new-release-branches.py ... --createfile
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
the `--createfile` argument is necessary, that file is needed for the
|
||||
next step.
|
||||
====
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Due to memory leak issue in pdc, we need to set the config in
|
||||
/etc/pdc.d/fedora.json \{ "fedora": \{ "host":
|
||||
"http://pdc-web02.iad2.fedoraproject.org/rest_api/v1/", "develop":
|
||||
false, "ssl-verify": false, } }
|
||||
====
|
||||
|
||||
==== dist-git
|
||||
|
||||
Now that pdc has the new release and each package has been branched in
|
||||
pdc we need to update dist-git in two steps:
|
||||
|
||||
* Create the new branch in git
|
||||
* Update the gitolite.conf to allow user to push to this new branch
|
||||
|
||||
For both of these actions you will need the file generated by pdc above.
|
||||
|
||||
===== Create the git branches
|
||||
|
||||
On `pkgs01.stg` (for testing) or `pkgs02` (for production), run:
|
||||
|
||||
....
|
||||
$ sudo -u pagure python /usr/local/bin/mass-branching-git.py <new branch name> <input file>
|
||||
....
|
||||
|
||||
Where `<new branch name>` will be something like `f28` and the
|
||||
`<input file>` the path to the file generated by pdc above.
|
||||
|
||||
==== Ansible
|
||||
|
||||
Apps in ansible need to be updated to be aware of a new branch.
|
||||
|
||||
===== Bodhi
|
||||
|
||||
Bodhi needs to be updated to add new release. This needs to be done in
|
||||
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2[bodhi2
|
||||
role] in infra ansible repo. This change includes, updating
|
||||
koji-sync-listener.py, new-updates-sync, pungi configs for both rpm and
|
||||
modular updates, bodhi templates.
|
||||
|
||||
|
||||
===== Enable Branched Compose
|
||||
|
||||
We need to enable the branched compose. This is done in
|
||||
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/releng[releng
|
||||
role] of infra ansbile repo
|
||||
|
||||
|
||||
===== Fedora Branched
|
||||
|
||||
Set FedoraBranched variable to True in infra ansible repo
|
||||
|
||||
|
||||
===== Koji hub
|
||||
|
||||
|
||||
===== Robosignatory
|
||||
|
||||
Robosignatory has two parts:
|
||||
|
||||
[arabic]
|
||||
. Disable branched signing, so that we can freeze branched until we get
|
||||
a compose
|
||||
. Adding new release
|
||||
|
||||
Both can be in
|
||||
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/robosignatory[robosignatory
|
||||
role] in infra ansible repo
|
||||
|
||||
|
||||
===== Push the changes
|
||||
|
||||
When done editing the files, commit, push and apply them via the
|
||||
corresponding ansible playbook:
|
||||
|
||||
....
|
||||
sudo rbac-playbook groups/koji-hub.yml
|
||||
sudo rbac-playbook groups/releng-compose.yml
|
||||
sudo rbac-playbook groups/bodhi-backend.yml
|
||||
sudo rbac-playbook openshift-apps/greenwave.yml
|
||||
sudo -i ansible-playbook /srv/web/infra/ansible/playbooks/groups/proxies.yml -t pkgdb2
|
||||
sudo rbac-playbook groups/mbs.yml -t mbs
|
||||
....
|
||||
|
||||
Ask someone in fedora infra to run the robosignatory playbook.
|
||||
|
||||
==== Koji
|
||||
|
||||
The koji build system needs to have some tag/target work done to handle
|
||||
builds from the new branch and to update where builds from rawhide go.
|
||||
|
||||
Run
|
||||
https://pagure.io/releng/blob/main/f/scripts/branching/make-koji-release-tags[make-koji-release-tags]
|
||||
script in https://pagure.io/releng[pagure releng] repo
|
||||
|
||||
==== Fedora Release
|
||||
|
||||
The `fedora-release` package needs to be updated in Rawhide and
|
||||
Branched.
|
||||
|
||||
Changes to `fedora-release.spec` in the *rawhide* branch:
|
||||
|
||||
[arabic]
|
||||
. Increment `%define dist_version`:
|
||||
+
|
||||
....
|
||||
-%define dist_version 35
|
||||
+%define dist_version 36
|
||||
....
|
||||
. Increment `Version:` and reset `Release:`:
|
||||
+
|
||||
....
|
||||
-Version: 35
|
||||
-Release: 0.3%{?eln:.eln%{eln}}
|
||||
+Version: 36
|
||||
+Release: 0.1%{?eln:.eln%{eln}}
|
||||
....
|
||||
. Add a `%changelog` entry:
|
||||
+
|
||||
....
|
||||
%changelog
|
||||
+* Tue Feb 23 2021 Mohan Boddu <mboddu@bhujji.com> - 36-0.1
|
||||
+- Setup for rawhide being F36
|
||||
....
|
||||
|
||||
Changes to `fedora-release.spec` in the *branched* branch:
|
||||
|
||||
[arabic]
|
||||
. Adjust `release_name` and unset `is_rawhide`:
|
||||
+
|
||||
....
|
||||
-%define release_name Rawhide
|
||||
-%define is_rawhide 1
|
||||
+%define release_name Thirty Five
|
||||
+%define is_rawhide 0
|
||||
....
|
||||
. Verify the correct number for `dist_version` and `Version:`:
|
||||
+
|
||||
....
|
||||
%define dist_version 35
|
||||
Version: 35
|
||||
....
|
||||
. Bump `Release:`:
|
||||
+
|
||||
....
|
||||
-Release: 0.3%{?eln:.eln%{eln}}
|
||||
+Release: 0.4%{?eln:.eln%{eln}}
|
||||
....
|
||||
. Add a `%changelog` entry:
|
||||
+
|
||||
....
|
||||
%changelog
|
||||
+* Tue Feb 23 2021 Mohan Boddu <mboddu@bhujji.com> - 35-0.4
|
||||
+- Branching F35 from rawhide
|
||||
....
|
||||
|
||||
==== Fedora Repos
|
||||
|
||||
The `fedora-repos` package needs to be updated in Rawhide, Branched, and
|
||||
also in all stable release branches (in order to receive new GPG keys
|
||||
and updated symlinks).
|
||||
|
||||
Changes to the *rawhide* branch (mostly in `fedora-repos.spec`):
|
||||
|
||||
[arabic]
|
||||
. Generate and add a _Rawhide+1_ GPG key file, then add it to the spec
|
||||
file:
|
||||
+
|
||||
....
|
||||
Source57: RPM-GPG-KEY-fedora-37-primary
|
||||
....
|
||||
. Update the `archmap` file and define architectures for _Rawhide+1_:
|
||||
+
|
||||
....
|
||||
+fedora-37-primary: x86_64 armhfp aarch64 ppc64le s390x
|
||||
....
|
||||
. Increment `%global rawhide_release`:
|
||||
+
|
||||
....
|
||||
-%global rawhide_release 35
|
||||
+%global rawhide_release 36
|
||||
....
|
||||
. Bump `Version:` and reset `Release:`:
|
||||
+
|
||||
....
|
||||
-Version: 35
|
||||
-Release: 0.2%{?eln:.eln%{eln}}
|
||||
+Version: 36
|
||||
+Release: 0.1%{?eln:.eln%{eln}}
|
||||
....
|
||||
. Add a `%changelog` entry:
|
||||
+
|
||||
....
|
||||
%changelog
|
||||
+* Tue Feb 23 2021 Tomas Hrcka <thrcka@redhat.com> - 36-0.1
|
||||
+- Setup for rawhide being F36
|
||||
....
|
||||
|
||||
Changes to the *branched* branch (mostly in `fedora-repos.spec`):
|
||||
|
||||
[arabic]
|
||||
. Copy the _Rawhide+1_ GPG key file from the _rawhide_ branch, then add
|
||||
it to the spec file:
|
||||
+
|
||||
....
|
||||
Source57: RPM-GPG-KEY-fedora-37-primary
|
||||
....
|
||||
. Copy the `archmap` file from the _rawhide_ branch.
|
||||
. Update `%global rawhide_release`:
|
||||
+
|
||||
....
|
||||
-%global rawhide_release 35
|
||||
+%global rawhide_release 36
|
||||
....
|
||||
. Enable `updates_testing_enabled`:
|
||||
+
|
||||
....
|
||||
-%global updates_testing_enabled 0
|
||||
+%global updates_testing_enabled 1
|
||||
....
|
||||
. Bump `Release:`:
|
||||
+
|
||||
....
|
||||
-Release: 0.2%{?eln:.eln%{eln}}
|
||||
+Release: 0.3%{?eln:.eln%{eln}}
|
||||
....
|
||||
. Add a `%changelog` entry:
|
||||
+
|
||||
....
|
||||
%changelog
|
||||
+* Tue Feb 23 2021 Tomas Hrcka <thrcka@redhat.com> - 35-0.3
|
||||
+- Update Rawhide definition, enable updates-testing for Branched
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Build `fedora-release` and `fedora-repos` packages for Branched release
|
||||
*before enabling the Rawhide gating*.
|
||||
====
|
||||
|
||||
Changes to the *stable* branches (mostly in `fedora-repos.spec`):
|
||||
|
||||
[arabic]
|
||||
. Copy the _Rawhide+1_ GPG key file from the _rawhide_ branch, then add
|
||||
it to the spec file:
|
||||
+
|
||||
....
|
||||
Source57: RPM-GPG-KEY-fedora-37-primary
|
||||
....
|
||||
. Copy the `archmap` file from the _rawhide_ branch.
|
||||
. Update `%global rawhide_release`:
|
||||
+
|
||||
....
|
||||
-%global rawhide_release 35
|
||||
+%global rawhide_release 36
|
||||
....
|
||||
. Bump `Release:`:
|
||||
+
|
||||
....
|
||||
-Release: 0.2%{?eln:.eln%{eln}}
|
||||
+Release: 0.3%{?eln:.eln%{eln}}
|
||||
....
|
||||
. Add a `%changelog` entry:
|
||||
+
|
||||
....
|
||||
%changelog
|
||||
+* Tue Feb 23 2021 Tomas Hrcka <thrcka@redhat.com> - 34-0.3
|
||||
+- Update Rawhide definition
|
||||
....
|
||||
|
||||
==== Bodhi
|
||||
|
||||
===== Linking Empty Repos
|
||||
|
||||
We need to link empty repos so that new-updates-sync wont complain about
|
||||
missing repos. The following commands should be run on
|
||||
*bodhi-backend01.phx2.fedoraproject.org*
|
||||
|
||||
....
|
||||
$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f32-updates
|
||||
$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f32-updates-testing
|
||||
$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f32-modular-updates
|
||||
$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f32-modular-updates-testing
|
||||
....
|
||||
|
||||
===== Creating Empty Repos
|
||||
|
||||
To create empty repos on the master mirror, run
|
||||
https://pagure.io/releng/blob/main/f/scripts/branching/create_empty_repos.sh[create_emtpy_repos.sh]
|
||||
from https://pagure.io/releng[pagure releng] repo. This should be run on
|
||||
*bodhi-backend01.phx2.fedoraproject.org*
|
||||
|
||||
....
|
||||
$ sudo -u ftpsync sh scripts/branching/create_empty_repos.sh 31
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Please verify the repo permissions that are created under
|
||||
/pub/fedora/linux/development/<fedora_release_number> and
|
||||
/pub/fedora-secondary/development/<fedora_release_number>. They should
|
||||
be owned by _ftpsync:ftpsync_
|
||||
====
|
||||
|
||||
===== Creating rawhide release
|
||||
|
||||
To create a rawhide release in bodhi, you need to run
|
||||
|
||||
....
|
||||
$ bodhi releases create --name "F36" --long-name "Fedora 36" --id-prefix FEDORA --version 36 --branch f36 --dist-tag f36 --stable-tag f36 --testing-tag f36-updates-testing --candidate-tag f36-updates-candidate --pending-stable-tag f36-updates-pending --pending-testing-tag f36-updates-testing-pending --pending-signing-tag f36-signing-pending --state pending --override-tag f36-override --create-automatic-updates --not-composed-by-bodhi
|
||||
....
|
||||
|
||||
To create a container release for rawhide in bodhi, you need to run
|
||||
|
||||
....
|
||||
$ bodhi releases create --name "F36C" --long-name "Fedora 36 Containers" --id-prefix FEDORA-CONTAINER --version 36 --branch f36 --dist-tag f36-container --stable-tag f36-container-updates --testing-tag f36-container-updates-testing --candidate-tag f36-container-updates-candidate --pending-stable-tag f36-container-updates-pending --pending-testing-tag f36-container-updates-testing-pending --state pending --override-tag f36-container-override
|
||||
....
|
||||
|
||||
To create a flatpak release for branched in bodhi, you need to run
|
||||
|
||||
....
|
||||
$ bodhi releases create --name "F35F" --long-name "Fedora 35 Flatpaks" --id-prefix FEDORA-FLATPAK --version 35 --branch f35 --dist-tag f35-flatpak --stable-tag f35-flatpak-updates --testing-tag f35-flatpak-updates-testing --candidate-tag f35-flatpak-updates-candidate --pending-stable-tag f35-flatpak-updates-pending --pending-testing-tag f35-flatpak-updates-testing-pending --state pending --override-tag f35-flatpak-override
|
||||
....
|
||||
|
||||
You need to run the `bodhi openshift` playbook, so that UI will know
|
||||
about the new release. Then, you need to restart
|
||||
*fm-consumer@config.service* and *bodhi-celery.service* services on
|
||||
*bodhi-backend01.phx2.fedoraproject.org*
|
||||
|
||||
....
|
||||
$ sudo rbac-playbook openshift-apps/bodhi.yml
|
||||
$ sudo systemctl restart fm-consumer@config.service bodhi-celery.service
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Build fedora-release, fedora-repos package for *rawhide after enabling
|
||||
the rawhide gating*
|
||||
====
|
||||
|
||||
===== Update rawhide koji repo
|
||||
|
||||
We need to point the _rawhide_ buildroot repo to the newly created
|
||||
rawhide buildroot. This way kojira doesn't make a newrepo for _rawhide_
|
||||
target as often as fxx-build (new rawhide buildroot).
|
||||
|
||||
Run the following command from any of the compose boxes
|
||||
|
||||
....
|
||||
$ cd /mnt/koji/repos/rawhide; rm -f latest; ln -s ../f34-build/latest
|
||||
./latest
|
||||
....
|
||||
|
||||
===== Update block_retired.py script
|
||||
|
||||
https://pagure.io/releng/blob/main/f/scripts/block_retired.py[block_retired.py]
|
||||
script in releng repo should be updated with rawhide release and also
|
||||
branched release should be added to the script.
|
||||
|
||||
Please look at this
|
||||
https://pagure.io/releng/c/9eb97f491f7a767ab8b90498adfa3b34ee235247?branch=main[block_retired.py
|
||||
commit] as an example.
|
||||
|
||||
===== Updating MirrorManager
|
||||
|
||||
We need to update the mirrormanager so that it will point rawhide to the
|
||||
new rawhide release.
|
||||
|
||||
Please follow the instructions in the
|
||||
https://pagure.io/fedora-infrastructure/issue/9239#comment-671446[fedora
|
||||
infra ticket] to update the database of mirrormanager.
|
||||
|
||||
===== Enable autosigning on branched release
|
||||
|
||||
Once the branched compose is composed, we need to re-enable
|
||||
robosignatory on branched release
|
||||
|
||||
===== ELN related work
|
||||
|
||||
Add the new rawhide key to eln pungi config. For example, look at this
|
||||
https://pagure.io/pungi-fedora/c/e993441164ee83374df7f463777f2bf1d456fd6d?branch=eln[pungi
|
||||
eln config commit]
|
||||
|
||||
Change the trigger notification for DistroBuildSync to the new Rawhide
|
||||
version. For example, look at this
|
||||
https://gitlab.com/redhat/centos-stream/ci-cd/distrosync/distrobuildsync-config/-/commit/1497d9aea42cf00af646b4a0f9f9ed1a7f0a477f[commit].
|
||||
|
||||
===== Branch new rawhide in Koschei
|
||||
|
||||
Branch new fedora rawhide in
|
||||
https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/koschei/#_branching_a_new_fedora_release[koschei].
|
||||
|
||||
==== Fedora Container Base Image
|
||||
|
||||
In order to enable builds for Container Base Images via the
|
||||
https://docs.pagure.org/releng/layered_image_build_service.html[Fedora
|
||||
Layered Image Build System] we will need to import a new image for
|
||||
Rawhide as well as for the new `fedora:rawhide` and `fedora:${RAWHIDE}`
|
||||
tags.
|
||||
|
||||
Check for the latest successful Rawhide Base Image composed image
|
||||
https://koji.fedoraproject.org/koji/packageinfo?packageID=21546[here].
|
||||
|
||||
On `compose-x86-01.phx2` run:
|
||||
|
||||
....
|
||||
# Update this to be the correct URL for your image
|
||||
$ BASEIMAGE_URL="https://kojipkgs.fedoraproject.org//packages/Fedora-Docker-Base/Rawhide/20170310.n.0/images/Fedora-Docker-Base-Rawhide-20170310.n.0.x86_64.tar.xz"
|
||||
|
||||
# Update this to whatever version number Rawhide now points to
|
||||
$ RAWHIDE="27"
|
||||
|
||||
# Load the latest, find it's image name
|
||||
$ sudo docker load < <(curl -s "${BASEIMAGE_URL}")
|
||||
$ sudo docker images | grep base-rawhide
|
||||
fedora-docker-base-rawhide-20170310.n.0.x86_64 latest ffd832a990ca 5 hours ago 201.8 MB
|
||||
|
||||
# Tag everything
|
||||
$ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 candidate-registry.fedoraproject.org/fedora:rawhide
|
||||
$ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 candidate-registry.fedoraproject.org/fedora:${RAWHIDE}
|
||||
$ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 registry.fedoraproject.org/fedora:rawhide
|
||||
$ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 registry.fedoraproject.org/fedora:${RAWHIDE
|
||||
|
||||
# Push the images
|
||||
$ sudo docker push candidate-registry.fedoraproject.org/fedora:rawhide
|
||||
$ sudo docker push candidate-registry.fedoraproject.org/fedora:${RAWHIDE}
|
||||
$ sudo docker push registry.fedoraproject.org/fedora:rawhide
|
||||
$ sudo docker push registry.fedoraproject.org/fedora:${RAWHIDE}
|
||||
|
||||
# Clean up after ourselves
|
||||
$ sudo docker rmi fedora-docker-base-rawhide-20170310.n.0.x86_64
|
||||
Untagged: fedora-docker-base-rawhide-20170310.n.0.x86_64:latest
|
||||
$ for i in $(sudo docker images -q -f 'dangling=true'); do sudo docker rmi $i; done
|
||||
....
|
||||
|
||||
===== Update sync script
|
||||
|
||||
In releng repository update
|
||||
https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh#_38[script].
|
||||
|
||||
And set current_rawhide variable.
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
FIXME: Need some love here
|
||||
====
|
||||
122
modules/release_guide/pages/sop_mass_rebuild_modules.adoc
Normal file
122
modules/release_guide/pages/sop_mass_rebuild_modules.adoc
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
== Mass Rebuild of Modules
|
||||
|
||||
=== Description
|
||||
|
||||
Periodically we do mass rebuilds of modules in rawhide during the
|
||||
development cycle. This SOP will outline the steps necessary to do this.
|
||||
|
||||
=== Assumptions
|
||||
|
||||
This assumes that the mass rebuild has already been approved and
|
||||
scheduled via release engineering and FESCo. Coordinate with
|
||||
infrastructure as well for any needed updates.
|
||||
|
||||
=== Considerations
|
||||
|
||||
* The most important thing to keep in mind while doing a mass rebuild is
|
||||
to communicate clearly what actions are being performed and the status
|
||||
of the rebuild.
|
||||
* Check in on scripts frequently to avoid a long stalled command from
|
||||
adding significant delays in completing the rebuild.
|
||||
|
||||
=== Actions
|
||||
|
||||
==== Preparatory Steps
|
||||
|
||||
The following steps should be completed after the
|
||||
https://docs.pagure.org/releng/sop_mass_rebuild_packages.html[mass
|
||||
rebuild of packages] is done.
|
||||
|
||||
. Update Scripts
|
||||
|
||||
The mass rebuild depends on two main scripts from the
|
||||
https://pagure.io/releng[releng git repository]. Each one requires some
|
||||
changes in variables for each new mass rebuild cycle.
|
||||
|
||||
____
|
||||
* {blank}
|
||||
+
|
||||
_mass-rebuild-modules.py_::
|
||||
** rebuildid
|
||||
* {blank}
|
||||
+
|
||||
_massrebuildsinfo.py_::
|
||||
** module_mass_rebuild_epoch
|
||||
** module_mass_rebuild_platform
|
||||
____
|
||||
|
||||
Change the following items:
|
||||
|
||||
* the `rebuildid` to match the release for which you are mass rebuilding
|
||||
modules as per in massrebuildsinfo.py
|
||||
* `module_mass_rebuild_epoch` mostly will be the epoch of mass rebuild
|
||||
of packages
|
||||
* `module_mass_rebuild_platform` should be the rawhide module platform
|
||||
|
||||
==== Starting the Mass Rebuild of Modules
|
||||
|
||||
The `mass-rebuild-modules.py` script takes care of:
|
||||
|
||||
* Discovering available available modules from PDC
|
||||
* Find the module info from mbs and check if a module build is submitted
|
||||
after the epoch date
|
||||
* Checking out modules from dist-git
|
||||
* Switching to appropriate stream
|
||||
* Find modulemd file
|
||||
* Use libmodulemd to determine if this module stream applies to this
|
||||
platform version
|
||||
* If needs rebuilding, committing the change
|
||||
* Push the commit
|
||||
* Submitting the build request through mbs
|
||||
|
||||
. Connect to the mass-rebuild Machine
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ssh compose-branched01.iad2.fedoraproject.org
|
||||
....
|
||||
____
|
||||
. Start a terminal multiplexer
|
||||
+
|
||||
____
|
||||
....
|
||||
$ tmux
|
||||
....
|
||||
____
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository].
|
||||
. Run the [.title-ref]#mass-rebuild-modules.py# script from
|
||||
_releng/scripts_
|
||||
+
|
||||
____
|
||||
....
|
||||
$ cd path/to/releng_repo/scripts
|
||||
$ ./mass-rebuild-modules.py <path_to_token_file> build --wait 2>&1 | tee ~/massbuildmodules.out
|
||||
....
|
||||
____
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
The token file should be located in infra's private ansible repo, or ask
|
||||
infra to get it to you using this
|
||||
https://pagure.io/fedora-infrastructure/issue/8048#comment-587789[process].
|
||||
====
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
The [.title-ref]#build# option is really important to pay attention,
|
||||
since the mass branching of modules will also use the same script, just
|
||||
changing the option to [.title-ref]#branch# and
|
||||
[.title-ref]#module_mass_branching_platform# in
|
||||
[.title-ref]#massrebuildsinfo.py#
|
||||
====
|
||||
|
||||
==== Post Mass Rebuild Tasks
|
||||
|
||||
Once the module mass rebuild is done, send an email to the
|
||||
devel-announce@ list
|
||||
|
||||
. Send the final notification to the
|
||||
_devel-announce@lists.fedoraproject.org_ list
|
||||
262
modules/release_guide/pages/sop_mass_rebuild_packages.adoc
Normal file
262
modules/release_guide/pages/sop_mass_rebuild_packages.adoc
Normal file
|
|
@ -0,0 +1,262 @@
|
|||
== Mass Rebuild
|
||||
|
||||
=== Description
|
||||
|
||||
Periodically we do mass rebuilds of rawhide during the development
|
||||
cycle. This SOP will outline the steps necessary to do this.
|
||||
|
||||
=== Assumptions
|
||||
|
||||
This assumes that the mass rebuild has already been approved and
|
||||
scheduled via release engineering and FESCo. Coordinate with
|
||||
infrastructure as well for any needed koji updates.
|
||||
|
||||
This also assumes that the mass rebuild does not need to be done in
|
||||
dependency order, and that the mass rebuild does not involve a ABI
|
||||
change.
|
||||
|
||||
=== Considerations
|
||||
|
||||
* The most important thing to keep in mind while doing a mass rebuild is
|
||||
to communicate clearly what actions are being performed and the status
|
||||
of the rebuild.
|
||||
* Check in on scripts frequently to avoid a long stalled command from
|
||||
adding significant delays in completing the rebuild.
|
||||
* Check with secondary arches, whether they up-to-date enough with
|
||||
primary, create rebuild tag and target when they are. It will then take
|
||||
care of rebuilds of the arch specific packages in appropriate kojis.
|
||||
|
||||
=== Actions
|
||||
|
||||
==== Preparatory Steps
|
||||
|
||||
The following steps may be completed in the weeks leading up to the
|
||||
scheduled mass rebuild.
|
||||
|
||||
. Create the Mass Rebuild Pagure Issue
|
||||
+
|
||||
____
|
||||
Create an issue on the https://pagure.io/releng/issues[Release
|
||||
Engineering issues page] that points at the schedule for the current
|
||||
release.
|
||||
|
||||
See https://pagure.io/releng/issue/6898[the Fedora 27 mass rebuild issue
|
||||
example].
|
||||
____
|
||||
. Set up the Mass Rebuild Wiki Page
|
||||
+
|
||||
____
|
||||
The mass rebuild wiki page should answer the following questions for
|
||||
maintainers:
|
||||
|
||||
* Why the mass rebuild is happening
|
||||
* How to opt out of the mass rebuild
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
See https://fedoraproject.org/wiki/Fedora_26_Mass_Rebuild[the Fedora 26
|
||||
Wiki example].
|
||||
====
|
||||
____
|
||||
. Send out the Mass Rebuild Notice
|
||||
+
|
||||
____
|
||||
Send out the same information posted on the wiki to the
|
||||
[.title-ref]#devel-announce@lists.fedoraproject.org# mailing list.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
See
|
||||
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[the
|
||||
Fedora 26 e-mail example].
|
||||
====
|
||||
____
|
||||
. Create a Tag to Contain the Mass Rebuild
|
||||
+
|
||||
____
|
||||
Mass rebuilds require their own tag to contain all related builds. The
|
||||
example assumes we are doing a rebuild for Fedora 26.
|
||||
|
||||
....
|
||||
$ koji add-tag f26-rebuild --parent f26
|
||||
....
|
||||
____
|
||||
. Request Package Auto-Signing for New Mass-Rebuild Tag
|
||||
+
|
||||
____
|
||||
File a ticket with https://pagure.io/fedora-infrastructure/issues[Fedora
|
||||
Infrastructure] requesting the new mass-rebuild tag be enabled for
|
||||
package auto-signing.
|
||||
____
|
||||
. Create the Koji Target for the Mass Rebuild
|
||||
+
|
||||
____
|
||||
Using the same [.title-ref]#f26-rebuild# tag created in the previous
|
||||
example:
|
||||
|
||||
....
|
||||
$ koji add-target f26-rebuild f26-build
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
*koji add-target* _target-name_ _buildroot-tag_ _destination-tag_
|
||||
describes the syntax format above. If the _destination-tag_ is not
|
||||
specified then it will be the same as the _target-name_.
|
||||
====
|
||||
____
|
||||
. Update Scripts
|
||||
+
|
||||
____
|
||||
The mass rebuild depends on four main scripts from the
|
||||
https://pagure.io/releng[releng git repository]. Each one requires some
|
||||
changes in variables for each new mass rebuild cycle.
|
||||
|
||||
* {blank}
|
||||
+
|
||||
mass-rebuild.py::
|
||||
** buildtag
|
||||
** targets
|
||||
** epoch
|
||||
** comment
|
||||
** target
|
||||
* {blank}
|
||||
+
|
||||
find-failures.py::
|
||||
** buildtag
|
||||
** desttag
|
||||
** epoch
|
||||
* mass-tag.py
|
||||
* {blank}
|
||||
+
|
||||
need-rebuild.py::
|
||||
** buildtag
|
||||
** target
|
||||
** updates
|
||||
** epoch
|
||||
____
|
||||
|
||||
Change the following items:
|
||||
|
||||
* the build tag, holding tag, and target tag should be updated to
|
||||
reflect the Fedora release you're building for
|
||||
* the `epoch` should be updated to the point at which all features that
|
||||
the mass rebuild is for have landed in the build system (and a newRepo
|
||||
task completed with those features)
|
||||
* the comment which is inserted into spec changelogs
|
||||
|
||||
==== Starting the Mass Rebuild
|
||||
|
||||
The `mass-rebuild.py` script takes care of:
|
||||
|
||||
* Discovering available packages in koji
|
||||
* Trimming out packages which have already been rebuilt
|
||||
* Checking out packages from git
|
||||
* Bumping the spec file
|
||||
* Committing the change
|
||||
* git tagging the change
|
||||
* Submitting the build request to Koji
|
||||
|
||||
. Connect to the mass-rebuild Machine
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ssh branched-composer.phx2.fedoraproject.org
|
||||
....
|
||||
____
|
||||
. Start a terminal multiplexer
|
||||
+
|
||||
____
|
||||
....
|
||||
$ tmux
|
||||
....
|
||||
____
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository].
|
||||
. Run the mass-rebuild.py script from _releng/scripts_
|
||||
+
|
||||
____
|
||||
....
|
||||
$ cd path/to/releng_repo/scripts
|
||||
$ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out
|
||||
....
|
||||
____
|
||||
|
||||
==== Monitoring Mass Rebuilds
|
||||
|
||||
The community has a very high interest in the status of rebuilds and
|
||||
many maintainers will want to know if their build failed right away. The
|
||||
`find-failures.py` and `need-rebuild.py` scripts are designed to update
|
||||
publicly available URLs for stakeholders to monitor.
|
||||
|
||||
. Connect to a Compose Machine
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ssh compose-x86-02.phx2.fedoraproject.org
|
||||
....
|
||||
____
|
||||
. Start a terminal multiplexer
|
||||
+
|
||||
____
|
||||
....
|
||||
$ tmux
|
||||
....
|
||||
____
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository]
|
||||
. {blank}
|
||||
+
|
||||
Set Up the Rebuild Failures Notification Web Site::
|
||||
The `find_failures.py` script discovers attempted builds that have
|
||||
failed. It lists those failed builds and sorts them by package owner.
|
||||
+
|
||||
....
|
||||
$ while true; do ./find_failures.py > f26-failures.html && cp f26-failures.html /mnt/koji/mass-rebuild/f26-failures.html; sleep 600; done
|
||||
....
|
||||
. Start a second pane in the terminal emulator
|
||||
. {blank}
|
||||
+
|
||||
Set up the Site for Packages that Need Rebuilt::
|
||||
The `need-rebuild.py` script discovers packages that have not yet been
|
||||
rebuilt and generates an html file listing them sorted by package
|
||||
owner. This gives external stakeholders a rough idea of how much work
|
||||
is remaining in the mass rebuild.
|
||||
+
|
||||
....
|
||||
$ while true; do ./need-rebuild.py > f26-need-rebuild.html && cp f26-need-rebuild.html /mnt/koji/mass-rebuild/f26-need-rebuild.html; sleep 600; done
|
||||
....
|
||||
|
||||
==== Post Mass Rebuild Tasks
|
||||
|
||||
Once the mass rebuild script completes, and all the pending builds have
|
||||
finished, the builds will need to be tagged. The `mass-tag.py` script
|
||||
will accomplish this task. The script will:
|
||||
|
||||
* Discover completed builds
|
||||
* Trim out builds that are older than the latest build for a given
|
||||
package
|
||||
* Tag remaining builds into their final destination (without generating
|
||||
email)
|
||||
|
||||
. Clone or checkout the latest copy of the
|
||||
https://pagure.io/releng[releng git repository]
|
||||
. Run the `mass-tag.py` script (requires koji kerberos authentication)
|
||||
+
|
||||
____
|
||||
....
|
||||
$ cd path/to/releng_repo/scripts
|
||||
$ ./mass-tag.py --source f36-rebuild --target f36
|
||||
....
|
||||
____
|
||||
. Send the final notification to the
|
||||
_devel-announce@lists.fedoraproject.org_ list
|
||||
+
|
||||
____
|
||||
The contents should look something like this
|
||||
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[example
|
||||
email].
|
||||
____
|
||||
17
modules/release_guide/pages/sop_package_blocking.adoc
Normal file
17
modules/release_guide/pages/sop_package_blocking.adoc
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
== Package Blocking
|
||||
|
||||
=== Description
|
||||
|
||||
If a
|
||||
https://fedoraproject.org/wiki/How_to_remove_a_package_at_end_of_life[package
|
||||
is removed (retired) from Fedora], for example because it was renamed,
|
||||
it needs to be blocked in Koji. This prevents creating new package
|
||||
builds and distribution of built RPMs. Packages are blocked in the
|
||||
listing of `tags`, due to inheritance it is enough to block packages at
|
||||
the oldest tag will make it unavailable also in upstream tags.
|
||||
|
||||
=== Action
|
||||
|
||||
The blocking of retired packages is done by the
|
||||
https://pagure.io/releng/blob/master/f/scripts/block_retired.py[block_retired.py]
|
||||
script as part of the daily Rawhide and Branched composes.
|
||||
121
modules/release_guide/pages/sop_package_unblocking.adoc
Normal file
121
modules/release_guide/pages/sop_package_unblocking.adoc
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
== Package Unblocking
|
||||
|
||||
=== Description
|
||||
|
||||
Packages are sometimes unblocked from Fedora, usually when a package had
|
||||
been orphaned and now has a new owner. When this happens, release
|
||||
engineering needs to "unblock" the package from koji tags.
|
||||
|
||||
=== Action
|
||||
|
||||
==== Find Unblock requests
|
||||
|
||||
Unblock requests are usually reported in the
|
||||
https://pagure.io/releng/issues[rel-eng issue tracker].
|
||||
|
||||
==== Perform the unblocking
|
||||
|
||||
First assign the ticket to yourself to show, that you are handling the
|
||||
request.
|
||||
|
||||
===== Discover proper place to unblock
|
||||
|
||||
The ticket should tell you which Fedora releases to unblock the package
|
||||
in. Typically it'll say "Fedora 13" or "F14". This means we need to
|
||||
unblock it at that Fedora level and all future tags. However depending
|
||||
on where the package was blocked we may have to do our unblock action at
|
||||
a different Fedora level.
|
||||
|
||||
To discover where a package is blocked, use the `list-pkgs` method of
|
||||
koji.
|
||||
|
||||
....
|
||||
$ koji list-pkgs --help
|
||||
Usage: koji list-pkgs [options]
|
||||
(Specify the --help global option for a list of other help options)
|
||||
|
||||
Options:
|
||||
-h, --help show this help message and exit
|
||||
--owner=OWNER Specify owner
|
||||
--tag=TAG Specify tag
|
||||
--package=PACKAGE Specify package
|
||||
--quiet Do not print header information
|
||||
--noinherit Don't follow inheritance
|
||||
--show-blocked Show blocked packages
|
||||
--show-dups Show superseded owners
|
||||
--event=EVENT# query at event
|
||||
--ts=TIMESTAMP query at timestamp
|
||||
--repo=REPO# query at event for a repo
|
||||
....
|
||||
|
||||
For example if we wanted to see where python-psco was blocked we would
|
||||
do:
|
||||
|
||||
....
|
||||
$ koji list-pkgs --package python-psyco --show-blocked
|
||||
Package Tag Extra Arches Owner
|
||||
----------------------- ----------------------- ---------------- ---------------
|
||||
python-psyco dist-f14 konradm [BLOCKED]
|
||||
python-psyco olpc2-ship2 shahms
|
||||
python-psyco olpc2-trial3 shahms
|
||||
...
|
||||
....
|
||||
|
||||
Here we can see that it was blocked at dist-f14. If we got a request
|
||||
that was to unblock it before f14, we can simply use the dist-f14 target
|
||||
to unblock. However if they want it unblocked after f14, we would use
|
||||
the earliest dist-f?? tag the user wants, such as dist-f15 if the user
|
||||
asked for it to be unblocked in Fedora 15+
|
||||
|
||||
===== Performing the unblock
|
||||
|
||||
To unblock a package for a tag, use the `unblock-pkg` method of Koji.
|
||||
|
||||
....
|
||||
$ koji unblock-pkg --help
|
||||
Usage: koji unblock-pkg [options] tag package [package2 ...]
|
||||
(Specify the --help global option for a list of other help options)
|
||||
|
||||
Options:
|
||||
-h, --help show this help message and exit
|
||||
....
|
||||
|
||||
For example, if we were asked to unblock python-psyco in F14 we would
|
||||
issue:
|
||||
|
||||
....
|
||||
$ koji unblock-pkg dist-f14 python-psyco
|
||||
....
|
||||
|
||||
Now the ticket can be closed.
|
||||
|
||||
=== Verification
|
||||
|
||||
To verify that the package was successfully unblocked use the
|
||||
`list-pkgs` koji command:
|
||||
|
||||
....
|
||||
$ koji list-pkgs --package python-psyco --show-blocked
|
||||
....
|
||||
|
||||
We should see the package listed as not blocked at dist-f14 or above:
|
||||
|
||||
....
|
||||
Package Tag Extra Arches Owner
|
||||
----------------------- ----------------------- ---------------- ---------------
|
||||
python-psyco olpc2-trial3 jkeating
|
||||
python-psyco olpc2-ship2 jkeating
|
||||
python-psyco olpc2-update1 jkeating
|
||||
python-psyco trashcan jkeating
|
||||
python-psyco f8-final jkeating
|
||||
...
|
||||
....
|
||||
|
||||
We should not see it listed as blocked in dist-f14 or any later Fedora
|
||||
tags.
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
* Watch the next day's rawhide/branched/whatever report for a slew of
|
||||
broken deps related to the package. We may have to re-block the package
|
||||
in order to fix the deps.
|
||||
7
modules/release_guide/pages/sop_pdc.adoc
Normal file
7
modules/release_guide/pages/sop_pdc.adoc
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
== Product Definition Center
|
||||
|
||||
This is a stub SOP for PDC. The doc really lives in the infra SOPs,
|
||||
here: https://infrastructure.fedoraproject.org/infra/docs/pdc.rst
|
||||
|
||||
Please update and maintain that SOP. Leave this document here as a
|
||||
breadcrumb to make it easier to find that doc.
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
== Process fedora-scm-requests tickets
|
||||
|
||||
=== Description
|
||||
|
||||
When a packager wants a new package added to Fedora or a new dist-git
|
||||
branch blessed, they need to go through the new package process and,
|
||||
once their package review is approved, they use the
|
||||
[.title-ref]#fedrepo-req# cli tool to file a ticket in the
|
||||
https://pagure.io/releng/fedora-scm-requests[fedora-scm-requests queue].
|
||||
|
||||
Periodically, (daily?) release engineering will need to review and
|
||||
process this queue using the [.title-ref]#fedrepo-req-admin# tool.
|
||||
|
||||
=== Setup
|
||||
|
||||
A release engineering will need to have several values set locally as
|
||||
well as sufficient permissions in a number of server-side systems.
|
||||
|
||||
. A pagure.io token. See the fedrepo-req README for instructions on
|
||||
where to get this.
|
||||
. src.fedoraproject.org token generated by [.title-ref]#pagure-admin#.
|
||||
Ask @pingou how to get one. If doing this yourself, go to pkgs01 and run
|
||||
[.title-ref]#PAGURE_CONFIG=/etc/pagure/pagure.cfg pagure-admin
|
||||
admin-token create -h# for more info.
|
||||
. pdc token. See the PDC SOP for getting one of these.
|
||||
|
||||
=== Action
|
||||
|
||||
. Run [.title-ref]#fedrepo-req-admin list# to list all open requests.
|
||||
. Run [.title-ref]#fedrepo-req-admin process N# to process a particular
|
||||
ticket.
|
||||
. Run [.title-ref]#fedrepo-req-admin processall# to iterate over all the
|
||||
tickets.
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
== Promoting Container Content
|
||||
|
||||
=== Description
|
||||
|
||||
Even though the promotion of content is aimed to be fully automated,
|
||||
sometimes there is the need or desire to promote content from the
|
||||
candidate registry to the stable registry. Below is how to accomplish
|
||||
this using https://github.com/projectatomic/skopeo[skopeo].
|
||||
|
||||
=== Action
|
||||
|
||||
This action should be performed on
|
||||
`compose-x86-01.phx2.fedoraproject.org` by an user who is a member of
|
||||
the `sysadmin-releng` https://admin.fedoraproject.org/accounts/[FAS]
|
||||
Group.
|
||||
|
||||
The container image should be provided by the requester, they will have
|
||||
the information from their container build in the
|
||||
https://docs.pagure.org/releng/layered_image_build_service.html[Fedora
|
||||
Layered Image Build System]. It will have a name resembling
|
||||
`candidate-registry.fedoraproject.org/f26/foo:0.1-1.f26container`.
|
||||
|
||||
Sync content between the candidate registry and production registry,
|
||||
effectively "promoting it". Substitute the `$IMAGE_NAME` in the
|
||||
following example with whatever the actual image name is that is
|
||||
provided. (This would be everything after
|
||||
`candidate-registry.fedoraproject.org`, so using the above example then
|
||||
`$IMAGE_NAME` would be `f26/foo:0.1-1.f26container`.)
|
||||
|
||||
....
|
||||
$ sudo skopeo copy \
|
||||
--src-cert-dir /etc/docker/certs.d/candidate-registry.fedoraproject.org/ \
|
||||
--dest-cert-dir /etc/docker/certs.d/registry.fedoraproject.org/ \
|
||||
docker://candidate-registry.fedoraproject.org/$IMAGE_NAME \
|
||||
docker://registry.fedoraproject.org/$IMAGE_NAME
|
||||
....
|
||||
|
||||
=== Verification
|
||||
|
||||
In order to verify, we need to inspect the stable registry, again with
|
||||
https://github.com/projectatomic/skopeo[skopeo] to ensure the image
|
||||
metadata exists.
|
||||
|
||||
....
|
||||
$ skopeo inspect docker://registry.fedoraproject.org/$IMAGE_NAME
|
||||
....
|
||||
|
||||
In this JSON output you will see a list element titled `RepoTags` and in
|
||||
there should be the `$VERSION-$RELEASE` listed there, following our
|
||||
example above this entry would be `0.1-1.f26container`.
|
||||
225
modules/release_guide/pages/sop_pushing_updates.adoc
Normal file
225
modules/release_guide/pages/sop_pushing_updates.adoc
Normal file
|
|
@ -0,0 +1,225 @@
|
|||
== Pushing Updates
|
||||
|
||||
=== Description
|
||||
|
||||
Fedora updates are typically pushed once a day. This SOP covers the
|
||||
steps involved.
|
||||
|
||||
==== Coordinate
|
||||
|
||||
Releng has a rotation of who pushes updates when. Please coordinate and
|
||||
only push updates when you are expected to or have notified other releng
|
||||
folks you are doing so. See:
|
||||
https://apps.fedoraproject.org/calendar/release-engineering/ for the
|
||||
list or on irc you can run `.pushduty` in any channel with zodbot to see
|
||||
who is on duty this week.
|
||||
|
||||
==== Login to machine to sign updates
|
||||
|
||||
Login to a machine that is configured for sigul client support and has
|
||||
the bodhi client installed. Currently, this machine is:
|
||||
`bodhi-backend01.phx2.fedoraproject.org`
|
||||
|
||||
==== Decide what releases you're going to push.
|
||||
|
||||
* If there is a Freeze ongoing, you SHOULD NOT push all stable requests
|
||||
for a branched release, only specific blocker or freeze exception
|
||||
requests that QA will request in a releng ticket.
|
||||
* If there is no Freeze ongoing you can push all Fedora and EPEL
|
||||
releases at the same time if you wish.
|
||||
* From time to time there may be urgent requests in some branches, you
|
||||
can only push those if requested. Note however that bodhi2 will
|
||||
automatically push branches with security updates before others.
|
||||
|
||||
==== Get a list of packages to push
|
||||
|
||||
....
|
||||
$ sudo -u apache bodhi-push --releases 'f26,f25,f24,epel-7,EL-6' --username <yourusername>
|
||||
<enter your password+2factorauth, then your fas password>
|
||||
....
|
||||
|
||||
Sometimes you see a warning "Warning: foobar-1.fcxx has unsigned builds
|
||||
and has been skipped" which means those updates are currently getting
|
||||
signed and it can verified by listing the tagged builds in
|
||||
fxx-signing-pending tag.
|
||||
|
||||
::::
|
||||
$ koji list-tagged fxx-signing-pending
|
||||
|
||||
You can say 'n' to the push at this point if you wish to sign packages
|
||||
(see below). Or you can keep this request open in a window while you
|
||||
sign the packages, then come back and say y.
|
||||
|
||||
List the releases above you wish to push from: 25 24 5 6 7, etc
|
||||
|
||||
You can also specify `--request=testing` to limit pushes. Valid types
|
||||
are `testing` or `stable`.
|
||||
|
||||
The list of updates should be in the cache directory named
|
||||
`Stable-$Branch` or `Testing-$Branch` for each of the Branches you
|
||||
wished to push.
|
||||
|
||||
During freezes you will need to do two steps: (If say, fedora 26
|
||||
branched was frozen):
|
||||
|
||||
....
|
||||
$ sudo -u apache bodhi-push --releases f26 --request=testing \
|
||||
--username <username>
|
||||
....
|
||||
|
||||
Then
|
||||
|
||||
....
|
||||
$ sudo -u apache bodhi-push --releases 'f25,f24,epel-7,EL-6' --username <username>
|
||||
....
|
||||
|
||||
During the Release Candidate compose phase we tag builds to be included
|
||||
into a -compose tag (e.g. f26-compose). When we have a candidate that
|
||||
has been signed off as gold we need to ensure that all builds tagged
|
||||
into the -compose tag have been pushed stable. Once we have pushed all
|
||||
-compose builds stable we then have to clone the base tag (e.g. f26) to
|
||||
a tag for the milestone for Alpha and Beta (e.g. f26-Alpha). After final
|
||||
release we need to lock the base tag and adjust the release status in
|
||||
bodhi so that updates now hit the -updates tag (e.g. f26-updates). Once
|
||||
we have cloned the tag or locked the tag and adjusted bodhi we are free
|
||||
to push stable updates again.
|
||||
|
||||
==== Pushing Stable updates during freeze
|
||||
|
||||
During feezes we need to push to stable builds included in the compose.
|
||||
QA will file a ticket with the nvrs to push.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
If you are pushing a bodhi update that contains multiple builds, you
|
||||
need only pass bodhi-push a single build nvr and all the others in that
|
||||
update will be detected and pushed along with it. However, if you are
|
||||
pushing multiple disjoint bodhi updates then each build will need to be
|
||||
listed individually.
|
||||
====
|
||||
|
||||
....
|
||||
$ sudo -u apache bodhi-push --builds '<nvr1>,<nvr2>,...' --username <username>
|
||||
....
|
||||
|
||||
===== There are no updates to push.
|
||||
|
||||
If you are getting the message `There are no updates to push.` or the
|
||||
list of packages you are seeing to push out for the Stable updates
|
||||
request is not correct compared to what you specified in the `--builds`
|
||||
section of the command above then one of two things likely happened.
|
||||
|
||||
. The update hasn't yet reached appropriate karma
|
||||
+
|
||||
This should be handled case-by-case, if the QA Team has requested this
|
||||
be pushed as stable to fix a blocker but there's not yet enough karma
|
||||
for an autostable prompt to occur then you should verify with QA that
|
||||
these are ready to go out even without karma. If they are, then log into
|
||||
the Bodhi WebUI and modify the karma threshold of the update to 1 and
|
||||
add karma (if necessary). This is not something we should do as normal
|
||||
practice and is considered an edge case. When update requests come to
|
||||
RelEng, it should have appropriate karma. Sometimes it doesn't and as
|
||||
long as QA approves, we need not block on it.
|
||||
. The update was never requested for stable
|
||||
+
|
||||
It's possible the update wasn't requested for stable, you can resolve
|
||||
this by running the following on one of the bodhi-backend systems:
|
||||
+
|
||||
....
|
||||
bodhi updates request <BODHI-REQUEST-ID> stable
|
||||
....
|
||||
|
||||
==== Perform the bodhi push
|
||||
|
||||
Say 'y' to push for the above command.
|
||||
|
||||
=== Verification
|
||||
|
||||
. Monitor Bodhi's composes with `bodhi-monitor-composes`
|
||||
+
|
||||
....
|
||||
$ sudo -u apache watch -d -n 60 bodhi-monitor-composes
|
||||
....
|
||||
. Monitor the systemd journal
|
||||
+
|
||||
....
|
||||
$ sudo journalctl -o short -u fedmsg-hub -l -f
|
||||
....
|
||||
. Check the processes
|
||||
+
|
||||
....
|
||||
$ ps axf|grep bodhi
|
||||
....
|
||||
. Watch for fedmsgs through the process. It will indicate what releases
|
||||
it's working on, etc. You may want to watch in `#fedora-fedmsg`.
|
||||
+
|
||||
....
|
||||
bodhi.masher.start -- kevin requested a mash of 48 updates
|
||||
bodhi.mashtask.start -- bodhi masher started a push
|
||||
bodhi.mashtask.mashing -- bodhi masher started mashing f23-updates
|
||||
bodhi.mashtask.mashing -- bodhi masher started mashing f22-updates-testing
|
||||
...
|
||||
bodhi.update.complete.stable -- moceap's wondershaper-1.2.1-5.fc23 bodhi update completed push to stable https://admin.fedoraproject.org/updates/FEDORA-2015-13052
|
||||
...
|
||||
bodhi.errata.publish -- Fedora 23 Update: wondershaper-1.2.1-5.fc23 https://admin.fedoraproject.org/updates/FEDORA-2015-13052
|
||||
bodhi.mashtask.complete -- bodhi masher successfully mashed f23-updates
|
||||
bodhi.mashtask.sync.wait -- bodhi masher is waiting for f22-updates-testing to hit the master mirror
|
||||
....
|
||||
. Seach for problems with a particular push:
|
||||
+
|
||||
....
|
||||
sudo journalctl --since=yesterday -o short -u fedmsg-hub | grep dist-6E-epel (or f22-updates, etc)
|
||||
....
|
||||
. Note: Bodhi will look at the things you have told it to push and see
|
||||
if any have security updates, those branches will be started first. It
|
||||
will then fire off threads (up to 3 at a time) and do the rest.
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
Pushes often fall over due to tagging issues or unsigned packages. Be
|
||||
prepared to work through the failures and restart pushes from time to
|
||||
time
|
||||
|
||||
....
|
||||
$ sudo -u apache bodhi-push --resume
|
||||
....
|
||||
|
||||
Bodhi will ask you which push(es) you want to resume.
|
||||
|
||||
=== Common issues / problems with pushes
|
||||
|
||||
* When the push fails due to new unsigned packages that were added after
|
||||
you started the process. re-run step 4a or 4b with just the package
|
||||
names that need to be signed, then resume.
|
||||
* When the push fails due to an old package that has no signature, run:
|
||||
`koji write-signed-rpm <gpgkeyid> <n-v-r>` and resume.
|
||||
* When the push fails due to a package not being tagged with
|
||||
updates-testing when being moved stable:
|
||||
`koji tag-pkg dist-<tag>-updates-testing <n-v-r>`
|
||||
* When signing fails, you may need to ask that the sigul bridge or
|
||||
server be restarted.
|
||||
* If the updates push fails with a:
|
||||
`OSError: [Errno 16] Device or resource busy: '/var/lib/mock/*-x86_64/root/var/tmp/rpm-ostree.*'`
|
||||
You need to umount any tmpfs mounts still open on the backend and resume
|
||||
the push.
|
||||
* If the updates push fails with:
|
||||
`"OSError: [Errno 39] Directory not empty: '/mnt/koji/mash/updates/*/../*.repocache/repodata/'`
|
||||
you need to restart fedmsg-hub on the backend and resume.
|
||||
* If the updates push fails with:
|
||||
`IOError: Cannot open /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml: File /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml doesn't exists or not a regular file`
|
||||
This issue will be resolved with NFSv4, but in the mean time it can be
|
||||
worked around by removing the [.title-ref]#.repocache# directory and
|
||||
resuming the push.
|
||||
`$ sudo rm -fr /mnt/koji/mash/updates/epel7.repocache`
|
||||
* If the Atomic OSTree compose fails with some sort of
|
||||
[.title-ref]#Device or Resource busy# error, then run
|
||||
[.title-ref]#mount# to see if there are any stray [.title-ref]#tmpfs#
|
||||
mounts still active:
|
||||
`tmpfs on /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq type tmpfs (rw,relatime,seclabel,mode=755)`
|
||||
You can then
|
||||
`$ sudo umount /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq`
|
||||
and resume the push.
|
||||
|
||||
Other issues should be addressed by releng or bodhi developers in
|
||||
`#fedora-releng`.
|
||||
95
modules/release_guide/pages/sop_rawhide_bodhi.adoc
Normal file
95
modules/release_guide/pages/sop_rawhide_bodhi.adoc
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
== Enabling Rawhide in Bodhi
|
||||
|
||||
=== Description
|
||||
|
||||
This SOP covers the steps needed to enable Rawhide in Bodhi.
|
||||
|
||||
==== Create the release in Bodhi
|
||||
|
||||
In oder to start creating updates in Bodhi for rawhide, the release
|
||||
needs to be created in Bodhi. Rawhide in Bodhi is represented by the
|
||||
Fedora version (ie Fedora 31), but it is set in the prerelease state.
|
||||
|
||||
===== Add the koji tags
|
||||
|
||||
....
|
||||
$ koji add-tag --parent f31 f31-updates-candidate
|
||||
$ koji add-tag --parent f31 f31-updates-testing
|
||||
$ koji add-tag --parent f31-updates-testing f31-updates-testing-pending
|
||||
$ koji edit-tag --perm autosign f31-updates-testing-pending
|
||||
$ koji add-tag --parent f31 f31-updates-pending
|
||||
$ koji add-tag --parent f31 f31-override
|
||||
....
|
||||
|
||||
===== Change the koji targets
|
||||
|
||||
....
|
||||
$ koji edit-target f31 --dest-tag f31-updates-candidate
|
||||
$ koji edit-target f31-candidate --dest-tag f31-updates-candidate
|
||||
$ koji edit-target rawhide --dest-tag f31-updates-candidate
|
||||
....
|
||||
|
||||
===== Create the release in bodhi
|
||||
|
||||
....
|
||||
$ bodhi releases create --name "F31" --long-name "Fedora 31" --id-prefix FEDORA --version 31 --branch f31 \
|
||||
--dist-tag f31 --stable-tag f31 --testing-tag f31-updates-testing --candidate-tag f31-updates-candidate \
|
||||
--pending-stable-tag f31-updates-pending --pending-testing-tag f31-updates-testing-pending \
|
||||
--state pending --override-tag f31-override --create-automatic-updates --not-composed-by-bodhi
|
||||
....
|
||||
|
||||
The important flags are [.title-ref]#--not-composed-by-bodhi# which
|
||||
tells bodhi not to include the rawhide updates in the nightly pushes and
|
||||
[.title-ref]#--create-automatic-updates# which tells bodhi to
|
||||
automatically create an update listen to koji tag (build tagged with the
|
||||
pending-testing-tag) messages.
|
||||
|
||||
===== Bodhi configuration
|
||||
|
||||
Bodhi is configured to required zero mandatory days in testing for the
|
||||
rawhide release. This is done in ansible
|
||||
roles/bodhi2/base/templates/production.ini.j2 with the following.
|
||||
|
||||
....
|
||||
f{{ FedoraRawhideNumber }}.pre_beta.mandatory_days_in_testing = 0
|
||||
....
|
||||
|
||||
===== Robosignatory configuration
|
||||
|
||||
Robosignatory needs to be configured to signed the rawhide builds before
|
||||
these builds are tested by the CI pipeline.
|
||||
|
||||
....
|
||||
{
|
||||
"from": "f31-updates-candidate",
|
||||
"to": "f31-updates-testing-pending",
|
||||
"key": "fedora-31",
|
||||
"keyid": "3c3359c4"
|
||||
},
|
||||
....
|
||||
|
||||
==== Branching Rawhide
|
||||
|
||||
When it is time to branch rawhide, a new release should be created (eg.
|
||||
F32) following the steps above. The old rawhide release (eg. F31) should
|
||||
stay configured as rawhide until we active Bodhi for it (2 weeks later).
|
||||
To activate Bodhi on the old rawhide (eg. F31) the existing release in
|
||||
bodhi should be modified has follow.
|
||||
|
||||
::::
|
||||
$ bodhi releases edit --name "F31" --stable-tag f31-updates
|
||||
--no-create-automatic-updates --composed-by-bodhi
|
||||
|
||||
===== Robosignatory configuration
|
||||
|
||||
At Bodhi activation time the Robosignatory configuration needs to be
|
||||
update to match the normal configuration of bodhi releases.
|
||||
|
||||
....
|
||||
{
|
||||
"from": "f31-signing-pending",
|
||||
"to": "f31-updates-testing-pending",
|
||||
"key": "fedora-31",
|
||||
"keyid": "3c3359c4"
|
||||
},
|
||||
....
|
||||
|
|
@ -0,0 +1,76 @@
|
|||
== Release the Fedora Container Base Image
|
||||
|
||||
=== Description
|
||||
|
||||
This SOP covers the steps involved in changing and releasing the Fedora
|
||||
Container Base image.
|
||||
|
||||
Fedora releases 2 container base images, [.title-ref]#fedora# and
|
||||
[.title-ref]#fedora-minimal#. These images are available on 3 registries
|
||||
[.title-ref]#registry.fedoraproject.org#, [.title-ref]#quay.io# and
|
||||
[.title-ref]#DockerHub# (fedora-minimal is not available in DockerHub).
|
||||
|
||||
==== Modify a base image (Kickstart)
|
||||
|
||||
Base images are built in koji using the [.title-ref]#image-factory#
|
||||
application to build the container image root filesystem (rootfs).
|
||||
Kickstart files are used to configure how the image is built and what is
|
||||
available in the image The solution consist of 3 Kickstarts.
|
||||
|
||||
https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-common.ks[fedora-container-common]
|
||||
|
||||
https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-base.ks[fedora-container-base]
|
||||
|
||||
https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-base-minimal.ks[fedora-container-base-minimal]
|
||||
|
||||
Changes made on the rawhide branch will results in the rawhide image,
|
||||
other branches (f30, f31) should be used to modify other releases.
|
||||
|
||||
==== Compose Configuration (Pungi)
|
||||
|
||||
The configuration used to compose the container images is available in
|
||||
the pungi-fedora repository.
|
||||
|
||||
For rawhide the configuration is in
|
||||
|
||||
https://pagure.io/pungi-fedora/blob/main/f/fedora.conf
|
||||
|
||||
While for other releases the configuration is in a dedicated file
|
||||
|
||||
https://pagure.io/pungi-fedora/blob/f31/f/fedora-container.conf
|
||||
|
||||
==== Release on registry.fedoraproject.org and quay.io
|
||||
|
||||
If you want to release the base image on registry.fp.o and quay.io you
|
||||
can use the following script.
|
||||
|
||||
https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh[sync-latest-container-base-image.sh]
|
||||
|
||||
You will need to run that script from on of the releng composer machines
|
||||
in the infrastructure in order to have the credentials.
|
||||
|
||||
If you do not have access to that machines, you can request the release
|
||||
by opening a ticket on the https://pagure.io/releng/issues[releng
|
||||
tracker].
|
||||
|
||||
The script can then be executed as follow
|
||||
|
||||
....
|
||||
$./sync-latest-container-base-image.sh 31
|
||||
$./sync-latest-container-base-image.sh 32
|
||||
....
|
||||
|
||||
This will take care of pushing the [.title-ref]#fedora# and
|
||||
[.title-ref]#fedora-minimal# images to both registries.
|
||||
|
||||
==== Release on DockerHub
|
||||
|
||||
Releasing on DockerHub is a little different since Fedora is an
|
||||
"offical" image there. In order to release new images there we have to
|
||||
update a Dockerfile and rootfs tarball on the following repo.
|
||||
|
||||
https://github.com/fedora-cloud/docker-brew-fedora[docker-brew-fedora].
|
||||
|
||||
For the details on how to run the script please see
|
||||
|
||||
https://github.com/fedora-cloud/docker-brew-fedora/blob/main/README.md[README].
|
||||
68
modules/release_guide/pages/sop_release_package_signing.adoc
Normal file
68
modules/release_guide/pages/sop_release_package_signing.adoc
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
== Release Package Signing
|
||||
|
||||
=== Description
|
||||
|
||||
For each of Fedora's public releases (Alpha, Beta, and Final) it is
|
||||
Release Engineering's responsibility to sign all packages with Fedora's
|
||||
GPG key. This provides confidence to Fedora's users about the
|
||||
authenticity of packages provided by Fedora.
|
||||
|
||||
The `/sop_create_release_signing_key` document explains the process for
|
||||
creating the GPG key used.
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
This script takes a very long time to run, as much as 4 or 5 days, so it
|
||||
needs to be started well in advance of when you need the packages all
|
||||
signed.
|
||||
|
||||
Signing all the packages will cause a lot of churn on the mirrors, so
|
||||
expect longer than usual compose and rsync times, as well as potential
|
||||
issues with proxies as file contents change but the name remains the
|
||||
same.
|
||||
|
||||
=== Action
|
||||
|
||||
. Log into a system with `sigul` and start a `screen` or `tmux` session.
|
||||
The signing process takes a long time--screen allows the process to
|
||||
continue if you session gets disconnected.
|
||||
+
|
||||
....
|
||||
$ screen -S sign
|
||||
....
|
||||
+
|
||||
or
|
||||
+
|
||||
....
|
||||
$ tmux new -s sign
|
||||
....
|
||||
. Check out the Release Engineering `git` repo
|
||||
+
|
||||
::::
|
||||
$ git clone https://pagure.io/releng.git
|
||||
. Change directories to the `scripts` directory to execute
|
||||
`sigulsign_unsigned.py`.
|
||||
+
|
||||
For example, to sign everything for Fedora 13 Alpha we would issue:
|
||||
+
|
||||
::::
|
||||
$ ./sigulsign_unsigned.py -vv --tag dist-f13 fedora-13
|
||||
+
|
||||
This signs the packages with verbose output so you can track progress
|
||||
incrementally.
|
||||
|
||||
=== Verification
|
||||
|
||||
Once the signing is done, use `rpmdev-checksig` to verify that a package
|
||||
has been signed. You can use the output of a recent rawhide compose to
|
||||
test. In this example we use a released Fedora 12 package:
|
||||
|
||||
....
|
||||
$ rpmdev-checksig /pub/fedora/linux/releases/12/Everything/i386/os/Packages/pungi-2.0.20-1.fc12.noarch.rpm
|
||||
/pub/fedora/linux/releases/12/Everything/i386/os/Packages/pungi-2.0.20-1.fc12.noarch.rpm: MISSING KEY - 57bbccba
|
||||
....
|
||||
|
||||
This output shows that the apckage was signed with key `57bbccba`, and
|
||||
that this key does not exist in your local rpm database. If the key did
|
||||
exist in the local rpm database it's likely there would be no output so
|
||||
it's best to run this on a system that does not have gpg keys imported.
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
== Remove dist-git branches
|
||||
|
||||
=== Description
|
||||
|
||||
Release Engineering is often asked by maintainers to remove branches in
|
||||
dist-git by maintainers.
|
||||
|
||||
=== Action
|
||||
|
||||
. Log into batcave01
|
||||
+
|
||||
....
|
||||
ssh <fas-username>@batcave01.iad2.fedoraproject.org
|
||||
....
|
||||
. Get root shell
|
||||
. Log into pkgs01.iad2.fedoraproject.org :
|
||||
+
|
||||
....
|
||||
ssh pkgs01.iad2.fedoraproject.org
|
||||
....
|
||||
. Change to the package's directory
|
||||
+
|
||||
....
|
||||
cd /srv/git/rpms/<package>.git/
|
||||
....
|
||||
. Remove the branch
|
||||
+
|
||||
....
|
||||
git branch -D <branchname> </pre>
|
||||
....
|
||||
|
||||
=== Verification
|
||||
|
||||
To verify just list the branches.
|
||||
|
||||
....
|
||||
git branch
|
||||
....
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
Make sure that the branch in question isn't one of our pre-created
|
||||
branches `f??/rawhide`, `olpc?/rawhide`, `el?/rawhide`
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
== Requesting Automation Users
|
||||
|
||||
[[sop_requesting_task_automation_user]]
|
||||
=== Description
|
||||
|
||||
When performing automated Release Engineering tasks using `RelEng
|
||||
Automation <_releng-automation>` you will sometimes find that you need
|
||||
to perform an action in the Infrastructure with `sudo` that does not yet
|
||||
have an automation user associated with it.
|
||||
|
||||
=== Actions
|
||||
|
||||
==== Requesting a new loopabull user
|
||||
|
||||
File a ticket with https://pagure.io/fedora-infrastructure/[Fedora
|
||||
Infrastructure] making sure to satisfy the following requirements:
|
||||
|
||||
* Provide a justification for these permissions being needed (What are
|
||||
you trying to do and why?)
|
||||
* Commands needing to be run with sudo
|
||||
* Destination server on which the commands need to be run
|
||||
* The `loopabull_` username requested to be created for this OR which
|
||||
`loopabull_` username needs it's pre-existing permissions enhanced
|
||||
|
||||
For reference:
|
||||
https://pagure.io/fedora-infrastructure/issue/5943[Example
|
||||
Infrastructure Ticket]
|
||||
|
|
@ -0,0 +1,83 @@
|
|||
== Retire Orphaned Packages
|
||||
|
||||
=== Description
|
||||
|
||||
Every release prior to the
|
||||
https://fedoraproject.org/wiki/Schedule[Feature Freeze/Branching]
|
||||
Release Engineering retires
|
||||
https://fedoraproject.org/wiki/Orphaned_package_that_need_new_maintainers[orphaned
|
||||
packages]. This keeps out unowned software and prevents future problems
|
||||
down the road.
|
||||
|
||||
=== Action
|
||||
|
||||
The orphan process takes place in stages:
|
||||
|
||||
. Detecting a list of orphans and the dependencies that will be broken
|
||||
if the orphans are removed.
|
||||
. Sending the list of potential orphans to devel@lists.fedoraproject.org
|
||||
for community review and removal from the orphan list.
|
||||
. Retriring packages nobody wants to adopt.
|
||||
|
||||
==== Detecting Orphans
|
||||
|
||||
A script called `find_unblocked_orphans.py` assists in the detection
|
||||
process. It should be run on a machine that has `koji` and
|
||||
`python-fedora` installed. It runs without options and takes a while to
|
||||
complete.
|
||||
|
||||
`find_unblocked_orphans.py` is available in the
|
||||
https://pagure.io/releng[Release Engineering git repository]
|
||||
|
||||
==== Announcing Packages to be retired
|
||||
|
||||
`find_unblocked_orphans.py` outputs text to stdout on the command line
|
||||
in a form suitable for the body of an email message.
|
||||
|
||||
....
|
||||
$ ./find-unblocked-orphans.py > email-message
|
||||
....
|
||||
|
||||
Email the output to the development list
|
||||
(`devel@lists.fedodraproject.org`) at least a month before the feature
|
||||
freeze, send mails with updated lists as necessary. This gives
|
||||
maintainers an opportunity to pick up orphans that are important to them
|
||||
or are required by other packages.
|
||||
|
||||
==== Retiring Orphans
|
||||
|
||||
Once maintainers have been given an opportunity to pick up orphaned
|
||||
packages, the remaining
|
||||
https://fedoraproject.org/wiki/How_to_remove_a_package_at_end_of_life[packages
|
||||
are retired]
|
||||
|
||||
===== Bugs
|
||||
|
||||
This procedure probably leaves open bugs for the retired packages
|
||||
behind. It is not within the scope of release engineering to take care
|
||||
of these. If bugs are closed, only bugs targeted at Rawhide should be
|
||||
affected, since other branches might still be maintained.
|
||||
|
||||
=== Verification
|
||||
|
||||
To verify that the packages were blocked correctly we can use the
|
||||
`latest-pkg` `koji` action.
|
||||
|
||||
....
|
||||
$ koji latest-pkg dist-f21 wdm
|
||||
....
|
||||
|
||||
This should return nothing, as the `wdm` package is blocked.
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
Generally we retire anything that doesn't leave broken dependencies. If
|
||||
there are orphans whose removal would result in broken dependencies a
|
||||
second warning should be sent to `devel@lists.fedoraproject.org` and to
|
||||
`<package>-owner@fedoraproject.org` for each dependent package.
|
||||
|
||||
Allow another couple of days for maintainers to take notice and fix
|
||||
these package so the package repository can be maintained without broken
|
||||
dependencies or needing to the package. It is not good to have broken
|
||||
package dependencies in our package repositories so every effort should
|
||||
be made to find owners or to fix the broken dependencies.
|
||||
29
modules/release_guide/pages/sop_signing_builds.adoc
Normal file
29
modules/release_guide/pages/sop_signing_builds.adoc
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
== Sign the packages
|
||||
|
||||
* This doc explains how to sign builds in the release(s).
|
||||
* Manual signing should rarely ever be needed anymore. Just make sure
|
||||
that robosignatory is setup for all tags that are created.
|
||||
* If a build seems to be stuck in the autosigning queue (one of the
|
||||
-pending or -signing-pending tags), just koji untag and koji tag the
|
||||
package. This will retrigger autosigning.
|
||||
* If bodhi is reporting a build as unsigned but the build is not in the
|
||||
-signing-pending tag, that means bodhi missed the tagging. Just run the
|
||||
following command to make the build get retagged again, giving Bodhi
|
||||
another change at seeing the signing
|
||||
+
|
||||
....
|
||||
$ koji move $dist-updates-testing-pending $dist-signing-pending $build
|
||||
....
|
||||
* If need be, sign builds using scripts/sigulsign_unsigned.py from
|
||||
releng git repo
|
||||
+
|
||||
_NOTE! This will NOT help if Bodhi marks a build as unsigned!_
|
||||
+
|
||||
....
|
||||
$ ./sigulsign_unsigned.py -vv --write-all \
|
||||
--sigul-batch-size=25 fedora-22 \
|
||||
$(cat /var/cache/sigul/Stable-F22 /var/cache/sigul/Testing-F22)
|
||||
....
|
||||
|
||||
(Make sure you sign each release with the right key... ie, 'fedora-19'
|
||||
key with F19 packages, or 'epel-6' with EL-6 packages)
|
||||
57
modules/release_guide/pages/sop_sigul_client_setup.adoc
Normal file
57
modules/release_guide/pages/sop_sigul_client_setup.adoc
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
== Sigul Client Setup
|
||||
|
||||
This document describes how to configure a sigul client. For more
|
||||
information on sigul, please see link:User-Mitr[User:Mitr]
|
||||
|
||||
=== Prerequisites
|
||||
|
||||
. Install `sigul` and its dependencies. It is available in both Fedora
|
||||
and EPEL:
|
||||
+
|
||||
On Fedora:
|
||||
+
|
||||
....
|
||||
dnf install sigul
|
||||
....
|
||||
+
|
||||
On RHEL/CentOS (Using EPEL):
|
||||
+
|
||||
....
|
||||
yum install sigul
|
||||
....
|
||||
. Ensure that your koji certificate and the link:Fedora-Cert[Fedora CA
|
||||
certificates] are present on the system you're running the sigul client
|
||||
from at the following locations:
|
||||
* `~/.fedora.cert`
|
||||
* `~/.fedora-server-ca.cert`
|
||||
* `~/.fedora-upload-ca.cert`
|
||||
. Admin privileges on koji are required to write signatures.
|
||||
|
||||
=== Configuration
|
||||
|
||||
. Run `sigul_setup_client`
|
||||
. Choose a password for your NSS database. By default this will be
|
||||
stored on-disk in `~/.sigul/client.conf`.
|
||||
. Choose an export password. You will only need to remember it until
|
||||
finishing `sigul_setup_client`.
|
||||
. Enter the DB password you chose earlier, then the export password. You
|
||||
should see the message `pk12util: PKCS12 IMPORT SUCCESSFUL`
|
||||
. Enter the DB password again. You should see the message `Done`.
|
||||
. Assuming that you are running the sigul client within phx2, edit
|
||||
`~/.sigul/client.conf` to include the following lines:
|
||||
|
||||
....
|
||||
[client]
|
||||
bridge-hostname: sign-bridge.phx2.fedoraproject.org
|
||||
server-hostname: sign-vault.phx2.fedoraproject.org
|
||||
....
|
||||
|
||||
=== Updating your Fedora certificate
|
||||
|
||||
When your Fedora certificate expires, after updating it run the
|
||||
following commands:
|
||||
|
||||
....
|
||||
$ certutil -d ~/.sigul -D -n sigul-client-cert
|
||||
$ sigul_setup_client
|
||||
....
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
== Stage Final Release for Mirrors
|
||||
|
||||
=== Description
|
||||
|
||||
When the release has been fully tested and approved at the "Go/No-Go"
|
||||
meeting it is ready for release to the Fedora mirrors.
|
||||
|
||||
=== Action
|
||||
|
||||
. Gather the needed info for running the staging script: Release
|
||||
Version: the numerical version number of the release `24` ComposeID: The
|
||||
ID of the Compose Label: Compsoe label for the location in stage
|
||||
`24_RC-1.2` for example Key: the name of teh release key `fedora-24` or
|
||||
`fedora-24-secondary` as examples Prerelease: 0 or 1 sets if the release
|
||||
goes in test/ or not Arch: <optional> For secondary arches, changes some
|
||||
internal locations
|
||||
+
|
||||
....
|
||||
$ scripts/stage-release.sh 24 Fedora-24-20160614.0 24_RC-1.2 fedora-24 0
|
||||
....
|
||||
. Sync the release to the Red Hat internal archive following internally
|
||||
documented
|
||||
|
||||
=== Check and set EOL on previous releases to reflect reality
|
||||
|
||||
. check PDC for active releases and respective EOL date
|
||||
. if needed run the adjust-eol-all.py script from releng repository to
|
||||
correct any mistakes
|
||||
|
||||
=== Verification
|
||||
|
||||
Verification is somewhat difficult as one cannot look at the content via
|
||||
the web server due to permissions. Typically we ask somebody from the
|
||||
Infrastructure team to give the tree a second set of eyes.
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
Hope the release is good!
|
||||
22
modules/release_guide/pages/sop_template.adoc
Normal file
22
modules/release_guide/pages/sop_template.adoc
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
== Standard Operating Procedure Template
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
This is a template file, this can be copied to a new filename in order
|
||||
to start a new document.
|
||||
|
||||
New SOP Documents should be added to the `.. toctree::` list in the
|
||||
`sop.rst` file.
|
||||
|
||||
These documents are formatted using sphinx-doc reStructure Text, for
|
||||
markup reference please see: http://sphinx-doc.org/rest.html
|
||||
====
|
||||
|
||||
=== Description
|
||||
|
||||
=== Action
|
||||
|
||||
=== Verification
|
||||
|
||||
=== Consider Before Running
|
||||
165
modules/release_guide/pages/sop_unretire.adoc
Normal file
165
modules/release_guide/pages/sop_unretire.adoc
Normal file
|
|
@ -0,0 +1,165 @@
|
|||
== Unretiring a package branch
|
||||
|
||||
=== Description
|
||||
|
||||
Sometimes, packagers request that we _unretire_ a package branch that
|
||||
has previously been retired.
|
||||
|
||||
This typically happens on the [.title-ref]#rawhide# branch, but could
|
||||
conceivably happen on any stable or arbitrary branch.
|
||||
|
||||
=== Action
|
||||
|
||||
==== Validate Package Ready for Unretirement
|
||||
|
||||
. Verify the package was not retired for any reason, such as legal or
|
||||
license issues, that would prevent it from being re-instated.
|
||||
. Ensure a bugzilla was filed to review the package for unretirement.
|
||||
. Verify with the the requestor exactly which tags they would like
|
||||
unblocked as part of the unretirement request.
|
||||
|
||||
==== Revert the Retirement Commit
|
||||
|
||||
. Connect to one of the compose systems.
|
||||
+
|
||||
____
|
||||
....
|
||||
$ ssh compose-x86-02.phx2.fedoraproject.org
|
||||
....
|
||||
____
|
||||
. Clone the git-dist package using the the proper release engineering
|
||||
credentials.
|
||||
+
|
||||
____
|
||||
....
|
||||
$ GIT_SSH=/usr/local/bin/relengpush fedpkg --user releng clone PACKAGENAME
|
||||
....
|
||||
____
|
||||
. Enter the directory of the cloned package and configure the git user
|
||||
information.
|
||||
+
|
||||
____
|
||||
....
|
||||
$ cd PACKAGENAME
|
||||
$ git config --local user.name "Fedora Release Engineering"
|
||||
$ git config --local user.email "releng@fedoraproject.org"
|
||||
....
|
||||
____
|
||||
. Git revert the [.title-ref]#dead.package# file commit in dist-git on
|
||||
the particular branch using its commit hash_id. Ensure the commit
|
||||
message contains a URL to the request in pagure.
|
||||
+
|
||||
____
|
||||
....
|
||||
$ git revert -s COMMIT_HASH_ID
|
||||
$ GIT_SSH=/usr/loca/bin/relengpush fedpkg --user releng push
|
||||
....
|
||||
____
|
||||
|
||||
==== Unblock the Package in Koji
|
||||
|
||||
. Check the current state of the branches in koji for the package.
|
||||
+
|
||||
____
|
||||
....
|
||||
$ koji list-pkgs --show-blocked --package=PACKAGENAME
|
||||
....
|
||||
____
|
||||
. Unblock each requested tag using koji.
|
||||
+
|
||||
....
|
||||
$ koji unblock-pkg TAGNAME PACKAGENAME
|
||||
....
|
||||
|
||||
==== Verify Package is Not Orphaned
|
||||
|
||||
. Check package ownership
|
||||
+
|
||||
Navigate to [.title-ref]#https://src.fedoraproject.org/# and check
|
||||
package owner.
|
||||
. If the package is orphaned, then give the package to the requestor
|
||||
using the [.title-ref]#give-package# script from the
|
||||
https://pagure.io/releng[Release Engineering Repo].
|
||||
+
|
||||
....
|
||||
$ ./scripts/distgit/give-package --package=PACKAGENAME --custodian=REQUESTOR
|
||||
....
|
||||
+
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
This script requires the user to be a member of the group
|
||||
[.title-ref]#cvsadmin# in FAS.
|
||||
====
|
||||
|
||||
==== Update Product Definition Center (PDC)
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
If there are more than one tag to be unblocked then the PDC update step
|
||||
should be completed for each tag and package.
|
||||
====
|
||||
|
||||
. Log into the https://pdc.fedoraproject.org/[Fedora PDC instance] using
|
||||
a FAS account.
|
||||
. Check PDC for the entry for the [.title-ref]#PACKAGENAME# in each
|
||||
[.title-ref]#TAG# that was unblocked in a previous step.
|
||||
+
|
||||
____
|
||||
....
|
||||
https://pdc.fedoraproject.org/rest_api/v1/component-branch-slas/?branch=TAG&global_component=PACKAGENAME
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
If no information is returned by this query then it is not in PDC and is
|
||||
likely not yet a branch. The requestor should use the
|
||||
[.title-ref]#fedpkg request-branch# utility to ask for a branch.
|
||||
====
|
||||
____
|
||||
. If the package existed within PDC then obtain a token from the PDC
|
||||
site while logged in by navigating to the
|
||||
[.title-ref]#https://pdc.fedoraproject.org/rest_api/v1/auth/token/obtain/#
|
||||
URL with the Firefox web browser.
|
||||
. Press F12 once the page has loaded and select the tab labeled
|
||||
[.title-ref]#Network#. Refresh the web page and find the line whose
|
||||
string in the file column matches
|
||||
[.title-ref]#/rest_api/v1/auth/token/obtain/#.
|
||||
. Right click on specified line and select Copy>Copy as cURL. Paste this
|
||||
into a terminal session and add [.title-ref]#-H "Accept:
|
||||
application/json"#. It should look something similar to the below:
|
||||
+
|
||||
____
|
||||
....
|
||||
$ curl 'https://pdc.fedoraproject.org/rest_api/v1/auth/token/obtain/' \
|
||||
-H 'Host: pdc.fedoraproject.org' \
|
||||
-H .0) Gecko/20100101 Firefox/57.0' \
|
||||
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
|
||||
-H 'Accept-Language: en-US,en;q=0.5' \
|
||||
--compressed \
|
||||
-H 'Cookie: csrftoken=CSRF_TOKEN_HASH; SERVERID=pdc-web01; mellon-saml-sesion-cookie=SAML_SESSION_HASH; sessionid=SESSION_ID_HASH' \
|
||||
-H 'Connection: keep-alive' \
|
||||
-H 'Upgrade-Insecure-Requests: 1' \
|
||||
-H 'Cache-Control: max-age=0' \
|
||||
-H "Accept: application/json"
|
||||
....
|
||||
____
|
||||
. Using the token obtained from the previous step run the
|
||||
[.title-ref]#adjust-eol.py# script from the
|
||||
https://pagure.io/releng[Release Engineering Repo].
|
||||
+
|
||||
____
|
||||
....
|
||||
$ PYTHONPATH=scripts/pdc/ scripts/pdc/adjust-eol.py fedora MYTOKEN PACKAGENAME rpm TAG default -y
|
||||
....
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
The local machine will have configuration information in the
|
||||
[.title-ref]#/etc/pdc.d/# directory. This is why _fedora_ can be passed
|
||||
as an argument instead of the full API endpoint URL.
|
||||
====
|
||||
____
|
||||
55
modules/release_guide/pages/sop_update_critpath.adoc
Normal file
55
modules/release_guide/pages/sop_update_critpath.adoc
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
== Update Critpath
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
Critpath = "Critical Path"
|
||||
|
||||
This is a collection of packages deemed "critical" to Fedora
|
||||
====
|
||||
|
||||
=== Description
|
||||
|
||||
PDC has information about which packages are critpath and which are not.
|
||||
A script that reads the yum repodata (critpath group in comps, and the
|
||||
package dependencies) is used to generate this. Since package
|
||||
dependencies change, this list should be updated periodically.
|
||||
|
||||
=== Action
|
||||
|
||||
. Release engineering scripts for updating critpath live in the
|
||||
https://pagure.io/releng[releng git repository].
|
||||
. Check the critpath.py script to see if the list of releases needs to
|
||||
be updated:
|
||||
+
|
||||
....
|
||||
for r in ['12', '13', '14', '15', '16', '17']: # 13, 14, ...
|
||||
releasepath[r] = 'releases/%s/Everything/$basearch/os/' % r
|
||||
updatepath[r] = 'updates/%s/$basearch/' % r
|
||||
|
||||
# Branched Fedora goes here
|
||||
branched = '18'
|
||||
....
|
||||
+
|
||||
The for loop has the version numbers for releases that have gone past
|
||||
final. branched has the release that's been branched from rawhide but
|
||||
not yet hit final. (These have different paths in the repository and may
|
||||
not have an updates directory, thus they're in separate sections).
|
||||
. Run the script with the release to generate info for (for a release
|
||||
that's hit final, this is the release number example: "17". For
|
||||
branched, it's "branched").
|
||||
+
|
||||
....
|
||||
./critpath.py --srpm -o critpath.txt branched
|
||||
....
|
||||
. Run the update script to add that to PDC:
|
||||
+
|
||||
....
|
||||
./update-critpath --user toshio f18 critpath.txt
|
||||
....
|
||||
+
|
||||
The username is your fas username. You must be in cvsadmin to be able to
|
||||
change this. The branch is the name of the dist-git branch. critpath.txt
|
||||
is the file that the output of critpath.py went into. The script needs a
|
||||
PDC token to talk to the server, configured in /etc/pdc.d/. See the PDC
|
||||
SOP for more info.
|
||||
54
modules/release_guide/pages/sop_update_releng_docs.adoc
Normal file
54
modules/release_guide/pages/sop_update_releng_docs.adoc
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
== Update RelEng Rendered Docs
|
||||
|
||||
=== Description
|
||||
|
||||
When an improvement happens to the Release Engineering documentation
|
||||
following the `contributing <contributing>` for the
|
||||
http://sphinx-doc.org/[Sphinx]
|
||||
https://en.wikipedia.org/wiki/ReStructuredText[reStructured Text] source
|
||||
found in `docs/source` within the https://pagure.io/releng[RelEng git
|
||||
repository] someone has to manually perform a process in order to update
|
||||
the documentation that is hosted in the https://pagure.io/pagure[pagure]
|
||||
documentation space for https://docs.pagure.org/releng/[Fedora RelEng
|
||||
docs].
|
||||
|
||||
=== Action
|
||||
|
||||
In order to render the documentation using
|
||||
http://sphinx-doc.org/[Sphinx], you need to first be sure to have the
|
||||
package installed:
|
||||
|
||||
....
|
||||
$ dnf install python-sphinx
|
||||
....
|
||||
|
||||
Then we'll need to clone the RelEng repository and the RelEng docs
|
||||
repository (the docs git repository is provided by pagure
|
||||
automatically). There is a script in the [.title-ref]#releng# repository
|
||||
that takes care of cleanly updating the documentation site for us.
|
||||
|
||||
....
|
||||
$ ./scripts/update-docs.sh
|
||||
....
|
||||
|
||||
The documentation is now live.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
This will require someone with permissions to push to the rawhide branch
|
||||
for the releng repository. If you are curious whom all has this ability,
|
||||
please refer to the `Main Page <index>` and contact someone from the
|
||||
"Team Composition"
|
||||
====
|
||||
|
||||
=== Verification
|
||||
|
||||
Visit the https://docs.pagure.org/releng/[Fedora RelEng docs] website
|
||||
and verify that the changes are reflected live on the docs site.
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
No considerations at this time. The docs git repository is simply a
|
||||
static html hosting space and we can just re-render the docs and push to
|
||||
it again if necessary.
|
||||
53
modules/release_guide/pages/sop_updating_comps.adoc
Normal file
53
modules/release_guide/pages/sop_updating_comps.adoc
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
== Updating Comps
|
||||
|
||||
=== Description
|
||||
|
||||
When we start a new Fedora development cycle (when we branch rawhide) we
|
||||
have to create a new comps file for the new release. This SOP covers
|
||||
that action.
|
||||
|
||||
=== Action
|
||||
|
||||
. clone the comps repo
|
||||
+
|
||||
....
|
||||
$ git clone ssh://git@pagure.io/fedora-comps.git
|
||||
....
|
||||
. Create the new comps file for next release:
|
||||
+
|
||||
....
|
||||
$ cp comps-f24.xml.in comps-f25.xml.in
|
||||
....
|
||||
. Edit Makefile to update comps-rawhide target
|
||||
+
|
||||
....
|
||||
- -comps-rawhide: comps-f24.xml
|
||||
- - @mv comps-f24.xml comps-rawhide.xml
|
||||
+comps-rawhide: comps-f25.xml
|
||||
+ @mv comps-f25.xml comps-rawhide.xml
|
||||
....
|
||||
. Add the new comps file to source control:
|
||||
+
|
||||
....
|
||||
$ git add comps-f25.xml.in
|
||||
....
|
||||
. Edit the list of translated comps files in po/POTFILES.in to reflect
|
||||
currently supported releases.
|
||||
+
|
||||
....
|
||||
-comps-f22.xml
|
||||
+comps-f25.xml
|
||||
....
|
||||
. Send it up:
|
||||
+
|
||||
::::
|
||||
$ git push
|
||||
|
||||
=== Verification
|
||||
|
||||
One can review the logs for rawhide compose after this change to make
|
||||
sure the right comps file was used.
|
||||
|
||||
=== Consider Before Running
|
||||
|
||||
Nothing yet.
|
||||
260
modules/release_guide/pages/troubleshooting.adoc
Normal file
260
modules/release_guide/pages/troubleshooting.adoc
Normal file
|
|
@ -0,0 +1,260 @@
|
|||
== Fedora Release Engineering Troubleshooting Guide
|
||||
|
||||
Fedora Release Engineering consists of many different systems, many
|
||||
different code bases and multiple tools. Needless to say, things can get
|
||||
pretty complex in a hurry. This aspect of Fedora Release Engineering is
|
||||
not very welcoming to newcomers who would like to get involved. This
|
||||
guide stands as a place to educate those new to the processes, systems,
|
||||
code bases, and tools. It also is to serve as a reference to those who
|
||||
aren't new but maybe are fortunate enough to not have needed to diagnose
|
||||
things in recent memory.
|
||||
|
||||
We certainly won't be able to document every single possible compontent
|
||||
in the systems that could go wrong but hopefully over time this document
|
||||
will stand as a proper knowledge base for reference and educational
|
||||
purposes on the topics listed below.
|
||||
|
||||
=== Compose
|
||||
|
||||
If something with a compose has gone wrong, there's a number of places
|
||||
to find information. Each of these are discussed below.
|
||||
|
||||
==== releng-cron list
|
||||
|
||||
The compose output logs are emailed to the releng-cron mailing list. It
|
||||
is good practice to check the
|
||||
https://lists.fedoraproject.org/archives/list/releng-cron@lists.fedoraproject.org/[releng-cron
|
||||
mailing list archives] and find the latest output and give it a look.
|
||||
|
||||
==== compose machines
|
||||
|
||||
If the
|
||||
https://lists.fedoraproject.org/archives/list/releng-cron@lists.fedoraproject.org/[releng-cron
|
||||
mailing list archives] doesn't prove to be useful, you can move on to
|
||||
checking the contents of the composes themselves on the primary compose
|
||||
machines in the Fedora Infrastructure. At the time of this writing,
|
||||
there are multiple machines based on the specific compose you are
|
||||
looking for:
|
||||
|
||||
* Two-Week Atomic Nightly Compose
|
||||
** `compose-x86-01.phx2.fedoraproject.org`
|
||||
* Branched Compose
|
||||
** `branched-composer.phx2.fedoraproject.org`
|
||||
* Rawhide Compose
|
||||
** `rawhide-composer.phx2.fedoraproject.org`
|
||||
|
||||
Depending on which specific compose you are in search of will depend on
|
||||
what full path you will end up inspecting:
|
||||
|
||||
* For Two Week Atomic you will find the compose output in
|
||||
`/mnt/fedora_koji/compose/`
|
||||
* For Release Candidate / Test Candidate composes you will find compose
|
||||
output in `/mnt/koji/compose/`
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
It's possible the mock logs are no longer available. The mock chroots
|
||||
are rewritten on subsequent compose runs.
|
||||
====
|
||||
|
||||
You can also check for mock logs if they are still persisting from the
|
||||
compose you are interested in. Find the specific mock chroot directory
|
||||
name (that will reside in `/var/lib/mock/`) you are looking for by
|
||||
checking the appropriate compose mock configuration (the following is
|
||||
only a sample provided at the time of this writing):
|
||||
|
||||
....
|
||||
$ ls /etc/mock/*compose*
|
||||
/etc/mock/fedora-22-compose-aarch64.cfg /etc/mock/fedora-branched-compose-aarch64.cfg
|
||||
/etc/mock/fedora-22-compose-armhfp.cfg /etc/mock/fedora-branched-compose-armhfp.cfg
|
||||
/etc/mock/fedora-22-compose-i386.cfg /etc/mock/fedora-branched-compose-i386.cfg
|
||||
/etc/mock/fedora-22-compose-x86_64.cfg /etc/mock/fedora-branched-compose-x86_64.cfg
|
||||
/etc/mock/fedora-23-compose-aarch64.cfg /etc/mock/fedora-rawhide-compose-aarch64.cfg
|
||||
/etc/mock/fedora-23-compose-armhfp.cfg /etc/mock/fedora-rawhide-compose-armhfp.cfg
|
||||
/etc/mock/fedora-23-compose-i386.cfg /etc/mock/fedora-rawhide-compose-i386.cfg
|
||||
/etc/mock/fedora-23-compose-x86_64.cfg /etc/mock/fedora-rawhide-compose-x86_64.cfg
|
||||
....
|
||||
|
||||
==== running the compose yourself
|
||||
|
||||
If you happen to strike out there and are still in need of debugging, it
|
||||
might be time to just go ahead and run the compose yourself. The exact
|
||||
command needed can be found in the cron jobs located on their respective
|
||||
compose machines, this information can be found in the
|
||||
`compose-machines` section. Also note that each respective compose
|
||||
command should be ran from it's respective compose machine as outlined
|
||||
in the `compose-machines` section.
|
||||
|
||||
An example is below, setting the compose directory as your
|
||||
`username-debug.1`, the `.1` at the end is common notation for an
|
||||
incremental run of a compose. If you need to do another, increment it to
|
||||
`.2` and continue. This is helpful to be able to compare composes.
|
||||
|
||||
[NOTE]
|
||||
.Note
|
||||
====
|
||||
It is recommended that the following command be run in
|
||||
https://www.gnu.org/software/screen/[screen] or
|
||||
https://tmux.github.io/[tmux]
|
||||
====
|
||||
|
||||
....
|
||||
$ TMPDIR=`mktemp -d /tmp/twoweek.XXXXXX` && cd $TMPDIR \
|
||||
&& git clone -n https://pagure.io/releng.git && cd releng && \
|
||||
git checkout -b stable twoweek-stable && \
|
||||
LANG=en_US.UTF-8 ./scripts/make-updates 23 updates $USER-debug.1
|
||||
....
|
||||
|
||||
The above command was pulled from the `twoweek-atomic` cron job with
|
||||
only the final parameter being altered. This is used as the name of the
|
||||
output directory.
|
||||
|
||||
The compose can take some time to run, so don't be alarmed if you don't
|
||||
see output in a while. This should provide you all the infromation
|
||||
needed to debug and/or diagnose further. When in doubt, as in
|
||||
`#fedora-releng` on `irc.libera.chat`.
|
||||
|
||||
=== Docker Layered Image Build Service
|
||||
|
||||
The
|
||||
https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service[Docker
|
||||
Layered Image Build Service] is built using a combination of
|
||||
technologies such as https://www.openshift.org/[OpenShift],
|
||||
https://github.com/projectatomic/osbs-client[osbs-client], and the
|
||||
https://github.com/release-engineering/koji-containerbuild[koji-containerbuild]
|
||||
plugin that when combined are often refered to as an OpenShift Build
|
||||
Service instance (OSBS). Something to note is that
|
||||
https://www.openshift.org/[OpenShift] is a
|
||||
http://kubernetes.io/[kubernetes] distribution with many features built
|
||||
on top of http://kubernetes.io/[kubernetes] such as the
|
||||
https://docs.openshift.org/latest/dev_guide/builds.html[build primitive]
|
||||
that is used as the basis for the build service. This information will
|
||||
hopefully shed light on some of the terminology and commands used below.
|
||||
|
||||
There are a few "common" scenarios in which build may fail or hang that
|
||||
will need some sort of inspection of the build system.
|
||||
|
||||
==== Build Appears to stall after being scheduled
|
||||
|
||||
In the event that a build scheduled through koji appears to be stalled
|
||||
and is not in a `free` state (i.e. - has been scheduled). An
|
||||
administrator will need to ssh into `osbs-master01` or
|
||||
`osbs-master01.stg` (depending stage vs production) and inspect the
|
||||
build itself.
|
||||
|
||||
....
|
||||
$ oc status
|
||||
In project default on server https://10.5.126.216:8443
|
||||
|
||||
svc/kubernetes - 172.30.0.1 ports 443, 53, 53
|
||||
|
||||
bc/cockpit-f24 custom build of git://....
|
||||
build #8 succeeded 7 weeks ago
|
||||
build #9 failed 33 minutes ago
|
||||
|
||||
$ oc describe build/cockpit-f24-9
|
||||
# lots of output about status of the specific build
|
||||
|
||||
$ oc logs build/cockpit-f24-9
|
||||
# extremely verbose logs, these should in normal circumstances be found in
|
||||
# the koji build log post build
|
||||
....
|
||||
|
||||
The information found in the commands above will generally identify the
|
||||
issue.
|
||||
|
||||
==== Build fails but there's no log output in the Koji Task
|
||||
|
||||
Sometimes there is a communications issue between Koji and OSBS which
|
||||
cause for a failure to be listed in Koji but without all the logs. These
|
||||
can be diagnosed by checking the `kojid` logs on the Koji builder listed
|
||||
in the task output.
|
||||
|
||||
For example:
|
||||
|
||||
....
|
||||
$ fedpkg container-build
|
||||
Created task: 90123598
|
||||
Task info: http://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90123598
|
||||
Watching tasks (this may be safely interrupted)...
|
||||
90123598 buildContainer (noarch): free
|
||||
90123598 buildContainer (noarch): free -> open (buildvm-04.stg.phx2.fedoraproject.org)
|
||||
90123599 createContainer (x86_64): free
|
||||
90123599 createContainer (x86_64): free -> open (buildvm-02.stg.phx2.fedoraproject.org)
|
||||
90123599 createContainer (x86_64): open (buildvm-02.stg.phx2.fedoraproject.org) -> closed
|
||||
0 free 1 open 1 done 0 failed
|
||||
90123598 buildContainer (noarch): open (buildvm-04.stg.phx2.fedoraproject.org) -> FAILED: Fault: <Fault 2001: 'Image build failed. OSBS build id: cockpit-f24-9'>
|
||||
0 free 0 open 1 done 1 failed
|
||||
|
||||
90123598 buildContainer (noarch) failed
|
||||
....
|
||||
|
||||
In this example the buildContiner task was scheduled and ran on
|
||||
`buildvm-04.stg` with the actual createContainer task being on
|
||||
`buildvm-02.stg`, and `buildvm-02.stg` is where we're going to want to
|
||||
begin looking for failures to communicate with OSBS as this is the point
|
||||
of contact with the external system.
|
||||
|
||||
Logs can be found in `/var/log/kojid.log` or if necessary, check the
|
||||
koji hub in question. Generally, you will want to start with the first
|
||||
point of contact with OSBS and "work your way back" so in the above
|
||||
example you would first check `buildvm-02.stg`, then move on to
|
||||
`buildvm-04.stg` if nothing useful was found in the logs of the previous
|
||||
machine, and again move on to the koji hub if neither of the builder
|
||||
machines involved provided useful log information.
|
||||
|
||||
==== Build fails because it can't get to a network resource
|
||||
|
||||
Sometimes there is a situation where the firewall rules get messed up on
|
||||
one of the OpenShift Nodes in the environment. This can cause output
|
||||
similar to the following:
|
||||
|
||||
....
|
||||
$ fedpkg container-build --scratch
|
||||
Created task: 90066343
|
||||
Task info: http://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90066343
|
||||
Watching tasks (this may be safely interrupted)...
|
||||
90066343 buildContainer (noarch): free
|
||||
90066343 buildContainer (noarch): free -> open (buildvm-03.stg.phx2.fedoraproject.org)
|
||||
90066344 createContainer (x86_64): open (buildvm-04.stg.phx2.fedoraproject.org)
|
||||
90066344 createContainer (x86_64): open (buildvm-04.stg.phx2.fedoraproject.org) -> FAILED: Fault: <Fault 2001: "Image build failed. Error in plugin distgit_fetch_artefacts: OSError(2, 'No such file or directory'). OSBS build id: scratch-20161102132628">
|
||||
0 free 1 open 0 done 1 failed
|
||||
90066343 buildContainer (noarch): open (buildvm-03.stg.phx2.fedoraproject.org) -> closed
|
||||
0 free 0 open 1 done 1 failed
|
||||
....
|
||||
|
||||
If we go to the OSBS Master and run the following commands, we will see
|
||||
the root symptom:
|
||||
|
||||
....
|
||||
# oc logs build/scratch-20161102132628
|
||||
Error from server: Get https://osbs-node02.stg.phx2.fedoraproject.org:10250/containerLogs/default/scratch-20161102132628-build/custom-build: dial tcp 10.5.126.213:10250: getsockopt: no route to host
|
||||
|
||||
# ping 10.5.126.213
|
||||
PING 10.5.126.213 (10.5.126.213) 56(84) bytes of data.
|
||||
64 bytes from 10.5.126.213: icmp_seq=1 ttl=64 time=0.299 ms
|
||||
64 bytes from 10.5.126.213: icmp_seq=2 ttl=64 time=0.299 ms
|
||||
64 bytes from 10.5.126.213: icmp_seq=3 ttl=64 time=0.253 ms
|
||||
64 bytes from 10.5.126.213: icmp_seq=4 ttl=64 time=0.233 ms
|
||||
^C
|
||||
--- 10.5.126.213 ping statistics ---
|
||||
4 packets transmitted, 4 received, 0% packet loss, time 3073ms
|
||||
rtt min/avg/max/mdev = 0.233/0.271/0.299/0.028 ms
|
||||
|
||||
# http get 10.5.126.213:10250
|
||||
|
||||
http: error: ConnectionError: HTTPConnectionPool(host='10.5.126.213', port=10250): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fdab064b320>: Failed to establish a new connection: [Errno 113] No route to host',)) while doing GET request to URL: http://10.5.126.213:10250/
|
||||
....
|
||||
|
||||
In the above output, we can see that we do actually have network
|
||||
connectivity to the Node but we can not connect to the OpenShift service
|
||||
that should be listening on port `10250`.
|
||||
|
||||
To fix this, you need to ssh into the OpenShift Node that you can't
|
||||
connect to via port `10250` and run the following commands. This should
|
||||
resolve the issue.
|
||||
|
||||
....
|
||||
iptables -F && iptables -t nat -F && systemctl restart docker && systemctl restart origin-node
|
||||
....
|
||||
114
modules/release_guide/pages/workflow_automation.adoc
Normal file
114
modules/release_guide/pages/workflow_automation.adoc
Normal file
|
|
@ -0,0 +1,114 @@
|
|||
== Fedora RelEng Workflow Automation
|
||||
|
||||
The Fedora RelEng Workflow Automation is a means to allow RelEng to
|
||||
define a pattern by which Release Engineering work is automated in an
|
||||
uniform fashion. The automation technology of choice is
|
||||
https://ansible.com/[ansible] and the "workflow engine" is powered by
|
||||
https://github.com/maxamillion/loopabull[loopabull], which is an event
|
||||
loop that allows us to pass the information contained within a
|
||||
http://www.fedmsg.com/en/latest/[fedmsg] and insert it into
|
||||
https://ansible.com/[ansible]
|
||||
https://docs.ansible.com/ansible/playbooks.html[playbooks]. This will
|
||||
effectively create an event driven workflow that can take action
|
||||
conditionally based on the contents of arbitrary
|
||||
http://www.fedmsg.com/en/latest/[fedmsg] data.
|
||||
|
||||
Background on the topic can be found in the
|
||||
https://fedoraproject.org/wiki/Changes/ReleaseEngineeringAutomationWorkflowEngine[Release
|
||||
Engineering Automation Workflow Engine] Change proposal, as well as in
|
||||
the https://pagure.io/releng-automation[releng-automation] pagure
|
||||
repository.
|
||||
|
||||
=== RelEng Workflow Automation Architecture
|
||||
|
||||
By using http://www.fedmsg.com/en/latest/[fedmsg] as the source of
|
||||
information feeding the event loop, we will configure
|
||||
https://github.com/maxamillion/loopabull[loopabull] to listen for
|
||||
specific
|
||||
https://fedora-fedmsg.readthedocs.io/en/latest/topics.html[fedmsg
|
||||
topics] which will correspond with https://ansible.com/[ansible]
|
||||
https://docs.ansible.com/ansible/playbooks.html[playbooks]. When one of
|
||||
the appropriate
|
||||
https://fedora-fedmsg.readthedocs.io/en/latest/topics.html[fedmsg
|
||||
topics] is encountered across the message bus, it's message payload is
|
||||
then injected into the corresponding playbook as an extra set of
|
||||
variables. A member of the Fedora Release Engineering Team can at that
|
||||
point use this as a means to perform whatever arbitrary action or series
|
||||
of actions they can otherwise perform with https://ansible.com/[ansible]
|
||||
(including what we can enable via custom
|
||||
https://docs.ansible.com/ansible/modules.html[modules]) based on the
|
||||
input of the message payload.
|
||||
|
||||
The general overview of the architecture is below as well as a
|
||||
description of how it works:
|
||||
|
||||
....
|
||||
+------------+
|
||||
| fedmsg |
|
||||
| |
|
||||
+---+--------+
|
||||
| ^
|
||||
| |
|
||||
| |
|
||||
| |
|
||||
| |
|
||||
| |
|
||||
V |
|
||||
+------------------+-----------------+
|
||||
| |
|
||||
| Release Engineering |
|
||||
| Workflow Automation Engine |
|
||||
| |
|
||||
| - RabbitMQ |
|
||||
| - fedmsg-rabbitmq-serializer |
|
||||
| - loopabull |
|
||||
| |
|
||||
+----------------+-------------------+
|
||||
|
|
||||
|
|
||||
|
|
||||
|
|
||||
V
|
||||
+-----------------------+
|
||||
| |
|
||||
| composer/bodhi/etc |
|
||||
| |
|
||||
+-----------------------+
|
||||
....
|
||||
|
||||
The flow of data will begin with an event somewhere in the
|
||||
https://fedoraproject.org/wiki/Infrastructure[Fedora Infrastructure]
|
||||
that sends a http://www.fedmsg.com/en/latest/[fedmsg] across the message
|
||||
bus, then the messages will be taken in and serialized in to a
|
||||
https://www.rabbitmq.com/[rabbitmq] worker queue using
|
||||
https://pagure.io/fedmsg-rabbitmq-serializer[fedmsg-rabbitmq-serializer].
|
||||
Then https://github.com/maxamillion/loopabull[loopabull] will be
|
||||
listening to the rabbitmq worker queue for tasks to come in. Once a
|
||||
message is recieved, it is processed and once it is either no-op'd or a
|
||||
corresponding ansible playbook is run to completion, the message will be
|
||||
`ack`'d and cleared from the worker queue. This will allow for us to
|
||||
scale loopabull instances independently from the message queue as well
|
||||
as ensure that work is not lost because of a downed or busy loopabull
|
||||
instance. Also, as a point of note, the loopabull service instances will
|
||||
be scaled using https://freedesktop.org/wiki/Software/systemd/[systemd]
|
||||
https://fedoramagazine.org/systemd-template-unit-files/[unit templates].
|
||||
|
||||
Once a playbook has been triggered, it will run tasks on remote systems
|
||||
on behalf of a loopabull automation user. These users can be privileged
|
||||
if need be, however the scope of their privilege is based on the purpose
|
||||
they serve. These user accounts are provisioned by the
|
||||
https://fedoraproject.org/wiki/Infrastructure[Fedora Infrastructure]
|
||||
Team based on the requirements of the
|
||||
`RelEng Task Automation User Request Standard Operating
|
||||
Procedure (SOP) <sop_requesting_task_automation_user>` document and
|
||||
tasks are subject to code and security audit.
|
||||
|
||||
=== Fedora Lib RelEng
|
||||
|
||||
https://pagure.io/flr[Fedora Lib RelEng] (flr), is a library and set of
|
||||
command line tools to expose the library that aims to provide re-usable
|
||||
code for common tasks that need to be done in Release Engineering.
|
||||
Combining this set of command line tools when necessary with the Release
|
||||
Engineering Automation pipeline allows for easy separation of
|
||||
permissions and responsibilities via sudo permissions on remote hosts.
|
||||
This is explained in more detail on the project's pagure page.
|
||||
Loading…
Add table
Add a link
Reference in a new issue