diff --git a/antora.yml b/antora.yml index 82dcce2..d9d1710 100644 --- a/antora.yml +++ b/antora.yml @@ -16,3 +16,4 @@ nav: - modules/ROOT/nav.adoc - modules/developer_guide/nav.adoc - modules/sysadmin_guide/nav.adoc +- modules/release_guide/nav.adoc diff --git a/modules/release_guide/nav.adoc b/modules/release_guide/nav.adoc new file mode 100644 index 0000000..0204b5b --- /dev/null +++ b/modules/release_guide/nav.adoc @@ -0,0 +1,9 @@ +* xref:release_process.adoc[Release process] +* xref:index.adoc[Release guide] +** xref:mass_rebuild.adoc[Mass Rebuild] +** xref:sop_file_ftbfs.adoc[File FTBFS bugs] +** xref:sop_mass_branching.adoc[Mass Branching] +** xref:sop_bodhi_activation.adoc[Updates testing activation] +** xref:beta_freeze.adoc[Beta freeze] +** xref:beta_release.adoc[Beta release] +** xref:final_freeze.adoc[Final freeze] diff --git a/modules/release_guide/pages/_partials/attributes.adoc b/modules/release_guide/pages/_partials/attributes.adoc new file mode 100644 index 0000000..8f28f68 --- /dev/null +++ b/modules/release_guide/pages/_partials/attributes.adoc @@ -0,0 +1,5 @@ +:year: 2023 +:rawhide: 39 +:branched: 38 +:current: 37 +:old_release: 36 diff --git a/modules/release_guide/pages/architecture.adoc b/modules/release_guide/pages/architecture.adoc new file mode 100644 index 0000000..5cb2ca0 --- /dev/null +++ b/modules/release_guide/pages/architecture.adoc @@ -0,0 +1,12 @@ +[[architecture]] +== Fedora Release Engineering Architecture + +Here you will find an index of documentation that describes the Fedora +Release Engineering Architecture. The goal of this is to provide an +overview of the environment and various systems that are used to build +it as a reference to members of the Fedora Community as well as +materials to help prospective Release Engineering Community members. + +=== Architecture Documents + +layered_image_build_service workflow_automation diff --git a/modules/release_guide/pages/beta_freeze.adoc b/modules/release_guide/pages/beta_freeze.adoc new file mode 100644 index 0000000..4ad2609 --- /dev/null +++ b/modules/release_guide/pages/beta_freeze.adoc @@ -0,0 +1 @@ +== Fedora Beta Freeze \ No newline at end of file diff --git a/modules/release_guide/pages/beta_release.adoc b/modules/release_guide/pages/beta_release.adoc new file mode 100644 index 0000000..962e8a1 --- /dev/null +++ b/modules/release_guide/pages/beta_release.adoc @@ -0,0 +1 @@ +== Fedora Beta Release \ No newline at end of file diff --git a/modules/release_guide/pages/contributing.adoc b/modules/release_guide/pages/contributing.adoc new file mode 100644 index 0000000..d52e183 --- /dev/null +++ b/modules/release_guide/pages/contributing.adoc @@ -0,0 +1,248 @@ +== Fedora Release Engineering Contributing Guide + +Fedora Release Engineering works with many different utilities that are +maintained in respective upstream locations. Fedora Release Engineering +maintains an "Upstream First" policy such that if there is a bug in an +utility or a feature needed, we pursue that upstream before entertaining +the idea of carrying a Fedora specific patch. + +Fedora Release Engineering also has a number of scripts and utilities +that are strictly Fedora centric in order to automate tasks and +processes as they relate to Fedora itself which is what is contained in +the https://pagure.io/releng[releng git repository] hosted on +https://pagure.io/pagure[Pagure]. If you would like to contribute to +something in this repository, please reference the +`contributing-to-releng` section. + +=== Contributing to releng + +In order to contribute to the releng http://www.git-scm.com[git] +repository (where the source reStructured Text version of these docs +live), you should first have a +https://admin.fedoraproject.org/accounts[Fedora Account System] (FAS) +account, log into https://pagure.io[pagure.io] and the fork the +https://pagure.io/releng[releng git repository]. + +Once you've forked the https://pagure.io/releng[releng git repository], +you will need to set the remote upstream git clone of your fork in order +to track the official releng repository. While not mandatory, it's +conventional to call the remote upstream the name `upstream` which can +be done with the following command while within the directory of your +local git clone of your fork. + +[source,bash] +---- +$ git remote add upstream https://pagure.io/releng.git +---- + +[NOTE] +.Note +==== +If you are not currently familiar with git, it is highly recommended to +visit git's upstream and familiarize yourself. + +http://www.git-scm.com/ +==== + +==== RelEng Developer Workflow + +There are many options for developer workflow, but the recommended +workflow for Fedora releng repository is known as a +"http://www.git-scm.com/book/en/v2/Git-Branching-Branching-Workflows#Topic-Branches[Topic +Branch]" based workflow in git nomenclature. This is how Fedora Release +Engineering contributors can submit changes to the +https://pagure.io/releng[releng git repository] for both code and +documentation. + +The general workflow is as follows: + +* Checkout `main` branch of the local git clone of your releng +repository clone. ++ +.... +$ git checkout main +.... +* Pull upstream and merge into local main to make sure that your main +branch is in line with the latest changes from upstream. Then push it to +your clone so that origin knows about the changes. ++ +.... +$ git pull --rebase upstream main +$ git push origin main +.... +* Create a topic branch from main. ++ +.... +$ git checkout -b my-topic-branch +.... +* Make changes in your topic branch and commit them to your topic +branch. ++ +.... +$ vim somefile.py + +.... make some change ... + +$ git add somefile.py +$ git commit -s -m "awesome patch to somefile.py" +.... +* This step is optional but recommended in order to avoid collisions +when submitting upstream. Here we will checkout main again and merge +`upstream/main` so that we can resolve any conflicts locally. ++ +.... +$ git checkout main +$ git pull --rebase upstream main +$ git push origin main +.... +* Rebase on main before submitting a pull request ++ +.... +$ git rebase main + +..... Resolve any conflicts if needed ...... +.... +* Push your topic branch to your fork's origin in pagure. ++ +.... +$ git push origin my-topic-branch +.... +* Open a pull request in Rel Eng Pagure. +https://pagure.io/releng/pull-requests + +==== Developer Workflow Tips and Tricks + +Below are some Fedora Release Engineering Developer Workflow Tips and +Tricks used by current members of the team in order to help assist with +development. + +===== pullupstream + +The following is an useful shell function to place in your `~/.bashrc` +to help automate certain aspects of the developer workflow. It will +allow you to merge in the upstream main or devel branch into your forked +repository for easily keeping in line with the upstream repository. + +The following is the bash function to be added to your `~/.bashrc` and +make sure to `source ~/.bashrc` after adding it in order to "enable" the +function. + +.... +pullupstream () { + if [[ -z "$1" ]]; then + printf "Error: must specify a branch name (e.g. - main, devel)\n" + else + pullup_startbranch=$(git describe --contains --all HEAD) + git checkout $1 + git pull --rebase upstream $1 + git push origin $1 + git checkout ${pullup_startbranch} + fi +} +.... + +With the function in place you can easily pull and merge in the releng +main branch even while using a topic branch as follows: + +.... +$ git status +On branch docs +nothing to commit, working directory clean + +$ pullupstream main +Switched to branch 'main' +Your branch is up-to-date with 'origin/main'. +Already up-to-date. +Everything up-to-date +Switched to branch 'docs' + +$ git status +On branch docs +nothing to commit, working directory clean +.... + +Now that you're back on your topic branch you can easily rebase on your +local main branch in order to resolve any merge conflicts that may come +up for clean pull request submission. + +.... +$ git rebase main +Current branch docs is up to date. +.... + +=== RelEng Upstream Tools + +Fedora Release Engineering uses many tools that exist in their own +upstream project space. These are tools that every Fedora Release +Engineer should be familiar with and in the event there is a bug or a +feature needed, we should participate in the respective upstream to +resolve the issue first before considering carrying a Fedora specific +patch. + +==== Tools List + +===== Tools Release Engineering is actively involved with upstream + +Below are a set of tools that are centric to the Release Engineering +team and our processes. We actively engage with upstreams of these +projects. For these tools, we recommend the same git contribution +workflow that is outlined above for this git repository. + +* https://pagure.io/koji[koji] -Build System used by Fedora +* https://pagure.io/mash[mash] -Tool that creates repositories from koji +tags, and solves them for multilib dependencies. +* https://pagure.io/pungi[pungi] -Fedora Compose tool +* https://github.com/release-engineering/product-definition-center[Product +Defintion Center (PDC)] -Repository and API for storing and querying +product metadata +* https://github.com/release-engineering/koji-containerbuild[koji-containerbuild] +-Koji plugin to integrate OSBS with koji + +===== Tools Release Engineering is actively mostly consumers of + +Below are the set of tools that the Release Engineering team either +consumes directly or as the side effect of other tools in the Release +Engineering Infrastructure. Tools here should always be engaged upstream +in the event of a bug or enhancement needed but are not tools that the +Release Engineering team is extremely active in their continued upstream +development and will defer to each upstream for recommended +contributions workflow. + +* https://pagure.io/fedpkg[fedpkg] -command line utility for Fedora (and +EPEL) developers. It interacts with dist-git, koji, rpmbuild, git, etc. +* https://pagure.io/rpkg[rpkg] -library for dealing with rpm packaging +in a git source control (used by fedpkg) +* https://github.com/release-engineering/dist-git[dist-git] -remote Git +repository specificaly designed to hold RPM package sources. +* http://createrepo.baseurl.org/[creatrepo] -A python program which +generate repodata from a set of rpm files. +* https://github.com/rpm-software-management/createrepo_c[createrepo_c] +-C implementation of createrepo +* https://github.com/clalancette/oz[oz] -set of programs and classes to +do automated installations of operating systems. +* http://imgfac.org/[imagefactory] -imagefactory builds images for a +variety of operating system/cloud combinations. +* https://pagure.io/sigul[sigul] -An automated gpg signing system +* https://github.com/rpm-software-management/mock/wiki[mock] -a tool for +building packages in prestine buildroots +* http://www.fedmsg.com/en/latest/[fedmsg] -Fedora Infrastructure +Message Bus +* https://github.com/rhinstaller/lorax[lorax] -tool to build install +trees and images +* http://www.openshift.org/[OpenShift] -Open Source Platform as a +Service by Red Hat +* https://github.com/projectatomic/osbs-client[OSBS] -set of utilities +that turn OpenShift into a layered image build system +* https://fedoraproject.org/wiki/Taskotron[taskotron] -a framework for +automated task execution. +* http://www.pulpproject.org/[pulp] -a platform for managing +repositories of content, such as software packages, and pushing that +content out to large numbers of consumer +* https://github.com/pulp/crane[crane] -Crane is a small read-only web +application that provides enough of the docker registry API to support +"docker pull" +* https://pagure.io/pagure[pagure] A git centered forge +* https://github.com/projectatomic/rpm-ostree[rpm-ostree] -Store RPMs in +OSTree repository, and atomically upgrade from commits +* https://wiki.gnome.org/Projects/OSTree[ostree] -a tool for managing +bootable, immutable, versioned filesystem trees. diff --git a/modules/release_guide/pages/final_freeze.adoc b/modules/release_guide/pages/final_freeze.adoc new file mode 100644 index 0000000..4b4cbab --- /dev/null +++ b/modules/release_guide/pages/final_freeze.adoc @@ -0,0 +1 @@ +== Fedora Final Freeze \ No newline at end of file diff --git a/modules/release_guide/pages/index.adoc b/modules/release_guide/pages/index.adoc new file mode 100644 index 0000000..6e60ae0 --- /dev/null +++ b/modules/release_guide/pages/index.adoc @@ -0,0 +1,150 @@ +== Fedora Release Engineering + +Contents: + +overview philosophy contributing troubleshooting architecture sop + +This page contains information about the Fedora Release Engineering +team. + +[[releng-contact-info]] +=== Contact Information + +* IRC: `#fedora-releng` on irc.libera.chat +* Mailing List: https://admin.fedoraproject.org/mailman/listinfo/rel-eng[rel-eng@lists.fedoraproject.org] +* Issue tracker: https://pagure.io/releng/new_issue[Fedora Releng Pagure Tickets] + +If you want the to get something done (e.g. moving packages to +buildroots or into frozen compositions) by the ReleaseEngineering Team, +please create a ticket in the issue tracker mentioned above. Please +enter your FAS-username or e-mail address in the respective textbox, to +make sure the team can contact you. + +[[index-team-composition]] +=== Team Composition +* https://fedoraproject.org/wiki/User:kevin[Kevin Fenzi (nirik)] +* https://fedoraproject.org/wiki/User:sharkcz[Dan Horák (sharkcz)](secondary arches) +* https://fedoraproject.org/wiki/User:pbrobinson[Peter Robinson(pbrobinson)] +* https://fedoraproject.org/wiki/User:maxamillion[Adam Miller(maxamillion)] +* https://fedoraproject.org/wiki/User:humaton[Tomas Hrcka(humaton)] + +Release Team members are approved by FESCo. However, FESCo has delegated +this power to the Release Team itself. If you want to join the team, +please read `join-releng`. + +=== What is Fedora Release Engineering? + +For a Broad Overview, see `overview `. + +=== Why we do things the way we do them + +For information on the Fedora Release Engineering Philosophy, see +`philosophy `. + +=== Fedora Release Engineering Leadership + +Mohan Boddu (mboddu on IRC, FAS username mohanboddu) + +Leadership is currently appointed by FESCo with input from the current +release team. + +=== Things we Do + +* {blank} ++ +Create official Fedora releases.:: + ** {blank} + + + Fedora Products;; + *** Cloud + *** Server + *** Workstation + ** Fedora Spins +* Report progress towards release from +https://fedoraproject.org/wiki/Releases/Branched[branched] creation on. +* Give reports to FESCo on changes to processes. +* If something is known to be controversial, we let FESCo know before +implementing otherwise implementation generally happens concurrently to +reporting. +* Set policy on freeze management +* Administrate the build system(s) +* Remove unmaintained packages from Fedora +* Push updated packages +* write and maintain tools to compose and push Fedora + +[[join-releng]] +=== Joining Release Engineering + +Much of rel-eng's communication is via IRC. One of the best ways to +initially get involved is to attend one of the meetings and say that +you're interested in doing some work during the open floor at the end of +the meeting. If you can't make the meeting times, you can also ping one +of us on IRC or sign up for the +https://admin.fedoraproject.org/mailman/listinfo/rel-eng[mailing list]. + +Since release engineering needs special access to systems essential to +Fedora people new to rel-eng will usually get access a little bit at a +time. Typically people won't immediately be granted the ability to sign +packages and push updates for example. A couple of tasks you could start +out with are troubleshooting why builds are failing (and if rel-eng +could take actions to fix it) as the requests are submitted to pagure or +help with scripts for various rel-eng tasks. + +There are also a number of tools that Fedora Release Engineering uses +and relies upon, working on improving these upstream to fascilitate with +new things that the Fedora Project is aiming to deliver is also a great +way to get involved with Fedora Rel-Eng. + +=== How we do it + +See our `Standard Operating Procedures ` for details on how we do +the things we do. + +Most discussions regarding release engineering will happen either in +[.title-ref]##fedora-releng# or on the releng mailing list. For +requests, please consult the `releng-contact-info` + +=== Meetings + +rel-eng holds regular meetings every Monday at 14:30 UTC in +[.title-ref]##fedora-meeting-3# on the Libera IRC network. + +* https://pagure.io/releng/issues?status=Open&tags=meeting[Meeting +agendas] are created from open tickets in pagure that contain the +meeting keyword. + +==== Meeting Minutes + +Minutes are posted to the rel-eng mailing list. They are also available +at the +https://meetbot.fedoraproject.org/sresults/?group_id=releng&type=team[Meetbot +team page for releng] + +There are also +https://fedoraproject.org/wiki/ReleaseEngineering/Meetings[historical +Meeting Minutes for 2007-04-16 to 2009-05-04]. + +=== Current activities + +See our https://pagure.io/releng/issues[ticket queue] for the things we +are currently working. + +See https://fedoraproject.org/wiki/Releases[Releases] for information +about Fedora releases, including schedules. + +=== Freeze Policies + +* https://fedoraproject.org/wiki/Milestone_freezes[Milestone (Alpha, +Beta, Final) freezes] +* https://fedoraproject.org/wiki/Software_String_Freeze_Policy[String +Freeze Policy] (Same time as Alpha Freeze) +* https://fedoraproject.org/wiki/Changes/Policy[Change freeze policy] +(that's 'Change' as in 'feature') +* https://fedoraproject.org/wiki/Updates_Policy[Updates Policy] (not +technically a freeze, but of interest) + +=== Indices and tables + +* `genindex` +* `modindex` +* `search` diff --git a/modules/release_guide/pages/layered_image_build_service.adoc b/modules/release_guide/pages/layered_image_build_service.adoc new file mode 100644 index 0000000..d0184d9 --- /dev/null +++ b/modules/release_guide/pages/layered_image_build_service.adoc @@ -0,0 +1,263 @@ +== Fedora Layered Image Build System + +The Fedora Layered Image Build System aims to provide an new type of +official binary artifact produced by Fedora. Currently, we produce two +main types of artifacts: RPMs, and images. The RPMs are created in +https://fedoraproject.org/wiki/Koji[Koji] from specfiles in dist-git. +The images come in different formats, but have in common creation in +Koji from kickstart files — this includes the official Fedora Docker +Base Image. This change introduces a new type of image, a +https://github.com/docker/docker/[Docker] Layered Image, which is +created from a Dockerfile and builds on top of that base image. + +=== Layered Image Build Service Architecture + +.... ++------------------------------+ +| Future Items to Integrate | ++------------------------------+ +| +--------------------------+ | +| |PDC Integration | | +| +--------------------------+ | +| |New Hotness | | +| +--------------------------+ | +| |Other??? | | +| +--------------------------+ | +| Magical Future | +| | ++------------------------------+ + + + + +------------------+ + |Fedora | + |Users/Contributors| + +--------+---------+ + ^ + | +----------------+ + | | Fedora | + +-------+-----------+ | Layered Image | + | Fedora Production | | Maintainers | + | Docker Registry | | | + +-------------------+ +-------+--------+ + ^ | + | V + | +-------------------+--+ + +-----------+------------+ | | + | RelEng Automation | | +------------------+ | + | (Release) | | |Dockerfile | | + +-----------+------------+ | +------------------+ | + ^ | |App "init" scripts| | + | | +------------------+ | + | | |Docs | | + +---------+------------+ | +------------------+ | + | Taskotron | | DistGit | + | Automated QA | | | + +---------+-------+----+ +-----------+----------+ + ^ ^ | + | | | + | | | + | | | + | +-------------------+ | + | | | + [docker images] | | + | | | + | [fedmsg] | ++---------------+-----------+ | | +| | | +---------------+ +| +----------------------+ | | | +| |Intermediate Docker | | +------+-------------------+ | +| |Images for OSBS Builds| | | | | +| +----------------------+ | | +----------------------+ | | +| |Layered Images | | | |containerbuild plugin | | | +| +----------------------+ | | +----------------------+ | | +| |docker|distribution | | | |rpmbuilds | | | +| +----------------------+ | | +----------------------+ | | +| Registry | | koji | | +| | | | | ++-----+----+----------------+ +-----+---+----+-----------+ | + | ^ | ^ ^ | + | | | | | | + | | +---------------------------+ | | | + | | | [osbs-client api] | | | + | | | +---------------------------+ | | + | | | | | | + | | | | | | + V | V | | | + +--+----+---+---+---+ | V + | | +-------------------+---+ + | +--------------+ | |fedpkg container-build + + | |OpenShift v3 | | +-----------------------+ + | +--------------+ | + | |Atomic Reactor| | + | +--------------+ | + | OSBS | + | | + +-------------------+ + + +[--------------------- Robosignatory ------------------------------------] +[ Every time an image is tagged or changes names, Robosignatory signs it ] +[ ] +[ NOTE: It's injection points in the diagram are ommitted for brevity ] +[------------------------------------------------------------------------] +.... + +=== Layered Image Build System Components + +The main aspects of the Layered Image Build System are: + +* Koji +** koji-containerbuild plugin +* OpenShift Origin v3 +* Atomic Reactor +* osbs-client tools +* A docker registry +** docker-distribution +* Taskotron +* fedmsg +* RelEng Automation + +The build system is setup such that Fedora Layered Image maintainers +will submit a build to Koji via the `fedpkg container-build` command a +`containers` namespace within +https://fedoraproject.org/wiki/Infrastructure/VersionControl/dist-git[DistGit]. +This will trigger the build to be scheduled in +https://www.openshift.org/[OpenShift] via +https://github.com/projectatomic/osbs-client[osbs-client] tooling, this +will create a custom +https://docs.openshift.org/latest/dev_guide/builds.html[OpenShift Build] +which will use the pre-made buildroot Docker image that we have created. +The https://github.com/projectatomic/atomic-reactor[Atomic Reactor] +(`atomic-reactor`) utility will run within the buildroot and prep the +build container where the actual build action will execute, it will also +maintain uploading the +https://fedoraproject.org/wiki/Koji/ContentGenerators[Content Generator] +metadata back to https://fedoraproject.org/wiki/Koji[Koji] and upload +the built image to the candidate docker registry. This will run on a +host with iptables rules restricting access to the docker bridge, this +is how we will further limit the access of the buildroot to the outside +world verifying that all sources of information come from Fedora. + +Completed layered image builds are hosted in a candidate docker registry +which is then used to pull the image and perform tests with +https://taskotron.fedoraproject.org/[Taskotron]. The taskotron tests are +triggered by a http://www.fedmsg.com/en/latest/[fedmsg] message that is +emitted from https://fedoraproject.org/wiki/Koji[Koji] once the build is +complete. Once the test is complete, taskotron will send fedmsg which is +then caught by the [.title-ref]#RelEng Automation# Engine that will run +the Automatic Release tasks in order to push the layered image into a +stable docker registry in the production space for end users to consume. + +Note that every time the layered image tagged to a new registry, +ultimately changing it's name, +https://pagure.io/robosignatory[Robosignatory] will automatically sign +the new image. This will also occur as part of the Automated Release +process as the image will be tagged from the candidate docker registry +into the production docker registry in order to "graduate" the image to +stable. + +==== Koji + +https://fedoraproject.org/wiki/Koji[Koji] is the Fedora Build System. + +==== koji-containerbuild plugin + +The +https://github.com/release-engineering/koji-containerbuild[koji-containerbuild] +plugin integrates Koji and OSBS so that builds can be scheduled by koji +and integrated into the build system with imports of metadata, logs, +build data, and build artifacts. + +==== OpenShift Origin v3 + +https://www.openshift.org/[OpenShift Origin v3] is an open source +Container Platform, built on top of http://kubernetes.io/[kubernetes] +and https://github.com/docker/docker/[Docker]. This provides many +aspects of the system needed including a build pipeline for Docker +images with custom build profiles, image streams, and triggers based on +events within the system. + +==== Atomic Reactor + +https://github.com/projectatomic/atomic-reactor[Atomic Reactor] is an +utility which allows for the building of containers from within other +containers providing hooks to trigger automation of builds as well as +plugins providing automatic integration many other utilities and +services. + +==== osbs-client tools + +https://github.com/projectatomic/osbs-client[osbs-client] tools allow +for users to access the build functionality of +https://www.openshift.org/[OpenShift Origin v3] using a simple set of +command line utilities. + +==== docker-registry + +A https://docs.docker.com/registry/[docker-registry] is a stateless, +highly scalable server side application that stores and lets you +distribute Docker images. + +There are many different implementations of docker-registries, two main +ones are supported by the Fedora Layered Image Build System. + +===== docker-distribution + +The https://github.com/docker/distribution/[docker-distribution] +registry is considered the Docker upstream "v2 registry" such that it +was used by upstream to implement the new version 2 specification of the +docker-registry. + +===== Fedora Production Registry + +Implementation details of this are still unknown at the time of this +writing and will be updated at a later date. For the current status and +implementation notes please visit the +https://fedoraproject.org/wiki/Changes/FedoraDockerRegistry[FedoraDockerRegistry] +page. + +==== Taskotron + +https://taskotron.fedoraproject.org/[Taskotron] is an automated task +execution framework, written on top of http://buildbot.net/[buildbot] +that currently executes many Fedora automated QA tasks and we will be +adding the Layered Image automated QA tasks. The tests themselves will +be held in DistGit and maintained by the Layered Image maintainers. + +==== RelEng Automation + +https://pagure.io/releng-automation[RelEng Automation] is an ongoing +effort to automate as much of the RelEng process as possible by using +http://ansible.com/[Ansible] and being driven by +http://www.fedmsg.com/en/latest/[fedmsg] via +https://github.com/maxamillion/loopabull[Loopabull] to execute Ansible +Playbooks based on fedmsg events. + +==== Robosignatory + +https://pagure.io/robosignatory[Robosignatory] is a fedmsg consumer that +automatically signs artifacts and will be used to automatically sign +docker layered images for verification by client tools as well as end +users. + +==== Future Integrations + +In the future various other components of the +https://fedoraproject.org/wiki/Infrastructure[Fedora Infrastructure] +will likely be incorporated. + +===== PDC + +https://pdc.fedoraproject.org/[PDC] is Fedora's implementation of +https://github.com/product-definition-center/product-definition-center[Product +Definition Center] which allows Fedora to maintain a database of each +Compose and all of it's contents in a way that can be queried and used +to make decisions in a programatic way. + +===== The New Hotness + +https://github.com/fedora-infra/the-new-hotness[The New Hotness] is a +http://www.fedmsg.com/en/latest/[fedmsg] consumer that listens to +release-monitoring.org and files bugzilla bugs in response (to notify +packagers that they can update their packages). diff --git a/modules/release_guide/pages/mass_rebuild.adoc b/modules/release_guide/pages/mass_rebuild.adoc new file mode 100644 index 0000000..1f4f984 --- /dev/null +++ b/modules/release_guide/pages/mass_rebuild.adoc @@ -0,0 +1,26 @@ += Mass rebuild information. + +== Description + +Periodically we do mass rebuilds of rawhide during the development cycle. This SOP will outline the steps necessary to do this. + +== Assumptions + +This assumes that the mass rebuild has already been approved and scheduled via release engineering and FESCo. Coordinate with infrastructure as well for any needed koji updates. + +This also assumes that the mass rebuild does not need to be done in dependency order, and that the mass rebuild does not involve a ABI change. + +== Considerations + +The most important thing to keep in mind while doing a mass rebuild is to communicate clearly what actions are being performed and the status of the rebuild. + +Check in on scripts frequently to avoid a long stalled command from adding significant delays in completing the rebuild. + +Check with secondary arches, whether they up-to-date enough with primary, create rebuild tag and target when they are. It will then take care of rebuilds of the arch specific packages in appropriate kojis. + +== Actions + +=== Preparatory Steps + +The following steps may be completed in the weeks leading up to the scheduled mass rebuild. + diff --git a/modules/release_guide/pages/mass_rebuild_modules.adoc b/modules/release_guide/pages/mass_rebuild_modules.adoc new file mode 100644 index 0000000..569347f --- /dev/null +++ b/modules/release_guide/pages/mass_rebuild_modules.adoc @@ -0,0 +1,122 @@ +== Mass Rebuild of Modules + +=== Description + +Periodically we do mass rebuilds of modules in rawhide during the +development cycle. This SOP will outline the steps necessary to do this. + +=== Assumptions + +This assumes that the mass rebuild has already been approved and +scheduled via release engineering and FESCo. Coordinate with +infrastructure as well for any needed updates. + +=== Considerations + +* The most important thing to keep in mind while doing a mass rebuild is +to communicate clearly what actions are being performed and the status +of the rebuild. +* Check in on scripts frequently to avoid a long stalled command from +adding significant delays in completing the rebuild. + +=== Actions + +==== Preparatory Steps + +The following steps should be completed after the +https://docs.pagure.org/releng/sop_mass_rebuild_packages.html[mass +rebuild of packages] is done. + +. Update Scripts + +The mass rebuild depends on two main scripts from the +https://pagure.io/releng[releng git repository]. Each one requires some +changes in variables for each new mass rebuild cycle. + +____ +* {blank} ++ +_mass-rebuild-modules.py_:: + ** rebuildid +* {blank} ++ +_massrebuildsinfo.py_:: + ** module_mass_rebuild_epoch + ** module_mass_rebuild_platform +____ + +Change the following items: + +* the `rebuildid` to match the release for which you are mass rebuilding +modules as per in massrebuildsinfo.py +* `module_mass_rebuild_epoch` mostly will be the epoch of mass rebuild +of packages +* `module_mass_rebuild_platform` should be the rawhide module platform + +==== Starting the Mass Rebuild of Modules + +The `mass-rebuild-modules.py` script takes care of: + +* Discovering available available modules from PDC +* Find the module info from mbs and check if a module build is submitted +after the epoch date +* Checking out modules from dist-git +* Switching to appropriate stream +* Find modulemd file +* Use libmodulemd to determine if this module stream applies to this +platform version +* If needs rebuilding, committing the change +* Push the commit +* Submitting the build request through mbs + +. Connect to the mass-rebuild Machine ++ +____ +.... +$ ssh compose-branched01.iad2.fedoraproject.org +.... +____ +. Start a terminal multiplexer ++ +____ +.... +$ tmux +.... +____ +. Clone or checkout the latest copy of the +https://pagure.io/releng[releng git repository]. +. Run the [.title-ref]#mass-rebuild-modules.py# script from +_releng/scripts_ ++ +____ +.... +$ cd path/to/releng_repo/scripts +$ ./mass-rebuild-modules.py build --wait 2>&1 | tee ~/massbuildmodules.out +.... +____ + +[NOTE] +.Note +==== +The token file should be located in infra's private ansible repo, or ask +infra to get it to you using this +https://pagure.io/fedora-infrastructure/issue/8048#comment-587789[process]. +==== + +[NOTE] +.Note +==== +The [.title-ref]#build# option is really important to pay attention, +since the mass branching of modules will also use the same script, just +changing the option to [.title-ref]#branch# and +[.title-ref]#module_mass_branching_platform# in +[.title-ref]#massrebuildsinfo.py# +==== + +==== Post Mass Rebuild Tasks + +Once the module mass rebuild is done, send an email to the +devel-announce@ list + +. Send the final notification to the +_devel-announce@lists.fedoraproject.org_ list diff --git a/modules/release_guide/pages/mass_rebuild_packages.adoc b/modules/release_guide/pages/mass_rebuild_packages.adoc new file mode 100644 index 0000000..5084ef9 --- /dev/null +++ b/modules/release_guide/pages/mass_rebuild_packages.adoc @@ -0,0 +1,262 @@ +== Mass Rebuild + +=== Description + +Periodically we do mass rebuilds of rawhide during the development +cycle. This SOP will outline the steps necessary to do this. + +=== Assumptions + +This assumes that the mass rebuild has already been approved and +scheduled via release engineering and FESCo. Coordinate with +infrastructure as well for any needed koji updates. + +This also assumes that the mass rebuild does not need to be done in +dependency order, and that the mass rebuild does not involve a ABI +change. + +=== Considerations + +* The most important thing to keep in mind while doing a mass rebuild is +to communicate clearly what actions are being performed and the status +of the rebuild. +* Check in on scripts frequently to avoid a long stalled command from +adding significant delays in completing the rebuild. +* Check with secondary arches, whether they up-to-date enough with +primary, create rebuild tag and target when they are. It will then take +care of rebuilds of the arch specific packages in appropriate kojis. + +=== Actions + +==== Preparatory Steps + +The following steps may be completed in the weeks leading up to the +scheduled mass rebuild. + +. Create the Mass Rebuild Pagure Issue ++ +____ +Create an issue on the https://pagure.io/releng/issues[Release +Engineering issues page] that points at the schedule for the current +release. + +See https://pagure.io/releng/issue/6898[the Fedora 27 mass rebuild issue +example]. +____ +. Set up the Mass Rebuild Wiki Page ++ +____ +The mass rebuild wiki page should answer the following questions for +maintainers: + +* Why the mass rebuild is happening +* How to opt out of the mass rebuild + +[NOTE] +.Note +==== +See https://fedoraproject.org/wiki/Fedora_26_Mass_Rebuild[the Fedora 26 +Wiki example]. +==== +____ +. Send out the Mass Rebuild Notice ++ +____ +Send out the same information posted on the wiki to the +[.title-ref]#devel-announce@lists.fedoraproject.org# mailing list. + +[NOTE] +.Note +==== +See +https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[the +Fedora 26 e-mail example]. +==== +____ +. Create a Tag to Contain the Mass Rebuild ++ +____ +Mass rebuilds require their own tag to contain all related builds. The +example assumes we are doing a rebuild for Fedora 26. + +.... +$ koji add-tag f26-rebuild --parent f26 +.... +____ +. Request Package Auto-Signing for New Mass-Rebuild Tag ++ +____ +File a ticket with https://pagure.io/fedora-infrastructure/issues[Fedora +Infrastructure] requesting the new mass-rebuild tag be enabled for +package auto-signing. +____ +. Create the Koji Target for the Mass Rebuild ++ +____ +Using the same [.title-ref]#f26-rebuild# tag created in the previous +example: + +.... +$ koji add-target f26-rebuild f26-build +.... + +[NOTE] +.Note +==== +*koji add-target* _target-name_ _buildroot-tag_ _destination-tag_ +describes the syntax format above. If the _destination-tag_ is not +specified then it will be the same as the _target-name_. +==== +____ +. Update Scripts ++ +____ +The mass rebuild depends on four main scripts from the +https://pagure.io/releng[releng git repository]. Each one requires some +changes in variables for each new mass rebuild cycle. + +* {blank} ++ +mass-rebuild.py:: + ** buildtag + ** targets + ** epoch + ** comment + ** target +* {blank} ++ +find-failures.py:: + ** buildtag + ** desttag + ** epoch +* mass-tag.py +* {blank} ++ +need-rebuild.py:: + ** buildtag + ** target + ** updates + ** epoch +____ + +Change the following items: + +* the build tag, holding tag, and target tag should be updated to +reflect the Fedora release you're building for +* the `epoch` should be updated to the point at which all features that +the mass rebuild is for have landed in the build system (and a newRepo +task completed with those features) +* the comment which is inserted into spec changelogs + +==== Starting the Mass Rebuild + +The `mass-rebuild.py` script takes care of: + +* Discovering available packages in koji +* Trimming out packages which have already been rebuilt +* Checking out packages from git +* Bumping the spec file +* Committing the change +* git tagging the change +* Submitting the build request to Koji + +. Connect to the mass-rebuild Machine ++ +____ +.... +$ ssh branched-composer.phx2.fedoraproject.org +.... +____ +. Start a terminal multiplexer ++ +____ +.... +$ tmux +.... +____ +. Clone or checkout the latest copy of the +https://pagure.io/releng[releng git repository]. +. Run the mass-rebuild.py script from _releng/scripts_ ++ +____ +.... +$ cd path/to/releng_repo/scripts +$ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out +.... +____ + +==== Monitoring Mass Rebuilds + +The community has a very high interest in the status of rebuilds and +many maintainers will want to know if their build failed right away. The +`find-failures.py` and `need-rebuild.py` scripts are designed to update +publicly available URLs for stakeholders to monitor. + +. Connect to a Compose Machine ++ +____ +.... +$ ssh compose-x86-02.phx2.fedoraproject.org +.... +____ +. Start a terminal multiplexer ++ +____ +.... +$ tmux +.... +____ +. Clone or checkout the latest copy of the +https://pagure.io/releng[releng git repository] +. {blank} ++ +Set Up the Rebuild Failures Notification Web Site:: + The `find_failures.py` script discovers attempted builds that have + failed. It lists those failed builds and sorts them by package owner. + + +.... +$ while true; do ./find_failures.py > f26-failures.html && cp f26-failures.html /mnt/koji/mass-rebuild/f26-failures.html; sleep 600; done +.... +. Start a second pane in the terminal emulator +. {blank} ++ +Set up the Site for Packages that Need Rebuilt:: + The `need-rebuild.py` script discovers packages that have not yet been + rebuilt and generates an html file listing them sorted by package + owner. This gives external stakeholders a rough idea of how much work + is remaining in the mass rebuild. + + +.... +$ while true; do ./need-rebuild.py > f26-need-rebuild.html && cp f26-need-rebuild.html /mnt/koji/mass-rebuild/f26-need-rebuild.html; sleep 600; done +.... + +==== Post Mass Rebuild Tasks + +Once the mass rebuild script completes, and all the pending builds have +finished, the builds will need to be tagged. The `mass-tag.py` script +will accomplish this task. The script will: + +* Discover completed builds +* Trim out builds that are older than the latest build for a given +package +* Tag remaining builds into their final destination (without generating +email) + +. Clone or checkout the latest copy of the +https://pagure.io/releng[releng git repository] +. Run the `mass-tag.py` script (requires koji kerberos authentication) ++ +____ +.... +$ cd path/to/releng_repo/scripts +$ ./mass-tag.py --source f36-rebuild --target f36 +.... +____ +. Send the final notification to the +_devel-announce@lists.fedoraproject.org_ list ++ +____ +The contents should look something like this +https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[example +email]. +____ diff --git a/modules/release_guide/pages/overview.adoc b/modules/release_guide/pages/overview.adoc new file mode 100644 index 0000000..46cc1f2 --- /dev/null +++ b/modules/release_guide/pages/overview.adoc @@ -0,0 +1,141 @@ +[[overview]] +== Fedora Release Engineering Overview + +[[overview-intro]] +=== Introduction + +The development of Fedora is a very open process, involving over a +thousand package maintainers (along with testers, translators, +documentation writers and so forth). These maintainers are responsible +for the bulk of Fedora distribution development. An elected +https://fedoraproject.org/wiki/Fedora_Engineering_Steering_Committee[committee +of people] provides some level of direction over the engineering aspects +of the project. + +The rapid pace of Fedora development leaves little time for polishing +the development system into a quality release. To solve this dilemma, +the Fedora project makes use of regular freezes and milestone (Alpha, +Beta, Final) releases of the distribution, as well as "branching" of our +trees to maintain different strains of development. + +Stable branches of the Fedora tree and associated +https://fedoraproject.org/wiki/Repositories[Repositories] are maintained +for each Fedora release. The +https://fedoraproject.org/wiki/Releases/Rawhide[Rawhide] rolling +development tree is the initial entry point for all Fedora development, +and the trunk from which all branches diverge. Releases are +https://fedoraproject.org/wiki/Releases/Branched[Branched] from Rawhide +some time before they are sent out as stable releases, and the milestone +releases (Alpha, Beta and Final) are all built from this Branched tree. + +Nightly snapshot images of various kinds are built from Rawhide and +Branched (when it exists) and made available for download from within +the trees on the https://mirrors.fedoraproject.org/[mirrors] or from the +https://fedoraproject.org/wiki/Koji[Koji] build system. + +The https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle[Fedora +Release Life Cycle] page is a good entry point for more details on these +processes. Some other useful references regarding the Fedora release +process include: + +* The https://fedoraproject.org/wiki/Changes/Policy[Release planning +process] +* The +https://fedoraproject.org/wiki/QA:Release_validation_test_plan[release +validation test plan] +* The https://fedoraproject.org/wiki/QA:Updates_Testing[updates-testing +process], via https://fedoraproject.org/wiki/Bodhi[Bodhi] and the +https://fedoraproject.org/wiki/Updates_Policy[Updates Policy] +* The https://fedoraproject.org/wiki/QA:SOP_compose_request[test compose +and release candidate system] +* The https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process[blocker +bug process] and +https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process[freeze +exception bug process] +* The [.title-ref]#Repositories# +* The https://fedoraproject.org/wiki/Bugs_and_feature_requests[Bugzilla +system] + +=== Final Release Checklist + +Various tasks need to be accomplished prior to a final Fedora release. +Release Engineering is responsible for many of them, as outlined here. + +==== Release Announcement + +The https://fedoraproject.org/wiki/Docs_Project[Fedora Documentation +Project] prepares release announcements for the final releases. A +https://bugzilla.redhat.com/bugzilla/enter_bug.cgi?product=Fedora%20Documentation&op_sys=Linux&target_milestone=---&bug_status=NEW&version=devel&component=release-notes&rep_platform=All&priority=normal&bug_severity=normal&assigned_to=relnotes%40fedoraproject.org&cc=&estimated_time_presets=0.0&estimated_time=0.0&bug_file_loc=http%3A%2F%2F&short_desc=RELNOTES%20-%20Summarize%20the%20release%20note%20suggestion%2Fcontent&comment=Provide%20details%20here.%20%20Do%20not%20change%20the%20blocking%20bug.&status_whiteboard=&keywords=&issuetrackers=&dependson=&blocked=151189&ext_bz_id=0&ext_bz_bug_id=&data=&description=&contenttypemethod=list&contenttypeselection=text%2Fplain&contenttypeentry=&maketemplate=Remember%20values%20as%20bookmarkable%20template&form_name=enter_bug[bug +needs to be filed] for this two weeks before the final release date. + +==== Mirror List Files + +A new set of mirror list files need to be created for the new release. +Email mailto:mirror-admin@fedoraproject.org[Fedora Mirror Admins] to +have these files created. These should be created at the Final Freeze +point but may redirect to Rawhide until final bits have been staged. + +=== Release Composing + +Fedora “releases” can be built by anyone with a fast machine of proper +arch and access to a package repository. All of the tools necessary to +build a release are available from the package repository. These tools +aim to provide a consistent way to build Fedora releases. A complete +release can actually be built with only a couple commands, including the +creation of network install images, offline install images ('DVDs'), +live images, disk images, install repositories, [[FedUp]] upgrade +images, and other bits. These commands are named pungi and +livecd-creator. + +[NOTE] +.Note +==== +There is work currently being done to replace livecd-creator with +https://github.com/rhinstaller/lorax/blob/master/src/sbin/livemedia-creator[livemedia-creator]. +==== + +==== Pungi + +https://pagure.io/pungi[Pungi] is the tool used to compose Fedora +releases. It requires being ran in a chroot of the package set that it +is composing. This ensures that the correct userland tools are used to +match up with the kernel that will be used to perform the installation. +It uses a comps file + yum repos to gather packages for the compose. The +https://fedoraproject.org/wiki/Koji[Koji] build system provides a way to +run tasks within chroots on the various arches, as well as the ability +to produce yum repos of packages from specific collections. + +==== Livecd-creator + +Livecd-creator is part of the +https://fedoraproject.org/wiki/FedoraLiveCD[livecd-tools] package in +Fedora and it is used to compose the live images as part of the Fedora +release. It is also used to compose many of the custom +https://spins.fedoraproject.org[Spins] or variants of Fedora. + +=== Distribution + +Once a compose has been completed, the composed tree of release media, +installation trees, and frozen +https://fedoraproject.org/wiki/Repositories[Repositories] needs to be +synchronized with the Fedora mirror system. [[MirrorManager]] has some +more details on the mirror system. Many of the images are also offered +via BitTorrent as an alternative method of downloading. + +==== Download Mirrors + +Depends on the Fedora Mirror System and infrastructure to populate them +privately. + +==== BitTorrent + +BitTorrent is currently served by http://torrent.fedoraproject.org. +Images are added to the system via this +https://infrastructure.fedoraproject.org/infra/docs/docs/sysadmin-guide/sops/torrentrelease.rst[Standard +Operating Procedure]. + +=== Acknowledgements + +This document was influenced by +http://www.freebsd.org/doc/en_US.ISO8859-1/articles/releng/article.html[release +engineering documents] from http://freebsd.org[FreeBSD]. diff --git a/modules/release_guide/pages/philosophy.adoc b/modules/release_guide/pages/philosophy.adoc new file mode 100644 index 0000000..42bec54 --- /dev/null +++ b/modules/release_guide/pages/philosophy.adoc @@ -0,0 +1,130 @@ +[[philosophy]] +== Fedora Release Engineering Philosophy + +Being an official part of Fedora means that it is composed and supported +by Fedora Release Engineering (releng). Releng bestows this status on +items that successfully makes its way through our Manufacturing Train +and satisfies all related policies. With this status the item is Open, +Integrated, Reproducible, Auditable, Definable, and Deliverable. (in no +particular order) This document provides definitions for each of those +terms, why they're important, and how we guarantee them. All parts of +Fedora should strive to be meet all parts of being official at all +times. + +=== Open + +It goes without saying that Fedora is built on +https://fedoraproject.org/wiki/Foundations[the four F's]. In releng we +are no different, we require everything be open, that is open source, +developed in the open, available for all to look at, use and contribute +to. All downstreams should be able to take our tooling and make their +own derivative of Fedora, either by rebuilding everything, perhaps with +different otions or just adding thier own marks and packages and +recomposing. At any time anyone should be able to see how Fedora is put +together and put together their own version of Fedora. + +=== Integrated + +Fedora is a huge project with a massive number of ever growing +deliverables. This means when we add new deliverables we need to have +the composing of them tightly integrated into the process. We ship +Fedora a whole unit, so we need to make it as a whole unit. Any new +tooling we use needs to be consistent with the existing tooling in how +it works. The tooling has to ensure that the output is Reproducible, +Auditable, Definable so that it can be Deliverable. + +=== Reproducible + +A reproducible component is one we can rebuild from scratch at any time +with less than a day's worth of effort. It implies we can look up all of +the source code, and know exactly what revisions to use from source +control. We know exactly what tools are used in the build environment to +transform the source code into product content (binaries). We also know +how to reproduce the same build environment to ensure the tools behave +like we expect. This aspect is why releng is in the business of +standardizing on build tools. + +Reproducible components are important because they make them a lot +easier to maintain. The Security Team takes advantage of this aspect of +an Product. Not knowing how to rebuild a subsystem in a product to apply +security fixes, or bug fixes, makes their job much more difficult. It +would be a significant risk to provide a product to users that we are +incapable (or merely unsure) of how to build again. Not knowing the +origin of source code is also a significant risk, which is why many of +our build environments are configured to prevent tools from dynamically +downloading content from the internet. + +The combination of Koji and fedpkg is what enables releng to rebuild a +component. fedpkg manages the source code in our dist-git system, and +Koji archives details about the build environment, the tools used, logs, +and of course the binaries themselves. The reproducibility aspect of a +product is the primary reason we require all products be built in Koji +if they seek to be an officially supported part of Fedora. + +=== Auditable + +Fedora and Red Hat expect auditable output too, which means releng knows +who built what, when, and where (and how, but that's reproducibility). +Being able to authoritatively say that something was built within Fedora +by people who have signed the FPCA is important for several reasons. One +big reason is it promotes and fosters accountability within the +comunity. It promotes ownership. Another one is more defensive: in the +event of a security breach, we have a lot of evidence and data prepared +to help us identify what content (if any) was compromised. If a kernel +RPM randomly shows up, and we have no records of building it and/or +shipping it, that should raise a lot of alarms pretty quickly! + +Red Hat's Infosec team and Fedora Security care about this aspect +deeply. We should never be in a position where we cannot definitively +answer why a piece of content is available to users. This aspect is also +a key part of the verification that is done when Fedora becomes RHEL. +All downstream consumers of Fedora expect that they can verify the code +and binaries that they consume from Fedora. + +Releng tracks this data in 2 systems, 1 of which we own: Koji and Bodhi. +Koji uses ssl certs tied to FAS and bodhi uses FAS for authentication to +provide a strong relationship between a user and the content. Koji +builds the content of course, the Bodhi tracks the bugs, documentation, +and enhancements associated with the content and actually does the +delivery. Bodhi maintains records of what was shipped when and where, +and who pushed it. + +=== Definable + +The ability to define and predict content is necessary as well. It is +important to know exactly what was included in a release. It helps +protect against shipping content that unnecessarily causes a support +burden. This aspect of a Fedora component helps support other aspects +like reproducibility. No need to reproduce software we do not have to +ship, right? Ensuring the product content is lean and trim may sound +obvious, but in the world of sprawling RPM dependencies, Maven +artifacts, and Ruby gems, it is actually rather easy to include content +during the course of a multi-month or multi-year development cycle. + +Furthermore, a definable component has the changes made to it vetted and +understood by multiple teams. They are not made in an ad-hoc manor or +without consent from FESCo, QA, releng, and the Product Working Groups +that contribute at the program level. This reduces change risk to the +user, which our users and downstreams like to hear. + +Many systems help make components definable. Releng uses Bugzilla, trac, +blocker bugs and bodhi to track additions and changes. + +=== Deliverable + +Official parts of Fedora are eligible to be delivered to `/pub/fedora/` +or `/pub/alt/releases` on https://dl.fedoraproject.org/pub/[Fedora +Download] and to get +https://admin.fedoraproject.org/mirrormanager[mirrorlists] in +https://github.com/fedora-infra/mirrormanager2[mirrormanager]. These +Distribution Platforms are maintained by Fedora Infrastructure and +releng. This is not a feature of the product content itself or how it +was built, but rather how it was delivered to users through releng's +processes. Those platforms are geographically replicated by the +volunteer mirror network. They provide a reliable and durable service +that ensures users can always reach Fedora for updates, even in the +event of a disaster affecting our data center in phx2. + +User support (user mailing lists and [.title-ref]##fedora#) and Fedora +Security team depend on these services. It is vital to that critical +security fixes and updates are always available to users. diff --git a/modules/release_guide/pages/release_process.adoc b/modules/release_guide/pages/release_process.adoc new file mode 100644 index 0000000..57fb11e --- /dev/null +++ b/modules/release_guide/pages/release_process.adoc @@ -0,0 +1,40 @@ +include::_partials/attributes.adoc[] + += Release process. + +== Rawhide starts Fedora Linux {branched} development +== Mass Rebuild: RPMs first, then modules +== Branch Fedora Linux {branched} from Rawhide +== Rawhide starts Fedora Linux {rawhide} development +== Update Greenwave policy product_versions to add branched release +== Post-branch Freeze (end date approximate) +== Bodhi updates-testing activation point +== Beta Freeze (starts at 1400 UTC) +== Beta Release Public Availability +== Final Freeze (starts at 1400 UTC) +== Final Release Public Availability (GA) +== Fedora Linux {old_release} EOL auto closure +== Beta release Go/No-Go Meeting +== Final release Go/No-Go Meeting +== Update Greenwave policy product_versions to remove EOL release +== Notifications for retirement of packages with security issues +== Retire packages with security issues +== fedora-repos update with the new keys is available in updates-testing +== Begin reminders for packages that haven't rebuilt in last 2 releases +== Re-sign F{branched} content with F{rawhide} key +== Retire Orphaned and Long-Time FTBFS Rawhide Packages +== Retire FTI packages in NEW state +== Stage Beta on master minor +== Bit-flip Beta to mirrors +== Notify Mirrors of Beta Release +== Set current release to active in Package database +== Retire all packages with broken deps +== Retire FTI packages in NEW state +== Enable Updates in Bodhi for the current release +== Stage Final on master minor +== Bit-flip Final to mirrors +== Update PDC with F{old_release} EOL date +== REMINDER: EOL release final update push in 1 week +== REMINDER: EOL release final update push in 1 day +== Notify Mirrors of Final Release +== Remove Beta images diff --git a/modules/release_guide/pages/sop.adoc b/modules/release_guide/pages/sop.adoc new file mode 100644 index 0000000..a735135 --- /dev/null +++ b/modules/release_guide/pages/sop.adoc @@ -0,0 +1,48 @@ +[[sop]] +== Fedora Release Engineering Standard Operating Procedures + +This page documents the Standard Operating Procedures for Fedora Release +Engineering. + +=== Purpose + +The SOP section contains a list of tasks Fedora Release Engineering team +provides for the project. Current Fedora Release Engineering team +members would can add tasks they know about and list steps to complete +the task as well as issues to consider. This is a great way to make sure +that individuals aren't the only ones that can fix a problem. + +=== The Community + +The SOP section is left in the public because it is our hope that others +in the community will add common problems, fix our steps and just in +general look over what we're doing and help us when we're doing +something stupid. We encourage anyone interested to go over our +processes with a fine tooth comb. It'll make us better and we'll +probably learn something in the process. Therefore please open a +https://pagure.io/releng/pull-requests[pull request] to suggest +improvements. + +=== Procedures + +Needed: #. Composing an official release #. Making a composed release +publicly available + +=== Standard Operating Procedures + +sop_adding_build_targets sop_adding_content_generator +sop_adding_new_release_engineer sop_adding_new_branch_sla +sop_adding_packages_compose_artifact sop_adding_side_build_targets +sop_branch_freeze sop_branching sop_breaking_development_freeze +sop_composing_fedora sop_clean_amis sop_create_release_signing_key +sop_deprecate_ftbfs_packages sop_end_of_life sop_eol_change +sop_mass_branching sop_bodhi_activation sop_mass_rebuild_packages +sop_mass_rebuild_modules sop_file_ftbfs sop_generating_openh264_composes +sop_package_blocking sop_package_unblocking +sop_process_dist_git_requests sop_promoting_container_content +sop_signing_builds sop_pushing_updates sop_release_package_signing +sop_remote_dist_git_branches sop_requesting_task_automation_users +sop_retire_orphaned_packages sop_sigul_client_setup +sop_stage_final_release_for_mirrors sop_unretire sop_update_critpath +sop_update_releng_docs sop_updating_comps sop_fedora_media_writer +sop_find_module_info sop_release_container_base_image diff --git a/modules/release_guide/pages/sop_adding_build_targets.adoc b/modules/release_guide/pages/sop_adding_build_targets.adoc new file mode 100644 index 0000000..a27157b --- /dev/null +++ b/modules/release_guide/pages/sop_adding_build_targets.adoc @@ -0,0 +1,180 @@ +== Adding Build Targets + +=== Description + +Each new release we create a build target for the next release. This SOP +will describe the steps necessary to prepare the new build target. + +=== Action + +Adding a build target is a complicated task. It involves updating koji, +git, and fedora-release package. + +[[adding_build_targets_koji]] +==== Koji + +In koji a couple collection tags need to be made, and a target created +to tie them together. We create a base collection tag named after the +release, and a build tag to hold a few things we use in the buildroots +that are not part of the distribution (glibc32/glibc64). Inheritance to +the previous release is used for ownership and package data, as well as +buildroot content data. + +The `add-tag`, `add-tag-inheritance`, `edit-tag`, and `add-target` +commands are used. + +.... +$ koji add-tag --help +Usage: koji add-tag [options] name +(Specify the --help global option for a list of other help options) + +Options: +-h, --help show this help message and exit +--parent=PARENT Specify parent +--arches=ARCHES Specify arches + + +$ koji add-tag-inheritance --help +Usage: koji add-tag-inheritance [options] tag parent-tag +(Specify the --help global option for a list of other help options) + +Options: +-h, --help show this help message and exit +--priority=PRIORITY Specify priority +--maxdepth=MAXDEPTH Specify max depth +--intransitive Set intransitive +--noconfig Set to packages only +--pkg-filter=PKG_FILTER +Specify the package filter +--force=FORCE Force adding a parent to a tag that already has that +parent tag + +$ koji edit-tag --help +Usage: koji edit-tag [options] name +(Specify the --help global option for a list of other help options) + +Options: + -h, --help show this help message and exit + --arches=ARCHES Specify arches + --perm=PERM Specify permission requirement + --no-perm Remove permission requirement + --lock Lock the tag + --unlock Unlock the tag + --rename=RENAME Rename the tag + +$ koji add-target --help +Usage: koji add-target name build-tag +(Specify the --help global option for a list of other help options) + +Options: +-h, --help show this help message and exit +.... + +For example if we wanted to create the Fedora 17 tags, we would do the +following: + +.... +koji add-tag --parent f16-updates f17 +koji add-tag --parent f17 f17-updates +koji add-tag --parent f17-updates f17-candidate +koji add-tag --parent f17-updates f17-updates-testing +koji add-tag --parent f17-updates-testing f17-updates-testing-pending +koji add-tag --parent f17-updates f17-updates-pending +koji add-tag --parent f17-updates f17-override +koji add-tag --parent f17-override --arches=i686,x86_64 f17-build +koji add-tag-inheritance --priority 1 f17-build f16-build +koji edit-tag --perm=fedora-override f17-override +koji edit-tag --lock f17-updates +koji add-target f17 f17-build +.... + +[NOTE] +.Note +==== +The `-pending` tags are used by +https://fedoraproject.org/wiki/Bodhi[Bodhi] and +https://fedoraproject.org/wiki/Taskotron[Taskotron] to track and test +proposed updates. These tags are not build targets and they don't get +made into repos, so proper inheritance isn't vital. +==== + +==== Git + +The pkgdb_sync_git_branches.py file which is hosted in Fedora +Infrastructure ansible +(roles/distgit/templates/pkgdb_sync_git_branches.py) needs to be updated +for the new target for making branches. + +Update `BRANCHES` with the new branch information. The branch name maps +to the branch that is its parent. + +.... +BRANCHES = {'el4': 'rawhide', 'el5': 'rawhide', 'el6': 'f12', + 'OLPC-2': 'f7', + 'rawhide': None, + 'fc6': 'rawhide', + 'f7': 'rawhide', + 'f8': 'rawhide', + 'f9': 'rawhide', + 'f10': 'rawhide', + 'f11': 'rawhide', + 'f12': 'rawhide', + 'f13': 'rawhide', 'f14': 'rawhide'} +.... + +and update `GITBRANCHES` with the translation from pkgdb branch string +to git branch string: + +.... +GITBRANCHES = {'EL-4': 'el4', 'EL-5': 'el5', 'EL-6': 'el6', 'OLPC-2': 'olpc2', + 'FC-6': 'fc6', 'F-7': 'f7', 'F-8': 'f8', 'F-9': 'f9', 'F-10': 'f10', + 'F-11': 'f11', 'F-12': 'f12', 'F-13': 'f13', 'F-14': 'f14', 'devel': 'rawhide'} +.... + +The genacls.pkgdb file also needs to be updated for active branches to +generate ACLs for. Update the `ACTIVE` variable. genacls.pkgdb lives in +puppet (modules/gitolite/files/distgit/genacls.pkgdb). The format is +pkgdb branch string to git branch string (until pkgdb uses git branch +strings): + +.... +ACTIVE = {'OLPC-2': 'olpc2/', 'EL-4': 'el4/', 'EL-5': 'el5/', + 'EL-6': 'el6/', 'F-11': 'f11/', 'F-12': 'f12/', 'F-13': 'f13/', + 'F-14': 'f14/', 'devel': 'rawhide'} +.... + +==== fedora-release + +Currently the fedora-release package provides the `%{?dist}` definitions +used during building of packages. When a new target is created, +fedora-release must be built for the collection with new dist +definitions. + +==== Comps + +* In the comps module in Fedora Hosted git +(ssh://git.fedorarhosted.org/git/comps.git), create and add a new comps +file based on the previous release. (Just copying it should be fine.) +Add the new file to `po/POTFILES.in`. +* When rawhide is retargeted in koji to point to the new release, update +the `Makefile` to target comps-rawhide.xml at the new version. +* Don't forget to `git push` your changes after committing. + +=== Verification + +Given the complexity of all the changes necessary to create a new build +target, the best way to verify is to attempt a build. Given that +fedora-release has to be built before anything else so that dist tags +translate correctly it is a good test case. For example, this was used +to test the new Fedora 15 target: + +* Use pkgdb to request an F-15 branch of fedora-release +* Use pkgdb2branch.py to actually make the branch +* Update fedora-release clone +* Adjust .spec file in rawhide for new dist defines +* commit/build +* Track build in koji to ensure proper tagging is used + +What this won't test is translations of dist at tag time given that +`fedora-release` doesn't use dist in it's Release. Verifying with a +second package that uses dist is a good idea. diff --git a/modules/release_guide/pages/sop_adding_content_generator.adoc b/modules/release_guide/pages/sop_adding_content_generator.adoc new file mode 100644 index 0000000..2563151 --- /dev/null +++ b/modules/release_guide/pages/sop_adding_content_generator.adoc @@ -0,0 +1,58 @@ +== Adding new koji content generator + +=== Description + +Koji added support for content generators some time ago. Basic premise +of content generators is that it lets us create build systems for new +types of content without affecting or changing core Koji code and in +some way simplify integration with rest of the release toolchain. More +information about content generators, background, requirements and more +can be found in Koji +https://docs.pagure.org/koji/content_generators/[content generator +documentation] + +For content generator to be able to create/import builds into Koji +following prerequisites have to be met: + +* Koji has to recognize the content generator type +* User doing the content generator import has to have permissions for +this action +* Any new content types have to be defined in Koji + +=== Questions to ask + +There are some questions that should be answered before the content +generator is enabled/added to Koji + +* Where is the content generator service running, what is its support +status etc? +* Is new type of content being added or is the content generator +providing different way to build content Koji already knows about? +* What is the expected size of content that will be imported into Koji? +* Does the content generator follow each of the requirements for writing +it from Koji documentation referenced above? + +=== Adding a new content generator into koji + +First we create the content generator and give a user permission to do +imports for it: + +.... +koji grant-cg-access --new +.... + +In many cases the content generator will be adding content with new +content type. This can be achieved simply by running: + +.... +koji call addBType +.... + +==== Explanation + +* username - is a name of user which will be doing the imports. In most +cases this will be a service-level account +* content generator name - this name has to be provided by the content +generator development team +* --new - this switch ensures we create the content generator if it +doesn't exist diff --git a/modules/release_guide/pages/sop_adding_new_branch_sla.adoc b/modules/release_guide/pages/sop_adding_new_branch_sla.adoc new file mode 100644 index 0000000..efade52 --- /dev/null +++ b/modules/release_guide/pages/sop_adding_new_branch_sla.adoc @@ -0,0 +1,28 @@ +== Adding New Branch SLAs + +=== Description + +In the ArbitraryBranching model, packagers can choose whatever SLAs they +want for the branches of their packages, but they must choose from a +subset of pre-defined SLAs stored in PDC, maintained by releng. + +This SOP describes the steps necessary for a release engineer to create +a new SLA. + +=== Action + +Adding a new SLA is simple. It involves running a script in the releng +repo, with an authorized token. There is a token available on +[.title-ref]#pdc-backend01# in the [.title-ref]#/etc/pdc.d/# directory. + +.... +$ ./scripts/pdc/insert-sla.py +Name of the SLA: wild_and_cavalier +Description of the SLA: Anything goes! This branch may rebase at any time. No stability guarantees provided. +.... + +=== Verification + +Verifying that the SLA is present is simple: visit the +https://pdc.fedoraproject.org/rest_api/v1/component-branch-slas/[appropriate +PDC endpoint] and verify that your newly-added SLA is present. diff --git a/modules/release_guide/pages/sop_adding_new_release_engineer.adoc b/modules/release_guide/pages/sop_adding_new_release_engineer.adoc new file mode 100644 index 0000000..2cf2c5a --- /dev/null +++ b/modules/release_guide/pages/sop_adding_new_release_engineer.adoc @@ -0,0 +1,160 @@ +== Adding a New Release Engineer + +=== Description + +People volunteer (or get assigned) to doing Fedora release engineering +from time to time. This SOP seeks to describe the process to add a new +release engineer so that they have the rights to accomplish their tasks, +know where to find the tasks, and are introduced to the existing +members. There are several groups that manage access to the respective +systems: + +* `cvsadmin`: Admin group for pkgdb2 (allows to retire/orphan all +packages etc), allows git write access via SSH to all packages in +dist-git +* `gitreleng`: Allows write access to release engineering git repo +* `signers`: Membership is necessary to use +https://fedoraproject.org/wiki/Sigul_Client_Setup_SOP[sigul]. +* `sysadmin`: Allows SSH access to bastion, the SSH gateway to the PHX2 +data center. SSH access to several other internal systems is only +possible from here. +* `sysadmin-cvs`: Allows shell access to pkgs01 (pkgs.fedoraproject.org) +* `sysadmin-releng`: Allows SSH access to autosign01, koji03, koji04, +releng04, relepel01 from bastion + +=== Action + +A new release engineer will access rights in a variety of systems, as +well as be introduced to the releng group. + +==== Git + +Fedora Release Engineering maintains a git repo of scripts. This can be +found in https://pagure.io/pagure[Pagure] at +ssh://git@pagure.io/releng.git. Write access to this group is controlled +by access to the 'gitreleng' FAS group. The new member's FAS username +will need to be added to this group. + +https://pagure.io/releng + +`FIXME: walkthrough group addition` + +==== FAS + +There is a releng group in FAS that release engineers are added to in +order to grant them various rights within the Fedora infrastructure. The +new member's FAS username will need to be added to this group. + +`FIXME: walkthrough group addition` + +==== Koji + +In order to perform certain (un)tagging actions a release engineer must +be an admin in koji. To grant a user admin rights one uses the +`grant-permission` command in koji. + +.... +$ koji grant-permission --help +Usage: koji grant-permission [ ...] +(Specify the --help global option for a list of other help options) + +Options: + -h, --help show this help message and exit +.... + +For example if we wanted to grant npetrov admin rights we would issue: + +.... +$ koji grant-permission admin npetrov +.... + +==== Sigul + +Sigul is our signing server system. They need to bet setup as a signer +if they are going to be signing packages for a release. + +For information on how to setup Sigul, please see: +https://fedoraproject.org/wiki/Sigul_Client_Setup_SOP[sigul] + +==== RelEng Docs Page + +The new release engineer should be added to the +ref:link:index-team-composition[Release Engineering membership list] + +==== rel-eng email list + +Release engineering ticket spam and discussion happens on our +https://admin.fedoraproject.org/mailman/listinfo/rel-eng[Mailing List]. +New releng people need to subscribe. + +==== IRC + +We ask that release engineers idle in [.title-ref]##fedora-releng# on +Libera to be available for queries by other infrastructure admins. +Idling on [.title-ref]##fedora-admin# on Libera is optional. It is noisy +little bit but people sometimes ask releng stuff. + +==== New member announcement + +When a new releng member starts, we announce it to the email list. This +lets the other admins know to expect a new name to show up in tickets +and on IRC. + +=== Verification + +==== Git + +You can verify group membership by sshing to a system that is setup with +FAS and using `getent` to verify membership in the gitreleng group: + +.... +$ ssh fedorapeople.org getent group gitreleng +gitreleng:x:101647:ausil,dwa,jwboyer,kevin,notting,pbabinca,sharkcz,skvidal,spot +.... + +You can verify that the new user is in the above list. + +==== FAS + +You can verify group membership by sshing to a system that is setup with +FAS and using `getent` to verify membership in the releng group: + +.... +$ ssh fedorapeople.org getent group releng +releng:x:101737:atowns,ausil,dwa,jwboyer,kevin,lmacken,notting,pbabinca,spot +.... + +You can verify that the new user is in the above list. + +==== Koji + +To verify that the releng member is an admin koji use the +`list-permissions` koji command: + +.... +$ koji list-permissions --user npetrov +admin +.... + +This shows that npetrov is an admin. + +==== Sigul + +* `FIXME` + +==== Wiki Page + +Verification is easy. Just look at the page. + +==== releng mailing list + +Verify by asking the user if they got the announcement email + +==== Announcement email + +See above + +=== Consider Before Running + +* Make sure the releng person has a solid grasp of the tasks we do and +where to get help if stuck diff --git a/modules/release_guide/pages/sop_adding_packages_compose_artifact.adoc b/modules/release_guide/pages/sop_adding_packages_compose_artifact.adoc new file mode 100644 index 0000000..f6ba618 --- /dev/null +++ b/modules/release_guide/pages/sop_adding_packages_compose_artifact.adoc @@ -0,0 +1,81 @@ +== Adding a package to a Release Artifact + +=== Description + +In the event that a Fedora contributor would like to have a package +added to an Artifact of a Compose (such as an installer ISO image, a +liveCD, Cloud Image, Vagrant, Docker, etc.) that is slated for Release, +the following procedures must be followed due to the interdependence of +different components within the distro layout. + +==== Background + +First, some information on where this all comes from and how things fit +together. + +There is the concept of the "Install Tree" which is the collection of +packages available at install time. This is a vast sub-set of the whole +of the Fedora Package Collection and it is the pool of possible packages +that is available to end users who choose to customize their install +from the https://fedoraproject.org/wiki/Anaconda[Anaconda] installer. It +is also the pool of possible packages that is available to the +https://pagure.io/fedora-kickstarts[fedora-kickstarts] kickstart files +that are used to generate various components of the compose via +https://pagure.io/pungi[pungi] which then produces the Release +Artifacts. + +The Install Tree itself is defined by the +https://pagure.io/fedora-comps[comps] groups so in order to add a net +new package to one of the Release Artifacts, the package must be placed +in an appropriate https://pagure.io/fedora-comps[comps] xml file. For +more information on what specifically defines the "appropriate +[.title-ref]#comps_# xml file" and what kinds of approvals or review +might be needed for adding new packages, please see +https://fedoraproject.org/wiki/How_to_use_and_edit_comps.xml_for_package_groups[this +HowTo]. + +=== Action + +We will need to edit the comps file specific to the Fedora release we +would like to target. For example, if we were to target Fedora 25 we +would edit `comps-f25.xml.in` found within the +https://pagure.io/fedora-comps[comps] git repository and this should be +modified based on the +https://fedoraproject.org/wiki/How_to_use_and_edit_comps.xml_for_package_groups#How_to_edit_comps[How +to edit comps] procedure. + +If the package that was added is a part of a pre-existing comps group +that is already used in the target Release Artifact's +https://pagure.io/fedora-kickstarts[fedora-kickstarts] kickstart file +then we are done. + +However, if there is a new comps group added then we need to include +that new comps group in the respective +https://pagure.io/fedora-kickstarts[fedora-kickstarts] kickstart file +similar to the following. + +.... +%packages +@mynewcompsgroup +.... + +Next we will need to tell https://pagure.io/pungi[pungi] Variants data +about the new group and it's relationship to the corresponding +https://sgallagh.wordpress.com/2016/03/18/sausage-factory-multiple-edition-handling-in-fedora/[Variant]. +This information is held in the https://pagure.io/pungi-fedora[Fedora +Pungi Configs] https://pagure.io/[pagure] git forge repository. The file +needed to be edited is `variants-fedora.xml` and can be viewed from a +web browser +https://pagure.io/pungi-fedora/blob/master/f/variants-fedora.xml[here]. + +Once this has been completed, we're all done. + +=== Verification + +Verify that the next compose is successful and that the change made +didn't cause any issues. This can be done from the +https://pdc.fedoraproject.org/compose/[Fedora Product Definition Center] +which is a central store of information about Composes and their +resulting artifacts. + +=== Consider Before Running diff --git a/modules/release_guide/pages/sop_adding_side_build_targets.adoc b/modules/release_guide/pages/sop_adding_side_build_targets.adoc new file mode 100644 index 0000000..720b608 --- /dev/null +++ b/modules/release_guide/pages/sop_adding_side_build_targets.adoc @@ -0,0 +1,208 @@ +== Adding Side Build Tags + +=== Description + +Bigger Features can take a while to stabilize and land or need a large +number of packages to be built against each other, this is easiest +served by having a separate build tag for the development work. This SOP +will describe the steps necessary to prepare the new build target. + +=== Action + +Engineers should be aware that adding side build targets incurs extra +newRepo tasks in the koji. + +==== Research Tag + +. Verify whether a tag already exists. ++ +The typical tag format is _PRODUCT_-_DESCRIPTOR_. The _DESCRIPTOR_ +should be something brief that clearly shows why the tag exists. ++ +[NOTE] +.Note +==== +Don't think too hard about what makes a good descriptor. The descriptor +for the XFCE version 4.8 side-build target was _xfce48_. KDE often +simply uses _kde_ as its descriptor. Use best judgement and if in doubt +ask in IRC on `#fedora-releng`. +==== ++ +EPEL6 + +.... +$ koji taginfo epel6-kde +.... ++ +EPEL7 + +.... +$ koji taginfo epel7-kde +.... ++ +Fedora + +.... +$ koji taginfo f28-kde +.... ++ +[NOTE] +.Note +==== +If the tag already exists, continue searching for an available tag by +appending `-2` and incrementing the number by one until an available tag +is found. For example, if `f28-kde` already exists then search for +`f28-kde-2`, `f28-kde-3`, and so on until a suitable tag is found. +==== +. Determine the appropriate architectures. ++ +EPEL6 + +.... +$ koji taginfo dist-6E-epel-build +.... ++ +EPEL7 + +.... +$ koji taginfo epel7-build +.... ++ +Fedora + +.... +$ koji taginfo f28-build +.... + +==== Create Side Build Target + +. Create the proper tag. ++ +Note the slightly different syntax depending on which product needs the +side-build target and the comma delimited list of architectures based on +the information from a previous step. ++ +EPEL6 + +.... +$ koji add-tag epel6-kde --parent=dist-6E-epel-build --arches=i686,x86_64,ppc64 +.... ++ +EPEL7 + +.... +$ koji add-tag epel7-kde --parent=epel7-build --arches=aarch64,x86_64,ppc64,ppc64le +.... ++ +Fedora + +.... +$ koji add-tag f28-kde --parent=f28-build --arches=armv7hl,i686,x86_64,aarch64,ppc64,ppc64le,s390x +.... +. Create the target. ++ +EPEL6 + +.... +$ koji add-target epel6-kde epel6-kde +.... ++ +EPEL7 + +.... +$ koji add-target epel7-kde epel7-kde +.... ++ +Fedora + +.... +$ koji add-target f28-kde f28-kde +.... +. Find the taskID for the newRepo task associated with the newly created +target. ++ +.... +$ koji list-tasks --method=newRepo +ID Pri Owner State Arch Name +25101143 15 kojira OPEN noarch newRepo (f28-kde) +.... +. Ensure the newRepo task is being run across all architectures. ++ +.... +$ koji watch-task 25101143 +Watching tasks (this may be safely interrupted)... +25101143 newRepo (f28-kde): open (buildvm-14.phx2.fedoraproject.org) +25101154 createrepo (i386): closed +25101150 createrepo (ppc64le): closed +25101152 createrepo (ppc64): closed +25101151 createrepo (aarch64): closed +25101149 createrepo (armhfp): closed +25101153 createrepo (s390x): open (buildvm-ppc64le-04.ppc.fedoraproject.org) +25101148 createrepo (x86_64): open (buildvm-aarch64-21.arm.fedoraproject.org) +.... +. Request Package Auto-Signing for New Tag ++ +File a ticket in https://pagure.io/fedora-infrastructure/issues[pagure +infrastructure] requesting the new tag be enabled for package +auto-signing. +. Update the Pagure Issue ++ +Update the issue according to the following template which assumes a +side target was made for KDE under Fedora 28. _TAG_NAME_ has been +created: ++ +____ +$ koji add-tag f28-kde --parent=f28-build +--arches=armv7hl,i686,x86_64,aarch64,ppc64,ppc64le,s390x + +$ koji add-target f28-kde f28-kde + +You can do builds with: + +$ fedpkg build --target=f28-kde + +Let us know when you are done and we will move all the builds into f28. +____ + +=== Cleanup + +Fedora Release Engineering is responsible for merging the packages from +the side-build target and tag back into the main tag. The requestor will +update the original ticket when ready for the procedure outlined below. + +. Remove the target ++ +.... +$ koji remove-target +.... +. Merge side build back to main target. ++ +Get the latest checkout from https://pagure.io/releng/[Fedora Release +Engineering Repository] and run the [.title-ref]#mass-tag.py# from the +scripts directory. ++ +.... +$ ./mass-tag.py --source --target > mass_tag.txt +.... ++ +[NOTE] +.Note +==== +The _MAIN_TAG_NAME_ for Fedora is typically the pending subtag, e.g. +`f28-pending` when bodhi is not managing updates. After bodhi is enabled +and managing updates then merge into `f28-updates-candidate`. +==== +. Paste Output to the Original Ticket ++ +Paste the output from mass-tag.py into the pagure/releng ticket to show +what packages were merged and what packages need rebuilding for those +who work on the buildroot. + +Tags are *never* removed. + +=== Consider Before Running + +* Is the amount of work to be done worth the cost of newRepo tasks. +* If there is only a small number of packages overrides may be better. +* Is there a mass-rebuild going on? no side tags are allowed while a +mass rebuild is underway diff --git a/modules/release_guide/pages/sop_bodhi_activation.adoc b/modules/release_guide/pages/sop_bodhi_activation.adoc new file mode 100644 index 0000000..e654ea8 --- /dev/null +++ b/modules/release_guide/pages/sop_bodhi_activation.adoc @@ -0,0 +1,238 @@ +== Bodhi Activation Point + +=== Description + +Bodhi must be activated after two weeks of +https://docs.pagure.org/releng/sop_mass_branching.html[Mass Branching] +at 14:00 UTC. + +=== Action + +==== Making koji changes + +Make the following koji tag changes + +.... +$ koji remove-tag-inheritance f33-updates-candidate f33 +$ koji remove-tag-inheritance f33-updates-testing f33 +$ koji remove-tag-inheritance f33-updates-pending f33 +$ koji remove-tag-inheritance f33-override f33 +$ koji add-tag-inheritance f33-updates-candidate f33-updates +$ koji add-tag-inheritance f33-updates-testing f33-updates +$ koji add-tag-inheritance f33-updates-pending f33-updates +$ koji add-tag-inheritance f33-override f33-updates +$ koji edit-tag --perm=admin f33 +.... + +==== Update bodhi rpm release + +Set the bodhi rpm to release to not to automatically create the update +and also bodhi knows to compose the updates + +.... +$ bodhi releases edit --name "F33" --stable-tag f33-updates --testing-repository updates-testing --package-manager dnf --no-create-automatic-updates --composed-by-bodhi +.... + +==== Add the modular release + +Run the following command on your own workstation to add the modular +release + +.... +$ bodhi releases create --name F33M --long-name "Fedora 33 Modular" --id-prefix FEDORA-MODULAR --version 33 --branch f33m --dist-tag f33-modular --stable-tag f33-modular-updates --testing-tag f33-modular-updates-testing --candidate-tag f33-modular-updates-candidate --pending-stable-tag f33-modular-updates-pending --pending-testing-tag f33-modular-updates-testing-pending --pending-signing-tag f33-modular-signing-pending --override-tag f33-modular-override --state pending --user mohanboddu +.... + +[WARNING] +.Warning +==== +Due to a https://github.com/fedora-infra/bodhi/issues/2177[bug] in +Bodhi, it is critical that Bodhi processes be restarted any time +`bodhi releases create` or `bodhi releases edit` are used. +==== + +[NOTE] +.Note +==== +Add the container and flatpak releases if they weren't already added to +bodhi +==== + +=== Ansible Changes + +==== Update vars + +Update the _FedoraBranchedBodhi_ and _RelEngFrozen_ vars in infra +ansible + +.... +diff --git a/vars/all/FedoraBranchedBodhi.yaml b/vars/all/FedoraBranchedBodhi.yaml +index aba8be2..606eb2e 100644 +--- a/vars/all/FedoraBranchedBodhi.yaml ++++ b/vars/all/FedoraBranchedBodhi.yaml +@@ -3,4 +3,4 @@ +# prebeta: After bodhi enablement/beta freeze and before beta release +# postbeta: After beta release and before final release +# current: After final release +-FedoraBranchedBodhi: preenable ++FedoraBranchedBodhi: prebeta +diff --git a/vars/all/RelEngFrozen.yaml b/vars/all/RelEngFrozen.yaml +index 5836689..87d85f3 100644 +--- a/vars/all/RelEngFrozen.yaml ++++ b/vars/all/RelEngFrozen.yaml +@@ -1 +1 @@ +-RelEngFrozen: False ++RelEngFrozen: True +.... + +==== Update Greenwave Policy + +Now edit the Greenwave policy to configure a policy for the new release +by editing `roles/openshift-apps/greenwave/templates/configmap.yml` in +the Infrastructure Ansible repository. + +.... +diff --git a/roles/openshift-apps/greenwave/templates/fedora.yaml b/roles/openshift-apps/greenwave/templates/fedora.yaml +index 7a76f61..d15e154 100644 +--- a/roles/openshift-apps/greenwave/templates/fedora.yaml ++++ b/roles/openshift-apps/greenwave/templates/fedora.yaml +@@ -84,6 +84,9 @@ rules: +--- !Policy +id: "no_requirements_testing" +product_versions: ++ - fedora-33-modular ++ - fedora-33-containers ++ - fedora-33-flatpaks + - fedora-32-modular + - fedora-32-containers + - fedora-32-flatpaks +@@ -107,6 +110,9 @@ rules: [] +--- !Policy +id: "no_requirements_for_stable" +product_versions: ++ - fedora-33-modular ++ - fedora-33-containers ++ - fedora-33-flatpaks + - fedora-32-modular + - fedora-32-containers + - fedora-32-flatpaks +@@ -133,6 +139,7 @@ id: "openqa_release_critical_tasks_for_testing" +product_versions: + - fedora-rawhide + - fedora-eln ++ - fedora-33 + - fedora-32 + - fedora-31 + - fedora-30 +@@ -147,6 +154,7 @@ id: "openqa_release_critical_tasks_for_stable" +product_versions: + - fedora-rawhide + - fedora-eln ++ - fedora-33 + - fedora-32 + - fedora-31 + - fedora-30 +.... + +==== Update Robosignatory Config + +Update the robosignatory config in the infra ansible repo as following + +.... +diff --git a/roles/robosignatory/templates/robosignatory.toml.j2 b/roles/robosignatory/templates/robosignatory.toml.j2 +index 16a6708..68f4251 100644 +--- a/roles/robosignatory/templates/robosignatory.toml.j2 ++++ b/roles/robosignatory/templates/robosignatory.toml.j2 +@@ -259,8 +259,8 @@ handlers = ["console"] + type = "modular" + + [[consumer_config.koji_instances.primary.tags]] +- from = "f33-modular-updates-candidate" +- to = "f33-modular" ++ from = "f33-modular-signing-pending" ++ to = "f33-modular-updates-testing-pending" + key = "{{ (env == 'production')|ternary('fedora-33', 'testkey') }}" + keyid = "{{ (env == 'production')|ternary('9570ff31', 'd300e724') }}" + type = "modular" +.... + +==== Run the playbooks + +.... +$ rbac-playbook openshift-apps/greenwave.yml +$ rbac-playbook openshift-apps/bodhi.yml +$ rbac-playbook groups/bodhi-backend.yml +$ rbac-playbook groups/releng-compose.yml +$ rbac-playbook manual/autosign.yml +.... + +Greenwave runs in OpenShift (as implied by the playbook paths), and so +the change will not be live right away when the playbook finishes. You +can monitor +https://greenwave-web-greenwave.app.os.fedoraproject.org/api/v1.0/policies +to wait for the new policy to appear (it should take a few minutes). + +==== Restart bodhi services + +Restart bodhi services to understand the bodhi new release on +bodhi-backend01 (Look at warning in +https://docs.pagure.org/releng/sop_bodhi_activation.html#action and the +bug is https://github.com/fedora-infra/bodhi/issues/2177) + +.... +$ sudo systemctl restart bodhi-celery +$ sudo systemctl restart fm-consumer@config +$ sudo systemctl restart koji-sync-listener +.... + +==== Send Announcement + +Email *devel-announce* and *test-announce* lists about Bodhi Activation. +Please find the body of the email below: + +.... +Hi all, + +Today's an important day on the Fedora 25 schedule[1], with several significant cut-offs. First of all today is the Bodhi activation point [2]. That means that from now all Fedora 25 packages must be submitted to updates-testing and pass the relevant requirements[3] before they will be marked as 'stable' and moved to the fedora repository. + +Today is also the Alpha freeze[4]. This means that only packages which fix accepted blocker or freeze exception bugs[5][6] will be marked as 'stable' and included in the Alpha composes. Other builds will remain in updates-testing until the Alpha release is approved, at which point the Alpha freeze is lifted and packages can move to 'stable' as usual until the Beta freeze. + +Today is also the Software String freeze[7], which means that strings marked for translation in Fedora-translated projects should not now be changed for Fedora 25. + +Finally, today is the 'completion deadline' Change Checkpoint[8], meaning that Fedora 25 Changes must now be 'feature complete or close enough to completion that a majority of its functionality can be tested'. + +Regards + + +[1] https://fedorapeople.org/groups/schedule/f-34/f-34-key-tasks.html +[2] https://fedoraproject.org/wiki/Updates_Policy#Bodhi_enabling +[3] https://fedoraproject.org/wiki/Updates_Policy#Branched_release +[4] https://fedoraproject.org/wiki/Milestone_freezes +[5] https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process +[6] https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process +[7] https://fedoraproject.org/wiki/ReleaseEngineering/StringFreezePolicy +[8] https://fedoraproject.org/wiki/Changes/Policy +.... + +=== Verification + +Compare koji tagging structure with older release + +.... +$ koji list-tag-inheritance --reverse +$ koji list-tag-inheritance --reverse +.... + +Compare the bodhi release with older release + +.... +$ bodhi releases info +$ bodhi releases info +.... + +Check for other variants like modular, container and flatpaks + +=== Consider Before Running + +No considerations at this time. The docs git repository is simply a +static html hosting space and we can just re-render the docs and push to +it again if necessary. diff --git a/modules/release_guide/pages/sop_branch_freeze.adoc b/modules/release_guide/pages/sop_branch_freeze.adoc new file mode 100644 index 0000000..75af952 --- /dev/null +++ b/modules/release_guide/pages/sop_branch_freeze.adoc @@ -0,0 +1,33 @@ +== Branching Freeze + +=== Introduction/Background + +When the next release is branched from rawhide, it initially composes +much like rawhide with nightly composes and no updates process. + +Once the Bodhi is activated, we will push updates to the branched and +the nightly composes will start to differ. But two weeks before the +scheduled release of either Beta or GA, we will start the freeze for +that release and stop pushing updates. + +* Send announcement to devel-announce mailing list noting that the alpha +change freeze is going to happen at least one day in advance. + +[NOTE] +.Note +==== +For updates pushers: In Change freeze only updates that fix accepted +blockers or Freeze break bugs are allowed into the main tree. Please +coordinate with QA for any stable updates pushes. Otherwise ONLY push +updates-testing. +==== + +[NOTE] +.Note +==== +For Final/GA release: During Final freeze, we dont want to block any +packages in koji as it will effect the RC composes. So, please update +the +https://pagure.io/releng/blob/master/f/scripts/block_retired.py[block_retired.py] +script and remove the branched release reference. +==== diff --git a/modules/release_guide/pages/sop_branching.adoc b/modules/release_guide/pages/sop_branching.adoc new file mode 100644 index 0000000..6804dc9 --- /dev/null +++ b/modules/release_guide/pages/sop_branching.adoc @@ -0,0 +1,33 @@ +== Branching + +=== Description + +This SOP covers how to make git and pkgdb branches for packages, either +for new packages that have passed review, or for existing packages that +need a new branch (e.g. EPEL). Release Engineering has written a script +to automate this process. + +=== Normal Action (automated) + +. On your local system (not on an infrastructure hosted system), be sure +you have the following packages installed: +* python-bugzilla +* python-configobj +* python-fedora +. Run "bugzilla login" and successfully receive an Authorization cookie. +. {blank} ++ +Clone the fedora-infrastructure tools repository::: +.... +git clone https://pagure.io/releng.git +.... +. In scripts/process-git-requests, run "process-git-requests". Answer +the prompts. + +=== Manual Action + +==== Creating a new branch for an existing package + +. ssh into `pkgs.fedoraproject.org` +. `pkgdb-client edit -u $YOURUSERNAME -b $NEWBRANCH --master=devel $NAMEOFPACKAGE` +. `pkgdb2branch.py $NAMEOFPACKAGE` diff --git a/modules/release_guide/pages/sop_breaking_development_freeze.adoc b/modules/release_guide/pages/sop_breaking_development_freeze.adoc new file mode 100644 index 0000000..6b3370f --- /dev/null +++ b/modules/release_guide/pages/sop_breaking_development_freeze.adoc @@ -0,0 +1,51 @@ +== Breaking Development Freeze + +`FIXME: NEED TO FIGURE OUT HOW TO FEDORA-VERSION-NEXT` + +=== Description + +Packages which require an exception to freeze policies must be run +through this SOP. + +The following freeze policies are set for the following significant +release milestones: + +* https://fedoraproject.org/wiki/Software_String_Freeze_Policy[String +Freeze] +* https://fedoraproject.org/wiki/Milestone_freezes[Beta Freeze] +* https://fedoraproject.org/wiki/Milestone_freezes[Final Freeze] + +See https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle[Fedora +Release Life Cycle] for a summary of all the freezes, dates, and +exception handling, or the release engineering +[https://fedorapeople.org/groups/schedule/f-\{\{FedoraVersionNumbernext}}-releng-tasks.html +calendar for the current release]. + +=== Action + +The commands to tag a package properly once it has been accepted: + +.... +$ koji move-pkg --force dist-f{{FedoraVersionNumber|next}}-updates-candidate dist-f{{FedoraVersionNumber|next}} +$ koji tag-pkg --force f{{FedoraVersionNumber|next}}- +.... + +Where is the package name, and is the first release +in which the package should land (e.g. alpha, beta, final). + +=== Verification + +The `koji` client reports success or failure. For secondary +verification, run these commands: + +.... +$ koji latest-pkg dist-f{{FedoraVersionNumber|next}} +$ koji latest-pkg dist-f{{FedoraVersionNumber|next}}-updates-candidate +.... + +=== Consider Before Running + +* Change agrees with stated policies (see links above) +* Approval for change has been granted under +https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process[Blocker Bug +Process] or [.title-ref]#Freeze Exception Bug Process# diff --git a/modules/release_guide/pages/sop_clean_amis.adoc b/modules/release_guide/pages/sop_clean_amis.adoc new file mode 100644 index 0000000..0be1e5e --- /dev/null +++ b/modules/release_guide/pages/sop_clean_amis.adoc @@ -0,0 +1,59 @@ +== Clean AMIs Process + +=== Description + +The Fedora AMIs are uploaded on a daily basis to Amazon Web Services. +Over time the number of AMIs piles up and have to be removed manually. +Manual removal comes with it's own set of issues where missing to delete +the AMIs is a viable issue. + +The goal of the script is to automate the process and continue regular +removal of the AMIs. The report of the script is pushed to a +https://pagure.io/ami-purge-report[Pagure repo] + +=== Action + +There is a script in the https://pagure.io/releng[Fedora RelEng repo] +named `clean-amis.py` under the `scripts` directory. + +The script runs as a cron job within the Fedora Infrastructure to delete +the old AMIs. The permission of the selected AMIs are changed to +private. This is to make sure that if someone from the community raises +an issue we have the option to get the AMI back to public. After 10 +days, if no complaints are raised the AMIs are deleted permanently. + +The complete process can be divided in couple of parts: + +* Fetching the data from datagrepper. Based on the [.title-ref]#--days# +param, the script starts fetching the fedmsg messages from datagrepper +for the specified timeframe i.e. for lasts [.title-ref]#n# days, where +[.title-ref]#n# is the value of [.title-ref]#--days# param. The queried +fedmsg topic [.title-ref]#fedimg.image.upload#. +* Selection of the AMIs: After the AMIs are parsed from datagrepper. The +AMIs are filtered to remove Beta, Two-week Atomic Host and GA released +AMIs. Composes with [.title-ref]#compose_type# set to +[.title-ref]#nightly# are picked up for deletion. Composes which contain +date in the [.title-ref]#compose label# are also picked up for deletion. +GA composes also have the compose_type set to production. So to +distinguish then we filter them if the compose_label have date in them. +The GA composes dont have date whereas they have the version in format +of X.Y +* Updated permissions of AMIs The permissions of the selected AMIs are +changed to private. +* Deletion of AMIs After 10 days, the private AMIs are deleted. + +In order to change the permissions of the AMIs use the command given +below, add [.title-ref]#--dry-run# argument test the command works. +Adding [.title-ref]#--dry-run# argument will print the AMIs to console. + +.... +AWS_ACCESS_KEY={{ ec2_image_delete_access_key_id }} AWS_SECRET_ACCESS_KEY={{ ec2_image_delete_access_key }} PAGURE_ACCESS_TOKEN={{ ami_purge_report_api_key }} ./clean-amis.py --change-perms --days 7 --permswaitperiod 5 +.... + +In order to delete the AMIs whose launch permissions have been removed, +add [.title-ref]#--dry-run# argument test the command works. Adding +[.title-ref]#--dry-run# argument will print the AMIs to console. + +.... +AWS_ACCESS_KEY={{ ec2_image_delete_access_key_id }} AWS_SECRET_ACCESS_KEY={{ ec2_image_delete_access_key }} PAGURE_ACCESS_TOKEN={{ ami_purge_report_api_key }} ./clean-amis.py --delete --days 17 --deletewaitperiod 10 +.... diff --git a/modules/release_guide/pages/sop_composing_fedora.adoc b/modules/release_guide/pages/sop_composing_fedora.adoc new file mode 100644 index 0000000..da22869 --- /dev/null +++ b/modules/release_guide/pages/sop_composing_fedora.adoc @@ -0,0 +1,622 @@ +== Composing Fedora + +=== Description + +All composes are defined by configuration files kept in the +https://pagure.io/pungi-fedora[pungi-fedora repository]. + +Composes fall into two categories. They may be release candidates +created on demand or nightly composes set to run at a scheduled time +each day. + +[cols=",,",options="header",] +|=== +|Compose Name |Configuration File |Compose Script +|Docker |fedora-docker.conf |docker-nightly.sh +|Cloud |fedora-cloud.conf |cloud-nightly.sh +|Modular |fedora-modular.conf |modular-nightly.sh +|Nightly |fedora.conf |nightly.sh +|=== + +When Quality Engineering (QE) requests a Release Candidate (RC) they do +so by opening an issue in the releng repository on pagure. Release +candidate composes are not currently automated. + +[cols=",,",options="header",] +|=== +|Compose Name |Configuration File |Compose Script +|Beta |fedora-beta.conf |release-candidate.sh +|GA |fedora-final.conf |release-candidate.sh +|=== + +=== Action + +The following procedures are for release candidates only. They do not +apply to the scheduled nightly composes. + +==== Review Compose Tags + +. List any pre-existing builds in the current compose tag ++ +.... +$ koji list-tagged f[release_version]-compose +.... +. Verify pre-existing builds are in compose tags ++ +The tagged builds from the previous composes should all be present in +the output from the previous step. Consult the request ticket for the +list of builds expected in this output. ++ +[NOTE] +.Note +==== +The very first run of an Beta, or GA compose should have no builds +listed under the compose tag. It is important to clear pre-existing +builds from the compose tag when moving between the Beta and RC +composes. Verify that these builds were removed. + +.... +$ koji list-tagged f[release_version]-compose +$ koji untag-build --all f[release_version]-compose [build1 build2 ...] +.... +==== ++ +[NOTE] +.Note +==== +The order in which packages are added into the +f[release_version]-compose tag matter. If the builds are untagged +erroneously then special attention should be given to adding them back +correctly. +==== +. Add builds specified by QE to the current compose tag ++ +.... +$ koji tag-build f[release_version]-compose [build1 build2 ...] +.... ++ +[NOTE] +.Note +==== +These steps may be completed on a local machine as long as the user has +appropriate permissions in the koji tool. +==== + +==== Package Signing before the Compose + +. Check for unsigned packages ++ +.... +$ koji list-tagged f[release_version]-signing-pending +.... ++ +[NOTE] +.Note +==== +If there are unsigned builds then wait for the automated queue to pick +them up and sign them. Contact a member of the Fedora infrastructure +team if the package signing has taken more than thirty minutes. +==== + +==== Running the Compose + +. Update the pungi-fedora config file Composes use a configuration file +to construct the compose. Each compose uses its own configuration. The +`global_release` variable should start from 1.1 and the second number +should increment each time a new compose is created. +* Beta - `fedora-beta.conf` +* RC - `fedora-final.conf` +. Log into the compose backend ++ +.... +$ ssh compose-x86-01.phx2.fedoraproject.org +.... +. Open a screen session ++ +.... +$ screen +.... +. Obtain the pungi-fedora branch for the current compose ++ +The first time any user account executes a compose the pungi-fedora git +repository must be cloned. The compose candidate script that invokes +pungi should be run from `compose-x86-01.iad2.fedoraproject.org`. ++ +.... +$ git clone ssh://git@pagure.io/pungi-fedora.git +.... ++ +Enter the pungi-fedora directory. ++ +.... +$ cd pungi-fedora +.... ++ +If the clone step above was not required then fully update the existing +repository checkout from pagure. ++ +.... +$ git fetch origin +$ git checkout f[release_version] +$ git pull origin f[release_version] +.... +. Run the compose ++ +.... +$ sudo ./release-candidate.sh [Beta|RC]-#.# +.... ++ +The numbering scheme begins with 1.1 and the second number is +incremented after each compose. ++ +[NOTE] +.Note +==== +Pungi requires numbers in the format #.# as an argument. It is because +of this that composes always start with the number 1 and the second +number is incremented with each compose. +==== ++ +[NOTE] +.Note +==== +If the compose fails with a directory missing error, then create the +compose directory with `mkdir /mnt/koji/compose/[release_version]` +==== + +==== Syncing the Compose + +We sync the compose to `/pub/alt/stage` to enable faster access to new +content for QA and the larger Fedora community. + +. Log into the compose backend ++ +.... +$ ssh compose-x86-01.iad2.fedoraproject.org +.... +. Open a screen session ++ +.... +$ screen +.... +. Check the status of the compose ++ +.... +$ cat /mnt/koji/compose/[release_version]/[compose_id]/STATUS +.... ++ +Do not continue with any further steps if the output above is `DOOMED`. +. Create the directory targeted for the copy : ++ +.... +$ sudo -u ftpsync mkdir -m 750 -p /pub/alt/stage/[release_version]_[release_label]-[#.#] +.... +. Locate the compose directory that will be the copy source : ++ +.... +$ ls /mnt/koji/compose/[release_version]/[compose_id] +.... ++ +[NOTE] +.Note +==== +Take care executing the synchronization if the next compose is already +running. Be sure to grab the correct directory. + +If in doubt, check +/mnt/koji/compose/[release_version]/[compose_id]/STATUS to be sure it is +finished. +==== +. Run the synchronization one-liner ++ +The synchronization of the completed compose to the public domain is +currently a one-liner shell script. Pay close attention to what needs +replaced in the example below. ++ +.... +$ sudo -u ftpsync sh -c 'for dir in Everything Cloud Container Kinoite Labs Modular Server Silverblue Spins Workstation metadata; do rsync -avhH /mnt/koji/compose/31/Fedora-31-20190911.0/compose/$dir/ /pub/alt/stage/31_Beta-1.1/$dir/ --link-dest=/pub/fedora/linux/development/31/Everything/ --link-dest=/pub/alt/stage/31_Beta-1.1/Everything/; done' +.... ++ +[NOTE] +.Note +==== +If multiple composes are run like 1.2, 1.3, add multiple --link-dest +arguments above with multiple composes +==== +. Set the permissions of the synced compose : ++ +.... +$ sudo -u ftpsync chmod 755 /pub/alt/stage/[release_version]_[release_label]-[#.#] +.... +. Update the issue in the releng pagure repository ++ +Once the compose and sync is complete the issue in pagure should be +updated and closed. ++ +Standard Ticket Verbage + +Compose is done and available from +https://kojipkgs.fedoraproject.org/compose/26/Fedora-26-20170328.0/compose/ +it has been synced to +http://dl.fedoraproject.org/pub/alt/stage/26_Alpha-1.4/ rpms have all be +hardlinked to /pub/fedora/linux/development/26/ + +===== Verification + +The method for verifying a compose has completed is checking +`/mnt/koji/compose/[release_version]/[compose_dir]/STATUS`. Any status +other than DOOMED is OK. + +=== Pre-Release Work + +==== Pushing Updates to Stable + +When the release is signed off on Thursday after the Go/No-Go meeting, +push the freeze and blocker to stable updates + +Generally the updates are requested stable by QA. If they are not +available, you can request them by following + +.... +$ bodhi updates request stable +.... + +Once the updates are requested stable, please push them to stable by +following the +https://docs.pagure.org/releng/sop_pushing_updates.html#pushing-stable-updates-during-freeze[bodhi +push to stable sop] + +==== koji tag changes + +Once the updates are pushed stable, we need to clone the koji tag for +beta release or lock the koji tag for final release. + +===== For Beta Release + +.... +$ koji clone-tag --all --latest-only f31 f31-Beta +$ koji clone-tag --all --latest-only f31-modular f31-Beta-modular +.... + +===== For Final Release + +.... +$ koji edit-tag --lock f31 +$ koji edit-tag --lock f31-modular +.... + +==== Bodhi Changes + +Set the bodhi release to `current` + +.... +$ bodhi releases edit --name F31 --state current +.... + +=== Changes for Final Release + +Once Final is GO, we need to perform different changes as that of Beta +release. + +==== Last Branched Compose + +Manually run a branched compose so that the GOLD content is same as the +nightly compose. This also helps in updating the silverblue refs as that +of the GOLD content. + +==== Update silverblue refs + +Please update the refs as per the following commands on +[.title-ref]#bodhi-backend01.phx2.fedoraproject.org# + +Run the following commands from +[.title-ref]#/mnt/koji/compose/ostree/repo# and +[.title-ref]#/mnt/koji/ostree/repo/# + +.... +$ sudo -u ftpsync ostree refs --create=fedora/31/x86_64/updates/silverblue fedora/31/x86_64/silverblue +$ sudo -u ftpsync ostree refs --create=fedora/31/aarch64/updates/silverblue fedora/31/aarch64/silverblue +$ sudo -u ftpsync ostree refs --create=fedora/31/ppc64le/updates/silverblue fedora/31/ppc64le/silverblue + +$ sudo ostree refs --delete fedora/31/x86_64/silverblue +$ sudo ostree refs --delete fedora/31/aarch64/silverblue +$ sudo ostree refs --delete fedora/31/ppc64le/silverblue + +$ sudo -u ftpsync ostree refs --alias --create=fedora/31/x86_64/silverblue fedora/31/x86_64/updates/silverblue +$ sudo -u ftpsync ostree refs --alias --create=fedora/31/aarch64/silverblue fedora/31/aarch64/updates/silverblue +$ sudo -u ftpsync ostree refs --alias --create=fedora/31/ppc64le/silverblue fedora/31/ppc64le/updates/silverblue +.... + +Run the following command only from [.title-ref]#/mnt/koji/ostree/repo/# + +.... +$ sudo ostree summary -u +.... + +[NOTE] +.Note +==== +Before pushing the updates to fxx-updates, run the last branched compose +so that both branched and rc composes have the same content. Once the +branched compose is done, then update the silverblue refs as mentioned +above. If the order is changed, that will screw up the refs +==== + +==== Disable Branched Compose + +Now that we have a final GOLD compose, we dont need nightly branched +composes anymore. This is disabled in +https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/releng[releng +role] in infra ansible repo and then running the playbook. + +.... +$ sudo rbac-playbook groups/releng-compose.yml +.... + +==== Lift RelEng freeze + +Lift the RelEng Freeze so that the updates will be pushed to stable. +This is done by editing +https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/vars/all/RelEngFrozen.yaml[RelEngFrozen +variable] in infra ansible repo and then run the bodhi playbook. + +.... +$ sudo rbac-playbook groups/bodhi-backend.yml +.... + +==== Other Changes + +These changes include enabling nightly container and cloud composes, +other variable changes in infra ansible repo, bodhi pungi config +changes, updates sync changes and others. + +Run the appropriate playbooks after the following changes + +.... +diff --git a/roles/releng/files/branched b/roles/releng/files/branched + index 966f5c3..1c0454f 100644 + --- a/roles/releng/files/branched + +++ b/roles/releng/files/branched + @@ -1,3 +1,3 @@ + # branched compose + -MAILTO=releng-cron@lists.fedoraproject.org + -15 7 * * * root TMPDIR=`mktemp -d /tmp/branched.XXXXXX` && cd $TMPDIR && git clone https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && /usr/local/bin/lock-wrapper branched-compose "PYTHONMALLOC=debug LANG=en_US.UTF-8 ./nightly.sh" && sudo -u ftpsync /usr/local/bin/update-fullfiletimelist -l /pub/fedora-secondary/update-fullfiletimelist.lock -t /pub fedora fedora-secondary + +#MAILTO=releng-cron@lists.fedoraproject.org + +#15 7 * * * root TMPDIR=`mktemp -d /tmp/branched.XXXXXX` && cd $TMPDIR && git clone https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && /usr/local/bin/lock-wrapper branched-compose "PYTHONMALLOC=debug LANG=en_US.UTF-8 ./nightly.sh" && sudo -u ftpsync /usr/local/bin/update-fullfiletimelist -l /pub/fedora-secondary/update-fullfiletimelist.lock -t /pub fedora fedora-secondary + diff --git a/roles/releng/files/cloud-updates b/roles/releng/files/cloud-updates + index a0ffbe8..287d57d 100644 + --- a/roles/releng/files/cloud-updates + +++ b/roles/releng/files/cloud-updates + @@ -6,6 +6,6 @@ MAILTO=releng-cron@lists.fedoraproject.org + MAILTO=releng-cron@lists.fedoraproject.org + 15 7 * * * root TMPDIR=`mktemp -d /tmp/CloudF29.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f29 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR + + -#Fedora 28 Cloud nightly compose + -#MAILTO=releng-cron@lists.fedoraproject.org + -#15 8 * * * root TMPDIR=`mktemp -d /tmp/CloudF28.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f28 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR + +#Fedora 31 Cloud nightly compose + +MAILTO=releng-cron@lists.fedoraproject.org + +15 8 * * * root TMPDIR=`mktemp -d /tmp/CloudF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR + diff --git a/roles/releng/files/container-updates b/roles/releng/files/container-updates + index d763149..5446840 100644 + --- a/roles/releng/files/container-updates + +++ b/roles/releng/files/container-updates + @@ -1,6 +1,6 @@ + -#Fedora 28 Container Updates nightly compose + -#MAILTO=releng-cron@lists.fedoraproject.org + -#45 5 * * * root TMPDIR=`mktemp -d /tmp/containerF28.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f28 && LANG=en_US.UTF-8 ./container-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR + +#Fedora 31 Container Updates nightly compose + +MAILTO=releng-cron@lists.fedoraproject.org + +45 5 * * * root TMPDIR=`mktemp -d /tmp/containerF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./container-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR + + # Fedora 30 Container Updates nightly compose + MAILTO=releng-cron@lists.fedoraproject.org + diff --git a/vars/all/00-FedoraCycleNumber.yaml b/vars/all/00-FedoraCycleNumber.yaml + index 22476b0..4bd0d46 100644 + --- a/vars/all/00-FedoraCycleNumber.yaml + +++ b/vars/all/00-FedoraCycleNumber.yaml + @@ -1 +1 @@ + -FedoraCycleNumber: 30 + +FedoraCycleNumber: 31 + + diff --git a/vars/all/FedoraBranched.yaml b/vars/all/FedoraBranched.yaml + index 42ac534..0bbcc1d 100644 + --- a/vars/all/FedoraBranched.yaml + +++ b/vars/all/FedoraBranched.yaml + @@ -1 +1 @@ + -FedoraBranched: True + +FedoraBranched: False + + diff --git a/vars/all/FedoraPreviousPrevious.yaml b/vars/all/FedoraPreviousPrevious.yaml + index a8e3d3b..a061e04 100644 + --- a/vars/all/FedoraPreviousPrevious.yaml + +++ b/vars/all/FedoraPreviousPrevious.yaml + @@ -1 +1 @@ + -FedoraPreviousPrevious: False + +FedoraPreviousPrevious: True + diff --git a/vars/all/Frozen.yaml b/vars/all/Frozen.yaml + index 97d3bc3..7578a88 100644 + --- a/vars/all/Frozen.yaml + +++ b/vars/all/Frozen.yaml + @@ -1 +1 @@ + -Frozen: True + +Frozen: False + + + diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 + index 688bade..28b524a 100644 + --- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 + +++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 + @@ -179,8 +179,8 @@ ostree = { + # In the case of testing, also inject the last stable updates + "https://kojipkgs{{ env_suffix }}.fedoraproject.org/compose/updates/f[[ release.version_int ]]-updates/compose/Everything/$basearch/os/", + [% endif %] + - # For f31 the compose location is going to be under /compose/branched/ + - [% if release.version_int == 31 %] + + # For F32 the compose location is going to be under /compose/branched/ + + [% if release.version_int == 32 %] + "https://kojipkgs{{ env_suffix }}.fedoraproject.org/compose/branched/latest-Fedora-[[ release.version_int ]]/compose/Everything/$basearch/os/" + [% else %] + "https://kojipkgs{{ env_suffix }}.fedoraproject.org/compose/[[ release.version_int ]]/latest-Fedora-[[ release.version_int ]]/compose/Everything/$basearch/os/" + + diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 + index 28b524a..640ddf0 100644 + --- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 + +++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 + @@ -193,8 +193,8 @@ ostree = { + "ostree_ref": "fedora/[[ release.version_int ]]/${basearch}/testing/silverblue", + [% endif %] + "tag_ref": False, + - "arches": ["x86_64"], + - "failable": ["x86_64"] + + "arches": ["x86_64", "ppc64le", "aarch64" ], + + "failable": ["x86_64", "ppc64le", "aarch64" ] + }, + ] + } + + + diff --git a/roles/bodhi2/backend/files/new-updates-sync b/roles/bodhi2/backend/files/new-updates-sync + index d08c893..2d0fb4d 100755 + --- a/roles/bodhi2/backend/files/new-updates-sync + +++ b/roles/bodhi2/backend/files/new-updates-sync + @@ -25,8 +25,9 @@ RELEASES = {'f31': {'topic': 'fedora', + 'modules': ['fedora', 'fedora-secondary'], + 'repos': {'updates': { + 'from': 'f31-updates', + - 'ostrees': [{'ref': 'fedora/31/x86_64/updates/silverblue', + - 'dest': OSTREEDEST}], + + 'ostrees': [{'ref': 'fedora/31/%(arch)s/updates/silverblue', + + 'dest': OSTREEDEST, + + 'arches': ['x86_64', 'ppc64le', 'aarch64']}], + 'to': [{'arches': ['x86_64', 'armhfp', 'aarch64', 'source'], + 'dest': os.path.join(FEDORADEST, '31', 'Everything')}, + {'arches': ['ppc64le', 's390x'], + @@ -34,8 +35,9 @@ RELEASES = {'f31': {'topic': 'fedora', + ]}, + 'updates-testing': { + 'from': 'f31-updates-testing', + - 'ostrees': [{'ref': 'fedora/31/x86_64/testing/silverblue', + - 'dest': OSTREEDEST}], + + 'ostrees': [{'ref': 'fedora/31/%(arch)s/testing/silverblue', + + 'dest': OSTREEDEST, + + 'arches': ['x86_64', 'ppc64le', 'aarch64']}], + 'to': [{'arches': ['x86_64', 'aarch64', 'armhfp', 'source'], + 'dest': os.path.join(FEDORADEST, 'testing', '31', 'Everything')}, + {'arches': ['ppc64le', 's390x'], + + + diff --git a/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json b/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json + index aac977e..9e0cbf2 100644 + --- a/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json + +++ b/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json + @@ -12,14 +12,14 @@ + "version": "devel" + }, + { + - "allow_retire": true, + + "allow_retire": false, + "branchname": "f31", + "date_created": "2014-05-14 12:36:15", + "date_updated": "2018-08-14 17:07:23", + "dist_tag": ".fc31", + "koji_name": "f31", + "name": "Fedora", + - "status": "Under Development", + + "status": "Active", + "version": "31" + }, + { +.... + +==== Bodhi config + +==== After Beta + +.... +diff --git a/vars/all/FedoraBranchedBodhi.yaml b/vars/all/FedoraBranchedBodhi.yaml +index 606eb2e..ca2ba61 100644 +--- a/vars/all/FedoraBranchedBodhi.yaml ++++ b/vars/all/FedoraBranchedBodhi.yaml +@@ -3,4 +3,4 @@ +#prebeta: After bodhi enablement/beta freeze and before beta release +#postbeta: After beta release and before final release +#current: After final release +-FedoraBranchedBodhi: prebeta ++FedoraBranchedBodhi: postbeta +.... + +==== After Final + +.... +diff --git a/vars/all/FedoraBranchedBodhi.yaml b/vars/all/FedoraBranchedBodhi.yaml +index 380f61d..76ba14d 100644 +--- a/vars/all/FedoraBranchedBodhi.yaml ++++ b/vars/all/FedoraBranchedBodhi.yaml +@@ -1,2 +1,2 @@ +#options are: prebeta, postbeta, current +-FedoraBranchedBodhi: postbeta ++FedoraBranchedBodhi: current +.... + +==== Mirroring + +Run [.title-ref]#stage-release.sh# script from +https://pagure.io/releng[releng repo] in pagure on +[.title-ref]#bodhi-backend01.phx2.fedoraproject.org#, this will sign the +checksums and will put the content on mirrors. + +===== For Beta + +.... +$ sh scripts/stage-release.sh 32_Beta Fedora-32-20200312.0 32_Beta-1.2 fedora-32 1 +.... + +===== For Final + +.... +$ sh scripts/stage-release.sh 32 Fedora-32-20200415.0 32_RC-1.1 fedora-32 0 +.... + +[NOTE] +.Note +==== +Make sure to grab the directory size usage numbers which is used to send +an email to [.title-ref]#mirror-admin@lists.fedoraproject.org# list. +==== + +==== Sync the signed checksums to stage + +We need to sync the signed checksums to /pub/alt/stage/ by running the +following command + +.... +$ for dir in Everything Cloud Container Labs Server Spins Workstation Silverblue Kinoite metadata; do sudo -u ftpsync rsync -avhH /mnt/koji/compose/37/Fedora-37-20221105.0/compose/$dir/ /pub/alt/stage/37_RC-1.7/$dir/ --link-dest=/pub/fedora/linux/releases/37/Everything/ --link-dest=/pub/alt/stage/37_RC-1.2/Everything/ --link-dest=/pub/alt/stage/37_RC-1.3/Everything --link-dest=/pub/alt/stage/37_RC-1.4/Everything --link-dest=/pub/alt/stage/37_RC-1.5/Everything --link-dest=/pub/alt/stage/37_RC-1.6/Everything --link-dest=/pub/alt/stage/37_RC-1.7/Everything; done +.... + +=== Move development to release folder with mirrormanager + +Two weeks after the release move bits from development to release +directory + +. {blank} ++ +ssh to the mm-backend01.iad2.fedoraproject.org:: + ::;; + $ ssh mm-backend01.iad2.fedoraproject.org +. {blank} ++ +get root:: + ::;; + $ sudo su +. {blank} ++ +run the mm2_move-devel-to-release:: + ::;; + $ mm2_move-devel-to-release --version=35 --category='Fedora Linux' + +=== Consider before Running + +Composes and file synchronizations should be run in a screen session on +a remote machine. This enables the operator to withstand network +connection issues. diff --git a/modules/release_guide/pages/sop_create_release_signing_key.adoc b/modules/release_guide/pages/sop_create_release_signing_key.adoc new file mode 100644 index 0000000..9e8f68a --- /dev/null +++ b/modules/release_guide/pages/sop_create_release_signing_key.adoc @@ -0,0 +1,536 @@ +== Create Release Signing Key + +=== Description + +At the beginning of each release under development a new package signing +key is created for it. This key is used to prove the authenticity of +packages built by Fedora and distributed by Fedora. This key will be +used to sign all packages for the public test and final releases. + +=== Action + +==== Sigul + +Sigul is the signing server which holds our keys. In order to make use +of a new key, the key will have to be created and access to the key will +have to be granted. The `new-key`, `grant-key-access`, and +`change-passphrase` commands are used. + +.... +$ sigul new-key --help +usage: client.py new-key [options] key + +Add a key + +options: + -h, --help show this help message and exit + --key-admin=USER Initial key administrator + --name-real=NAME_REAL + Real name of key subject + --name-comment=NAME_COMMENT + A comment about of key subject + --name-email=NAME_EMAIL + E-mail of key subject + --expire-date=YYYY-MM-DD + Key expiration date + +$ sigul grant-key-access --help +usage: client.py grant-key-access key user + +Grant key access to a user + +options: + -h, --help show this help message and exit + +$ sigul change-passphrase --help +usage: client.py change-passphrase key + +Change key passphrase + +options: + -h, --help show this help message and exit +.... + +For example if we wanted to create the Fedora 23 signing key, we would +do the following: + +. Log into a system configured to run sigul client. +. Create the key using a strong passphrase when prompted ++ +.... +$ sigul new-key --key-admin ausil --key-type gnupg \ + --gnupg-name-real Fedora \ + --gnupg-name-comment 23 \ + --gnupg-name-email fedora-23-primary@fedoraproject.org fedora-23 +.... ++ +For EPEL ++ +.... +$ sigul new-key --key-admin ausil --key-type gnupg \ + --gnupg-name-real "Fedora EPEL" \ + --gnupg-name-comment 7 \ + --gnupg-name-email epel@fedoraproject.org epel-7 +.... +. Wait a while for entropy. This can take several minutes. +. For Fedora, also create the IMA signing key ++ +.... +$ sigul new-key --key-admin ausil --key-type ECC fedora-23-ima +.... +. Grant key access to Fedora Account holders who will be signing +packages and protect it with a temporary a passphrase. For example, +`CHANGEME.`. Do the same with the -ima key for Fedora. ++ +.... +$ sigul grant-key-access fedora-23 kevin +.... + +[NOTE] +.Note +==== +*IMPORTANT:* Grant the access to autopen user as it's required for +robosignatory autosigning and then restart robosignatory service +==== + +. Provide the key name and temporary passphrase to signers. If they +don't respond, revoke access until they are ready to change their +passphrase. Signers can change their passphrase using the +`change-passphrase` command: ++ +.... +$ sigul change-passphrase fedora-23 +.... +. When your sigul cert expires, you will need to run: ++ +.... +certutil -d ~/.sigul -D -n sigul-client-cert +.... ++ +to remove the old cert, then ++ +.... +sigul_setup_client +.... ++ +to add a new one. + +==== fedora-repos + +The fedora-repos package houses a copy of the public key information. +This is used by rpm to verify the signature on files encountered. +Currently the fedora-repos package has a single key file named after the +version of the key and the arch the key is for. To continue our example, +the file would be named `RPM-GPG-KEY-fedora-27-primary` which is the +primary arch key for Fedora 27. To create this file, use the +`get-public-key` command from sigul: + +.... +$ sigul get-public-key fedora-27 > RPM-GPG-KEY-fedora-27-primary +.... + +Add this file to the repo and update the archmap file for the new +release. + +.... +$ git add RPM-GPG-KEY-fedora-27-primary +.... + +Then make a new fedora-repos build for rawhide +(`FIXME: this should be its own SOP`) + +==== getfedora.org + +getfedora.org/keys lists information about all of our keys. We need to +let the websites team know we have created a new key so that they can +add it to the list. + +We do this by filing an issues in their pagure instance +https://pagure.io/fedora-websites/ we should point them at this SOP + +===== Web team SOP + +.... +# from git repo root +cd fedoraproject.org/ +curl $KEYURL > /tmp/newkey +$EDITOR update-gpg-keys # Add key ID of recently EOL'd version to obsolete_keys +./update-gpg-key /tmp/newkey +gpg static/fedora.gpg # used to verify the new keyring +# it should look something like this: +# pub 4096R/57BBCCBA 2009-07-29 Fedora (12) +# pub 4096R/E8E40FDE 2010-01-19 Fedora (13) +# pub 4096R/97A1071F 2010-07-23 Fedora (14) +# pub 1024D/217521F6 2007-03-02 Fedora EPEL +# sub 2048g/B6610DAF 2007-03-02 [expires: 2017-02-27] +# it must only have the two supported versions of fedora, rawhide and EPEL +# also verify that static/$NEWKEY.txt exists +$EDITOR data/content/{keys,verify}.html # see git diff 1840f96~ 1840f96 +.... + +==== sigulsign_unsigned + +`sigulsign_unsigned.py` is the script Release Engineers use to sign +content in koji. This script has a hardcoded list of keys and aliases to +the keys that needs to be updated when we create new keys. + +Add the key details to the `KEYS` dictionary near the top of the +`sigulsign_unsigned.py` script. It lives in Release Engineering's git +repo at `ssh://git@pagure.io/releng.git` in the `scripts` directory. You +will need to know the key ID to insert the correct information: + +.... +$ gpg +.... + +==== Public Keyservers + +We upload the key to the public key servers when we create the keys. To +do this, we need to get the ascii key block from sigul, determine the +key ID, import they key into our local keyring, and then upload it to +the key servers. + +.... +$ sigul get-public-key fedora-13 > fedora-13 +$ gpg fedora-13 (The ID is the "E8E40FDE" part of 4096R/E8E40FDE) +$ gpg --import fedora-13 +$ gpg --send-keys E8E40FDE +.... + +==== pungi-fedora + +The nightly compose configs come from the pungi-fedora project on +https://pagure.io We need to create a pull request to pull in the new +key. + +.... +$ git clone ssh://git@pagure.io//pungi-fedora.git +$ cd pungi-fedora +$ vim *conf + +$ git commit -m 'Add new key' +$ git push +$ file a Pull Request +.... + +==== Koji + +Koji has a garbage collection utility that will find builds that meet +criteria to be removed to save space. Part of that criteria has to do +with whether or not the build has been signed with a key. If the +collection utility doesn't know about a key it will ignore the build. +Thus as we create new keys we need to inform the utility of these keys +or else builds can pile up. The configuration for the garbage collection +lives within ansible. + +On a clone of the infrastructure ansible git repo edit the +roles/koji_hub/templates/koji-gc.conf.j2 file: + +.... +diff --git a/roles/koji_hub/templates/koji-gc.conf.j2 b/roles/koji_hub/templates/koji-gc.conf.j2 +index 9ecb750..9c48a8e 100644 +--- a/roles/koji_hub/templates/koji-gc.conf.j2 ++++ b/roles/koji_hub/templates/koji-gc.conf.j2 +@@ -35,6 +35,7 @@ key_aliases = + 81B46521 fedora-24 + FDB19C98 fedora-25 + 64DAB85D fedora-26 ++ F5282EE4 fedora-27 + 217521F6 fedora-epel + 0608B895 fedora-epel-6 + 352C64E5 fedora-epel-7 +@@ -52,6 +53,7 @@ unprotected_keys = + fedora-24 + fedora-25 + fedora-26 ++ fedora-27 + fedora-extras + redhat-beta + fedora-epel +@@ -91,6 +93,7 @@ policy = + sig fedora-24 && age < 12 weeks :: keep + sig fedora-25 && age < 12 weeks :: keep + sig fedora-26 && age < 12 weeks :: keep ++ sig fedora-27 && age < 12 weeks :: keep + sig fedora-epel && age < 12 weeks :: keep + sig fedora-epel-6 && age < 12 weeks :: keep + sig fedora-epel-7 && age < 12 weeks :: keep +.... + +In this case the fedora-epel key was added to the list of key aliases, +then referenced in the list of unprotected_keys, and finally a policy +was created for how long to keep builds signed with this key. + +Once you've made your change commit and push. The buildsystem will pick +up this change the next time puppet refreshes. + +=== Verification + +We can verify that the key was created in sigul, the correct users have +access to the key, the key was added to the fedora-release package, that +the website was updated with the right key, that sigulsign_unsigned was +properly updated, and that the key was successfully updated to the +public key servers. + +==== sigul + +Use the `list-keys` command to verify that the key was indeed added to +sigul: + +.... +$ sigul list-keys +Administrator's password: +fedora-10 +fedora-10-testing +fedora-11 +fedora-12 +fedora-13 +.... + +Our new key should be on the list. This command expects *your* +administrative password. + +Use the `list-key-users` command to verify all the signers have access: + +.... +$ sigul list-key-users fedora-13 +Key passphrase: +jkeating +jwboyer +.... + +This command expects *your* key passphrase for the key in question. + +==== fedora-release + +To verify that the key was added to this package correctly, download the +latest build from koji and run rpm2cpio on it, then run gpg on the key +file: + +.... +$ koji download-build --arch noarch --latest f27 fedora-repos +fedora-repos-rawhide-27-0.1.noarch.rpm | 7.3 kB 00:00:00 +fedora-repos-27-0.1.noarch.rpm | 87 kB 00:00:00 +$ rpmdev-extract fedora-repos-27-0.1.noarch.rpm +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-27-fedora +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-ppc +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-ppc +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-arm +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-armhfp +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-mips +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-secondary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-14-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-14-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-14-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-arm +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-armhfp +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-ppc +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-s390 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-s390x +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-secondary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-arm +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-armhfp +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-ppc +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-s390 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-s390x +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-secondary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-arm +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-armhfp +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-ppc +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-s390 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-s390x +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-secondary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-arm +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-armhfp +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-ppc +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-s390 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-s390x +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-secondary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-armhfp +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-ppc +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-s390 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-s390x +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-secondary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-armhfp +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-ppc +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-s390 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-s390x +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-secondary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-aarch64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-armhfp +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-ppc64le +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-s390 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-s390x +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-secondary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-aarch64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-armhfp +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-ppc64le +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-s390 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-s390x +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-secondary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-aarch64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-armhfp +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-ppc64le +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-s390 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-s390x +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-secondary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-aarch64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-armhfp +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-ppc64le +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-s390x +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-secondary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-aarch64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-armhfp +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-ppc64le +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-s390x +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-secondary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-aarch64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-armhfp +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-ppc64le +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-s390x +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-secondary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-aarch64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-armhfp +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-ppc64le +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-s390x +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-7-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-ppc +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-primary-original +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-x86_64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-i386 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-ia64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-ppc +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-ppc64 +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-primary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-primary-original +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-secondary +fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-x86_64 +fedora-repos-27-0.1.noarch/etc/yum.repos.d +fedora-repos-27-0.1.noarch/etc/yum.repos.d/fedora-cisco-openh264.repo +fedora-repos-27-0.1.noarch/etc/yum.repos.d/fedora-updates-testing.repo +fedora-repos-27-0.1.noarch/etc/yum.repos.d/fedora-updates.repo +fedora-repos-27-0.1.noarch/etc/yum.repos.d/fedora.repo + +$ gpg2 fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-primary +pub rsa4096 2017-02-21 [SCE] + 860E19B0AFA800A1751881A6F55E7430F5282EE4 +uid Fedora 27 (27) + pub 4096R/E8E40FDE 2010-01-19 Fedora (13) +.... + +You may wish to do this in a tempoary directory to make cleaning it up +easy. + +==== getfedora.org + +One can simply browse to https://getfedora.org/keys to verify that the +key has been uploaded. + +==== sigulsign_unsigned + +The best way to test whether or not the key has been added correctly is +to sign a package using the key, like our newly built fedora-repos +package. + +.... +$ ./sigulsign_unsigned.py fedora-13 fedora-release-13-0.3 +Passphrase for fedora-13: +.... + +The command should exit cleanly. + +==== Public key servers + +One can use the search-keys command from gpg to locate the +key on the public server: + +.... +$ gpg2 --search-keys "Fedora (13)" +gpg: searching for "Fedora (13)" from hkp server subkeys.pgp.net +(1) Fedora (13) + 4096 bit RSA key E8E40FDE, created: 2010-01-19 +... +.... + +==== Koji + +Log into koji02.phx2.fedoraproject.org by way of +bastion.fedoraproject.org. + +Verify that `/etc/koji-gc/koji-gc.conf` has the new key in it. + +=== Consider Before Running + +Nothing at this time. diff --git a/modules/release_guide/pages/sop_deprecate_ftbfs_packages.adoc b/modules/release_guide/pages/sop_deprecate_ftbfs_packages.adoc new file mode 100644 index 0000000..d2416a1 --- /dev/null +++ b/modules/release_guide/pages/sop_deprecate_ftbfs_packages.adoc @@ -0,0 +1,123 @@ +== Deprecate FTBFS Packages + +=== Description + +[NOTE] +.Note +==== +FTBFS = "Fails To Build From Source" +==== + +Every release prior to the Feature Freeze we deprecate all packages that +https://fedoraproject.org/wiki/Fails_to_build_from_source[FTBFS]. This +keeps out software that no longer builds from source, and prevents +future problems down the road. + +=== Action + +The FTBFS process takes place in stages: + +. Detecting a list of FTBFS packages and the dependencies that will be +broken if they are removed. +. Sending the list of potential deprecated FTBFS packages to +devel@lists.fedoraproject.org for community review and removal from the +FTBFS list by fixing the package. +. Removing packages confirmed as FTBFS from the Fedora package +repositories. + +==== Detecting FTBFS + +We will remove packages that have failed to build for at least two +release cycles. For example, in preparation for Fedora 21 branching, +packages which FTBFS since the Fedora 19 cycle (i.e. packages that have +a dist tag of fc18 or earlier) will be considered candidates for +removal. Adjust +https://pagure.io/releng/blob/main/f/scripts/find_FTBFS.py[find_FTBFS.py] +and run it to get a list of candidate packages. + +Given a candidate list from above, rel-eng should attempt to build each +of the candidate packages using koji. Should package building now +succeed, the package may be removed from the candidate list. + +==== Announcing Packages to be Deprecated + +Email the output to the development list +(`devel@lists.fedodraproject.org`) at least a week before the feature +freeze. This gives maintainers an opportunity to fix packages that are +important to them. Follow-up on the list where necessary. + +==== Retiring FTBFS packages + +Once maintainers have been given an opportunity to pick up and fix FTBFS +packages, the remaining packages are `retired` by blocking them, and +creating the `dead.package` file in git. + +===== GIT and Package DB + +Required permissions: provenpackage for GIT, cvsadmin for Package DB. + +We just have to remove the existing files from the `rawhide` branch and +replace them with a `dead.package` file whose contents describe why the +package is dead. Also the package needs to be marked as retired in +PackageDB. Fedpkg takes care of this: + +For example, if we wished to clean up git for the roxterm package we +would: + +.... +$ fedpkg clone roxterm +$ cd roxterm +$ fedpkg retire "Retired on $(date -I), because it failed to build for two releases (FTBFS Cleanup)." +.... + +===== Koji + +Required permissions: admin in koji if the automatic blocking fails. + +Blocking should happen automatically a few minutes after the packags was +retired in PackageDB. If it does not, use the `block-pkg` `koji` command +is used to do the blocking. + +Koji accepts multiple package names as input and thus we can use the +FTBFS package list as input. Deprecated packages are only blocked from +the latest `f##` tag. For example, if we wanted to `deprecate` (block) +`sbackup, roxterm,` and `uisp` from rawhide during the development of +Fedora 21 we would run the following command: + +.... +$ koji block-pkg f21 sbackup roxterm uisp +.... + +===== Bugs + +This procedure probably leaves open bugs for the deprecated packages +behind. It is not within the scope of releng to take care of these. If +bugs are closed, only bugs targeted at Rawhide should be affected, since +other branches might still be maintained. + +=== Verification + +To verify that the packages were blocked correctly we can use the +`latest-pkg` `koji` action. + +.... +$ koji latest-pkg dist-f16 wdm +.... + +This should return nothing, as the `wdm` package is blocked. + +Also check that package DB shows that the package is retired and that +the rawhide branch contains only a dead.package file. + +=== Consider Before Running + +Generally we block anything that doesn't leave broken dependencies. If +there are packages whose removal would result in broken dependencies a +second warning should be sent to devel@lists.fedoraproject.org and to +-owner@fedoraproject.org for each dependent package. + +Allow another couple of days for maintainers to take notice and fix +these package so the package repository can be maintained without broken +dependencies or needing to deprecate the package. It is not good to have +broken package dependencies in our package repositories so every effort +should be made to find owners or to fix the broken dependencies. diff --git a/modules/release_guide/pages/sop_end_of_life.adoc b/modules/release_guide/pages/sop_end_of_life.adoc new file mode 100644 index 0000000..064d48a --- /dev/null +++ b/modules/release_guide/pages/sop_end_of_life.adoc @@ -0,0 +1,537 @@ +== End Of Life + +=== Description + +Each release of Fedora is maintained as laid out in the +https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Maintenance_Schedule[maintenance +schedule]. At the conclusion of the maintenance period, a Fedora release +enters `end of life` status. This procedure describes the tasks +necessary to move a release to that status. + +=== Actions + +==== Set date + +* {blank} ++ +Releng responsibilities::: + ** Follow guidelines of + https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Maintenance_Schedule[maintenance + schedule] + ** Take into account any infrastructure or other supporting project + resource contention + ** Announce the closure of the release to the package maintainers. + +==== Reminder announcement + +Send an email to devel@, devel-announce@, test-announce@, announce@ +lists as remainder about the release EOL. + +===== Announcement Content + +.... +Hello all, + +Fedora 31 will go end of life for updates and support on 24th of +November 2020. No further updates, including security updates, will be +available for Fedora 31 after the said date. All the updates of Fedora +31 being pushed to stable will be stopped as well. + +Fedora 32 will continue to receive updates until approximately one +month after the release of Fedora 34. The maintenance schedule of +Fedora releases is documented on the Fedora Project wiki [0]. The +fedora Project wiki also contains instructions [1] on how to upgrade +from a previous release of Fedora to a version receiving updates. + +Regards, +Mohan Boddu. + +[0]https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Maintenance_Schedule +[1]https://fedoraproject.org/wiki/Upgrading?rd=DistributionUpgrades +.... + +==== Koji tasks + +* Disable builds by removing targets + +.... +$ koji remove-target f31 +$ koji remove-target f31-candidate +$ koji remove-target f31-container-candidate +$ koji remove-target f31-flatpak-candidate +$ koji remove-target f31-infra +$ koji remove-target f31-coreos-continuous +$ koji remove-target f31-rebuild +$ koji remote-target #any side targets that are still around +.... + +* Purge from disk the signed copies of rpms that are signed with the +EOL'd release key. To acheive this, add the release key to +*koji_cleanup_signed.py* script in https://pagure.io/releng[releng] repo +and the script on compose-branched01.iad2.fedoraproject.org + +.... +./scripts/koji_cleanup_signed.py +.... + +==== PDC tasks + +* Set PDC *active* value for the release to *False* + +.... +curl -u: -H 'Authorization: Token ' -H 'Accept: application/json' -H 'Content-Type:application/json' -X PATCH -d '{"active":"false"}' https://pdc.fedoraproject.org/rest_api/v1/releases/fedora-31/ +.... + +* Set the EOL dates in PDC for all the components to the release EOL +date if they are not already set. Run the following script from +https://pagure.io/releng[releng] repo + +.... +python scripts/pdc/adjust-eol-all.py f31 2020-11-24 +.... + +==== Bodhi tasks + +* Run the following bodhi commands to set the releases state to +*archived* + +.... +$ bodhi releases edit --name "F31" --state archived +$ bodhi releases edit --name "F31M" --state archived +$ bodhi releases edit --name "F31C" --state archived +$ bodhi releases edit --name "F31F" --state archived +.... + +[WARNING] +.Warning +==== +Due to a https://github.com/fedora-infra/bodhi/issues/2177[bug] in +Bodhi, it is critical that Bodhi processes be restarted any time +`bodhi releases create` or `bodhi releases edit` are used. +==== + +* On bodhi-backend01.iad2.fedoraproject.org, run the following commands + +.... +$ sudo systemctl restart fm-consumer@config.service +$ sudo systemctl restart bodhi-celery.service +.... + +==== Fedora Infra Ansible Changes + +* We need to make changes to bodhi, koji, mbs, releng, autosign roles in +ansible repo. + +[source,diff] +---- +From 73dc8a1042a190f1b88bf78e110d44753cfa7962 Mon Sep 17 00:00:00 2001 +From: Mohan Boddu +Date: Nov 24 2020 17:19:23 +0000 +Subject: F31 EOL + + +Signed-off-by: Mohan Boddu + +--- + +diff --git a/roles/bodhi2/backend/files/new-updates-sync b/roles/bodhi2/backend/files/new-updates-sync +index a143047..d8c8a73 100755 +--- a/roles/bodhi2/backend/files/new-updates-sync ++++ b/roles/bodhi2/backend/files/new-updates-sync +@@ -113,50 +113,6 @@ RELEASES = {'f33': {'topic': 'fedora', + 'dest': os.path.join(FEDORAALTDEST, 'testing', '32', 'Modular')} + ]}} + }, +- 'f31': {'topic': 'fedora', +- 'version': '31', +- 'modules': ['fedora', 'fedora-secondary'], +- 'repos': {'updates': { +- 'from': 'f31-updates', +- 'ostrees': [{'ref': 'fedora/31/%(arch)s/updates/silverblue', +- 'dest': OSTREEDEST, +- 'arches': ['x86_64', 'ppc64le', 'aarch64']}], +- 'to': [{'arches': ['x86_64', 'armhfp', 'aarch64', 'source'], +- 'dest': os.path.join(FEDORADEST, '31', 'Everything')}, +- {'arches': ['ppc64le', 's390x'], +- 'dest': os.path.join(FEDORAALTDEST, '31', 'Everything')} +- ]}, +- 'updates-testing': { +- 'from': 'f31-updates-testing', +- 'ostrees': [{'ref': 'fedora/31/%(arch)s/testing/silverblue', +- 'dest': OSTREEDEST, +- 'arches': ['x86_64', 'ppc64le', 'aarch64']}], +- 'to': [{'arches': ['x86_64', 'aarch64', 'armhfp', 'source'], +- 'dest': os.path.join(FEDORADEST, 'testing', '31', 'Everything')}, +- {'arches': ['ppc64le', 's390x'], +- 'dest': os.path.join(FEDORAALTDEST, 'testing', '31', 'Everything')} +- ]}} +- }, +- 'f31m': {'topic': 'fedora', +- 'version': '31m', +- 'modules': ['fedora', 'fedora-secondary'], +- 'repos': {'updates': { +- 'from': 'f31-modular-updates', +- 'ostrees': [], +- 'to': [{'arches': ['x86_64', 'aarch64', 'armhfp', 'source'], +- 'dest': os.path.join(FEDORADEST, '31', 'Modular')}, +- {'arches': ['ppc64le', 's390x'], +- 'dest': os.path.join(FEDORAALTDEST, '31', 'Modular')} +- ]}, +- 'updates-testing': { +- 'from': 'f31-modular-updates-testing', +- 'ostrees': [], +- 'to': [{'arches': ['x86_64', 'aarch64', 'armhfp', 'source'], +- 'dest': os.path.join(FEDORADEST, 'testing', '31', 'Modular')}, +- {'arches': ['ppc64le', 's390x'], +- 'dest': os.path.join(FEDORAALTDEST, 'testing', '31', 'Modular')} +- ]}} +- }, + 'epel8': {'topic': 'epel', + 'version': '8', + 'modules': ['epel'], +diff --git a/roles/bodhi2/backend/tasks/main.yml b/roles/bodhi2/backend/tasks/main.yml +index a4b2a2b..d84f86a 100644 +--- a/roles/bodhi2/backend/tasks/main.yml ++++ b/roles/bodhi2/backend/tasks/main.yml +@@ -76,7 +76,7 @@ + # bodhi2/backend/files/koji_sync_listener.py + # This cronjob runs only once a day. The listener script runs reactively. + cron: name="owner-sync" minute="15" hour="4" user="root" +- job="/usr/local/bin/lock-wrapper owner-sync '/usr/local/bin/owner-sync-pagure f34 f34-container f34-modular f33 f33-container f33-modular f33-flatpak f32 f32-container f32-modular f32-flatpak f31 f31-container f31-flatpak f31-modular epel8 epel8-playground epel8-modular epel7 dist-6E-epel module-package-list modular'" ++ job="/usr/local/bin/lock-wrapper owner-sync '/usr/local/bin/owner-sync-pagure f34 f34-container f34-modular f33 f33-container f33-modular f33-flatpak f32 f32-container f32-modular f32-flatpak epel8 epel8-playground epel8-modular epel7 dist-6E-epel module-package-list modular'" + cron_file=update-koji-owner + when: env == "production" + tags: +diff --git a/roles/bodhi2/backend/templates/koji_sync_listener.toml b/roles/bodhi2/backend/templates/koji_sync_listener.toml +index 753adc0..41954ca 100644 +--- a/roles/bodhi2/backend/templates/koji_sync_listener.toml ++++ b/roles/bodhi2/backend/templates/koji_sync_listener.toml +@@ -48,10 +48,6 @@ taglist = [ + "f32-container", + "f32-modular", + "f32-flatpak", +- "f31", +- "f31-container", +- "f31-flatpak", +- "f31-modular", + "epel8", + "epel8-playground", + "epel8-modular", +diff --git a/roles/koji_hub/templates/hub.conf.j2 b/roles/koji_hub/templates/hub.conf.j2 +index 2f8b716..4816dba 100644 +--- a/roles/koji_hub/templates/hub.conf.j2 ++++ b/roles/koji_hub/templates/hub.conf.j2 +@@ -187,6 +187,5 @@ sidetag = + tag f34-build :: allow + tag f33-build :: allow + tag f32-build :: allow +- tag f31-build :: allow + tag eln-build :: allow + all :: deny +diff --git a/roles/mbs/common/files/default-modules.production/platform-f31.yaml b/roles/mbs/common/files/default-modules.production/platform-f31.yaml +deleted file mode 100644 +index 0608f93..0000000 +--- a/roles/mbs/common/files/default-modules.production/platform-f31.yaml ++++ /dev/null +@@ -1,28 +0,0 @@ +-data: +- description: Fedora 31 traditional base +- license: +- module: [MIT] +- name: platform +- profiles: +- buildroot: +- rpms: [bash, bzip2, coreutils, cpio, diffutils, fedora-release, findutils, gawk, +- glibc-minimal-langpack, grep, gzip, info, make, patch, redhat-rpm-config, +- rpm-build, sed, shadow-utils, tar, unzip, util-linux, which, xz] +- srpm-buildroot: +- rpms: [bash, fedora-release, fedpkg-minimal, glibc-minimal-langpack, gnupg2, +- redhat-rpm-config, rpm-build, shadow-utils] +- stream: f31 +- summary: Fedora 31 traditional base +- context: 00000000 +- version: 1 +- xmd: +- mbs: +- buildrequires: {} +- commit: f31 +- requires: {} +- koji_tag: module-f31-build +- mse: TRUE +- virtual_streams: [fedora] +-document: modulemd +-version: 1 +- +diff --git a/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json b/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json +index d2f9a89..0eae499 100644 +--- a/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json ++++ b/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json +@@ -41,7 +41,7 @@ + "dist_tag": ".fc31", + "koji_name": "f31", + "name": "Fedora", +- "status": "Active", ++ "status": "EOL", + "version": "31" + }, + { +diff --git a/roles/releng/files/cloud-updates b/roles/releng/files/cloud-updates +index 0a37b49..ebb807c 100644 +--- a/roles/releng/files/cloud-updates ++++ b/roles/releng/files/cloud-updates +@@ -7,5 +7,5 @@ MAILTO=releng-cron@lists.fedoraproject.org +15 7 * * * root TMPDIR=`mktemp -d /tmp/CloudF32.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f32 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR + +# Fedora 31 Cloud nightly compose +-MAILTO=releng-cron@lists.fedoraproject.org +-15 8 * * * root TMPDIR=`mktemp -d /tmp/CloudF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR ++#MAILTO=releng-cron@lists.fedoraproject.org ++#15 8 * * * root TMPDIR=`mktemp -d /tmp/CloudF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR +diff --git a/roles/releng/files/container-updates b/roles/releng/files/container-updates +index d3a0e14..6932bec 100644 +--- a/roles/releng/files/container-updates ++++ b/roles/releng/files/container-updates +@@ -1,6 +1,6 @@ +#Fedora 31 Container Updates nightly compose +-MAILTO=releng-cron@lists.fedoraproject.org +-45 5 * * * root TMPDIR=`mktemp -d /tmp/containerF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./container-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR ++#MAILTO=releng-cron@lists.fedoraproject.org ++#45 5 * * * root TMPDIR=`mktemp -d /tmp/containerF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./container-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR + +# Fedora 33 Container Updates nightly compose +MAILTO=releng-cron@lists.fedoraproject.org +diff --git a/roles/robosignatory/templates/robosignatory.toml.j2 b/roles/robosignatory/templates/robosignatory.toml.j2 +index 41ab24c..60295c1 100644 +--- a/roles/robosignatory/templates/robosignatory.toml.j2 ++++ b/roles/robosignatory/templates/robosignatory.toml.j2 +@@ -92,30 +92,6 @@ handlers = ["console"] + + # Temporary tags + +- [[consumer_config.koji_instances.primary.tags]] +- from = "f31-kde" +- to = "f31-kde" +- key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}" +- keyid = "{{ (env == 'production')|ternary('3c3359c4', 'd300e724') }}" +- +- [[consumer_config.koji_instances.primary.tags]] +- from = "f31-gnome" +- to = "f31-gnome" +- key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}" +- keyid = "{{ (env == 'production')|ternary('3c3359c4', 'd300e724') }}" +- +- [[consumer_config.koji_instances.primary.tags]] +- from = "f31-python" +- to = "f31-python" +- key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}" +- keyid = "{{ (env == 'production')|ternary('3c3359c4', 'd300e724') }}" +- +- [[consumer_config.koji_instances.primary.tags]] +- from = "f30-kde" +- to = "f30-kde" +- key = "{{ (env == 'production')|ternary('fedora-30', 'testkey') }}" +- keyid = "{{ (env == 'production')|ternary('cfc659b9', 'd300e724') }}" +- + # Infra tags + + [[consumer_config.koji_instances.primary.tags]] +@@ -143,12 +119,6 @@ handlers = ["console"] + keyid = "{{ (env == 'production')|ternary('47dd8ef9', 'd300e724') }}" + + [[consumer_config.koji_instances.primary.tags]] +- from = "f31-infra-candidate" +- to = "f31-infra-stg" +- key = "{{ (env == 'production')|ternary('fedora-infra', 'testkey') }}" +- keyid = "{{ (env == 'production')|ternary('47dd8ef9', 'd300e724') }}" +- +- [[consumer_config.koji_instances.primary.tags]] + from = "f32-infra-candidate" + to = "f32-infra-stg" + key = "{{ (env == 'production')|ternary('fedora-infra', 'testkey') }}" +@@ -170,18 +140,6 @@ handlers = ["console"] + # Gated coreos-pool tag + + [[consumer_config.koji_instances.primary.tags]] +- from = "f30-coreos-signing-pending" +- to = "coreos-pool" +- key = "{{ (env == 'production')|ternary('fedora-30', 'testkey') }}" +- keyid = "{{ (env == 'production')|ternary('cfc659b9', 'd300e724') }}" +- +- [[consumer_config.koji_instances.primary.tags]] +- from = "f31-coreos-signing-pending" +- to = "coreos-pool" +- key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}" +- keyid = "{{ (env == 'production')|ternary('3c3359c4', 'd300e724') }}" +- +- [[consumer_config.koji_instances.primary.tags]] + from = "f32-coreos-signing-pending" + to = "coreos-pool" + key = "{{ (env == 'production')|ternary('fedora-32', 'testkey') }}" +@@ -297,19 +255,6 @@ handlers = ["console"] + keyid = "{{ (env == 'production')|ternary('12c944d0', 'd300e724') }}" + type = "modular" + +- [[consumer_config.koji_instances.primary.tags]] +- from = "f31-signing-pending" +- to = "f31-updates-testing-pending" +- key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}" +- keyid = "{{ (env == 'production')|ternary('3c3359c4', 'd300e724') }}" +- +- [[consumer_config.koji_instances.primary.tags]] +- from = "f31-modular-signing-pending" +- to = "f31-modular-updates-testing-pending" +- key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}" +- keyid = "{{ (env == 'production')|ternary('3c3359c4', 'd300e724') }}" +- type = "modular" +- + #epel8 modular tags + [[consumer_config.koji_instances.primary.tags]] + from = "epel8-modular-signing-pending" +@@ -417,12 +362,6 @@ handlers = ["console"] + key = "{{ (env == 'production')|ternary('fedora-32', 'testkey') }}" + keyid = "{{ (env == 'production')|ternary('12c944d0', 'd300e724') }}" + +- [[consumer_config.koji_instances.primary.tags]] +- from = "f31-openh264" +- to = "f31-openh264" +- key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}" +- keyid = "{{ (env == 'production')|ternary('3c3359c4', 'd300e724') }}" +- + # f33-rebuild + [[consumer_config.koji_instances.primary.tags]] + from = "f33-rebuild" +---- + +* Run the associated playbooks on _batcave_ + +.... +$ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/bodhi-backend.yml +$ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/koji-hub.yml +$ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/mbs.yml +$ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/releng-compose.yml +$ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/proxies -t pkgdb2 +$ sudo ansible-playbook /srv/web/infra/ansible/playbooks/manual/autosign.yml +$ sudo ansible-playbook /srv/web/infra/ansible/playbooks/openshift-apps/bodhi.yml +.... + +==== MBS Platform Retirement + +* To retire the platform in mbs, run the following command on +mbs-backend01.iad2.fedoraproject.org + +.... +$ sudo mbs-manager retire platform:f31 +.... + +==== Final announcement + +* Send the final announcement to devel@, devel-announce@, +test-announce@, announce@ lists + +===== Announcement content + +.... +Hello all, + +As of the 24th of November 2020, Fedora 31 has reached its end of life +for updates and support. No further updates, including security +updates, will be available for Fedora 31. All the updates that are +currently in testing won't get pushed to stable. Fedora 32 will +continue to receive updates until approximately one month after the +release of Fedora 34. The maintenance schedule of Fedora releases is +documented on the Fedora Project wiki [0]. The Fedora Project wiki +also contains instructions [1] on how to upgrade from a previous +release of Fedora to a version receiving updates. + +Mohan Boddu. + +[0] https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Maintenance_Schedule +[1] https://docs.fedoraproject.org/en-US/quick-docs/dnf-system-upgrade/ +.... + +===== Update eol wiki page + +https://fedoraproject.org/wiki/End_of_life update with release and +number of days. + +===== Move the EOL release to archive + +. log into to bodhi-backend01 and become root ++ +____ +.... +$ ssh bodhi-backend01.iad2.fedoraproject.org +$ sudo su +$ su - ftpsync +.... +____ +. then change into the releases directory. ++ +____ +.... +$ cd /pub/fedora/linux/releases +.... +____ +. check to see that the target directory doesnt already exist. ++ +____ +.... +$ ls /pub/archive/fedora/linux/releases/ +.... +____ +. do a recursive rsync to update any changes in the trees since the +previous copy. ++ +____ +.... +$ rsync -avAXSHP ./35/ /pub/archive/fedora/linux/releases/35/ +.... +____ +. we now do the updates and updates/testing in similar ways. ++ +____ +.... +$ cd ../updates/ +$ rsync -avAXSHP 35/ /pub/archive/fedora/linux/updates/35/ +$ cd testing +$ rsync -avAXSHP 35/ /pub/archive/fedora/linux/updates/testing/35/ +.... +____ +. do the same with fedora-secondary. +. announce to the mirror list this has been done and that in 2 weeks you +will move the old trees to archives. +. in two weeks, log into mm-backend01 and run the archive script ++ +____ +.... +$ sudo -u mirrormanager mm2_move-to-archive --originalCategory="Fedora Linux" --archiveCategory="Fedora Archive" --directoryRe='/35/Everything' +.... +____ +. if there are problems, the postgres DB may have issues and so you need +to get a DBA to update the backend to fix items. +. wait an hour or so then you can remove the files from the main tree. ++ +____ +.... +$ ssh bodhi-backend01 +$ cd /pub/fedora/linux +$ cd releases/35 +$ ls # make sure you have stuff here +$ rm -rf * +$ ln ../20/README . +$ cd ../../updates/35 +$ ls #make sure you have stuff here +$ rm -rf * +$ ln ../20/README . +$ cd ../testing/35 +$ ls # make sure you have stuff here +$ rm -rf * +$ ln ../20/README . +.... +____ + +=== Consider Before Running + +* Resource contention in infrastructure, such as outages +* Extenuating circumstances for specific planned updates, if any +* Send the reminder announcement, if it isn't sent already diff --git a/modules/release_guide/pages/sop_eol_change.adoc b/modules/release_guide/pages/sop_eol_change.adoc new file mode 100644 index 0000000..d225b4e --- /dev/null +++ b/modules/release_guide/pages/sop_eol_change.adoc @@ -0,0 +1,98 @@ +== Adjust EOLs and SLs on branches + +[NOTE] +.Note +==== +This SOP is about adjust EOLs and SLs on so-called "arbitrary" branches. +Unretiring a package _also_ involves changing the EOL on a branch, but +you won't find information on that here. See the unretirement SOP for +more info there. +==== + +=== Description + +With "arbitrary branching", modules can include streams for an RPM that +are not directly associated with a Fedora release. Modules _themselves_ +can have branches not directly associated with a Fedora release. For +instance, our [.title-ref]#python3# module has a [.title-ref]#3.6# +branch. The SLs on that module branch all go EOL on 2018-06-01. +*@torsava*, one of the +https://src.fedoraproject.org/modules/python3k[python3 module +maintainers], may request that this branch have its EOL extended until +2018-12-01. + +When a maintainer wants to change EOL on a rpm, module, or container +branch, they need to file a https://pagure.io/releng/issues[releng +ticket] requesting it. They have no way to do it on their own. Releng +must review the request, and then process it. + +=== Policy + +Here are some _policy_ guidelines to help you (releng) make decisions +about these tickets + +* Clarify. Does the maintainer want the EOL lengthed for an rpm? Or for +a module? Or for a container? If they just say "increase the EOL for +[.title-ref]#httpd#, please", it is not clear which thing they're really +talking about. +* Expect that maintainers generally know _when_ their EOL should go +until. You don't need to go and research upstream mailing lists to +figure out what makes sense. Politely asking the maintainer for some +background information on _why_ the EOL should be changed is good to +record in the ticket for posterity. Bonus points if they can provide +references to upstream discussions that make the request make sense. +* EOLs should _almost always_ only be extended into the future. +Shortening an EOL should only be done with care. There might be modules +out there that depend on an rpm branch with a conflicting EOL of their +own. If a _shorter_ EOL is requested for an rpm branch, you should +verify that no modules that depend on it have a conflicting EOL. If a +_shorter_ EOL is requested for a module branch, you should verify that +no other modules require it that have a conflicting EOL. +* EOLs should not be arbitrary dates. At Flock 2017, we +https://pagure.io/fedrepo_req/issue/100[decided on using two standard +dates] for EOLs to help make things less crazy. You should use December +1st or June 1st of any given year for all EOL dates. +* Many branches will _often_ have multiple SLs all with the _same_ EOL. +I.e., the branch is fully supported until such time as it is totally +retired. There is no gray area. However, it is _possible_ to have +branches with piecemeal SLs and EOLs. A branch may support +[.title-ref]#bug_fixes# until time X but will further support +[.title-ref]#security_fixes# until time Y. This is nicely flexible for +the maintainers, but also introduces complexity. If a maintainer +requests piecemeal SL EOLs, ask to make sure they really want this kind +of complexity. + +=== Action + +We have a script in the releng repo: + +.... +$ PYTHONPATH=scripts/pdc python3 scripts/pdc/adjust-eol.py -h +.... + +[NOTE] +.Note +==== +Run it with [.title-ref]#python3#. It imports [.title-ref]#fedrepo_req# +which is python3 by default. Installing [.title-ref]#python2# +dependencies should be possible when needed. +==== + +Here is an example of using it to increase the SL of the +[.title-ref]#3.6# branch of the [.title-ref]#python3# module (not the +rpm branch): + +.... +$ PYTHONPATH=scripts/pdc python3 scripts/pdc/adjust-eol.py \ + fedora \ + SOME_TOKEN \ + python3 \ + module \ + 3.6 \ + 2018-12-01 +Connecting to PDC args.server 'fedora' with token 'a9a1e4cbca122c21580d1fe4646e603a770c5280' +Found 2 existing SLs in PDC. +Adjust eol of (module)python3#3.6 security_fixes:2018-06-01 to 2018-12-01? [y/N]: y +Adjust eol of (module)python3#3.6 bug_fixes:2018-06-01 to 2018-12-01? [y/N]: y +Set eol to 2018-12-01 on ['(module)python3#3.6 security_fixes:2018-06-01', '(module)python3#3.6 bug_fixes:2018-06-01'] +.... diff --git a/modules/release_guide/pages/sop_fedora_media_writer.adoc b/modules/release_guide/pages/sop_fedora_media_writer.adoc new file mode 100644 index 0000000..f06ed36 --- /dev/null +++ b/modules/release_guide/pages/sop_fedora_media_writer.adoc @@ -0,0 +1,156 @@ +== Fedora Media Writer Building and Signing + +=== Description + +Whenever a new version of Fedora Media Writer is available, it is +required to build and code sign it. + +=== Action + +==== Windows + +{empty}#. Get a windows code signing key from private ansible +repository. It is in the ansible-private/files/code-signing/ directory + +. Convert the .cert formatted certificate to .pxf format: ++ +.... +$ openssl pkcs12 -export -in code-signing.crt -inkey code-signing.key -out authenticode.pfx +.... +. Clone the Fedora Media Writer git repo: ++ +.... +$ git clone https://github.com/MartinBriza/MediaWriter.git +.... +. Checkout the release tag for which the executable to be created: ++ +.... +$ git checkout tags/ +.... +. The script to build and sign the executable is available at +dist/win/build.sh +. Get the mingw-mediawriter from koji. Make sure the version is the one +that needs building and signing. +. Install the remaining packages that are listed under PACKAGES variable +in build.sh script. +. Export CERTPATH to the location where the .pfx file is located and +make sure its named as authenticode.pfx and export CERTPASS to the file +that contains the password used in creating .pvk file. +. Run the build.sh script: ++ +.... +$ ./build.sh +.... + +=== Verification + +The FedoraMediaWriter-win32-.exe is located under +dist/win/ directory. + +==== OS X: + +==== Build: + +. install xcode 8.1 from apple store. +. {blank} ++ +install qt for mac from::: + http://download.qt.io/official_releases/qt/5.7/5.7.0/qt-opensource-mac-x64-clang-5.7.0.dmg +. Open a terminal and run the following commands ++ +:::: + $ git clone https://github.com/MartinBriza/MediaWriter $ cd + MediaWriter $ mkdir build $ cd build $ + $QT_PREFIX/$QT_VERSION/clang64/bin/qmake .. $ make $ cp + build/helper/mac/helper.app/Contents/MacOS/helper + build/app/FedoraMediaWriter.app/Contents/MacOS/ $ cd build/app $ + $QT_PREFIX/$QT_VERSION/clang_64/bin/macdeployqt "Fedora Media + Writer.app" -executable="Fedora Media + Writer.app/Contents/MacOS/helper" -qmldir="../../app" + +==== Prepare certificates + +This only needs to happen once per build machine, and prepares the +certificates by requesting them from Apple. + +. Install Xcode from the Mac App store +. Start Xcode +. Press Command-, (or in the menu bar click Xcode -> Preferences) +. Go to Accounts tab +. Click the plus button and sign in +. Select the new account +. Select the correct team +. Click View Details +. Under "Signing Identities", find "Developer ID Application" +. Click Create +. Wait until the button disappears +. Done + +==== Sign and DMG + +. Open a terminal +. cd to the root directory of the FMW project +. Run the following bash script: ++ +____ +.... +#/bin/bash + +security -v unlock-keychain login.keychain + +# First sign all dynamic libraries (dylib's) +cd "build/app/Fedora Media Writer.app" +for dylib in $(find . -name "*dylib") +do +codesign -s "Developer ID Application: Fedora Gilmore" -v $dylib +done +# Now sign framework bundles +for framework in $(find . -name "*framework") +do +codesign -s "Developer ID Application: Fedora Gilmore" -v $framework +done + +# Sign the two binaries +codesign -s "Developer ID Application: Fedora Gilmore" -v Contents/MacOS/helper +codesign -s "Developer ID Application: Fedora Gilmore" -v "Contents/MacOS/Fedora Media Writer" + +# Sign the app bundle +codesign -s "Developer ID Application: Fedora Gilmore" -v . + +# Create the dmg +cd .. +rm -f FedoraMediaWriter-osx-*.dmg + +hdiutil create -srcfolder "Fedora Media Writer.app" -format UDCO -imagekey zlib-level=9 -scrub \ + -volname FedoraMediaWriter-osx FedoraMediaWriter-osx-$(git describe --tags).dmg +.... +____ + +==== Account Email(OS X) + +____ +:::: + releng@fedoraproject.org +____ + +==== Account Holders(OS X) + +. Primary: Dennis Gilmore(ausil) +. Backup: Kevin Fenzi(kevin) +. Manager/bill-payer: Paul Frields(pfrields) + +=== Sync binaries to the web + +copy both files to /srv/web/fmw on sundries01 create symlinks to the +FedoraMediaWriter-win32-latest.exe and FedoraMediaWriter-osx-latest.dmg + +=== Consider Before Running + +Nothing yet. + +=== Issue with signing + +If the build is done but it is not signed then try editing the +`build.sh` and add -askpass argument for all the osslsigncode commands +and run the script, when it asks for the password you can enter the +password that was used in creating .pvk file. diff --git a/modules/release_guide/pages/sop_file_ftbfs.adoc b/modules/release_guide/pages/sop_file_ftbfs.adoc new file mode 100644 index 0000000..df11054 --- /dev/null +++ b/modules/release_guide/pages/sop_file_ftbfs.adoc @@ -0,0 +1,76 @@ +== File FTBFS + +=== Description + +[NOTE] +.Note +==== +FTBFS = "Fails To Build From Source" +==== + +After every mass rebuild, we file FTBFS bugs for the packages that +failed to build during mass rebuild. + +This should be run after the +https://docs.pagure.org/releng/sop_mass_rebuild_packages.html#post-mass-rebuild-tasks[mass +rebuild builds are merged into main tag]. + +=== Action + +The FTBFS bugs are filed in bugzilla. + +. {blank} ++ +Create a bugzilla bug for FTBFS:: + * use the https://bugzilla.redhat.com/show_bug.cgi?id=1750908[previous + FTBFS bugzilla bug example] if its not created +. {blank} ++ +Set alias for RAWHIDEFTBFS:: + * remove RAWHIDEFTBFS alias from the previous FTBFS bugzilla + * set RAWHIDEFTBFS alias on the new rawhide version FTBFS bugzilla + * set the alias on RAWHIDEFailsToInstall bugzilla in same fashion +. {blank} ++ +Install [.title-ref]#python-bugzilla-cli# on your local machine if its +not installed:: +.... +$ sudo dnf install python-bugzilla-cli +.... +. {blank} ++ +Update the [.title-ref]#massrebuildsinfo.py#:: + * epoch + * buildtag + * destag + * tracking_bug + + + [NOTE] + .Note + ==== + Most of these values are already updated during mass rebuild, only one + that might need updating is [.title-ref]#tracking_bug# + ==== +. {blank} ++ +Update the [.title-ref]#mass_rebuild_file_bugs.py#:: + * rebuildid +. {blank} ++ +Login into bugzilla in the terminal using [.title-ref]#bugzilla login# +command:: +.... +$ bugzilla login +.... + + + [NOTE] + .Note + ==== + Login as [.title-ref]#releng@fedoraproject.org# + ==== +. {blank} ++ +Run [.title-ref]#mass_rebuild_file_bugs.py# locally:: +.... +$ python mass_rebuild_file_bugs.py +.... diff --git a/modules/release_guide/pages/sop_find_module_info.adoc b/modules/release_guide/pages/sop_find_module_info.adoc new file mode 100644 index 0000000..9965eb8 --- /dev/null +++ b/modules/release_guide/pages/sop_find_module_info.adoc @@ -0,0 +1,28 @@ +== Finding Module Information + +=== Description + +When users submit builds to the Module Build Service (MBS), it in turn +submits builds to Koji. Sometimes, you are looking at a koji build, and +you want to know what module-build it is a part of. + +=== Caveat + +It requires that the build has been completed and has been tagged, until +https://pagure.io/fm-orchestrator/issue/375 is complete. + +=== Setup + +Run the following: + +.... +$ sudo dnf install python-arrow python-requests koji +.... + +=== Action + +Run the following: + +.... +$ scripts/mbs/koji-module-info.py $BUILD_ID +.... diff --git a/modules/release_guide/pages/sop_generating_openh264_composes.adoc b/modules/release_guide/pages/sop_generating_openh264_composes.adoc new file mode 100644 index 0000000..1b5972c --- /dev/null +++ b/modules/release_guide/pages/sop_generating_openh264_composes.adoc @@ -0,0 +1,185 @@ +== Generating Openh264 Composes + +=== Description + +Openh264 repos are a special case and we need to generate the composes +for it in a different way. We use ODCS to generate the private compose +and send the rpms to Cisco to publish them on their CDN. We publish the +repodata on our side. + +[WARNING] +.Warning +==== +We do not have all the appropriate legal rights to distribute these +packages, so we need to be extra carefull to make sure they are never +distributed via our build system or websites +==== + +=== Action + +==== Permissions needed + +You will need some ODCS permissions in order to request private composes +and composes from tags. You can set this in infra/ansible in +inventory/group_vars/odcs in the odcs_allowed_clients_users variable. +See other releng users entries for format. + +==== Get the odcs token + +In order to generate an odcs compose, you need a openidc token. + +Run the odcs-token.py under `scripts/odcs/` from pagure releng +repository to generate the token. + +.... +$ ./odcs-token.py +.... + +==== Make sure rpms are written out with the right signature + +.... +$ koji write-signed-rpm eb10b464 openh264-2.2.0-1.fc38 +.... + +Where the key for that branch is listed, then the open264 package and +version. + +==== Generate a private odcs compose + +With the token generated above, generate the odcs private compose + +.... +$ python odcs-private-compose.py +.... + +`koji_tag`: fxx-openh264 (Openh264 builds are tagged to fxx-openh264 +tags where [.title-ref]#xx# represents the fedora release) + +`signingkeyid`: The short hash of the key for this Fedora branch. + +The composes are stored under `/srv/odcs/private/` dir on +`odcs-backend-releng01.iad2.fedoraproject.org` + +==== Pull the compose to your local machine + +We need to extract the rpms and tar them to send them to Cisco. In order +to that, first of all we need to pull the compose to our local machine. + +===== Move the compose to your home dir on odcs-backend-releng01.iad2.fedoraproject.org + +Since the compose is owned by [.title-ref]#odcs-server# pull it into +your home dir + +.... +$ mkdir ~/32-openh264 +$ sudo rsync -avhHP /srv/odcs/private/odcs-3835/ ~/32-openh264/ +$ sudo chown -R mohanboddu:mohanboddu ~/32-openh264/ +.... + +===== Sync the compose to your local machine + +Pull in the compose from your home dir on odcs releng backend to your +local machine into a temp working dir + +.... +$ mkdir openh264-20200813 +$ scp -rv odcs-backend-releng01.iad2.fedoraproject.org:/home/fedora/mohanboddu/32-openh264/ openh264-20200813/ +.... + +===== Make the changes needed + +Please follow the following commands to make the necessary tar files to +send to Cisco + +.... +$ cd openh264-20200813 +$ mkdir 32-rpms +# Copy rpms including devel rpms +$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*rpm 32-rpms/ +# Copy debuginfo rpms +$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*/*rpm 32-rpms/ +# copy the src.rpm +$ cp -rv 32-openh264/compose/Temporary/*/*/*/*/*src.rpm 32-rpms/ +$ cd 32-rpms +# Create the tar file with the rpms +$ tar -cJvf ../fedora-32-openh264-rpms.tar.xz *rpm +.... + +We need to send this tar file to Cisco along with the list of rpms in +each tarball. + +===== Syncing the compose to sundries01 + +Once we get a confirmation from Cisco that the rpms are updated on their +CDN, verify them by using curl. For example: + +.... +$ curl -I http://ciscobinary.openh264.org/openh264-2.1.1-1.fc32.x86_64.rpm +.... + +Now push these composes to *sundries01.iad2.fedoraproject.org* and +*mm-backend01.iad2.fedoraproject.org* + +On sundries01 we need to sync to a directory that is owned by _apache_, +so first we sync to the home directory on sundries01. Same with +mm-backend01 as the directory is owned by _root_. + +Create a temp working directory on sundries01 + +.... +$ ssh sundries01.iad2.fedoraproject.org +$ mkdir openh264-20200825 +.... + +Create a temp working directory on mm-backend01 + +.... +$ ssh mm-backend01.iad2.fedoraproject.org +$ mkdir openh264-20200825 +.... + +Then from your local machine, sync the compose + +.... +$ cd openh264-20200825 +$ rsync -avhHP 32-openh264 sundries01.iad2.fedoraproject.org:/home/fedora/mohanboddu/openh264-20200825 +$ rsync -avhHP 32-openh264 mm-backend01.iad2.fedoraproject.org:/home/fedora/mohanboddu/openh264-20200825 +.... + +On sundries01 + +.... +$ cd openh264-20200825 +$ sudo rsync -avhHP 32-openh264/compose/Temporary/ /srv/web/codecs.fedoraproject.org/openh264/32/ +.... + +On mm-backend01 + +.... +$ cd openh264-20200825 +$ sudo rsync -avhHP 32-openh264/compose/Temporary/ /srv/codecs.fedoraproject.org/openh264/32/ +.... + +===== Extra info + +Normally that should be it, but in some cases you may want to push +things out faster than normal, and here's a few things you can do to do +that: + +On mm-backend01.iad2.fedoraproject.org you can run: + +.... +# sudo -u mirrormanager /usr/local/bin/umdl-required codecs /var/log/mirrormanager/umdl-required.log +.... + +This will have mirrormanager scan the codecs dir and update it if it's +changed. + +On batcave01.iad2.fedoraproject.org you can use ansible to force all the +proxies to sync the codec content from sundries01: + +.... +# nsible -a '/usr/bin/rsync --delete -a --no-owner --no-group sundries01::codecs.fedoraproject.org/ /srv/web/codecs.fedoraproject.org/' proxies +.... + +Mirrorlist servers should update every 15min. diff --git a/modules/release_guide/pages/sop_mass_branching.adoc b/modules/release_guide/pages/sop_mass_branching.adoc new file mode 100644 index 0000000..2d70b60 --- /dev/null +++ b/modules/release_guide/pages/sop_mass_branching.adoc @@ -0,0 +1,520 @@ +== Mass Branching + +=== Description + +At each alpha freeze we branch the pending release away from `devel/` +which allows rawhide to move on while the pending release goes into +bugfix and polish mode. + +=== Action + +==== Update fedscm-admin + +Add the new release to fedscm-admin and create an update and send it to +limb. + +Please take a look at the +https://pagure.io/fedscm-admin/c/7862d58b5982803dbe4c47e0262c6ce78bc903db?branch=main[fedscm-admin +commit]. + +==== Repos to branch + +All the following listed repos needs updating, including adding a new +branch for branched release and updating rawhide branch with new release +values. + +[arabic] +. https://pagure.io/pungi-fedora +. https://pagure.io/fedora-kickstarts +. https://pagure.io/fedora-comps +. https://pagure.io/fedora-lorax-templates/ +. https://pagure.io/releng/fedora-module-defaults/ +. https://pagure.io/workstation-ostree-config/ +. https://src.fedoraproject.org/rpms/fedora-release +. https://src.fedoraproject.org/rpms/fedora-repos + +==== PDC + +The "product-release" needs to be created in PDC. + +In the `scripts/pdc/` directory, run: + +.... +$ python create-product-release.py fedora $TOKEN Fedora $NEW_VERSION +.... + +On `pdc-backend01.stg` (for testing) or `pdc-backend01` (for production) +clone (or update an existing one) the releng repo: + +.... +git clone https://pagure.io/releng.git +.... + +In the `scripts/pdc/` directory, run (see the `--help` of the script for +information on how to run it): + +.... +$ python create-new-release-branches.py ... --createfile +.... + +[NOTE] +.Note +==== +the `--createfile` argument is necessary, that file is needed for the +next step. +==== + +[NOTE] +.Note +==== +Due to memory leak issue in pdc, we need to set the config in +/etc/pdc.d/fedora.json \{ "fedora": \{ "host": +"http://pdc-web02.iad2.fedoraproject.org/rest_api/v1/", "develop": +false, "ssl-verify": false, } } +==== + +==== dist-git + +Now that pdc has the new release and each package has been branched in +pdc we need to update dist-git in two steps: + +* Create the new branch in git +* Update the gitolite.conf to allow user to push to this new branch + +For both of these actions you will need the file generated by pdc above. + +===== Create the git branches + +On `pkgs01.stg` (for testing) or `pkgs02` (for production), run: + +.... +$ sudo -u pagure python /usr/local/bin/mass-branching-git.py +.... + +Where `` will be something like `f28` and the +`` the path to the file generated by pdc above. + +==== Ansible + +Apps in ansible need to be updated to be aware of a new branch. + +===== Bodhi + +Bodhi needs to be updated to add new release. This needs to be done in +https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2[bodhi2 +role] in infra ansible repo. This change includes, updating +koji-sync-listener.py, new-updates-sync, pungi configs for both rpm and +modular updates, bodhi templates. + + +===== Enable Branched Compose + +We need to enable the branched compose. This is done in +https://pagure.io/fedora-infra/ansible/blob/main/f/roles/releng[releng +role] of infra ansbile repo + + +===== Fedora Branched + +Set FedoraBranched variable to True in infra ansible repo + + +===== Koji hub + + +===== Robosignatory + +Robosignatory has two parts: + +[arabic] +. Disable branched signing, so that we can freeze branched until we get +a compose +. Adding new release + +Both can be in +https://pagure.io/fedora-infra/ansible/blob/main/f/roles/robosignatory[robosignatory +role] in infra ansible repo + + +===== Push the changes + +When done editing the files, commit, push and apply them via the +corresponding ansible playbook: + +.... +sudo rbac-playbook groups/koji-hub.yml +sudo rbac-playbook groups/releng-compose.yml +sudo rbac-playbook groups/bodhi-backend.yml +sudo rbac-playbook openshift-apps/greenwave.yml +sudo -i ansible-playbook /srv/web/infra/ansible/playbooks/groups/proxies.yml -t pkgdb2 +sudo rbac-playbook groups/mbs.yml -t mbs +.... + +Ask someone in fedora infra to run the robosignatory playbook. + +==== Koji + +The koji build system needs to have some tag/target work done to handle +builds from the new branch and to update where builds from rawhide go. + +Run +https://pagure.io/releng/blob/main/f/scripts/branching/make-koji-release-tags[make-koji-release-tags] +script in https://pagure.io/releng[pagure releng] repo + +==== Fedora Release + +The `fedora-release` package needs to be updated in Rawhide and +Branched. + +Changes to `fedora-release.spec` in the *rawhide* branch: + +[arabic] +. Increment `%define dist_version`: ++ +.... +-%define dist_version 35 ++%define dist_version 36 +.... +. Increment `Version:` and reset `Release:`: ++ +.... +-Version: 35 +-Release: 0.3%{?eln:.eln%{eln}} ++Version: 36 ++Release: 0.1%{?eln:.eln%{eln}} +.... +. Add a `%changelog` entry: ++ +.... +%changelog ++* Tue Feb 23 2021 Mohan Boddu - 36-0.1 ++- Setup for rawhide being F36 +.... + +Changes to `fedora-release.spec` in the *branched* branch: + +[arabic] +. Adjust `release_name` and unset `is_rawhide`: ++ +.... +-%define release_name Rawhide +-%define is_rawhide 1 ++%define release_name Thirty Five ++%define is_rawhide 0 +.... +. Verify the correct number for `dist_version` and `Version:`: ++ +.... +%define dist_version 35 +Version: 35 +.... +. Bump `Release:`: ++ +.... +-Release: 0.3%{?eln:.eln%{eln}} ++Release: 0.4%{?eln:.eln%{eln}} +.... +. Add a `%changelog` entry: ++ +.... +%changelog ++* Tue Feb 23 2021 Mohan Boddu - 35-0.4 ++- Branching F35 from rawhide +.... + +==== Fedora Repos + +The `fedora-repos` package needs to be updated in Rawhide, Branched, and +also in all stable release branches (in order to receive new GPG keys +and updated symlinks). + +Changes to the *rawhide* branch (mostly in `fedora-repos.spec`): + +[arabic] +. Generate and add a _Rawhide+1_ GPG key file, then add it to the spec +file: ++ +.... +Source57: RPM-GPG-KEY-fedora-37-primary +.... +. Update the `archmap` file and define architectures for _Rawhide+1_: ++ +.... ++fedora-37-primary: x86_64 armhfp aarch64 ppc64le s390x +.... +. Increment `%global rawhide_release`: ++ +.... +-%global rawhide_release 35 ++%global rawhide_release 36 +.... +. Bump `Version:` and reset `Release:`: ++ +.... +-Version: 35 +-Release: 0.2%{?eln:.eln%{eln}} ++Version: 36 ++Release: 0.1%{?eln:.eln%{eln}} +.... +. Add a `%changelog` entry: ++ +.... +%changelog ++* Tue Feb 23 2021 Tomas Hrcka - 36-0.1 ++- Setup for rawhide being F36 +.... + +Changes to the *branched* branch (mostly in `fedora-repos.spec`): + +[arabic] +. Copy the _Rawhide+1_ GPG key file from the _rawhide_ branch, then add +it to the spec file: ++ +.... +Source57: RPM-GPG-KEY-fedora-37-primary +.... +. Copy the `archmap` file from the _rawhide_ branch. +. Update `%global rawhide_release`: ++ +.... +-%global rawhide_release 35 ++%global rawhide_release 36 +.... +. Enable `updates_testing_enabled`: ++ +.... +-%global updates_testing_enabled 0 ++%global updates_testing_enabled 1 +.... +. Bump `Release:`: ++ +.... +-Release: 0.2%{?eln:.eln%{eln}} ++Release: 0.3%{?eln:.eln%{eln}} +.... +. Add a `%changelog` entry: ++ +.... +%changelog ++* Tue Feb 23 2021 Tomas Hrcka - 35-0.3 ++- Update Rawhide definition, enable updates-testing for Branched +.... + +[NOTE] +.Note +==== +Build `fedora-release` and `fedora-repos` packages for Branched release +*before enabling the Rawhide gating*. +==== + +Changes to the *stable* branches (mostly in `fedora-repos.spec`): + +[arabic] +. Copy the _Rawhide+1_ GPG key file from the _rawhide_ branch, then add +it to the spec file: ++ +.... +Source57: RPM-GPG-KEY-fedora-37-primary +.... +. Copy the `archmap` file from the _rawhide_ branch. +. Update `%global rawhide_release`: ++ +.... +-%global rawhide_release 35 ++%global rawhide_release 36 +.... +. Bump `Release:`: ++ +.... +-Release: 0.2%{?eln:.eln%{eln}} ++Release: 0.3%{?eln:.eln%{eln}} +.... +. Add a `%changelog` entry: ++ +.... +%changelog ++* Tue Feb 23 2021 Tomas Hrcka - 34-0.3 ++- Update Rawhide definition +.... + +==== Bodhi + +===== Linking Empty Repos + +We need to link empty repos so that new-updates-sync wont complain about +missing repos. The following commands should be run on +*bodhi-backend01.phx2.fedoraproject.org* + +.... +$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f32-updates +$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f32-updates-testing +$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f32-modular-updates +$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f32-modular-updates-testing +.... + +===== Creating Empty Repos + +To create empty repos on the master mirror, run +https://pagure.io/releng/blob/main/f/scripts/branching/create_empty_repos.sh[create_emtpy_repos.sh] +from https://pagure.io/releng[pagure releng] repo. This should be run on +*bodhi-backend01.phx2.fedoraproject.org* + +.... +$ sudo -u ftpsync sh scripts/branching/create_empty_repos.sh 31 +.... + +[NOTE] +.Note +==== +Please verify the repo permissions that are created under +/pub/fedora/linux/development/ and +/pub/fedora-secondary/development/. They should +be owned by _ftpsync:ftpsync_ +==== + +===== Creating rawhide release + +To create a rawhide release in bodhi, you need to run + +.... +$ bodhi releases create --name "F36" --long-name "Fedora 36" --id-prefix FEDORA --version 36 --branch f36 --dist-tag f36 --stable-tag f36 --testing-tag f36-updates-testing --candidate-tag f36-updates-candidate --pending-stable-tag f36-updates-pending --pending-testing-tag f36-updates-testing-pending --pending-signing-tag f36-signing-pending --state pending --override-tag f36-override --create-automatic-updates --not-composed-by-bodhi +.... + +To create a container release for rawhide in bodhi, you need to run + +.... +$ bodhi releases create --name "F36C" --long-name "Fedora 36 Containers" --id-prefix FEDORA-CONTAINER --version 36 --branch f36 --dist-tag f36-container --stable-tag f36-container-updates --testing-tag f36-container-updates-testing --candidate-tag f36-container-updates-candidate --pending-stable-tag f36-container-updates-pending --pending-testing-tag f36-container-updates-testing-pending --state pending --override-tag f36-container-override +.... + +To create a flatpak release for branched in bodhi, you need to run + +.... +$ bodhi releases create --name "F35F" --long-name "Fedora 35 Flatpaks" --id-prefix FEDORA-FLATPAK --version 35 --branch f35 --dist-tag f35-flatpak --stable-tag f35-flatpak-updates --testing-tag f35-flatpak-updates-testing --candidate-tag f35-flatpak-updates-candidate --pending-stable-tag f35-flatpak-updates-pending --pending-testing-tag f35-flatpak-updates-testing-pending --state pending --override-tag f35-flatpak-override +.... + +You need to run the `bodhi openshift` playbook, so that UI will know +about the new release. Then, you need to restart +*fm-consumer@config.service* and *bodhi-celery.service* services on +*bodhi-backend01.phx2.fedoraproject.org* + +.... +$ sudo rbac-playbook openshift-apps/bodhi.yml +$ sudo systemctl restart fm-consumer@config.service bodhi-celery.service +.... + +[NOTE] +.Note +==== +Build fedora-release, fedora-repos package for *rawhide after enabling +the rawhide gating* +==== + +===== Update rawhide koji repo + +We need to point the _rawhide_ buildroot repo to the newly created +rawhide buildroot. This way kojira doesn't make a newrepo for _rawhide_ +target as often as fxx-build (new rawhide buildroot). + +Run the following command from any of the compose boxes + +.... + $ cd /mnt/koji/repos/rawhide; rm -f latest; ln -s ../f34-build/latest + ./latest +.... + +===== Update block_retired.py script + +https://pagure.io/releng/blob/main/f/scripts/block_retired.py[block_retired.py] +script in releng repo should be updated with rawhide release and also +branched release should be added to the script. + +Please look at this +https://pagure.io/releng/c/9eb97f491f7a767ab8b90498adfa3b34ee235247?branch=main[block_retired.py +commit] as an example. + +===== Updating MirrorManager + +We need to update the mirrormanager so that it will point rawhide to the +new rawhide release. + +Please follow the instructions in the +https://pagure.io/fedora-infrastructure/issue/9239#comment-671446[fedora +infra ticket] to update the database of mirrormanager. + +===== Enable autosigning on branched release + +Once the branched compose is composed, we need to re-enable +robosignatory on branched release + +===== ELN related work + +Add the new rawhide key to eln pungi config. For example, look at this +https://pagure.io/pungi-fedora/c/e993441164ee83374df7f463777f2bf1d456fd6d?branch=eln[pungi +eln config commit] + +Change the trigger notification for DistroBuildSync to the new Rawhide +version. For example, look at this +https://gitlab.com/redhat/centos-stream/ci-cd/distrosync/distrobuildsync-config/-/commit/1497d9aea42cf00af646b4a0f9f9ed1a7f0a477f[commit]. + +===== Branch new rawhide in Koschei + +Branch new fedora rawhide in +https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/koschei/#_branching_a_new_fedora_release[koschei]. + +==== Fedora Container Base Image + +In order to enable builds for Container Base Images via the +https://docs.pagure.org/releng/layered_image_build_service.html[Fedora +Layered Image Build System] we will need to import a new image for +Rawhide as well as for the new `fedora:rawhide` and `fedora:${RAWHIDE}` +tags. + +Check for the latest successful Rawhide Base Image composed image +https://koji.fedoraproject.org/koji/packageinfo?packageID=21546[here]. + +On `compose-x86-01.phx2` run: + +.... +# Update this to be the correct URL for your image +$ BASEIMAGE_URL="https://kojipkgs.fedoraproject.org//packages/Fedora-Docker-Base/Rawhide/20170310.n.0/images/Fedora-Docker-Base-Rawhide-20170310.n.0.x86_64.tar.xz" + +# Update this to whatever version number Rawhide now points to +$ RAWHIDE="27" + +# Load the latest, find it's image name +$ sudo docker load < <(curl -s "${BASEIMAGE_URL}") +$ sudo docker images | grep base-rawhide +fedora-docker-base-rawhide-20170310.n.0.x86_64 latest ffd832a990ca 5 hours ago 201.8 MB + +# Tag everything +$ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 candidate-registry.fedoraproject.org/fedora:rawhide +$ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 candidate-registry.fedoraproject.org/fedora:${RAWHIDE} +$ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 registry.fedoraproject.org/fedora:rawhide +$ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 registry.fedoraproject.org/fedora:${RAWHIDE + +# Push the images +$ sudo docker push candidate-registry.fedoraproject.org/fedora:rawhide +$ sudo docker push candidate-registry.fedoraproject.org/fedora:${RAWHIDE} +$ sudo docker push registry.fedoraproject.org/fedora:rawhide +$ sudo docker push registry.fedoraproject.org/fedora:${RAWHIDE} + +# Clean up after ourselves +$ sudo docker rmi fedora-docker-base-rawhide-20170310.n.0.x86_64 +Untagged: fedora-docker-base-rawhide-20170310.n.0.x86_64:latest +$ for i in $(sudo docker images -q -f 'dangling=true'); do sudo docker rmi $i; done +.... + +===== Update sync script + +In releng repository update +https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh#_38[script]. + +And set current_rawhide variable. + +=== Consider Before Running + +[NOTE] +.Note +==== +FIXME: Need some love here +==== diff --git a/modules/release_guide/pages/sop_mass_rebuild_modules.adoc b/modules/release_guide/pages/sop_mass_rebuild_modules.adoc new file mode 100644 index 0000000..569347f --- /dev/null +++ b/modules/release_guide/pages/sop_mass_rebuild_modules.adoc @@ -0,0 +1,122 @@ +== Mass Rebuild of Modules + +=== Description + +Periodically we do mass rebuilds of modules in rawhide during the +development cycle. This SOP will outline the steps necessary to do this. + +=== Assumptions + +This assumes that the mass rebuild has already been approved and +scheduled via release engineering and FESCo. Coordinate with +infrastructure as well for any needed updates. + +=== Considerations + +* The most important thing to keep in mind while doing a mass rebuild is +to communicate clearly what actions are being performed and the status +of the rebuild. +* Check in on scripts frequently to avoid a long stalled command from +adding significant delays in completing the rebuild. + +=== Actions + +==== Preparatory Steps + +The following steps should be completed after the +https://docs.pagure.org/releng/sop_mass_rebuild_packages.html[mass +rebuild of packages] is done. + +. Update Scripts + +The mass rebuild depends on two main scripts from the +https://pagure.io/releng[releng git repository]. Each one requires some +changes in variables for each new mass rebuild cycle. + +____ +* {blank} ++ +_mass-rebuild-modules.py_:: + ** rebuildid +* {blank} ++ +_massrebuildsinfo.py_:: + ** module_mass_rebuild_epoch + ** module_mass_rebuild_platform +____ + +Change the following items: + +* the `rebuildid` to match the release for which you are mass rebuilding +modules as per in massrebuildsinfo.py +* `module_mass_rebuild_epoch` mostly will be the epoch of mass rebuild +of packages +* `module_mass_rebuild_platform` should be the rawhide module platform + +==== Starting the Mass Rebuild of Modules + +The `mass-rebuild-modules.py` script takes care of: + +* Discovering available available modules from PDC +* Find the module info from mbs and check if a module build is submitted +after the epoch date +* Checking out modules from dist-git +* Switching to appropriate stream +* Find modulemd file +* Use libmodulemd to determine if this module stream applies to this +platform version +* If needs rebuilding, committing the change +* Push the commit +* Submitting the build request through mbs + +. Connect to the mass-rebuild Machine ++ +____ +.... +$ ssh compose-branched01.iad2.fedoraproject.org +.... +____ +. Start a terminal multiplexer ++ +____ +.... +$ tmux +.... +____ +. Clone or checkout the latest copy of the +https://pagure.io/releng[releng git repository]. +. Run the [.title-ref]#mass-rebuild-modules.py# script from +_releng/scripts_ ++ +____ +.... +$ cd path/to/releng_repo/scripts +$ ./mass-rebuild-modules.py build --wait 2>&1 | tee ~/massbuildmodules.out +.... +____ + +[NOTE] +.Note +==== +The token file should be located in infra's private ansible repo, or ask +infra to get it to you using this +https://pagure.io/fedora-infrastructure/issue/8048#comment-587789[process]. +==== + +[NOTE] +.Note +==== +The [.title-ref]#build# option is really important to pay attention, +since the mass branching of modules will also use the same script, just +changing the option to [.title-ref]#branch# and +[.title-ref]#module_mass_branching_platform# in +[.title-ref]#massrebuildsinfo.py# +==== + +==== Post Mass Rebuild Tasks + +Once the module mass rebuild is done, send an email to the +devel-announce@ list + +. Send the final notification to the +_devel-announce@lists.fedoraproject.org_ list diff --git a/modules/release_guide/pages/sop_mass_rebuild_packages.adoc b/modules/release_guide/pages/sop_mass_rebuild_packages.adoc new file mode 100644 index 0000000..5084ef9 --- /dev/null +++ b/modules/release_guide/pages/sop_mass_rebuild_packages.adoc @@ -0,0 +1,262 @@ +== Mass Rebuild + +=== Description + +Periodically we do mass rebuilds of rawhide during the development +cycle. This SOP will outline the steps necessary to do this. + +=== Assumptions + +This assumes that the mass rebuild has already been approved and +scheduled via release engineering and FESCo. Coordinate with +infrastructure as well for any needed koji updates. + +This also assumes that the mass rebuild does not need to be done in +dependency order, and that the mass rebuild does not involve a ABI +change. + +=== Considerations + +* The most important thing to keep in mind while doing a mass rebuild is +to communicate clearly what actions are being performed and the status +of the rebuild. +* Check in on scripts frequently to avoid a long stalled command from +adding significant delays in completing the rebuild. +* Check with secondary arches, whether they up-to-date enough with +primary, create rebuild tag and target when they are. It will then take +care of rebuilds of the arch specific packages in appropriate kojis. + +=== Actions + +==== Preparatory Steps + +The following steps may be completed in the weeks leading up to the +scheduled mass rebuild. + +. Create the Mass Rebuild Pagure Issue ++ +____ +Create an issue on the https://pagure.io/releng/issues[Release +Engineering issues page] that points at the schedule for the current +release. + +See https://pagure.io/releng/issue/6898[the Fedora 27 mass rebuild issue +example]. +____ +. Set up the Mass Rebuild Wiki Page ++ +____ +The mass rebuild wiki page should answer the following questions for +maintainers: + +* Why the mass rebuild is happening +* How to opt out of the mass rebuild + +[NOTE] +.Note +==== +See https://fedoraproject.org/wiki/Fedora_26_Mass_Rebuild[the Fedora 26 +Wiki example]. +==== +____ +. Send out the Mass Rebuild Notice ++ +____ +Send out the same information posted on the wiki to the +[.title-ref]#devel-announce@lists.fedoraproject.org# mailing list. + +[NOTE] +.Note +==== +See +https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[the +Fedora 26 e-mail example]. +==== +____ +. Create a Tag to Contain the Mass Rebuild ++ +____ +Mass rebuilds require their own tag to contain all related builds. The +example assumes we are doing a rebuild for Fedora 26. + +.... +$ koji add-tag f26-rebuild --parent f26 +.... +____ +. Request Package Auto-Signing for New Mass-Rebuild Tag ++ +____ +File a ticket with https://pagure.io/fedora-infrastructure/issues[Fedora +Infrastructure] requesting the new mass-rebuild tag be enabled for +package auto-signing. +____ +. Create the Koji Target for the Mass Rebuild ++ +____ +Using the same [.title-ref]#f26-rebuild# tag created in the previous +example: + +.... +$ koji add-target f26-rebuild f26-build +.... + +[NOTE] +.Note +==== +*koji add-target* _target-name_ _buildroot-tag_ _destination-tag_ +describes the syntax format above. If the _destination-tag_ is not +specified then it will be the same as the _target-name_. +==== +____ +. Update Scripts ++ +____ +The mass rebuild depends on four main scripts from the +https://pagure.io/releng[releng git repository]. Each one requires some +changes in variables for each new mass rebuild cycle. + +* {blank} ++ +mass-rebuild.py:: + ** buildtag + ** targets + ** epoch + ** comment + ** target +* {blank} ++ +find-failures.py:: + ** buildtag + ** desttag + ** epoch +* mass-tag.py +* {blank} ++ +need-rebuild.py:: + ** buildtag + ** target + ** updates + ** epoch +____ + +Change the following items: + +* the build tag, holding tag, and target tag should be updated to +reflect the Fedora release you're building for +* the `epoch` should be updated to the point at which all features that +the mass rebuild is for have landed in the build system (and a newRepo +task completed with those features) +* the comment which is inserted into spec changelogs + +==== Starting the Mass Rebuild + +The `mass-rebuild.py` script takes care of: + +* Discovering available packages in koji +* Trimming out packages which have already been rebuilt +* Checking out packages from git +* Bumping the spec file +* Committing the change +* git tagging the change +* Submitting the build request to Koji + +. Connect to the mass-rebuild Machine ++ +____ +.... +$ ssh branched-composer.phx2.fedoraproject.org +.... +____ +. Start a terminal multiplexer ++ +____ +.... +$ tmux +.... +____ +. Clone or checkout the latest copy of the +https://pagure.io/releng[releng git repository]. +. Run the mass-rebuild.py script from _releng/scripts_ ++ +____ +.... +$ cd path/to/releng_repo/scripts +$ ./mass-rebuild.py 2>&1 | tee ~/massbuild.out +.... +____ + +==== Monitoring Mass Rebuilds + +The community has a very high interest in the status of rebuilds and +many maintainers will want to know if their build failed right away. The +`find-failures.py` and `need-rebuild.py` scripts are designed to update +publicly available URLs for stakeholders to monitor. + +. Connect to a Compose Machine ++ +____ +.... +$ ssh compose-x86-02.phx2.fedoraproject.org +.... +____ +. Start a terminal multiplexer ++ +____ +.... +$ tmux +.... +____ +. Clone or checkout the latest copy of the +https://pagure.io/releng[releng git repository] +. {blank} ++ +Set Up the Rebuild Failures Notification Web Site:: + The `find_failures.py` script discovers attempted builds that have + failed. It lists those failed builds and sorts them by package owner. + + +.... +$ while true; do ./find_failures.py > f26-failures.html && cp f26-failures.html /mnt/koji/mass-rebuild/f26-failures.html; sleep 600; done +.... +. Start a second pane in the terminal emulator +. {blank} ++ +Set up the Site for Packages that Need Rebuilt:: + The `need-rebuild.py` script discovers packages that have not yet been + rebuilt and generates an html file listing them sorted by package + owner. This gives external stakeholders a rough idea of how much work + is remaining in the mass rebuild. + + +.... +$ while true; do ./need-rebuild.py > f26-need-rebuild.html && cp f26-need-rebuild.html /mnt/koji/mass-rebuild/f26-need-rebuild.html; sleep 600; done +.... + +==== Post Mass Rebuild Tasks + +Once the mass rebuild script completes, and all the pending builds have +finished, the builds will need to be tagged. The `mass-tag.py` script +will accomplish this task. The script will: + +* Discover completed builds +* Trim out builds that are older than the latest build for a given +package +* Tag remaining builds into their final destination (without generating +email) + +. Clone or checkout the latest copy of the +https://pagure.io/releng[releng git repository] +. Run the `mass-tag.py` script (requires koji kerberos authentication) ++ +____ +.... +$ cd path/to/releng_repo/scripts +$ ./mass-tag.py --source f36-rebuild --target f36 +.... +____ +. Send the final notification to the +_devel-announce@lists.fedoraproject.org_ list ++ +____ +The contents should look something like this +https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[example +email]. +____ diff --git a/modules/release_guide/pages/sop_package_blocking.adoc b/modules/release_guide/pages/sop_package_blocking.adoc new file mode 100644 index 0000000..dfa0fb9 --- /dev/null +++ b/modules/release_guide/pages/sop_package_blocking.adoc @@ -0,0 +1,17 @@ +== Package Blocking + +=== Description + +If a +https://fedoraproject.org/wiki/How_to_remove_a_package_at_end_of_life[package +is removed (retired) from Fedora], for example because it was renamed, +it needs to be blocked in Koji. This prevents creating new package +builds and distribution of built RPMs. Packages are blocked in the +listing of `tags`, due to inheritance it is enough to block packages at +the oldest tag will make it unavailable also in upstream tags. + +=== Action + +The blocking of retired packages is done by the +https://pagure.io/releng/blob/master/f/scripts/block_retired.py[block_retired.py] +script as part of the daily Rawhide and Branched composes. diff --git a/modules/release_guide/pages/sop_package_unblocking.adoc b/modules/release_guide/pages/sop_package_unblocking.adoc new file mode 100644 index 0000000..d2e3ed5 --- /dev/null +++ b/modules/release_guide/pages/sop_package_unblocking.adoc @@ -0,0 +1,121 @@ +== Package Unblocking + +=== Description + +Packages are sometimes unblocked from Fedora, usually when a package had +been orphaned and now has a new owner. When this happens, release +engineering needs to "unblock" the package from koji tags. + +=== Action + +==== Find Unblock requests + +Unblock requests are usually reported in the +https://pagure.io/releng/issues[rel-eng issue tracker]. + +==== Perform the unblocking + +First assign the ticket to yourself to show, that you are handling the +request. + +===== Discover proper place to unblock + +The ticket should tell you which Fedora releases to unblock the package +in. Typically it'll say "Fedora 13" or "F14". This means we need to +unblock it at that Fedora level and all future tags. However depending +on where the package was blocked we may have to do our unblock action at +a different Fedora level. + +To discover where a package is blocked, use the `list-pkgs` method of +koji. + +.... +$ koji list-pkgs --help +Usage: koji list-pkgs [options] +(Specify the --help global option for a list of other help options) + +Options: + -h, --help show this help message and exit + --owner=OWNER Specify owner + --tag=TAG Specify tag + --package=PACKAGE Specify package + --quiet Do not print header information + --noinherit Don't follow inheritance + --show-blocked Show blocked packages + --show-dups Show superseded owners + --event=EVENT# query at event + --ts=TIMESTAMP query at timestamp + --repo=REPO# query at event for a repo +.... + +For example if we wanted to see where python-psco was blocked we would +do: + +.... +$ koji list-pkgs --package python-psyco --show-blocked +Package Tag Extra Arches Owner +----------------------- ----------------------- ---------------- --------------- +python-psyco dist-f14 konradm [BLOCKED] +python-psyco olpc2-ship2 shahms +python-psyco olpc2-trial3 shahms +... +.... + +Here we can see that it was blocked at dist-f14. If we got a request +that was to unblock it before f14, we can simply use the dist-f14 target +to unblock. However if they want it unblocked after f14, we would use +the earliest dist-f?? tag the user wants, such as dist-f15 if the user +asked for it to be unblocked in Fedora 15+ + +===== Performing the unblock + +To unblock a package for a tag, use the `unblock-pkg` method of Koji. + +.... +$ koji unblock-pkg --help +Usage: koji unblock-pkg [options] tag package [package2 ...] +(Specify the --help global option for a list of other help options) + +Options: + -h, --help show this help message and exit +.... + +For example, if we were asked to unblock python-psyco in F14 we would +issue: + +.... +$ koji unblock-pkg dist-f14 python-psyco +.... + +Now the ticket can be closed. + +=== Verification + +To verify that the package was successfully unblocked use the +`list-pkgs` koji command: + +.... +$ koji list-pkgs --package python-psyco --show-blocked +.... + +We should see the package listed as not blocked at dist-f14 or above: + +.... +Package Tag Extra Arches Owner +----------------------- ----------------------- ---------------- --------------- +python-psyco olpc2-trial3 jkeating +python-psyco olpc2-ship2 jkeating +python-psyco olpc2-update1 jkeating +python-psyco trashcan jkeating +python-psyco f8-final jkeating +... +.... + +We should not see it listed as blocked in dist-f14 or any later Fedora +tags. + +=== Consider Before Running + +* Watch the next day's rawhide/branched/whatever report for a slew of +broken deps related to the package. We may have to re-block the package +in order to fix the deps. diff --git a/modules/release_guide/pages/sop_pdc.adoc b/modules/release_guide/pages/sop_pdc.adoc new file mode 100644 index 0000000..eb233c2 --- /dev/null +++ b/modules/release_guide/pages/sop_pdc.adoc @@ -0,0 +1,7 @@ +== Product Definition Center + +This is a stub SOP for PDC. The doc really lives in the infra SOPs, +here: https://infrastructure.fedoraproject.org/infra/docs/pdc.rst + +Please update and maintain that SOP. Leave this document here as a +breadcrumb to make it easier to find that doc. diff --git a/modules/release_guide/pages/sop_process_dist_git_requests.adoc b/modules/release_guide/pages/sop_process_dist_git_requests.adoc new file mode 100644 index 0000000..729c1b9 --- /dev/null +++ b/modules/release_guide/pages/sop_process_dist_git_requests.adoc @@ -0,0 +1,33 @@ +== Process fedora-scm-requests tickets + +=== Description + +When a packager wants a new package added to Fedora or a new dist-git +branch blessed, they need to go through the new package process and, +once their package review is approved, they use the +[.title-ref]#fedrepo-req# cli tool to file a ticket in the +https://pagure.io/releng/fedora-scm-requests[fedora-scm-requests queue]. + +Periodically, (daily?) release engineering will need to review and +process this queue using the [.title-ref]#fedrepo-req-admin# tool. + +=== Setup + +A release engineering will need to have several values set locally as +well as sufficient permissions in a number of server-side systems. + +. A pagure.io token. See the fedrepo-req README for instructions on +where to get this. +. src.fedoraproject.org token generated by [.title-ref]#pagure-admin#. +Ask @pingou how to get one. If doing this yourself, go to pkgs01 and run +[.title-ref]#PAGURE_CONFIG=/etc/pagure/pagure.cfg pagure-admin +admin-token create -h# for more info. +. pdc token. See the PDC SOP for getting one of these. + +=== Action + +. Run [.title-ref]#fedrepo-req-admin list# to list all open requests. +. Run [.title-ref]#fedrepo-req-admin process N# to process a particular +ticket. +. Run [.title-ref]#fedrepo-req-admin processall# to iterate over all the +tickets. diff --git a/modules/release_guide/pages/sop_promoting_container_content.adoc b/modules/release_guide/pages/sop_promoting_container_content.adoc new file mode 100644 index 0000000..146bffc --- /dev/null +++ b/modules/release_guide/pages/sop_promoting_container_content.adoc @@ -0,0 +1,50 @@ +== Promoting Container Content + +=== Description + +Even though the promotion of content is aimed to be fully automated, +sometimes there is the need or desire to promote content from the +candidate registry to the stable registry. Below is how to accomplish +this using https://github.com/projectatomic/skopeo[skopeo]. + +=== Action + +This action should be performed on +`compose-x86-01.phx2.fedoraproject.org` by an user who is a member of +the `sysadmin-releng` https://admin.fedoraproject.org/accounts/[FAS] +Group. + +The container image should be provided by the requester, they will have +the information from their container build in the +https://docs.pagure.org/releng/layered_image_build_service.html[Fedora +Layered Image Build System]. It will have a name resembling +`candidate-registry.fedoraproject.org/f26/foo:0.1-1.f26container`. + +Sync content between the candidate registry and production registry, +effectively "promoting it". Substitute the `$IMAGE_NAME` in the +following example with whatever the actual image name is that is +provided. (This would be everything after +`candidate-registry.fedoraproject.org`, so using the above example then +`$IMAGE_NAME` would be `f26/foo:0.1-1.f26container`.) + +.... +$ sudo skopeo copy \ + --src-cert-dir /etc/docker/certs.d/candidate-registry.fedoraproject.org/ \ + --dest-cert-dir /etc/docker/certs.d/registry.fedoraproject.org/ \ + docker://candidate-registry.fedoraproject.org/$IMAGE_NAME \ + docker://registry.fedoraproject.org/$IMAGE_NAME +.... + +=== Verification + +In order to verify, we need to inspect the stable registry, again with +https://github.com/projectatomic/skopeo[skopeo] to ensure the image +metadata exists. + +.... +$ skopeo inspect docker://registry.fedoraproject.org/$IMAGE_NAME +.... + +In this JSON output you will see a list element titled `RepoTags` and in +there should be the `$VERSION-$RELEASE` listed there, following our +example above this entry would be `0.1-1.f26container`. diff --git a/modules/release_guide/pages/sop_pushing_updates.adoc b/modules/release_guide/pages/sop_pushing_updates.adoc new file mode 100644 index 0000000..982596d --- /dev/null +++ b/modules/release_guide/pages/sop_pushing_updates.adoc @@ -0,0 +1,225 @@ +== Pushing Updates + +=== Description + +Fedora updates are typically pushed once a day. This SOP covers the +steps involved. + +==== Coordinate + +Releng has a rotation of who pushes updates when. Please coordinate and +only push updates when you are expected to or have notified other releng +folks you are doing so. See: +https://apps.fedoraproject.org/calendar/release-engineering/ for the +list or on irc you can run `.pushduty` in any channel with zodbot to see +who is on duty this week. + +==== Login to machine to sign updates + +Login to a machine that is configured for sigul client support and has +the bodhi client installed. Currently, this machine is: +`bodhi-backend01.phx2.fedoraproject.org` + +==== Decide what releases you're going to push. + +* If there is a Freeze ongoing, you SHOULD NOT push all stable requests +for a branched release, only specific blocker or freeze exception +requests that QA will request in a releng ticket. +* If there is no Freeze ongoing you can push all Fedora and EPEL +releases at the same time if you wish. +* From time to time there may be urgent requests in some branches, you +can only push those if requested. Note however that bodhi2 will +automatically push branches with security updates before others. + +==== Get a list of packages to push + +.... +$ sudo -u apache bodhi-push --releases 'f26,f25,f24,epel-7,EL-6' --username + +.... + +Sometimes you see a warning "Warning: foobar-1.fcxx has unsigned builds +and has been skipped" which means those updates are currently getting +signed and it can verified by listing the tagged builds in +fxx-signing-pending tag. + +:::: + $ koji list-tagged fxx-signing-pending + +You can say 'n' to the push at this point if you wish to sign packages +(see below). Or you can keep this request open in a window while you +sign the packages, then come back and say y. + +List the releases above you wish to push from: 25 24 5 6 7, etc + +You can also specify `--request=testing` to limit pushes. Valid types +are `testing` or `stable`. + +The list of updates should be in the cache directory named +`Stable-$Branch` or `Testing-$Branch` for each of the Branches you +wished to push. + +During freezes you will need to do two steps: (If say, fedora 26 +branched was frozen): + +.... +$ sudo -u apache bodhi-push --releases f26 --request=testing \ + --username +.... + +Then + +.... +$ sudo -u apache bodhi-push --releases 'f25,f24,epel-7,EL-6' --username +.... + +During the Release Candidate compose phase we tag builds to be included +into a -compose tag (e.g. f26-compose). When we have a candidate that +has been signed off as gold we need to ensure that all builds tagged +into the -compose tag have been pushed stable. Once we have pushed all +-compose builds stable we then have to clone the base tag (e.g. f26) to +a tag for the milestone for Alpha and Beta (e.g. f26-Alpha). After final +release we need to lock the base tag and adjust the release status in +bodhi so that updates now hit the -updates tag (e.g. f26-updates). Once +we have cloned the tag or locked the tag and adjusted bodhi we are free +to push stable updates again. + +==== Pushing Stable updates during freeze + +During feezes we need to push to stable builds included in the compose. +QA will file a ticket with the nvrs to push. + +[NOTE] +.Note +==== +If you are pushing a bodhi update that contains multiple builds, you +need only pass bodhi-push a single build nvr and all the others in that +update will be detected and pushed along with it. However, if you are +pushing multiple disjoint bodhi updates then each build will need to be +listed individually. +==== + +.... +$ sudo -u apache bodhi-push --builds ',,...' --username +.... + +===== There are no updates to push. + +If you are getting the message `There are no updates to push.` or the +list of packages you are seeing to push out for the Stable updates +request is not correct compared to what you specified in the `--builds` +section of the command above then one of two things likely happened. + +. The update hasn't yet reached appropriate karma ++ +This should be handled case-by-case, if the QA Team has requested this +be pushed as stable to fix a blocker but there's not yet enough karma +for an autostable prompt to occur then you should verify with QA that +these are ready to go out even without karma. If they are, then log into +the Bodhi WebUI and modify the karma threshold of the update to 1 and +add karma (if necessary). This is not something we should do as normal +practice and is considered an edge case. When update requests come to +RelEng, it should have appropriate karma. Sometimes it doesn't and as +long as QA approves, we need not block on it. +. The update was never requested for stable ++ +It's possible the update wasn't requested for stable, you can resolve +this by running the following on one of the bodhi-backend systems: ++ +.... +bodhi updates request stable +.... + +==== Perform the bodhi push + +Say 'y' to push for the above command. + +=== Verification + +. Monitor Bodhi's composes with `bodhi-monitor-composes` ++ +.... +$ sudo -u apache watch -d -n 60 bodhi-monitor-composes +.... +. Monitor the systemd journal ++ +.... +$ sudo journalctl -o short -u fedmsg-hub -l -f +.... +. Check the processes ++ +.... +$ ps axf|grep bodhi +.... +. Watch for fedmsgs through the process. It will indicate what releases +it's working on, etc. You may want to watch in `#fedora-fedmsg`. ++ +.... +bodhi.masher.start -- kevin requested a mash of 48 updates +bodhi.mashtask.start -- bodhi masher started a push +bodhi.mashtask.mashing -- bodhi masher started mashing f23-updates +bodhi.mashtask.mashing -- bodhi masher started mashing f22-updates-testing +... +bodhi.update.complete.stable -- moceap's wondershaper-1.2.1-5.fc23 bodhi update completed push to stable https://admin.fedoraproject.org/updates/FEDORA-2015-13052 +... +bodhi.errata.publish -- Fedora 23 Update: wondershaper-1.2.1-5.fc23 https://admin.fedoraproject.org/updates/FEDORA-2015-13052 +bodhi.mashtask.complete -- bodhi masher successfully mashed f23-updates +bodhi.mashtask.sync.wait -- bodhi masher is waiting for f22-updates-testing to hit the master mirror +.... +. Seach for problems with a particular push: ++ +.... +sudo journalctl --since=yesterday -o short -u fedmsg-hub | grep dist-6E-epel (or f22-updates, etc) +.... +. Note: Bodhi will look at the things you have told it to push and see +if any have security updates, those branches will be started first. It +will then fire off threads (up to 3 at a time) and do the rest. + +=== Consider Before Running + +Pushes often fall over due to tagging issues or unsigned packages. Be +prepared to work through the failures and restart pushes from time to +time + +.... +$ sudo -u apache bodhi-push --resume +.... + +Bodhi will ask you which push(es) you want to resume. + +=== Common issues / problems with pushes + +* When the push fails due to new unsigned packages that were added after +you started the process. re-run step 4a or 4b with just the package +names that need to be signed, then resume. +* When the push fails due to an old package that has no signature, run: +`koji write-signed-rpm ` and resume. +* When the push fails due to a package not being tagged with +updates-testing when being moved stable: +`koji tag-pkg dist--updates-testing ` +* When signing fails, you may need to ask that the sigul bridge or +server be restarted. +* If the updates push fails with a: +`OSError: [Errno 16] Device or resource busy: '/var/lib/mock/*-x86_64/root/var/tmp/rpm-ostree.*'` +You need to umount any tmpfs mounts still open on the backend and resume +the push. +* If the updates push fails with: +`"OSError: [Errno 39] Directory not empty: '/mnt/koji/mash/updates/*/../*.repocache/repodata/'` +you need to restart fedmsg-hub on the backend and resume. +* If the updates push fails with: +`IOError: Cannot open /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml: File /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml doesn't exists or not a regular file` +This issue will be resolved with NFSv4, but in the mean time it can be +worked around by removing the [.title-ref]#.repocache# directory and +resuming the push. +`$ sudo rm -fr /mnt/koji/mash/updates/epel7.repocache` +* If the Atomic OSTree compose fails with some sort of +[.title-ref]#Device or Resource busy# error, then run +[.title-ref]#mount# to see if there are any stray [.title-ref]#tmpfs# +mounts still active: +`tmpfs on /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq type tmpfs (rw,relatime,seclabel,mode=755)` +You can then +`$ sudo umount /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq` +and resume the push. + +Other issues should be addressed by releng or bodhi developers in +`#fedora-releng`. diff --git a/modules/release_guide/pages/sop_rawhide_bodhi.adoc b/modules/release_guide/pages/sop_rawhide_bodhi.adoc new file mode 100644 index 0000000..f799051 --- /dev/null +++ b/modules/release_guide/pages/sop_rawhide_bodhi.adoc @@ -0,0 +1,95 @@ +== Enabling Rawhide in Bodhi + +=== Description + +This SOP covers the steps needed to enable Rawhide in Bodhi. + +==== Create the release in Bodhi + +In oder to start creating updates in Bodhi for rawhide, the release +needs to be created in Bodhi. Rawhide in Bodhi is represented by the +Fedora version (ie Fedora 31), but it is set in the prerelease state. + +===== Add the koji tags + +.... +$ koji add-tag --parent f31 f31-updates-candidate +$ koji add-tag --parent f31 f31-updates-testing +$ koji add-tag --parent f31-updates-testing f31-updates-testing-pending +$ koji edit-tag --perm autosign f31-updates-testing-pending +$ koji add-tag --parent f31 f31-updates-pending +$ koji add-tag --parent f31 f31-override +.... + +===== Change the koji targets + +.... +$ koji edit-target f31 --dest-tag f31-updates-candidate +$ koji edit-target f31-candidate --dest-tag f31-updates-candidate +$ koji edit-target rawhide --dest-tag f31-updates-candidate +.... + +===== Create the release in bodhi + +.... +$ bodhi releases create --name "F31" --long-name "Fedora 31" --id-prefix FEDORA --version 31 --branch f31 \ + --dist-tag f31 --stable-tag f31 --testing-tag f31-updates-testing --candidate-tag f31-updates-candidate \ + --pending-stable-tag f31-updates-pending --pending-testing-tag f31-updates-testing-pending \ + --state pending --override-tag f31-override --create-automatic-updates --not-composed-by-bodhi +.... + +The important flags are [.title-ref]#--not-composed-by-bodhi# which +tells bodhi not to include the rawhide updates in the nightly pushes and +[.title-ref]#--create-automatic-updates# which tells bodhi to +automatically create an update listen to koji tag (build tagged with the +pending-testing-tag) messages. + +===== Bodhi configuration + +Bodhi is configured to required zero mandatory days in testing for the +rawhide release. This is done in ansible +roles/bodhi2/base/templates/production.ini.j2 with the following. + +.... +f{{ FedoraRawhideNumber }}.pre_beta.mandatory_days_in_testing = 0 +.... + +===== Robosignatory configuration + +Robosignatory needs to be configured to signed the rawhide builds before +these builds are tested by the CI pipeline. + +.... +{ + "from": "f31-updates-candidate", + "to": "f31-updates-testing-pending", + "key": "fedora-31", + "keyid": "3c3359c4" +}, +.... + +==== Branching Rawhide + +When it is time to branch rawhide, a new release should be created (eg. +F32) following the steps above. The old rawhide release (eg. F31) should +stay configured as rawhide until we active Bodhi for it (2 weeks later). +To activate Bodhi on the old rawhide (eg. F31) the existing release in +bodhi should be modified has follow. + +:::: + $ bodhi releases edit --name "F31" --stable-tag f31-updates + --no-create-automatic-updates --composed-by-bodhi + +===== Robosignatory configuration + +At Bodhi activation time the Robosignatory configuration needs to be +update to match the normal configuration of bodhi releases. + +.... +{ + "from": "f31-signing-pending", + "to": "f31-updates-testing-pending", + "key": "fedora-31", + "keyid": "3c3359c4" +}, +.... diff --git a/modules/release_guide/pages/sop_release_container_base_image.adoc b/modules/release_guide/pages/sop_release_container_base_image.adoc new file mode 100644 index 0000000..bb5c569 --- /dev/null +++ b/modules/release_guide/pages/sop_release_container_base_image.adoc @@ -0,0 +1,76 @@ +== Release the Fedora Container Base Image + +=== Description + +This SOP covers the steps involved in changing and releasing the Fedora +Container Base image. + +Fedora releases 2 container base images, [.title-ref]#fedora# and +[.title-ref]#fedora-minimal#. These images are available on 3 registries +[.title-ref]#registry.fedoraproject.org#, [.title-ref]#quay.io# and +[.title-ref]#DockerHub# (fedora-minimal is not available in DockerHub). + +==== Modify a base image (Kickstart) + +Base images are built in koji using the [.title-ref]#image-factory# +application to build the container image root filesystem (rootfs). +Kickstart files are used to configure how the image is built and what is +available in the image The solution consist of 3 Kickstarts. + +https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-common.ks[fedora-container-common] + +https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-base.ks[fedora-container-base] + +https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-base-minimal.ks[fedora-container-base-minimal] + +Changes made on the rawhide branch will results in the rawhide image, +other branches (f30, f31) should be used to modify other releases. + +==== Compose Configuration (Pungi) + +The configuration used to compose the container images is available in +the pungi-fedora repository. + +For rawhide the configuration is in + +https://pagure.io/pungi-fedora/blob/main/f/fedora.conf + +While for other releases the configuration is in a dedicated file + +https://pagure.io/pungi-fedora/blob/f31/f/fedora-container.conf + +==== Release on registry.fedoraproject.org and quay.io + +If you want to release the base image on registry.fp.o and quay.io you +can use the following script. + +https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh[sync-latest-container-base-image.sh] + +You will need to run that script from on of the releng composer machines +in the infrastructure in order to have the credentials. + +If you do not have access to that machines, you can request the release +by opening a ticket on the https://pagure.io/releng/issues[releng +tracker]. + +The script can then be executed as follow + +.... +$./sync-latest-container-base-image.sh 31 +$./sync-latest-container-base-image.sh 32 +.... + +This will take care of pushing the [.title-ref]#fedora# and +[.title-ref]#fedora-minimal# images to both registries. + +==== Release on DockerHub + +Releasing on DockerHub is a little different since Fedora is an +"offical" image there. In order to release new images there we have to +update a Dockerfile and rootfs tarball on the following repo. + +https://github.com/fedora-cloud/docker-brew-fedora[docker-brew-fedora]. + +For the details on how to run the script please see + +https://github.com/fedora-cloud/docker-brew-fedora/blob/main/README.md[README]. diff --git a/modules/release_guide/pages/sop_release_package_signing.adoc b/modules/release_guide/pages/sop_release_package_signing.adoc new file mode 100644 index 0000000..95cc706 --- /dev/null +++ b/modules/release_guide/pages/sop_release_package_signing.adoc @@ -0,0 +1,68 @@ +== Release Package Signing + +=== Description + +For each of Fedora's public releases (Alpha, Beta, and Final) it is +Release Engineering's responsibility to sign all packages with Fedora's +GPG key. This provides confidence to Fedora's users about the +authenticity of packages provided by Fedora. + +The `/sop_create_release_signing_key` document explains the process for +creating the GPG key used. + +=== Consider Before Running + +This script takes a very long time to run, as much as 4 or 5 days, so it +needs to be started well in advance of when you need the packages all +signed. + +Signing all the packages will cause a lot of churn on the mirrors, so +expect longer than usual compose and rsync times, as well as potential +issues with proxies as file contents change but the name remains the +same. + +=== Action + +. Log into a system with `sigul` and start a `screen` or `tmux` session. +The signing process takes a long time--screen allows the process to +continue if you session gets disconnected. ++ +.... +$ screen -S sign +.... ++ +or ++ +.... +$ tmux new -s sign +.... +. Check out the Release Engineering `git` repo ++ +:::: + $ git clone https://pagure.io/releng.git +. Change directories to the `scripts` directory to execute +`sigulsign_unsigned.py`. ++ +For example, to sign everything for Fedora 13 Alpha we would issue: ++ +:::: + $ ./sigulsign_unsigned.py -vv --tag dist-f13 fedora-13 ++ +This signs the packages with verbose output so you can track progress +incrementally. + +=== Verification + +Once the signing is done, use `rpmdev-checksig` to verify that a package +has been signed. You can use the output of a recent rawhide compose to +test. In this example we use a released Fedora 12 package: + +.... +$ rpmdev-checksig /pub/fedora/linux/releases/12/Everything/i386/os/Packages/pungi-2.0.20-1.fc12.noarch.rpm +/pub/fedora/linux/releases/12/Everything/i386/os/Packages/pungi-2.0.20-1.fc12.noarch.rpm: MISSING KEY - 57bbccba +.... + +This output shows that the apckage was signed with key `57bbccba`, and +that this key does not exist in your local rpm database. If the key did +exist in the local rpm database it's likely there would be no output so +it's best to run this on a system that does not have gpg keys imported. diff --git a/modules/release_guide/pages/sop_remote_dist_git_branches.adoc b/modules/release_guide/pages/sop_remote_dist_git_branches.adoc new file mode 100644 index 0000000..0ff5f89 --- /dev/null +++ b/modules/release_guide/pages/sop_remote_dist_git_branches.adoc @@ -0,0 +1,43 @@ +== Remove dist-git branches + +=== Description + +Release Engineering is often asked by maintainers to remove branches in +dist-git by maintainers. + +=== Action + +. Log into batcave01 ++ +.... +ssh @batcave01.iad2.fedoraproject.org +.... +. Get root shell +. Log into pkgs01.iad2.fedoraproject.org : ++ +.... +ssh pkgs01.iad2.fedoraproject.org +.... +. Change to the package's directory ++ +.... +cd /srv/git/rpms/.git/ +.... +. Remove the branch ++ +.... +git branch -D +.... + +=== Verification + +To verify just list the branches. + +.... +git branch +.... + +=== Consider Before Running + +Make sure that the branch in question isn't one of our pre-created +branches `f??/rawhide`, `olpc?/rawhide`, `el?/rawhide` diff --git a/modules/release_guide/pages/sop_requesting_task_automation_users.adoc b/modules/release_guide/pages/sop_requesting_task_automation_users.adoc new file mode 100644 index 0000000..adf0aab --- /dev/null +++ b/modules/release_guide/pages/sop_requesting_task_automation_users.adoc @@ -0,0 +1,27 @@ +== Requesting Automation Users + +[[sop_requesting_task_automation_user]] +=== Description + +When performing automated Release Engineering tasks using `RelEng +Automation <_releng-automation>` you will sometimes find that you need +to perform an action in the Infrastructure with `sudo` that does not yet +have an automation user associated with it. + +=== Actions + +==== Requesting a new loopabull user + +File a ticket with https://pagure.io/fedora-infrastructure/[Fedora +Infrastructure] making sure to satisfy the following requirements: + +* Provide a justification for these permissions being needed (What are +you trying to do and why?) +* Commands needing to be run with sudo +* Destination server on which the commands need to be run +* The `loopabull_` username requested to be created for this OR which +`loopabull_` username needs it's pre-existing permissions enhanced + +For reference: +https://pagure.io/fedora-infrastructure/issue/5943[Example +Infrastructure Ticket] diff --git a/modules/release_guide/pages/sop_retire_orphaned_packages.adoc b/modules/release_guide/pages/sop_retire_orphaned_packages.adoc new file mode 100644 index 0000000..b2d046b --- /dev/null +++ b/modules/release_guide/pages/sop_retire_orphaned_packages.adoc @@ -0,0 +1,83 @@ +== Retire Orphaned Packages + +=== Description + +Every release prior to the +https://fedoraproject.org/wiki/Schedule[Feature Freeze/Branching] +Release Engineering retires +https://fedoraproject.org/wiki/Orphaned_package_that_need_new_maintainers[orphaned +packages]. This keeps out unowned software and prevents future problems +down the road. + +=== Action + +The orphan process takes place in stages: + +. Detecting a list of orphans and the dependencies that will be broken +if the orphans are removed. +. Sending the list of potential orphans to devel@lists.fedoraproject.org +for community review and removal from the orphan list. +. Retriring packages nobody wants to adopt. + +==== Detecting Orphans + +A script called `find_unblocked_orphans.py` assists in the detection +process. It should be run on a machine that has `koji` and +`python-fedora` installed. It runs without options and takes a while to +complete. + +`find_unblocked_orphans.py` is available in the +https://pagure.io/releng[Release Engineering git repository] + +==== Announcing Packages to be retired + +`find_unblocked_orphans.py` outputs text to stdout on the command line +in a form suitable for the body of an email message. + +.... +$ ./find-unblocked-orphans.py > email-message +.... + +Email the output to the development list +(`devel@lists.fedodraproject.org`) at least a month before the feature +freeze, send mails with updated lists as necessary. This gives +maintainers an opportunity to pick up orphans that are important to them +or are required by other packages. + +==== Retiring Orphans + +Once maintainers have been given an opportunity to pick up orphaned +packages, the remaining +https://fedoraproject.org/wiki/How_to_remove_a_package_at_end_of_life[packages +are retired] + +===== Bugs + +This procedure probably leaves open bugs for the retired packages +behind. It is not within the scope of release engineering to take care +of these. If bugs are closed, only bugs targeted at Rawhide should be +affected, since other branches might still be maintained. + +=== Verification + +To verify that the packages were blocked correctly we can use the +`latest-pkg` `koji` action. + +.... +$ koji latest-pkg dist-f21 wdm +.... + +This should return nothing, as the `wdm` package is blocked. + +=== Consider Before Running + +Generally we retire anything that doesn't leave broken dependencies. If +there are orphans whose removal would result in broken dependencies a +second warning should be sent to `devel@lists.fedoraproject.org` and to +`-owner@fedoraproject.org` for each dependent package. + +Allow another couple of days for maintainers to take notice and fix +these package so the package repository can be maintained without broken +dependencies or needing to the package. It is not good to have broken +package dependencies in our package repositories so every effort should +be made to find owners or to fix the broken dependencies. diff --git a/modules/release_guide/pages/sop_signing_builds.adoc b/modules/release_guide/pages/sop_signing_builds.adoc new file mode 100644 index 0000000..b97b08b --- /dev/null +++ b/modules/release_guide/pages/sop_signing_builds.adoc @@ -0,0 +1,29 @@ +== Sign the packages + +* This doc explains how to sign builds in the release(s). +* Manual signing should rarely ever be needed anymore. Just make sure +that robosignatory is setup for all tags that are created. +* If a build seems to be stuck in the autosigning queue (one of the +-pending or -signing-pending tags), just koji untag and koji tag the +package. This will retrigger autosigning. +* If bodhi is reporting a build as unsigned but the build is not in the +-signing-pending tag, that means bodhi missed the tagging. Just run the +following command to make the build get retagged again, giving Bodhi +another change at seeing the signing ++ +.... +$ koji move $dist-updates-testing-pending $dist-signing-pending $build +.... +* If need be, sign builds using scripts/sigulsign_unsigned.py from +releng git repo ++ +_NOTE! This will NOT help if Bodhi marks a build as unsigned!_ ++ +.... +$ ./sigulsign_unsigned.py -vv --write-all \ + --sigul-batch-size=25 fedora-22 \ + $(cat /var/cache/sigul/Stable-F22 /var/cache/sigul/Testing-F22) +.... + +(Make sure you sign each release with the right key... ie, 'fedora-19' +key with F19 packages, or 'epel-6' with EL-6 packages) diff --git a/modules/release_guide/pages/sop_sigul_client_setup.adoc b/modules/release_guide/pages/sop_sigul_client_setup.adoc new file mode 100644 index 0000000..28bcb0f --- /dev/null +++ b/modules/release_guide/pages/sop_sigul_client_setup.adoc @@ -0,0 +1,57 @@ +== Sigul Client Setup + +This document describes how to configure a sigul client. For more +information on sigul, please see link:User-Mitr[User:Mitr] + +=== Prerequisites + +. Install `sigul` and its dependencies. It is available in both Fedora +and EPEL: ++ +On Fedora: ++ +.... +dnf install sigul +.... ++ +On RHEL/CentOS (Using EPEL): ++ +.... +yum install sigul +.... +. Ensure that your koji certificate and the link:Fedora-Cert[Fedora CA +certificates] are present on the system you're running the sigul client +from at the following locations: +* `~/.fedora.cert` +* `~/.fedora-server-ca.cert` +* `~/.fedora-upload-ca.cert` +. Admin privileges on koji are required to write signatures. + +=== Configuration + +. Run `sigul_setup_client` +. Choose a password for your NSS database. By default this will be +stored on-disk in `~/.sigul/client.conf`. +. Choose an export password. You will only need to remember it until +finishing `sigul_setup_client`. +. Enter the DB password you chose earlier, then the export password. You +should see the message `pk12util: PKCS12 IMPORT SUCCESSFUL` +. Enter the DB password again. You should see the message `Done`. +. Assuming that you are running the sigul client within phx2, edit +`~/.sigul/client.conf` to include the following lines: + +.... +[client] +bridge-hostname: sign-bridge.phx2.fedoraproject.org +server-hostname: sign-vault.phx2.fedoraproject.org +.... + +=== Updating your Fedora certificate + +When your Fedora certificate expires, after updating it run the +following commands: + +.... +$ certutil -d ~/.sigul -D -n sigul-client-cert +$ sigul_setup_client +.... diff --git a/modules/release_guide/pages/sop_stage_final_release_for_mirrors.adoc b/modules/release_guide/pages/sop_stage_final_release_for_mirrors.adoc new file mode 100644 index 0000000..b98dcbe --- /dev/null +++ b/modules/release_guide/pages/sop_stage_final_release_for_mirrors.adoc @@ -0,0 +1,38 @@ +== Stage Final Release for Mirrors + +=== Description + +When the release has been fully tested and approved at the "Go/No-Go" +meeting it is ready for release to the Fedora mirrors. + +=== Action + +. Gather the needed info for running the staging script: Release +Version: the numerical version number of the release `24` ComposeID: The +ID of the Compose Label: Compsoe label for the location in stage +`24_RC-1.2` for example Key: the name of teh release key `fedora-24` or +`fedora-24-secondary` as examples Prerelease: 0 or 1 sets if the release +goes in test/ or not Arch: For secondary arches, changes some +internal locations ++ +.... +$ scripts/stage-release.sh 24 Fedora-24-20160614.0 24_RC-1.2 fedora-24 0 +.... +. Sync the release to the Red Hat internal archive following internally +documented + +=== Check and set EOL on previous releases to reflect reality + +. check PDC for active releases and respective EOL date +. if needed run the adjust-eol-all.py script from releng repository to +correct any mistakes + +=== Verification + +Verification is somewhat difficult as one cannot look at the content via +the web server due to permissions. Typically we ask somebody from the +Infrastructure team to give the tree a second set of eyes. + +=== Consider Before Running + +Hope the release is good! diff --git a/modules/release_guide/pages/sop_template.adoc b/modules/release_guide/pages/sop_template.adoc new file mode 100644 index 0000000..a04a85d --- /dev/null +++ b/modules/release_guide/pages/sop_template.adoc @@ -0,0 +1,22 @@ +== Standard Operating Procedure Template + +[NOTE] +.Note +==== +This is a template file, this can be copied to a new filename in order +to start a new document. + +New SOP Documents should be added to the `.. toctree::` list in the +`sop.rst` file. + +These documents are formatted using sphinx-doc reStructure Text, for +markup reference please see: http://sphinx-doc.org/rest.html +==== + +=== Description + +=== Action + +=== Verification + +=== Consider Before Running diff --git a/modules/release_guide/pages/sop_unretire.adoc b/modules/release_guide/pages/sop_unretire.adoc new file mode 100644 index 0000000..bb1b183 --- /dev/null +++ b/modules/release_guide/pages/sop_unretire.adoc @@ -0,0 +1,165 @@ +== Unretiring a package branch + +=== Description + +Sometimes, packagers request that we _unretire_ a package branch that +has previously been retired. + +This typically happens on the [.title-ref]#rawhide# branch, but could +conceivably happen on any stable or arbitrary branch. + +=== Action + +==== Validate Package Ready for Unretirement + +. Verify the package was not retired for any reason, such as legal or +license issues, that would prevent it from being re-instated. +. Ensure a bugzilla was filed to review the package for unretirement. +. Verify with the the requestor exactly which tags they would like +unblocked as part of the unretirement request. + +==== Revert the Retirement Commit + +. Connect to one of the compose systems. ++ +____ +.... +$ ssh compose-x86-02.phx2.fedoraproject.org +.... +____ +. Clone the git-dist package using the the proper release engineering +credentials. ++ +____ +.... +$ GIT_SSH=/usr/local/bin/relengpush fedpkg --user releng clone PACKAGENAME +.... +____ +. Enter the directory of the cloned package and configure the git user +information. ++ +____ +.... +$ cd PACKAGENAME +$ git config --local user.name "Fedora Release Engineering" +$ git config --local user.email "releng@fedoraproject.org" +.... +____ +. Git revert the [.title-ref]#dead.package# file commit in dist-git on +the particular branch using its commit hash_id. Ensure the commit +message contains a URL to the request in pagure. ++ +____ +.... +$ git revert -s COMMIT_HASH_ID +$ GIT_SSH=/usr/loca/bin/relengpush fedpkg --user releng push +.... +____ + +==== Unblock the Package in Koji + +. Check the current state of the branches in koji for the package. ++ +____ +.... +$ koji list-pkgs --show-blocked --package=PACKAGENAME +.... +____ +. Unblock each requested tag using koji. ++ +.... +$ koji unblock-pkg TAGNAME PACKAGENAME +.... + +==== Verify Package is Not Orphaned + +. Check package ownership ++ +Navigate to [.title-ref]#https://src.fedoraproject.org/# and check +package owner. +. If the package is orphaned, then give the package to the requestor +using the [.title-ref]#give-package# script from the +https://pagure.io/releng[Release Engineering Repo]. ++ +.... +$ ./scripts/distgit/give-package --package=PACKAGENAME --custodian=REQUESTOR +.... ++ +[NOTE] +.Note +==== +This script requires the user to be a member of the group +[.title-ref]#cvsadmin# in FAS. +==== + +==== Update Product Definition Center (PDC) + +[NOTE] +.Note +==== +If there are more than one tag to be unblocked then the PDC update step +should be completed for each tag and package. +==== + +. Log into the https://pdc.fedoraproject.org/[Fedora PDC instance] using +a FAS account. +. Check PDC for the entry for the [.title-ref]#PACKAGENAME# in each +[.title-ref]#TAG# that was unblocked in a previous step. ++ +____ +.... +https://pdc.fedoraproject.org/rest_api/v1/component-branch-slas/?branch=TAG&global_component=PACKAGENAME +.... + +[NOTE] +.Note +==== +If no information is returned by this query then it is not in PDC and is +likely not yet a branch. The requestor should use the +[.title-ref]#fedpkg request-branch# utility to ask for a branch. +==== +____ +. If the package existed within PDC then obtain a token from the PDC +site while logged in by navigating to the +[.title-ref]#https://pdc.fedoraproject.org/rest_api/v1/auth/token/obtain/# +URL with the Firefox web browser. +. Press F12 once the page has loaded and select the tab labeled +[.title-ref]#Network#. Refresh the web page and find the line whose +string in the file column matches +[.title-ref]#/rest_api/v1/auth/token/obtain/#. +. Right click on specified line and select Copy>Copy as cURL. Paste this +into a terminal session and add [.title-ref]#-H "Accept: +application/json"#. It should look something similar to the below: ++ +____ +.... +$ curl 'https://pdc.fedoraproject.org/rest_api/v1/auth/token/obtain/' \ +-H 'Host: pdc.fedoraproject.org' \ +-H .0) Gecko/20100101 Firefox/57.0' \ +-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \ +-H 'Accept-Language: en-US,en;q=0.5' \ +--compressed \ +-H 'Cookie: csrftoken=CSRF_TOKEN_HASH; SERVERID=pdc-web01; mellon-saml-sesion-cookie=SAML_SESSION_HASH; sessionid=SESSION_ID_HASH' \ +-H 'Connection: keep-alive' \ +-H 'Upgrade-Insecure-Requests: 1' \ +-H 'Cache-Control: max-age=0' \ +-H "Accept: application/json" +.... +____ +. Using the token obtained from the previous step run the +[.title-ref]#adjust-eol.py# script from the +https://pagure.io/releng[Release Engineering Repo]. ++ +____ +.... +$ PYTHONPATH=scripts/pdc/ scripts/pdc/adjust-eol.py fedora MYTOKEN PACKAGENAME rpm TAG default -y +.... + +[NOTE] +.Note +==== +The local machine will have configuration information in the +[.title-ref]#/etc/pdc.d/# directory. This is why _fedora_ can be passed +as an argument instead of the full API endpoint URL. +==== +____ diff --git a/modules/release_guide/pages/sop_update_critpath.adoc b/modules/release_guide/pages/sop_update_critpath.adoc new file mode 100644 index 0000000..95f1d5a --- /dev/null +++ b/modules/release_guide/pages/sop_update_critpath.adoc @@ -0,0 +1,55 @@ +== Update Critpath + +[NOTE] +.Note +==== +Critpath = "Critical Path" + +This is a collection of packages deemed "critical" to Fedora +==== + +=== Description + +PDC has information about which packages are critpath and which are not. +A script that reads the yum repodata (critpath group in comps, and the +package dependencies) is used to generate this. Since package +dependencies change, this list should be updated periodically. + +=== Action + +. Release engineering scripts for updating critpath live in the +https://pagure.io/releng[releng git repository]. +. Check the critpath.py script to see if the list of releases needs to +be updated: ++ +.... +for r in ['12', '13', '14', '15', '16', '17']: # 13, 14, ... + releasepath[r] = 'releases/%s/Everything/$basearch/os/' % r + updatepath[r] = 'updates/%s/$basearch/' % r + +# Branched Fedora goes here +branched = '18' +.... ++ +The for loop has the version numbers for releases that have gone past +final. branched has the release that's been branched from rawhide but +not yet hit final. (These have different paths in the repository and may +not have an updates directory, thus they're in separate sections). +. Run the script with the release to generate info for (for a release +that's hit final, this is the release number example: "17". For +branched, it's "branched"). ++ +.... +./critpath.py --srpm -o critpath.txt branched +.... +. Run the update script to add that to PDC: ++ +.... +./update-critpath --user toshio f18 critpath.txt +.... ++ +The username is your fas username. You must be in cvsadmin to be able to +change this. The branch is the name of the dist-git branch. critpath.txt +is the file that the output of critpath.py went into. The script needs a +PDC token to talk to the server, configured in /etc/pdc.d/. See the PDC +SOP for more info. diff --git a/modules/release_guide/pages/sop_update_releng_docs.adoc b/modules/release_guide/pages/sop_update_releng_docs.adoc new file mode 100644 index 0000000..71c0790 --- /dev/null +++ b/modules/release_guide/pages/sop_update_releng_docs.adoc @@ -0,0 +1,54 @@ +== Update RelEng Rendered Docs + +=== Description + +When an improvement happens to the Release Engineering documentation +following the `contributing ` for the +http://sphinx-doc.org/[Sphinx] +https://en.wikipedia.org/wiki/ReStructuredText[reStructured Text] source +found in `docs/source` within the https://pagure.io/releng[RelEng git +repository] someone has to manually perform a process in order to update +the documentation that is hosted in the https://pagure.io/pagure[pagure] +documentation space for https://docs.pagure.org/releng/[Fedora RelEng +docs]. + +=== Action + +In order to render the documentation using +http://sphinx-doc.org/[Sphinx], you need to first be sure to have the +package installed: + +.... +$ dnf install python-sphinx +.... + +Then we'll need to clone the RelEng repository and the RelEng docs +repository (the docs git repository is provided by pagure +automatically). There is a script in the [.title-ref]#releng# repository +that takes care of cleanly updating the documentation site for us. + +.... +$ ./scripts/update-docs.sh +.... + +The documentation is now live. + +[NOTE] +.Note +==== +This will require someone with permissions to push to the rawhide branch +for the releng repository. If you are curious whom all has this ability, +please refer to the `Main Page ` and contact someone from the +"Team Composition" +==== + +=== Verification + +Visit the https://docs.pagure.org/releng/[Fedora RelEng docs] website +and verify that the changes are reflected live on the docs site. + +=== Consider Before Running + +No considerations at this time. The docs git repository is simply a +static html hosting space and we can just re-render the docs and push to +it again if necessary. diff --git a/modules/release_guide/pages/sop_updating_comps.adoc b/modules/release_guide/pages/sop_updating_comps.adoc new file mode 100644 index 0000000..b5860ba --- /dev/null +++ b/modules/release_guide/pages/sop_updating_comps.adoc @@ -0,0 +1,53 @@ +== Updating Comps + +=== Description + +When we start a new Fedora development cycle (when we branch rawhide) we +have to create a new comps file for the new release. This SOP covers +that action. + +=== Action + +. clone the comps repo ++ +.... +$ git clone ssh://git@pagure.io/fedora-comps.git +.... +. Create the new comps file for next release: ++ +.... +$ cp comps-f24.xml.in comps-f25.xml.in +.... +. Edit Makefile to update comps-rawhide target ++ +.... +- -comps-rawhide: comps-f24.xml +- - @mv comps-f24.xml comps-rawhide.xml ++comps-rawhide: comps-f25.xml ++ @mv comps-f25.xml comps-rawhide.xml +.... +. Add the new comps file to source control: ++ +.... +$ git add comps-f25.xml.in +.... +. Edit the list of translated comps files in po/POTFILES.in to reflect +currently supported releases. ++ +.... +-comps-f22.xml ++comps-f25.xml +.... +. Send it up: ++ +:::: + $ git push + +=== Verification + +One can review the logs for rawhide compose after this change to make +sure the right comps file was used. + +=== Consider Before Running + +Nothing yet. diff --git a/modules/release_guide/pages/troubleshooting.adoc b/modules/release_guide/pages/troubleshooting.adoc new file mode 100644 index 0000000..68e7bda --- /dev/null +++ b/modules/release_guide/pages/troubleshooting.adoc @@ -0,0 +1,260 @@ +== Fedora Release Engineering Troubleshooting Guide + +Fedora Release Engineering consists of many different systems, many +different code bases and multiple tools. Needless to say, things can get +pretty complex in a hurry. This aspect of Fedora Release Engineering is +not very welcoming to newcomers who would like to get involved. This +guide stands as a place to educate those new to the processes, systems, +code bases, and tools. It also is to serve as a reference to those who +aren't new but maybe are fortunate enough to not have needed to diagnose +things in recent memory. + +We certainly won't be able to document every single possible compontent +in the systems that could go wrong but hopefully over time this document +will stand as a proper knowledge base for reference and educational +purposes on the topics listed below. + +=== Compose + +If something with a compose has gone wrong, there's a number of places +to find information. Each of these are discussed below. + +==== releng-cron list + +The compose output logs are emailed to the releng-cron mailing list. It +is good practice to check the +https://lists.fedoraproject.org/archives/list/releng-cron@lists.fedoraproject.org/[releng-cron +mailing list archives] and find the latest output and give it a look. + +==== compose machines + +If the +https://lists.fedoraproject.org/archives/list/releng-cron@lists.fedoraproject.org/[releng-cron +mailing list archives] doesn't prove to be useful, you can move on to +checking the contents of the composes themselves on the primary compose +machines in the Fedora Infrastructure. At the time of this writing, +there are multiple machines based on the specific compose you are +looking for: + +* Two-Week Atomic Nightly Compose +** `compose-x86-01.phx2.fedoraproject.org` +* Branched Compose +** `branched-composer.phx2.fedoraproject.org` +* Rawhide Compose +** `rawhide-composer.phx2.fedoraproject.org` + +Depending on which specific compose you are in search of will depend on +what full path you will end up inspecting: + +* For Two Week Atomic you will find the compose output in +`/mnt/fedora_koji/compose/` +* For Release Candidate / Test Candidate composes you will find compose +output in `/mnt/koji/compose/` + +[NOTE] +.Note +==== +It's possible the mock logs are no longer available. The mock chroots +are rewritten on subsequent compose runs. +==== + +You can also check for mock logs if they are still persisting from the +compose you are interested in. Find the specific mock chroot directory +name (that will reside in `/var/lib/mock/`) you are looking for by +checking the appropriate compose mock configuration (the following is +only a sample provided at the time of this writing): + +.... +$ ls /etc/mock/*compose* + /etc/mock/fedora-22-compose-aarch64.cfg /etc/mock/fedora-branched-compose-aarch64.cfg + /etc/mock/fedora-22-compose-armhfp.cfg /etc/mock/fedora-branched-compose-armhfp.cfg + /etc/mock/fedora-22-compose-i386.cfg /etc/mock/fedora-branched-compose-i386.cfg + /etc/mock/fedora-22-compose-x86_64.cfg /etc/mock/fedora-branched-compose-x86_64.cfg + /etc/mock/fedora-23-compose-aarch64.cfg /etc/mock/fedora-rawhide-compose-aarch64.cfg + /etc/mock/fedora-23-compose-armhfp.cfg /etc/mock/fedora-rawhide-compose-armhfp.cfg + /etc/mock/fedora-23-compose-i386.cfg /etc/mock/fedora-rawhide-compose-i386.cfg + /etc/mock/fedora-23-compose-x86_64.cfg /etc/mock/fedora-rawhide-compose-x86_64.cfg +.... + +==== running the compose yourself + +If you happen to strike out there and are still in need of debugging, it +might be time to just go ahead and run the compose yourself. The exact +command needed can be found in the cron jobs located on their respective +compose machines, this information can be found in the +`compose-machines` section. Also note that each respective compose +command should be ran from it's respective compose machine as outlined +in the `compose-machines` section. + +An example is below, setting the compose directory as your +`username-debug.1`, the `.1` at the end is common notation for an +incremental run of a compose. If you need to do another, increment it to +`.2` and continue. This is helpful to be able to compare composes. + +[NOTE] +.Note +==== +It is recommended that the following command be run in +https://www.gnu.org/software/screen/[screen] or +https://tmux.github.io/[tmux] +==== + +.... +$ TMPDIR=`mktemp -d /tmp/twoweek.XXXXXX` && cd $TMPDIR \ + && git clone -n https://pagure.io/releng.git && cd releng && \ + git checkout -b stable twoweek-stable && \ + LANG=en_US.UTF-8 ./scripts/make-updates 23 updates $USER-debug.1 +.... + +The above command was pulled from the `twoweek-atomic` cron job with +only the final parameter being altered. This is used as the name of the +output directory. + +The compose can take some time to run, so don't be alarmed if you don't +see output in a while. This should provide you all the infromation +needed to debug and/or diagnose further. When in doubt, as in +`#fedora-releng` on `irc.libera.chat`. + +=== Docker Layered Image Build Service + +The +https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service[Docker +Layered Image Build Service] is built using a combination of +technologies such as https://www.openshift.org/[OpenShift], +https://github.com/projectatomic/osbs-client[osbs-client], and the +https://github.com/release-engineering/koji-containerbuild[koji-containerbuild] +plugin that when combined are often refered to as an OpenShift Build +Service instance (OSBS). Something to note is that +https://www.openshift.org/[OpenShift] is a +http://kubernetes.io/[kubernetes] distribution with many features built +on top of http://kubernetes.io/[kubernetes] such as the +https://docs.openshift.org/latest/dev_guide/builds.html[build primitive] +that is used as the basis for the build service. This information will +hopefully shed light on some of the terminology and commands used below. + +There are a few "common" scenarios in which build may fail or hang that +will need some sort of inspection of the build system. + +==== Build Appears to stall after being scheduled + +In the event that a build scheduled through koji appears to be stalled +and is not in a `free` state (i.e. - has been scheduled). An +administrator will need to ssh into `osbs-master01` or +`osbs-master01.stg` (depending stage vs production) and inspect the +build itself. + +.... +$ oc status +In project default on server https://10.5.126.216:8443 + +svc/kubernetes - 172.30.0.1 ports 443, 53, 53 + +bc/cockpit-f24 custom build of git://.... + build #8 succeeded 7 weeks ago + build #9 failed 33 minutes ago + +$ oc describe build/cockpit-f24-9 +# lots of output about status of the specific build + +$ oc logs build/cockpit-f24-9 +# extremely verbose logs, these should in normal circumstances be found in +# the koji build log post build +.... + +The information found in the commands above will generally identify the +issue. + +==== Build fails but there's no log output in the Koji Task + +Sometimes there is a communications issue between Koji and OSBS which +cause for a failure to be listed in Koji but without all the logs. These +can be diagnosed by checking the `kojid` logs on the Koji builder listed +in the task output. + +For example: + +.... +$ fedpkg container-build +Created task: 90123598 +Task info: http://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90123598 +Watching tasks (this may be safely interrupted)... +90123598 buildContainer (noarch): free +90123598 buildContainer (noarch): free -> open (buildvm-04.stg.phx2.fedoraproject.org) + 90123599 createContainer (x86_64): free + 90123599 createContainer (x86_64): free -> open (buildvm-02.stg.phx2.fedoraproject.org) + 90123599 createContainer (x86_64): open (buildvm-02.stg.phx2.fedoraproject.org) -> closed + 0 free 1 open 1 done 0 failed +90123598 buildContainer (noarch): open (buildvm-04.stg.phx2.fedoraproject.org) -> FAILED: Fault: + 0 free 0 open 1 done 1 failed + +90123598 buildContainer (noarch) failed +.... + +In this example the buildContiner task was scheduled and ran on +`buildvm-04.stg` with the actual createContainer task being on +`buildvm-02.stg`, and `buildvm-02.stg` is where we're going to want to +begin looking for failures to communicate with OSBS as this is the point +of contact with the external system. + +Logs can be found in `/var/log/kojid.log` or if necessary, check the +koji hub in question. Generally, you will want to start with the first +point of contact with OSBS and "work your way back" so in the above +example you would first check `buildvm-02.stg`, then move on to +`buildvm-04.stg` if nothing useful was found in the logs of the previous +machine, and again move on to the koji hub if neither of the builder +machines involved provided useful log information. + +==== Build fails because it can't get to a network resource + +Sometimes there is a situation where the firewall rules get messed up on +one of the OpenShift Nodes in the environment. This can cause output +similar to the following: + +.... +$ fedpkg container-build --scratch +Created task: 90066343 +Task info: http://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90066343 +Watching tasks (this may be safely interrupted)... +90066343 buildContainer (noarch): free +90066343 buildContainer (noarch): free -> open (buildvm-03.stg.phx2.fedoraproject.org) + 90066344 createContainer (x86_64): open (buildvm-04.stg.phx2.fedoraproject.org) + 90066344 createContainer (x86_64): open (buildvm-04.stg.phx2.fedoraproject.org) -> FAILED: Fault: + 0 free 1 open 0 done 1 failed +90066343 buildContainer (noarch): open (buildvm-03.stg.phx2.fedoraproject.org) -> closed + 0 free 0 open 1 done 1 failed +.... + +If we go to the OSBS Master and run the following commands, we will see +the root symptom: + +.... +# oc logs build/scratch-20161102132628 +Error from server: Get https://osbs-node02.stg.phx2.fedoraproject.org:10250/containerLogs/default/scratch-20161102132628-build/custom-build: dial tcp 10.5.126.213:10250: getsockopt: no route to host + +# ping 10.5.126.213 +PING 10.5.126.213 (10.5.126.213) 56(84) bytes of data. +64 bytes from 10.5.126.213: icmp_seq=1 ttl=64 time=0.299 ms +64 bytes from 10.5.126.213: icmp_seq=2 ttl=64 time=0.299 ms +64 bytes from 10.5.126.213: icmp_seq=3 ttl=64 time=0.253 ms +64 bytes from 10.5.126.213: icmp_seq=4 ttl=64 time=0.233 ms +^C +--- 10.5.126.213 ping statistics --- +4 packets transmitted, 4 received, 0% packet loss, time 3073ms +rtt min/avg/max/mdev = 0.233/0.271/0.299/0.028 ms + +# http get 10.5.126.213:10250 + +http: error: ConnectionError: HTTPConnectionPool(host='10.5.126.213', port=10250): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] No route to host',)) while doing GET request to URL: http://10.5.126.213:10250/ +.... + +In the above output, we can see that we do actually have network +connectivity to the Node but we can not connect to the OpenShift service +that should be listening on port `10250`. + +To fix this, you need to ssh into the OpenShift Node that you can't +connect to via port `10250` and run the following commands. This should +resolve the issue. + +.... +iptables -F && iptables -t nat -F && systemctl restart docker && systemctl restart origin-node +.... diff --git a/modules/release_guide/pages/workflow_automation.adoc b/modules/release_guide/pages/workflow_automation.adoc new file mode 100644 index 0000000..181c95a --- /dev/null +++ b/modules/release_guide/pages/workflow_automation.adoc @@ -0,0 +1,114 @@ +== Fedora RelEng Workflow Automation + +The Fedora RelEng Workflow Automation is a means to allow RelEng to +define a pattern by which Release Engineering work is automated in an +uniform fashion. The automation technology of choice is +https://ansible.com/[ansible] and the "workflow engine" is powered by +https://github.com/maxamillion/loopabull[loopabull], which is an event +loop that allows us to pass the information contained within a +http://www.fedmsg.com/en/latest/[fedmsg] and insert it into +https://ansible.com/[ansible] +https://docs.ansible.com/ansible/playbooks.html[playbooks]. This will +effectively create an event driven workflow that can take action +conditionally based on the contents of arbitrary +http://www.fedmsg.com/en/latest/[fedmsg] data. + +Background on the topic can be found in the +https://fedoraproject.org/wiki/Changes/ReleaseEngineeringAutomationWorkflowEngine[Release +Engineering Automation Workflow Engine] Change proposal, as well as in +the https://pagure.io/releng-automation[releng-automation] pagure +repository. + +=== RelEng Workflow Automation Architecture + +By using http://www.fedmsg.com/en/latest/[fedmsg] as the source of +information feeding the event loop, we will configure +https://github.com/maxamillion/loopabull[loopabull] to listen for +specific +https://fedora-fedmsg.readthedocs.io/en/latest/topics.html[fedmsg +topics] which will correspond with https://ansible.com/[ansible] +https://docs.ansible.com/ansible/playbooks.html[playbooks]. When one of +the appropriate +https://fedora-fedmsg.readthedocs.io/en/latest/topics.html[fedmsg +topics] is encountered across the message bus, it's message payload is +then injected into the corresponding playbook as an extra set of +variables. A member of the Fedora Release Engineering Team can at that +point use this as a means to perform whatever arbitrary action or series +of actions they can otherwise perform with https://ansible.com/[ansible] +(including what we can enable via custom +https://docs.ansible.com/ansible/modules.html[modules]) based on the +input of the message payload. + +The general overview of the architecture is below as well as a +description of how it works: + +.... ++------------+ +| fedmsg | +| | ++---+--------+ + | ^ + | | + | | + | | + | | + | | + V | ++------------------+-----------------+ +| | +| Release Engineering | +| Workflow Automation Engine | +| | +| - RabbitMQ | +| - fedmsg-rabbitmq-serializer | +| - loopabull | +| | ++----------------+-------------------+ + | + | + | + | + V ++-----------------------+ +| | +| composer/bodhi/etc | +| | ++-----------------------+ +.... + +The flow of data will begin with an event somewhere in the +https://fedoraproject.org/wiki/Infrastructure[Fedora Infrastructure] +that sends a http://www.fedmsg.com/en/latest/[fedmsg] across the message +bus, then the messages will be taken in and serialized in to a +https://www.rabbitmq.com/[rabbitmq] worker queue using +https://pagure.io/fedmsg-rabbitmq-serializer[fedmsg-rabbitmq-serializer]. +Then https://github.com/maxamillion/loopabull[loopabull] will be +listening to the rabbitmq worker queue for tasks to come in. Once a +message is recieved, it is processed and once it is either no-op'd or a +corresponding ansible playbook is run to completion, the message will be +`ack`'d and cleared from the worker queue. This will allow for us to +scale loopabull instances independently from the message queue as well +as ensure that work is not lost because of a downed or busy loopabull +instance. Also, as a point of note, the loopabull service instances will +be scaled using https://freedesktop.org/wiki/Software/systemd/[systemd] +https://fedoramagazine.org/systemd-template-unit-files/[unit templates]. + +Once a playbook has been triggered, it will run tasks on remote systems +on behalf of a loopabull automation user. These users can be privileged +if need be, however the scope of their privilege is based on the purpose +they serve. These user accounts are provisioned by the +https://fedoraproject.org/wiki/Infrastructure[Fedora Infrastructure] +Team based on the requirements of the +`RelEng Task Automation User Request Standard Operating +Procedure (SOP) ` document and +tasks are subject to code and security audit. + +=== Fedora Lib RelEng + +https://pagure.io/flr[Fedora Lib RelEng] (flr), is a library and set of +command line tools to expose the library that aims to provide re-usable +code for common tasks that need to be done in Release Engineering. +Combining this set of command line tools when necessary with the Release +Engineering Automation pipeline allows for easy separation of +permissions and responsibilities via sudo permissions on remote hosts. +This is explained in more detail on the project's pagure page.