296 lines
9 KiB
Text
296 lines
9 KiB
Text
include::_partials/attributes.adoc[]
|
|
|
|
= End Of Life
|
|
|
|
== Description
|
|
|
|
Each release of Fedora is maintained as laid out in the
|
|
https://docs.fedoraproject.org/en-US/releases/lifecycle/#_maintenance_schedule[maintenance
|
|
schedule]. At the conclusion of the maintenance period, a Fedora release
|
|
enters `end of life` status. This procedure describes the tasks
|
|
necessary to move a release to that status. To get the correct EOL date visit
|
|
https://fedorapeople.org/groups/schedule/f-{branched}/f-{branched}-key-tasks.html[release schedule] for the last release.
|
|
|
|
== Actions
|
|
|
|
=== Reminder announcement
|
|
|
|
Around a week before the EOL date, end an email to `devel@`, `devel-announce@`, `test@`, `test-announce@`, `announce@` lists as remainder about the release EOL.
|
|
Use the https://pagure.io/releng/blob/main/f/mail-templates/08-end-of-life.txt[template] from release engineering repo.
|
|
|
|
[NOTE]
|
|
====
|
|
Please ensure that the reminder email is scheduled to be sent one week before the end of life.
|
|
====
|
|
|
|
=== `fedora-appstream-metadata` package update
|
|
|
|
If necessary, update the
|
|
https://src.fedoraproject.org/rpms/fedora-appstream-metadata[`fedora-appstream-metadata`]
|
|
package with the new EOL date. Follow the instructions from the repo README.
|
|
|
|
Build the package, create an update and make sure it lands in the stable repo
|
|
before proceeding.
|
|
|
|
This is not necessary if the package already has the correct EOL date.
|
|
|
|
=== Koji tasks
|
|
|
|
* Disable builds by removing targets
|
|
|
|
[source,subs="attributes+"]
|
|
....
|
|
koji remove-target f{old_release}
|
|
koji remove-target f{old_release}-candidate
|
|
koji remove-target f{old_release}-container-candidate
|
|
koji remove-target f{old_release}-flatpak-candidate
|
|
koji remove-target f{old_release}-infra
|
|
koji remove-target f{old_release}-coreos-continuous
|
|
koji remove-target f{old_release}-rebuild
|
|
....
|
|
|
|
* Remove sidetags
|
|
|
|
Remove any existing sidetags from the eol release.
|
|
|
|
[source,subs="attributes+"]
|
|
....
|
|
for i in `koji list-sidetags --basetag f{old_release}`
|
|
do
|
|
koji remove-sidetag $i
|
|
done
|
|
....
|
|
|
|
You need to do this differently because just removing the target of
|
|
a sidetag will leave it in a weird state where it cannot be removed.
|
|
|
|
|
|
* Purge from disk the signed copies of rpms that are signed with the
|
|
EOL'd release key. To achieve this, add the release key to
|
|
*koji_cleanup_signed.py* script in https://pagure.io/releng[releng] repo
|
|
and run the script on compose-branched01.rdu3.fedoraproject.org:
|
|
|
|
....
|
|
tmux
|
|
sudo su
|
|
cd /root/releng
|
|
./scripts/koji_cleanup_signed.py
|
|
....
|
|
|
|
[NOTE]
|
|
====
|
|
Running this script in tmux or screen is recommended as it takes a long time to run.
|
|
====
|
|
|
|
=== Final stable push
|
|
|
|
Do a final stable push for the release in Bodhi, following the
|
|
xref:releng_misc_guide:sop_pushing_updates.adoc[Pushing Updates SOP].
|
|
|
|
=== Bodhi tasks
|
|
|
|
Run the following bodhi commands to set the releases state to
|
|
*archived*
|
|
|
|
[source,subs="attributes+"]
|
|
....
|
|
bodhi releases edit --name "F{old_release}" --state archived
|
|
bodhi releases edit --name "F{old_release}C" --state archived
|
|
bodhi releases edit --name "F{old_release}F" --state archived
|
|
....
|
|
|
|
=== Fedora Infra Ansible Changes
|
|
|
|
We need to make some changes and then run several roles in the ansible repo.
|
|
|
|
==== Update FedoraPreviousPrevious.yaml in ansible repository
|
|
|
|
Edit /vars/all/FedoraPreviousPrevious.yaml and set the value to `False`.
|
|
|
|
==== Adjust the EOL for gnome-software
|
|
|
|
Edit /roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json in ansible
|
|
and set the release to status "EOL".
|
|
|
|
==== Adjust the greenwave gating policy
|
|
|
|
Edit /roles/openshift-apps/greenwave/templates/fedora.yaml
|
|
and find the policies with \_upgrade_ in their name, near the bottom.
|
|
You should see a comment noting these are not run on "the oldest stable release".
|
|
The oldest release listed in the `product_versions` section for each of these policies should be one higher than the release you are EOLing.
|
|
Remove the lines for these releases, so the oldest release listed is now *two* higher than the release you are EOLing.
|
|
You may also remove all lines for the release you are EOLing, but this is not critical.
|
|
|
|
==== Remove eol release from robosignatory config
|
|
|
|
Edit roles/robosignatory/templates/robosignatory.toml.j2
|
|
and remove stansas that use "fedora-{old_release}" in them.
|
|
|
|
==== Run the playbooks
|
|
|
|
First, push the above edits. Then run the associated playbooks on _batcave01_:
|
|
|
|
....
|
|
sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/bodhi-backend.yml
|
|
sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/koji-hub.yml
|
|
sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/releng-compose.yml
|
|
sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/proxies.yml -t pkgdb2
|
|
sudo ansible-playbook /srv/web/infra/ansible/playbooks/openshift-apps/bodhi.yml
|
|
sudo ansible-playbook /srv/web/infra/ansible/playbooks/openshift-apps/greenwave.yml
|
|
....
|
|
|
|
You will need to have someone enable sshd on autosign02 and then run:
|
|
|
|
....
|
|
sudo ansible-playbook /srv/web/infra/ansible/playbooks/manual/autosign.yml
|
|
....
|
|
|
|
And then someone with the robosignatory passphrase will need to restart it.
|
|
|
|
[NOTE]
|
|
====
|
|
Another way to run the playbook is using rbac-playbook, in case you don't have sysadmin-main rights or can't become root.
|
|
Syntax: sudo rbac-playbook groups/bodhi-backend.yml
|
|
====
|
|
|
|
=== Final announcement
|
|
|
|
Send the final announcement to `devel@`, `devel-announce@`, `test@`, `test-announce@`, `announce@` lists.
|
|
Use the https://pagure.io/releng/blob/main/f/mail-templates/08-end-of-life.txt[template] from release engineering repo.
|
|
|
|
=== Other tasks
|
|
|
|
==== Remove EOL release from Koschei
|
|
|
|
The release going EOL must be removed from Koschei to prevent a flood of failed scratch builds.
|
|
See the xref:sysadmin_guide:koschei.adoc#_removing_an_eol_fedora_release[Koschei sysadmin guide] for details.
|
|
|
|
==== Remove release from fedfind source of truth
|
|
|
|
The Quality team maintains a source of truth for release information used by the fedfind tool
|
|
on fedorapeople (people01) in `/srv/groups/qa/metadata/release.json`.
|
|
If you have access to this file, you can remove the release going EOL from the file's list of stable releases.
|
|
Otherwise, alert the quality team to do so.
|
|
|
|
=== Stage the EOL release to archive
|
|
|
|
. Log into to bodhi-backend01 and become ftpsync, via root
|
|
+
|
|
____
|
|
....
|
|
ssh bodhi-backend01.rdu3.fedoraproject.org
|
|
sudo su
|
|
su - ftpsync
|
|
....
|
|
____
|
|
. Then change into the releases directory.
|
|
+
|
|
____
|
|
....
|
|
cd /pub/fedora/linux/releases
|
|
....
|
|
____
|
|
. Check to see that the target directory doesn't already exist.
|
|
+
|
|
____
|
|
....
|
|
ls /pub/archive/fedora/linux/releases/
|
|
....
|
|
____
|
|
. Do a recursive rsync to update any changes in the trees since the previous copy.
|
|
+
|
|
____
|
|
[source,subs="attributes+"]
|
|
....
|
|
rsync -avAXSHP ./{old_release}/ /pub/archive/fedora/linux/releases/{old_release}/
|
|
....
|
|
____
|
|
. Similarly, sync updates and updates/testing.
|
|
+
|
|
____
|
|
[source,subs="attributes+"]
|
|
....
|
|
cd ../updates/
|
|
rsync -avAXSHP {old_release}/ /pub/archive/fedora/linux/updates/{old_release}/
|
|
cd testing
|
|
rsync -avAXSHP {old_release}/ /pub/archive/fedora/linux/updates/testing/{old_release}/
|
|
....
|
|
____
|
|
. Do the same with fedora-secondary.
|
|
+
|
|
____
|
|
[source,subs="attributes+"]
|
|
....
|
|
cd /pub/fedora-secondary/releases/
|
|
rsync -avAXSHP ./{old_release}/ /pub/archive/fedora-secondary/releases/{old_release}/
|
|
cd ../updates/
|
|
rsync -avAXSHP {old_release}/ /pub/archive/fedora-secondary/updates/{old_release}/
|
|
cd testing
|
|
rsync -avAXSHP {old_release}/ /pub/archive/fedora-secondary/updates/testing/{old_release}/
|
|
....
|
|
____
|
|
. Announce to the mirror list this has been done and that in 2 weeks you will
|
|
move the old trees to archives.
|
|
|
|
=== Move the EOL release to archive
|
|
|
|
. In two weeks, run the following playbook from batcave:
|
|
____
|
|
[source,subs="attributes+"]
|
|
....
|
|
sudo ansible-playbook -v /srv/web/infra/ansible/playbooks/manual/mirrormanager/move-to-archive.yml --extra-vars="product='Fedora' version='{old_release}'"
|
|
....
|
|
____
|
|
. If there are problems, the postgres DB may have issues and so you need
|
|
to get a DBA to update the backend to fix items.
|
|
. Wait an hour or so then you can remove the files from the main tree.
|
|
|
|
[NOTE]
|
|
====
|
|
We need to do a cleanup here for both fedora and fedora-secondary for releases, updates, and updates/testing for the {old_release} release.
|
|
====
|
|
|
|
____
|
|
[source,subs="attributes+"]
|
|
....
|
|
ssh bodhi-backend01
|
|
cd /pub/fedora/linux/releases/{old_release}
|
|
ls
|
|
# make sure you have stuff here
|
|
rm -rf *
|
|
ln ../20/README .
|
|
cd /pub/fedora/linux/updates/{old_release}
|
|
ls
|
|
# make sure you have stuff here
|
|
rm -rf *
|
|
ln ../20/README .
|
|
cd /pub/fedora/linux/updates/testing/{old_release}
|
|
ls
|
|
# make sure you have stuff here
|
|
rm -rf *
|
|
ln ../20/README .
|
|
|
|
#for fedora-secondary
|
|
cd /pub/fedora-secondary/releases/{old_release}
|
|
ls
|
|
# make sure you have stuff here
|
|
rm -rf *
|
|
ln ../20/README .
|
|
cd /pub/fedora-secondary/updates/{old_release}
|
|
ls
|
|
# make sure you have stuff here
|
|
rm -rf *
|
|
ln ../20/README .
|
|
cd /pub/fedora-secondary/updates/testing/{old_release}
|
|
ls
|
|
# make sure you have stuff here
|
|
rm -rf *
|
|
ln ../20/README .
|
|
....
|
|
____
|
|
|
|
== Consider Before Running
|
|
|
|
* Resource contention in infrastructure, such as outages
|
|
* Extenuating circumstances for specific planned updates, if any
|
|
* Send the reminder announcement, if it isn't sent already
|