infra-docs-fpo/modules/release_guide/pages/release_eol.adoc

include::_partials/attributes.adoc[]

= End Of Life

== Description

Each release of Fedora is maintained as laid out in the
https://docs.fedoraproject.org/en-US/releases/lifecycle/#_maintenance_schedule[maintenance
schedule]. At the conclusion of the maintenance period, a Fedora release
enters `end of life` status. This procedure describes the tasks
necessary to move a release to that status. To get the correct EOL date visit
https://fedorapeople.org/groups/schedule/f-{branched}/f-{branched}-key-tasks.html[release schedule] for the last release.

== Actions

=== Reminder announcement

Around a week before the EOL date, end an email to `devel@`, `devel-announce@`, `test@`, `test-announce@`, `announce@` lists as remainder about the release EOL.
Use the https://pagure.io/releng/blob/main/f/mail-templates/08-end-of-life.txt[template] from release engineering repo.

[NOTE]
====
Please ensure that the reminder email is scheduled to be sent one week before the end of life.
====

=== `fedora-appstream-metadata` package update

If necessary, update the
https://src.fedoraproject.org/rpms/fedora-appstream-metadata[`fedora-appstream-metadata`]
package with the new EOL date. Follow the instructions from the repo README.

Build the package, create an update and make sure it lands in the stable repo
before proceeding.

This is not necessary if the package already has the correct EOL date.

=== Koji tasks

* Disable builds by removing targets

[source,subs="attributes+"]
....
koji remove-target f{old_release}
koji remove-target f{old_release}-candidate
koji remove-target f{old_release}-container-candidate
koji remove-target f{old_release}-flatpak-candidate
koji remove-target f{old_release}-infra
koji remove-target f{old_release}-coreos-continuous
koji remove-target f{old_release}-rebuild
....

* Remove sidetags

Remove any existing sidetags from the eol release.

[source,subs="attributes+"]
....
for i in `koji list-sidetags --basetag f{old_release}`
do
	koji remove-sidetag $i
done
....

You need to do this differently because just removing the target of
a sidetag will leave it in a weird state where it cannot be removed.


* Purge from disk the signed copies of rpms that are signed with the
EOL'd release key. To achieve this, add the release key to
*koji_cleanup_signed.py* script in https://pagure.io/releng[releng] repo
and run the script on compose-branched01.rdu3.fedoraproject.org:

....
tmux
sudo su
cd /root/releng
./scripts/koji_cleanup_signed.py
....

[NOTE]
====
Running this script in tmux or screen is recommended as it takes a long time to run.
====

=== Final stable push

Do a final stable push for the release in Bodhi, following the
xref:releng_misc_guide:sop_pushing_updates.adoc[Pushing Updates SOP].

=== Bodhi tasks

Run the following bodhi commands to set the releases state to
*archived*

[source,subs="attributes+"]
....
bodhi releases edit --name "F{old_release}" --state archived
bodhi releases edit --name "F{old_release}C" --state archived
bodhi releases edit --name "F{old_release}F" --state archived
....

=== Fedora Infra Ansible Changes

We need to make some changes and then run several roles in the ansible repo.

==== Update FedoraPreviousPrevious.yaml in ansible repository

Edit /vars/all/FedoraPreviousPrevious.yaml and set the value to `False`.

==== Adjust the EOL for gnome-software

Edit /roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json in ansible
and set the release to status "EOL".

==== Adjust the greenwave gating policy

Edit /roles/openshift-apps/greenwave/templates/fedora.yaml
and find the policies with \_upgrade_ in their name, near the bottom.
You should see a comment noting these are not run on "the oldest stable release".
The oldest release listed in the `product_versions` section for each of these policies should be one higher than the release you are EOLing.
Remove the lines for these releases, so the oldest release listed is now *two* higher than the release you are EOLing.
You may also remove all lines for the release you are EOLing, but this is not critical.

==== Remove eol release from robosignatory config

Edit roles/robosignatory/templates/robosignatory.toml.j2
and remove stansas that use "fedora-{old_release}" in them.

==== Run the playbooks

First, push the above edits. Then run the associated playbooks on _batcave01_:

....
sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/bodhi-backend.yml
sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/koji-hub.yml
sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/releng-compose.yml
sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/proxies.yml -t pkgdb2
sudo ansible-playbook /srv/web/infra/ansible/playbooks/openshift-apps/bodhi.yml
sudo ansible-playbook /srv/web/infra/ansible/playbooks/openshift-apps/greenwave.yml
....

You will need to have someone enable sshd on autosign02 and then run:

....
sudo ansible-playbook /srv/web/infra/ansible/playbooks/manual/autosign.yml
....

And then someone with the robosignatory passphrase will need to restart it.

[NOTE]
====
Another way to run the playbook is using rbac-playbook, in case you don't have sysadmin-main rights or can't become root.
Syntax: sudo rbac-playbook groups/bodhi-backend.yml
====

=== Final announcement

Send the final announcement to `devel@`, `devel-announce@`, `test@`, `test-announce@`, `announce@` lists.
Use the https://pagure.io/releng/blob/main/f/mail-templates/08-end-of-life.txt[template] from release engineering repo.

=== Other tasks

==== Remove EOL release from Koschei

The release going EOL must be removed from Koschei to prevent a flood of failed scratch builds.
See the xref:sysadmin_guide:koschei.adoc#_removing_an_eol_fedora_release[Koschei sysadmin guide] for details.

==== Remove release from fedfind source of truth

The Quality team maintains a source of truth for release information used by the fedfind tool
on fedorapeople (people01) in `/srv/groups/qa/metadata/release.json`.
If you have access to this file, you can remove the release going EOL from the file's list of stable releases.
Otherwise, alert the quality team to do so.

=== Stage the EOL release to archive

. Log into to bodhi-backend01 and become ftpsync, via root
+
____
....
ssh bodhi-backend01.rdu3.fedoraproject.org
sudo su
su - ftpsync
....
____
. Then change into the releases directory.
+
____
....
cd /pub/fedora/linux/releases
....
____
. Check to see that the target directory doesn't already exist.
+
____
....
ls /pub/archive/fedora/linux/releases/
....
____
. Do a recursive rsync to update any changes in the trees since the previous copy.
+
____
[source,subs="attributes+"]
....
rsync -avAXSHP ./{old_release}/ /pub/archive/fedora/linux/releases/{old_release}/
....
____
. Similarly, sync updates and updates/testing.
+
____
[source,subs="attributes+"]
....
cd ../updates/
rsync -avAXSHP {old_release}/ /pub/archive/fedora/linux/updates/{old_release}/
cd testing
rsync -avAXSHP {old_release}/ /pub/archive/fedora/linux/updates/testing/{old_release}/
....
____
. Do the same with fedora-secondary.
+
____
[source,subs="attributes+"]
....
cd /pub/fedora-secondary/releases/
rsync -avAXSHP ./{old_release}/ /pub/archive/fedora-secondary/releases/{old_release}/
cd ../updates/
rsync -avAXSHP {old_release}/ /pub/archive/fedora-secondary/updates/{old_release}/
cd testing
rsync -avAXSHP {old_release}/ /pub/archive/fedora-secondary/updates/testing/{old_release}/
....
____
. Announce to the mirror list this has been done and that in 2 weeks you will
move the old trees to archives.

=== Move the EOL release to archive

. In two weeks, run the following playbook from batcave:
____
[source,subs="attributes+"]
....
sudo ansible-playbook -v /srv/web/infra/ansible/playbooks/manual/mirrormanager/move-to-archive.yml --extra-vars="product='Fedora' version='{old_release}'"
....
____
. If there are problems, the postgres DB may have issues and so you need
to get a DBA to update the backend to fix items.
. Wait an hour or so then you can remove the files from the main tree.

[NOTE]
====
We need to do a cleanup here for both fedora and fedora-secondary for releases, updates, and updates/testing for the {old_release} release.
====

____
[source,subs="attributes+"]
....
ssh bodhi-backend01
cd /pub/fedora/linux/releases/{old_release}
ls
# make sure you have stuff here
rm -rf *
ln ../20/README .
cd /pub/fedora/linux/updates/{old_release}
ls
# make sure you have stuff here
rm -rf *
ln ../20/README .
cd /pub/fedora/linux/updates/testing/{old_release}
ls
# make sure you have stuff here
rm -rf *
ln ../20/README .

#for fedora-secondary
cd /pub/fedora-secondary/releases/{old_release}
ls
# make sure you have stuff here
rm -rf *
ln ../20/README .
cd /pub/fedora-secondary/updates/{old_release}
ls
# make sure you have stuff here
rm -rf *
ln ../20/README .
cd /pub/fedora-secondary/updates/testing/{old_release}
ls
# make sure you have stuff here
rm -rf *
ln ../20/README .
....
____

== Consider Before Running

* Resource contention in infrastructure, such as outages
* Extenuating circumstances for specific planned updates, if any
* Send the reminder announcement, if it isn't sent already