infra-docs-fpo/modules/release_guide/pages/sop_mass_branching.adoc

include::_partials/attributes.adoc[]

= Mass Branching

== Description

At each alpha freeze we branch the pending release away from `devel/`
which allows rawhide (currently F{rawhide}) to move on while the pending release goes into
bugfix and polish mode.

You will find below the list of steps to follow to branch a new Fedora release.

== Mass resigning

When we branch off of rawhide, the branched release packages are already signed by
the F\{release} key, but we need to resign everything in rawhide for the new F+1 key.
ie, When we branch f{rawhide} off rawhide, all it's packages are already signed by the f{rawhide}
key, but we need to resign everything with the f{rawhide_next} key for rawhide.

* Add a new config for the new key to robosignatory. Something like:

```
            [[consumer_config.koji_instances.primary.tags]]
            from = "f{rawhide}"
            to = "f{rawhide}"
            key = "{{ (env == 'production')|ternary('fedora-{rawhide_next}', 'testkey') }}"
            keyid = "{{ (env == 'production')|ternary('a15B79cc', 'd300e724') }}"
            {% if env == "production" %}
            file_signing_key = "fedora-{rawhide_next}-ima"
            {% endif %}
```
This allows robosignatory to sign packages in the f{rawhide} tag with the f{rawhide_next} key.
* git clone https://pagure.io/releng
* confirm the new key fingerprint is in scripts/sigulsign_unsigned.py
* run sigulsign_unsigned.py to gather list of packages to sign:
```
./sigulsign_unsigned.py --just-list --tag f{rawhide} fedora-{rawhide_next} | grep src | sed -e 's|.src||' > unsigned-packages
```
You should get a list of all the source packages by name.
* copy unsigned-packages list to autosign01
* on autosign01 run in a tmux session:
```
sudo -su robosignatory
passphrase=$(systemd-ask-password "Please enter passphrase for 'autosign' key: ")
(enter the autosign passphrase)
keyctl add user "sigul:autosign" "${passphrase}" @s
for i in `cat unsigned-packages`
do
  echo $i;
  robosignatory sign-tag primary $i f{rawhide};
  sleep 1;
done
```
This will iterate over all packages and sign them with the new f{rawhide_next} key.
Once complete, re-run the ./sigulsign_unsigned.py command to confirm all are signed.
On branching day, remove the robosignatory config for this resigning.

== Send announcement

One day before the mass branching, we send out announcemt because during mass branching,
new koji builds for rawhide are disabled.

== Create Bugzilla Component

To ensure accurate tracking and management of issues for each Fedora release, a new Bugzilla component should be created for the branched version, as this step is currently not covered in the documentation. A user with access to the 'fedora-pm' Bugzilla group needs to perform the following actions:

* Log in to Bugzilla.
* Navigate to the "Administration" section.
* Under "Components," locate "Fedora."
* Select "Edit Versions" to view the existing components.
* Add the new branched version to the list.

This process allows for proper issue categorization and ensures all bugs related to the new Fedora release are properly tracked.

== Disable rawhide builds in koji


Previously, disabling all builds in Koji involved configuring an outage as demonstrated in this pull request. However, starting with the Fedora 41 release, the approach was refined to block external submissions in Koji by setting a custom IP restriction.

The recommended method to achieve this is by adding firewall rules to both koji01 and koji02 servers, effectively blocking connections from proxy01 and proxy10. This can be achieved with the following iptables commands:

[source,bash,subs="attributes"]
----
iptables -I INPUT -m tcp -p tcp --dport 80 -s proxy01.rdu3.fedoraproject.org -j REJECT
iptables -I INPUT -m tcp -p tcp --dport 80 -s proxy10.rdu3.fedoraproject.org -j REJECT
----

These commands reject incoming traffic on port 80 from the specified proxies, preventing external submissions. Internal connections routed via proxy101 and proxy110 will continue to function as expected.

To reverse the firewall changes and allow external submissions again, use:

[source,bash,subs="attributes"]
----
iptables -D INPUT -m tcp -p tcp --dport 80 -s proxy01.rdu3.fedoraproject.org -j REJECT
iptables -D INPUT -m tcp -p tcp --dport 80 -s proxy10.rdu3.fedoraproject.org -j REJECT

----

This change should be implemented on both koji01 and koji02 machine.

=== Cancel all running build for rawhide

List all running tasks and select onle those relevant for current branching.

[source,bash,subs="attributes"]
----
$ koji list-builds --state=0 --type=rpm | grep fc{branched} | awk '{print $1}'
----

Cancel each of those tasks

```
$ koji cancel <build>
```

== Repos that needs to be updated for branching

All the following listed repos needs updating, including adding a new
branch for branched release and updating rawhide branch with new release
values.

* https://pagure.io/pungi-fedora
* https://pagure.io/fedora-kickstarts
* https://pagure.io/fedora-comps
* https://pagure.io/fedora-lorax-templates/
* https://pagure.io/workstation-ostree-config/
** Follow the procedure detailed in the repo README under the `Branching instructions for new Fedora releases` section.
* https://src.fedoraproject.org/rpms/fedora-release
* https://src.fedoraproject.org/rpms/fedora-repos

== Releng Changes

As part of the mass branching process, changes were made to the `find_unblocked_orphans.py` script to align with the latest Fedora release versions.

=== Key Changes
- Updated `koji_tag` to reference the new Fedora version:
  - Changed from `f{branched}` to `f{rawhide}`.
- Updated `pagure_branch` to reflect the new branching:
  - Changed from `f{stable}` to `f{branched}`.
- Ensured `koji_hub` references remain consistent across updates.

== dist-git

Now we need to update dist-git in two steps:

* Create the new branch in git
* Update the `gitolite.conf` to allow users to push to this new branch

For both of these actions, we need a file listing all **active** packages. 
Previously, this was generated by **PDC**, but since **PDC is decommissioned**, 
we now use a script to fetch active packages by filtering out retired and never-imported ones.

=== Generate the Active Package List

This script should be run in the **pkg machine**, which can be accessed via SSH from `batcave01.rdu3.fedoraproject.org`.

```
$ ssh batcave01.rdu3.fedoraproject.org
$ sudo ssh pkgs01.rdu3.fedoraproject.org  # For production
$ sudo ssh pkgs01.stg.rdu3.fedoraproject.org  # For testing
```

Before running the script, you need to **obtain the script from the Fedora RelEng repository**. You can do this in one of two ways:

**Option 1: Clone the RelEng repository**
```bash
$ git clone https://pagure.io/releng.git
$ cd releng/scripts/branching
$ chmod +x get_all_active_packages_branching.sh
```

This will generate the file components_f{branched}.txt, which contains all valid, active, and imported packages, prefixed with rpm/.

=== Create the git branches

On `pkgs01.stg` (for testing) or `pkgs01` (for production), run:

```
$ sudo -u pagure python /usr/local/bin/mass-branching-git.py <new branch name> <input file>
```

Where `<new branch name>` will be like `f{rawhide}` and the
`<input file>` the path to the file generated by pdc above.

== Ansible

Apps in https://pagure.io/fedora-infra/ansible[ansible] need to be updated to be aware of a new branch.

=== Bodhi

Bodhi needs to be updated to add new release. This needs to be done in
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2[bodhi2 role] in
infra ansible repo. This change includes, updating `koji-sync-listener.py`,
`new-updates-sync`, pungi configs for rpm updates, bodhi templates.

* `roles/bodhi2/backend/files/new-updates-sync`
* `roles/bodhi2/backend/tasks/main.yml`
* `roles/bodhi2/backend/templates/pungi.rpm.conf.j2`
* `roles/bodhi2/backend/templates/koji_sync_listener.toml`

Please check these files from the https://pagure.io/fedora-infra/ansible/c/549e5d3ace41c04fdbef9d81f359f16c2fe0c2fa?branch=main[commit] for your reference.

Also, set the EOL date for the new release to match the one in the
https://fedorapeople.org/groups/schedule/f-{branched}/f-{branched}-key-tasks.html[Fedora schedule]:

[source,subs="attributes+"]
....
$ bodhi releases edit --name F{branched} --eol YYYY-MM-DD
$ bodhi releases edit --name F{branched}C --eol YYYY-MM-DD
$ bodhi releases edit --name F{branched}F --eol YYYY-MM-DD
....

=== Poddlers (formerly known as toddlers)

==== Add new SLA to the poddlers App

Use https://pagure.io/fedora-infra/ansible/pull-request/2449[this PR] for reference and add new version to the config.

- Restart poddlers playbook and pod from OpenShift.
- **New Build Trigger:**


  ```sh
  oc -n poddlers get bc  # List existing build configs
  oc -n poddlers start-build bc/<build_name>
  ```


=== Fedora Branched

. Set https://pagure.io/fedora-infra/ansible/blob/main/f/vars/all/FedoraBranched.yaml[FedoraBranched] to True.
. Set https://pagure.io/fedora-infra/ansible/blob/main/f/vars/all/FedoraBranchedBodhi.yaml[FedoraBranchedBodhi] to preenable.

Please check the file `FedoraBranched.yaml` and `FedoraBranchedBodhi.yaml` from the https://pagure.io/fedora-infra/ansible/c/549e5d3ace41c04fdbef9d81f359f16c2fe0c2fa?branch=main[commit] for your reference.

=== Koji hub

Update the koji hub config to allow side tags for new koji rawhide (currently f{rawhide}) tag.

Please check the file `roles/koji_hub/templates/hub.conf.j2` from the https://pagure.io/fedora-infra/ansible/c/549e5d3ace41c04fdbef9d81f359f16c2fe0c2fa?branch=main[commit] for your reference.

Also, update the `roles/koji_hub/templates/koji-gc.conf.j2` file to list the new release
keys and retention.

=== Robosignatory

Robosignatory has two parts, which can be found in
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/robosignatory[robosignatory
role] in infra ansible repo.:

. Disable branched signing, so that we can freeze branched until we get a compose.
. Adding new release.

Please check the file `roles/robosignatory/templates/robosignatory.toml.j2` from the https://pagure.io/fedora-infra/ansible/c/a7dcdbd794e728f7b80b1afc6c142c0196b8c1a3?branch=main[commit] for your reference.

=== Push the changes

When done editing the files, commit, push and apply them via the corresponding
ansible playbooks:

```
$ sudo rbac-playbook groups/koji-hub.yml
$ sudo rbac-playbook groups/releng-compose.yml
$ sudo rbac-playbook openshift-apps/bodhi.yml
$ sudo rbac-playbook groups/bodhi-backend.yml
$ sudo rbac-playbook openshift-apps/greenwave.yml
$ sudo -i ansible-playbook /srv/web/infra/ansible/playbooks/groups/proxies.yml -t pkgdb2
```

[NOTE]
====
Avoid running the `koji-hub.yaml` playbook after turning off Koji builders to prevent overwriting manual changes.
====


Ask someone in fedora infra to run the robosignatory playbook.

== Koji

The koji build system needs to have some tag/target work done to handle
builds from the new branch and to update where builds from rawhide go.

Run
https://pagure.io/releng/blob/main/f/scripts/branching/make-koji-release-tags[make-koji-release-tags]
script in https://pagure.io/releng[pagure releng] repo

== Fedora Release

The `fedora-release` package needs to be updated in Rawhide and
Branched.

Changes to `fedora-release.spec` in the *rawhide* branch:

(can also check this https://src.fedoraproject.org/rpms/fedora-release/c/841f1fb8e5f7ff2b11a5e0905f15e765da405edd?branch=rawhide[commit] for reference)

[arabic]
. Increment `%define dist_version` to {rawhide}:
+
[source,subs="attributes"]
----
%define dist_version {rawhide}
----

. Increment `Version:` and reset `Release:`:
+
[source,subs="attributes"]
----
Version:        {rawhide}
Release:        0.1%{?eln:.eln%\{eln}}
----

. Add a `%changelog` entry:
+
[source,subs="attributes"]
----
%changelog
* Day Mon DD YYYY Name <Email> - {rawhide}-0.1
- Setup for rawhide being F{rawhide}
----

Changes to `fedora-release.spec` in the *branched* (currently {branched}) branch:

(can also check this https://src.fedoraproject.org/rpms/fedora-release/c/a2c289b44335eec89efaddee02d807b048edd51f?branch=f37[commit] for reference)

[arabic]
. Adjust `release_name` and unset `is_rawhide`:
+
[source,subs="attributes"]
----
%define release_name {branched_name}
%define is_rawhide 0
----

. Verify the correct number for `dist_version` and `Version:`:
+
[source,subs="attributes"]
----
%define dist_version {branched}
Version:        {branched}
----

. Bump `Release:`:
+
[source,subs="attributes"]
----
Release:        0.4%{?eln:.eln%\{eln}}
----

. Add a `%changelog` entry:
+
[source,subs="attributes"]
----
%changelog
* Day Mon DD YYYY Name <Email> - {branched}-0.4
- Branching F{branched} from rawhide
----

== Fedora Repos

The `fedora-repos` package needs to be updated in Rawhide, Branched, and
also in all stable release branches (in order to receive new GPG keys
and updated symlinks).

Changes to the *rawhide* branch (mostly in `fedora-repos.spec`):

(can also check this https://src.fedoraproject.org/rpms/fedora-repos/c/043010400f007c9f7167c9411d4463f4c2f08b9e?branch=rawhide[commit] for reference)

[arabic]
. Generate and add a _Rawhide+1_ which is {rawhide_next} GPG key file, then add it to the spec
file:
+
[source,subs="attributes"]
----
Source57:       RPM-GPG-KEY-fedora-{rawhide_next}-primary
----

. Update the `archmap` file and define architectures for _Rawhide+1_:
+
[source,subs="attributes"]
----
fedora-{rawhide_next}-primary: x86_64 armhfp aarch64 ppc64le s390x
----

. Increment `%global rawhide_release`:
+
[source,subs="attributes"]
----
%global rawhide_release {rawhide}
----

. Bump `Version:` and reset `Release:`:
+
[source,subs="attributes"]
----
Version:        {rawhide}
Release:        0.1%{?eln:.eln%\{eln}}
----

. Add a `%changelog` entry:
+
[source,subs="attributes"]
----
%changelog
* Day Mon DD YYYY Name <Email> - {rawhide}-0.1
- Setup for rawhide being F{rawhide}
----

Changes to the *branched* branch (mostly in `fedora-repos.spec`):

(can also check this https://src.fedoraproject.org/rpms/fedora-repos/c/cd678e7b4aa59e234f32c9697198f58d9c89d862?branch=f38[commit] for reference)

[arabic]
. Copy the _Rawhide+1_ which is {rawhide_next} GPG key file from the _rawhide_ branch, then add
it to the spec file:
+
[source,subs="attributes"]
----
Source57:       RPM-GPG-KEY-fedora-{rawhide_next}-primary
----

. Copy the `archmap` file from the _rawhide_ branch.
. Update `%global rawhide_release`:
+
[source,subs="attributes"]
----
 %global rawhide_release {rawhide}
----

. Enable `updates_testing_enabled`:
+
[source,subs="attributes"]
----
%global updates_testing_enabled 1
----

. Bump `Release`:
+
[source,subs="attributes"]
----
Release:        0.3%{?eln:.eln%\{eln}} +
----
. Add a `%changelog` entry:
+
[source,subs="attributes"]
----
%changelog
*Day Mon DD YYYY Name <Email> - {branched}-0.3 +
- Update Rawhide definition, enable updates-testing for Branched +
----


[NOTE]
====
Build `fedora-release` and `fedora-repos` packages for Branched release
*before enabling the Rawhide gating*.
====

[NOTE]
====
Consider using sidetags for `fedora-release` and `fedora-repos` updates; these updates should be bundled with updates to `mock-core-configs` and `distribution-gpg-keys` for a unified push.
====

Changes to the *stable* branches (mostly in `fedora-repos.spec`):

[arabic]
. Copy the _Rawhide+1_ GPG key which is {rawhide_next} file from the _rawhide_ branch, then add
it to the spec file:
+
[source,subs="attributes"]
----
Source57:       RPM-GPG-KEY-fedora-{rawhide_next}-primary
----

. Copy the `archmap` file from the _rawhide_ branch.
. Update `%global rawhide_release`:
+
[source,subs="attributes"]
----
 %global rawhide_release {rawhide}
----

. Bump `Release:`:
+
[source,subs="attributes"]
----
Release:        0.3%{?eln:.eln%\{eln}}
----

. Add a `%changelog` entry:
+
[source,subs="attributes"]
----
%changelog
*Day Mon DD YYYY Name <Email> - {current}-0.3
- Update Rawhide definition
----

== Bodhi

=== Linking Empty Repos

We need to link empty repos so that new-updates-sync wont complain about
missing repos. The following commands should be run on
*bodhi-backend01.phx2.fedoraproject.org*


[source,bash,subs="attributes"]
----
$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f{branched}-updates
$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f{branched}-updates-testing
----

=== Creating Empty Repos

To create empty repos on the master mirror, run
https://pagure.io/releng/blob/main/f/scripts/branching/create_empty_repos.sh[create_emtpy_repos.sh]
from https://pagure.io/releng[pagure releng] repo. This should be run on
*bodhi-backend01.phx2.fedoraproject.org*

[source,bash,subs="attributes"]
----
$ sudo -u ftpsync sh scripts/branching/create_empty_repos.sh {branched}
----

[NOTE]
====
Update the link in /mnt/koji/repos/rawhide/latest as per https://pagure.io/releng/issue/12255.
====

[NOTE]
====
Please verify the repo permissions that are created under
/pub/fedora/linux/development/<fedora_release_number> and
/pub/fedora-secondary/development/<fedora_release_number>. They should
be owned by _ftpsync:ftpsync_
Check directory permissions (should be "0755") to ensure new composes synchronize correctly.
====

=== Creating rawhide release

To create a rawhide release in bodhi, you need to run:

[source,bash,subs="attributes"]
----
$ bodhi releases create \
  --name "F{rawhide}" --long-name "Fedora {rawhide}" \
  --id-prefix FEDORA --version {rawhide} --branch f{rawhide} \
  --dist-tag f{rawhide} \
  --stable-tag f{rawhide} \
  --testing-tag f{rawhide}-updates-testing \
  --candidate-tag f{rawhide}-updates-candidate \
  --pending-stable-tag f{rawhide}-updates-pending \
  --pending-testing-tag f{rawhide}-updates-testing-pending \
  --pending-signing-tag f{rawhide}-signing-pending \
  --state pending \
  --override-tag f{rawhide}-override \
  -create-automatic-updates \
  --not-composed-by-bodhi
----

To create a container release for rawhide in bodhi, you need to run:

[source,bash,subs="attributes"]
----
$ bodhi releases create \
  --name "F{rawhide}C" --long-name "Fedora {rawhide} Containers" \
  --id-prefix FEDORA-CONTAINER --version {rawhide} --branch f{rawhide} \
  --dist-tag f{rawhide}-container \
  --stable-tag f{rawhide}-container-updates \
  --testing-tag f{rawhide}-container-updates-testing \
  --candidate-tag f{rawhide}-container-updates-candidate \
  --pending-stable-tag f{rawhide}-container-updates-pending \
  --pending-testing-tag f{rawhide}-container-updates-testing-pending \
  --state pending \
  --override-tag f{rawhide}-container-override
----

To create a flatpak release for branched in bodhi, you need to run:

[source,bash,subs="attributes"]
----
$ bodhi releases create \
  --name "F{branched}F" --long-name "Fedora {branched} Flatpaks" \
  --id-prefix FEDORA-FLATPAK --version {branched} --branch f{branched} \
  --dist-tag f{branched}-flatpak \
  --stable-tag f{branched}-flatpak-updates \
  --testing-tag f{branched}-flatpak-updates-testing \
  --candidate-tag f{branched}-flatpak-updates-candidate \
  --pending-stable-tag f{branched}-flatpak-updates-pending \
  --pending-testing-tag f{branched}-flatpak-updates-testing-pending \
  --state pending \
  --override-tag f{branched}-flatpak-override
----

You need to run the `bodhi openshift` playbook, so that UI will know
about the new release. Then, you need to restart
*fm-consumer@config.service* and *bodhi-celery.service* services on
*bodhi-backend01.phx2.fedoraproject.org*:

```
$ sudo rbac-playbook openshift-apps/bodhi.yml
$ sudo systemctl restart fm-consumer@config.service bodhi-celery.service
```

[NOTE]
====
Build fedora-release, fedora-repos package for *rawhide after enabling
the rawhide gating*
====

=== Update rawhide koji repo

We need to point the _rawhide_ buildroot repo to the newly created
rawhide buildroot. This way kojira doesn't make a newrepo for _rawhide_
target as often as fxx-build (new rawhide buildroot).

Run the following commands from any of the compose boxes:

[source,bash,subs="attributes"]
----
$ cd /mnt/koji/repos/rawhide
$ rm -f latest
$ ln -s ../f{rawhide}-build/latest ./latest
----

=== Updating MirrorManager

We need to update the mirrormanager so that it will point rawhide to the
new rawhide release.

Please follow the instructions in the
https://pagure.io/fedora-infrastructure/issue/9239#comment-671446[fedora
infra ticket] to update the database of mirrormanager.

=== Enable autosigning on branched release

Once the branched compose is composed, we need to re-enable
robosignatory on branched release

=== ELN related work

Add the new rawhide key to eln pungi config. For example, look at this
https://pagure.io/pungi-fedora/c/e993441164ee83374df7f463777f2bf1d456fd6d?branch=eln[pungi
eln config commit]

Change the trigger notification for DistroBuildSync to the new Rawhide
version. For example, look at this
https://gitlab.com/redhat/centos-stream/ci-cd/distrosync/distrobuildsync-config/-/commit/1497d9aea42cf00af646b4a0f9f9ed1a7f0a477f[commit].

=== Branch new rawhide in Koschei

Branch new fedora rawhide in
https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/koschei/#_branching_a_new_fedora_release[koschei].

[NOTE]
====
Temporarily disable the rawhide cron job during branching PRs to ensure a branched compose is created. Re-enable rawhide after this.
====

[NOTE]
====
In fedora-repos package build for new branched version enable the update-testing repository immediately upon branching.
====

== Consider Before Running

NOTE: FIXME: Need some love here