include::_partials/attributes.adoc[]

= EPEL Minor Version Mass Branching

== Description

Originally, the EPEL repository had packages that built against the latest RHEL minor, but since EPEL{epel_major}, the EPEL repository now builds against the latest Centos Stream release. To keep consistency for RHEL releases, EPEL repository goes through a mass branching process to generate a new minor release branch. This process let's the developers continue to maintain their packages against Centos Stream without risking to break packages that already work on the current active RHEL minor target.

== Send announcement

One day before the mass branching, we send out announcement because during mass branching,
new koji builds for EPEL are disabled.

=== Remove Targets

In Fedora it is recommended to https://docs.fedoraproject.org/en-US/infra/release_guide/sop_mass_branching/#_disable_rawhide_builds_in_koji[stop connections to koji through firewall]. 
But it was https://pagure.io/infra-docs-fpo/pull-request/357[decided not to require it] for this process.

In the case of EPEL, just removing the targets is enough to disable it's usage for the time being while the branching process it's being done.

[source,bash,subs="attributes"]
----
$ koji remove-target epel{epel_major}-candidate
$ koji remove-target epel{epel_major}
----

=== Cancel all running build for EPEL{epel_major}

List all running tasks and select only those relevant for current branching.

[source,bash,subs="attributes"]
----
$ koji list-builds --state=0 --type=rpm | grep el{epel_major}_{epel_minor} | awk '{print $1}'
----

Cancel each of those tasks

```
$ koji cancel <build>
```

== dist-git

Now we need to update dist-git in two steps:

* Create the new branch in git

For both of these actions you will need the list of active repos in EPEL{epel_major}.

=== Get current EPEL active packages

To get the list of all the current active packages, you must run the `scripts/branching-epe/get_all_active_packages_branching.sh` script.

[source,bash,subs="attributes"]
----
sh scripts/branching-epel/get_all_active_packages_branching.sh {epel_major}
----

The results will be saved to the components_epel{epel_new}.txt

=== Create the git branches

On `pkgs01.stg` (for testing) or `pkgs02` (for production), run:

[source,bash,subs="attributes"]
----
$ sudo -u pagure python /usr/local/bin/mass-branching-git.py --branch-from  epel{epel_major} epel{epel_current} components_epel{epel_major}.txt 
----

The first argument is the new epel version and the
second one is the path to the file generated in the previous step.

== Ansible

Apps in https://pagure.io/fedora-infra/ansible[ansible] need to be updated
to be aware of the new branch.

The following sections describe the things that need to be updated, but it's recommended to use https://pagure.io/fedora-infra/ansible/pull-request/2473#request_diff[this PR]
for reference.

=== Bodhi

Bodhi needs to be updated to add new release. This needs to be done in
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2[bodhi2 role] in
infra ansible repo. This change includes, updating `koji-sync-listener.py`,
`new-updates-sync`, pungi configs for rpm updates, bodhi templates.

* `roles/bodhi2/backend/files/new-updates-sync`
* `roles/bodhi2/backend/tasks/main.yml`
* `roles/bodhi2/backend/templates/koji_sync_listener.toml`

[NOTE]
====
In this step in Fedora, `roles/bodhi2/backend/templates/pungi.rpm.conf.j2`
would also need to be modified. But here it's not required since the signing key doesn't
change between minor versions
====

=== Toddlers

==== Add new SLA to the toddlers App

Make sure that the SLAs are added. Check https://pagure.io/fedora-infra/ansible/pull-request/2190#request_diff[this PR] 
for reference on what needs to be available.

=== Koji hub

Update the koji hub config to allow side tags for new koji epel{epel_major} minor (currently epel{epel_new}) tag.

* `roles/koji_hub/templates/hub.conf.j2`

=== Robosignatory

All minor branches use the same key, so it's not required to generate a new one, but it's important to update the configuration files so that the propper keys are used

* `roles/robosignatory/templates/robosignatory.toml.j2`

=== Push the changes

When done editing the files, commit, push and apply them via the corresponding
ansible playbooks:

[source,bash,subs="attributes"]
----
$ sudo rbac-playbook groups/batcave.yml
$ sudo rbac-playbook groups/koji-hub.yml
$ sudo rbac-playbook groups/bodhi-backend.yml
$ sudo rbac-playbook manual/autosign.yml
----

== Koji

The koji build system needs to have some tag/target work done to handle
builds from the new branch and to update where builds from epel{epel_major} go.

Run
https://pagure.io/releng/blob/main/f/scripts/branching-epel/make-koji-release-tags[make-koji-release-tags]
script in https://pagure.io/releng[pagure releng] repo

[NOTE]
====
Make sure that things tagged on epel{epel_current}-testing are tagged correctly.
https://bodhi.fedoraproject.org/updates/?search=&status=pending&status=testing&releases=EPEL-{epel_current}
====

== Bodhi

=== Linking Empty Repos

We need to link empty repos so that new-updates-sync wont complain about
missing repos. The following commands should be run on
*bodhi-backend01.phx2.fedoraproject.org*

[source,bash,subs="attributes"]
----
$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel{epel_new}
$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel{epel_new}-testing
----

=== Creating Empty Repos

To create empty repos on the master mirror, run
https://pagure.io/releng/blob/main/f/scripts/branching-epel/create_empty_repos.sh[create_empty_repos.sh]
from https://pagure.io/releng[pagure releng] repo. This should be run on
*bodhi-backend01.phx2.fedoraproject.org*

[source,bash,subs="attributes"]
----
$ sudo -u ftpsync sh scripts/branching-epel/create_empty_repos.sh {epel_current}
----

=== Creating EPEL{epel_new} release

To create a new minor release in bodhi, you need to run:

[source,bash,subs="attributes"]
----
$ bodhi releases create \
    --name "EPEL-{epel_new}" \
    --long-name "Fedora EPEL {epel_new}" \
    --version {epel_major} \
    --branch epel{epel_major} \
    --id-prefix FEDORA-EPEL \
    --dist-tag epel{epel_new} \
    --stable-tag epel{epel_new} \
    --testing-tag epel{epel_new}-testing \
    --candidate-tag epel{epel_new}-testing-candidate \
    --pending-signing-tag epel{epel_new}-signing-pending \
    --pending-testing-tag epel{epel_new}-testing-pending \
    --pending-stable-tag epel{epel_new}-pending \
    --override-tag epel{epel_new}-override \
    --mail-template fedora_epel_legacy_errata_template \
    --state current \
    --composed-by-bodhi \
    --no-create-automatic-updates
----

=== Recreate targets

Once mass-branching is complete and the new release exists in bodhi,
re-create these targets with the following commands:

[source,bash,subs="attributes"]
----
$ koji add-target epel${release_major}           epel${release}-build epel${release}-testing-candidate
$ koji add-target epel${release_major}-candidate epel${release}-build epel${release}-testing-candidate
----

=== Updating MirrorManager

MirrorManager will load the changes automatically, you can check that everything is in order throught the following command:

[source,bash,subs="attributes"]
----
$ curl 'https://mirrors.fedoraproject.org/metalink?arch=x86_64&repo=epel-{epel_current}'
----

=== Branch new EPEL{epel_major} minor in Koschei

Branch new fedora EPEL{epel_major} minor in
https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/koschei/#_branching_a_new_fedora_release[koschei].
