Infrastructure/arc
3
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

fix parsing errors and sphinx warnings

Browse source 
Signed-off-by: Ryan Lerch <rlerch@redhat.com>
This commit is contained in:
Ryan Lercho 2023-11-16 08:02:56 +10:00 • committed by zlopez
parent 8fb9b2fdf0
commit ba720c3d77
98 changed files with 4799 additions and 4788 deletions
Download patch file Download diff file Expand all files Collapse all files

72
docs/registry_to_quay/fedora_images.rst
View file

@ -3,22 +3,23 @@
Fedora Base/Minimal/CoreOS container images
===========================================
This document contains information about current workflow used to build and publish these container
images. It also describes changes needed to make it work with `quay.io <https://quay.io>`_.
This document contains information about current workflow used to build and publish
these container images. It also describes changes needed to make it work with `quay.io
<https://quay.io>`_.
Current workflow
----------------
The base/minimal/CoreOS images are currently built using OSBS. The OSBS submits builds to
`candidate-registry.fedoraproject.org <https://candidate-registry.fedoraproject.org/>`_ using
dockercfg and user can create an update in Bodhi. After testing period is finished or the update
gets enough karma it is moved by bodhi to
`registry.fedoraproject_org <https://registry.fedoraproject.org/>`_.
The base/minimal/CoreOS images are currently built using OSBS. The OSBS submits builds
to `candidate-registry.fedoraproject.org
<https://candidate-registry.fedoraproject.org/>`_ using dockercfg and user can create an
update in Bodhi. After testing period is finished or the update gets enough karma it is
moved by bodhi to `registry.fedoraproject_org <https://registry.fedoraproject.org/>`_.
This will change in F39 where the images will be built by
`koji-flatpak <https://pagure.io/koji-flatpak>`_ plugin. Releng script will then submit the builds
to both `registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ and
`quay.io <https://quay.io>`_.
This will change in F39 where the images will be built by `koji-flatpak
<https://pagure.io/koji-flatpak>`_ plugin. Releng script will then submit the builds to
both `registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ and `quay.io
<https://quay.io>`_.
.. image:: ../_static/container_workflow_registry.drawio.png
:target: ../_images/container_workflow_registry.drawio.png
@ -31,37 +32,40 @@ work with current setup. It will go through pipeline step by step and describe w
needs to be changed in case of base/minimal/CoreOS images workflow.
OSBS
^^^^
~~~~
OSBS is using Openshift local container registry, which is configured by
`dockercfg <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
file and pointing to candidate-registry. There is dockercfg for
`production <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_ and `staging <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-staging-secret.j2>`_.
This dockercfg is used as a secret in OSBS and it's deployed in
`setup-worker-namespace <https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/osbs/setup-worker-namespace.yml>`_ playbook.
To make this work with `quay.io <https://quay.io>`_ we need to replace the dockercfg with new
one pointing to `quay.io <https://quay.io>`_.
OSBS is using Openshift local container registry, which is configured by `dockercfg
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
file and pointing to candidate-registry. There is dockercfg for `production
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
and `staging
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-staging-secret.j2>`_.
This dockercfg is used as a secret in OSBS and it's deployed in `setup-worker-namespace
<https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/osbs/setup-worker-namespace.yml>`_
playbook. To make this work with `quay.io <https://quay.io>`_ we need to replace the
dockercfg with new one pointing to `quay.io <https://quay.io>`_.
.. note::
OSBS could have trouble reaching to `quay.io <https://quay.io>`_, this needs to be solved
for this to work.
OSBS could have trouble reaching to `quay.io <https://quay.io>`_, this needs to be
solved for this to work.
Releng script
^^^^^^^^^^^^^
~~~~~~~~~~~~~
In the new workflow introduced in Fedora 39+ is a Release Engineering script that pushes the image
to registry. This script could be found in
`releng repository <https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh>`_
and it's already pushing images to `quay.io <https://quay.io>`_.
In the new workflow introduced in Fedora 39+ is a Release Engineering script that pushes
the image to registry. This script could be found in `releng repository
<https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh>`_ and
it's already pushing images to `quay.io <https://quay.io>`_.
Bodhi
^^^^^
~~~~~
Bodhi is pushing the container builds from
`candidate-registry.fedoraproject.org <https://candidate-registry.fedoraproject.org/>`_ to
`registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ and this is configured in
`production.ini template <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2>`_
Bodhi is pushing the container builds from `candidate-registry.fedoraproject.org
<https://candidate-registry.fedoraproject.org/>`_ to `registry.fedoraproject_org
<https://registry.fedoraproject.org/>`_ and this is configured in `production.ini
template
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2>`_
in ansible.
To make it work with quay.io we just need to change `container.destination_registry` and
@ -69,5 +73,5 @@ To make it work with quay.io we just need to change `container.destination_regis
.. note::
Bodhi wouldn't be part of the base/minimal/CoreOS images workflow in Fedora 39 and later.\
So no need to change anything here.
Bodhi wouldn't be part of the base/minimal/CoreOS images workflow in Fedora 39 and
later.So no need to change anything here.

90
docs/registry_to_quay/flatpaks.rst
View file

@ -3,20 +3,21 @@
Fedora Flatpaks
===============
This document contains information about current workflow used to build and publish Fedora flatpaks.
It also describes changes needed to make it work with `quay.io <https://quay.io>`_.
This document contains information about current workflow used to build and publish
Fedora flatpaks. It also describes changes needed to make it work with `quay.io
<https://quay.io>`_.
Flatpaks are the biggest tenant occupying `registry.fedoraproject.org <https://registry.fedoraproject.org/>`_
right now.
Flatpaks are the biggest tenant occupying `registry.fedoraproject.org
<https://registry.fedoraproject.org/>`_ right now.
Current workflow
----------------
The flatpaks are build in koji using OSBS. The OSBS then submits the build to
`candidate-registry.fedoraproject.org <https://candidate-registry.fedoraproject.org/>`_
and packager can create an update in Bodhi. After testing period is finished or the update
gets enough karma it is moved by bodhi to
`registry.fedoraproject_org <https://registry.fedoraproject.org/>`_.
and packager can create an update in Bodhi. After testing period is finished or the
update gets enough karma it is moved by bodhi to `registry.fedoraproject_org
<https://registry.fedoraproject.org/>`_.
.. image:: ../_static/flatpaks_workflow_registry.drawio.png
:target: ../_images/flatpaks_workflow_registry.drawio.png
@ -24,61 +25,68 @@ gets enough karma it is moved by bodhi to
Changes for quay.io
-------------------
This section will describe changes that are needed to make quay.io work with current setup.
It will go through pipeline step by step and describe what needs to be changed in case of
flatpaks workflow.
This section will describe changes that are needed to make quay.io work with current
setup. It will go through pipeline step by step and describe what needs to be changed in
case of flatpaks workflow.
OSBS
^^^^
OSBS is using Openshift local container registry, which is configured by
`dockercfg <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
file and pointing to candidate-registry. There is dockercfg for
`production <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_ and `staging <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-staging-secret.j2>`_.
This dockercfg is used as a secret in OSBS and it's deployed in
`setup-worker-namespace <https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/osbs/setup-worker-namespace.yml>`_ playbook.
To make this work with quay.io we need to replace the dockercfg with new one pointing to quay.io.
~~~~
OSBS is using Openshift local container registry, which is configured by `dockercfg
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
file and pointing to candidate-registry. There is dockercfg for `production
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
and `staging
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-staging-secret.j2>`_.
This dockercfg is used as a secret in OSBS and it's deployed in `setup-worker-namespace
<https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/osbs/setup-worker-namespace.yml>`_
playbook. To make this work with quay.io we need to replace the dockercfg with new one
pointing to quay.io.
.. note::
OSBS could have trouble reaching to quay.io, this needs to be solved for this to work.
OSBS could have trouble reaching to quay.io, this needs to be solved for this to
work.
Koji
^^^^
There is currently a initiative that is trying to remove OSBS from flatpak build pipeline. At
the time of writing this document this is only enabled for F39. But as the F38 flaptak runtime
will be considered EOL the `koji-flatpak <https://pagure.io/koji-flatpak>`_ module will be the only
way.
~~~~
The koji-flatpak module is much direct, there is just one
`configuration file <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/koji_builder/templates/flatpak.conf>`_
used to specify where the koji builders should push the build container. The `candidate-registry`
variable is set in `tasks/main`.
There is currently a initiative that is trying to remove OSBS from flatpak build
pipeline. At the time of writing this document this is only enabled for F39. But as the
F38 flaptak runtime will be considered EOL the `koji-flatpak
<https://pagure.io/koji-flatpak>`_ module will be the only way.
Currently the builders don't have access to quay.io. The firewall settings needs to be adjusted
so they could push containers.
The koji-flatpak module is much direct, there is just one `configuration file
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/koji_builder/templates/flatpak.conf>`_
used to specify where the koji builders should push the build container. The
`candidate-registry` variable is set in `tasks/main`.
Currently the builders don't have access to quay.io. The firewall settings needs to be
adjusted so they could push containers.
Bodhi
^^^^^
~~~~~
Bodhi is pushing the container builds from candidate-registry.fedoraproject.org to
registry.fedoraproject.org and this is configured in
`production.ini template <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2>`_
registry.fedoraproject.org and this is configured in `production.ini template
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2>`_
in ansible.
To make it work with quay.io we just need to change `container.destination_registry` and
`container.source_registry` variables.
Proxies
^^^^^^^
~~~~~~~
We need to redirect all the pull requests for registry.fedoraproject.org to quay.io. This could be
done by `mod_proxy / mod_rewrite` and redirect all requests to
We need to redirect all the pull requests for registry.fedoraproject.org to quay.io.
This could be done by `mod_proxy / mod_rewrite` and redirect all requests to
https://registry.fedoraproject.org/v2/REPO/ to https://quay/v2/fedora/REPO/
flatpak-indexer
^^^^^^^^^^^^^^^
~~~~~~~~~~~~~~~
`flatpak-indexer <https://github.com/owtaylor/flatpak-indexer>`_ is used to read metadata from containers
and generate index in format flatpak client understands. It also creates deltas for updates. It's deployed in OpenShift
and the changes have to be done in
`configuration <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/flatpak-indexer/templates/configmap.yml>`_.
`flatpak-indexer <https://github.com/owtaylor/flatpak-indexer>`_ is used to read
metadata from containers and generate index in format flatpak client understands. It
also creates deltas for updates. It's deployed in OpenShift and the changes have to be
done in `configuration
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/flatpak-indexer/templates/configmap.yml>`_.

68
docs/registry_to_quay/index.rst
View file

@ -4,48 +4,56 @@ Migrate registry.fedoraproject.org to quay.io
Purpose
-------
This investigation is looking into service `quay.io <https://quay.io/>`_ and how can we utilize
it for the projects hosted on `registry.fedoraproject.org <https://registry.fedoraproject.org/>`_. It should also resolve if this is work should be initiative.
This investigation is looking into service `quay.io <https://quay.io/>`_ and how can we
utilize it for the projects hosted on `registry.fedoraproject.org
<https://registry.fedoraproject.org/>`_. It should also resolve if this is work should
be initiative.
Resources
---------
* Fedora Infrastructure ticket: https://pagure.io/fedora-infrastructure/issue/10386
- Fedora Infrastructure ticket: https://pagure.io/fedora-infrastructure/issue/10386
Requirements
------------
* Multi Arch containers (Already supported on quay.io)
* OCI images (Already supported on quay.io)
* Web interface (Quay.io already has web interface)
* Integrate Quay.io in current workflow
* Must not affect users of images
* `candidate-registry <https://candidate-registry.fedoraproject.org/>`_ must be moved as well
- Multi Arch containers (Already supported on quay.io)
- OCI images (Already supported on quay.io)
- Web interface (Quay.io already has web interface)
- Integrate Quay.io in current workflow
- Must not affect users of images
- `candidate-registry <https://candidate-registry.fedoraproject.org/>`_ must be moved as
well
Nice to have
------------
* Staging namespace on quay.io
- Staging namespace on quay.io
Risks
-----
* There is a `API endpoint throttling <https://docs.quay.io/issues/429.html>`_ per second on IP address on quay.io. This could cause issues in the future, but current load should be easily handled.
- There is a `API endpoint throttling <https://docs.quay.io/issues/429.html>`_ per
second on IP address on quay.io. This could cause issues in the future, but current
load should be easily handled.
Statistics
----------
These statistical data were retrieved from `oci-registry.iad2.fedoraproject.org` machine logs.
The logs on the machine are kept only for few days. These data were obtained from 16th September 2023 to 21st September 2023 (it corresponds with release of Fedora 39 Beta).
These statistical data were retrieved from `oci-registry.iad2.fedoraproject.org` machine
logs. The logs on the machine are kept only for few days. These data were obtained from
16th September 2023 to 21st September 2023 (it corresponds with release of Fedora 39
Beta).
* Number of downloads per day: 800 - 1 000
* Number of requests per day: 350 000 - 400 000
- Number of downloads per day: 800 - 1 000
- Number of requests per day: 350 000 - 400 000
Investigation
-------------
The investigation is separated to multiple parts based on the current artifacts hosted on registry:
Fedora flatpaks, Fedora container images (base, minimal, coreos) and toolbox images.
The investigation is separated to multiple parts based on the current artifacts hosted
on registry: Fedora flatpaks, Fedora container images (base, minimal, coreos) and
toolbox images.
.. toctree::
:maxdepth: 1
@ -54,30 +62,32 @@ Fedora flatpaks, Fedora container images (base, minimal, coreos) and toolbox ima
toolbox
fedora_images
Conclusions
-----------
Migrating `registry.fedoraproject.org <https://registry.fedoraproject.org/>`_ to `quay.io <https://quay.io/>`_
doesn't seem to be initiative worthy. Especially with changes happening in Fedora 39.
There needs to be changes done to all three build pipelines we investigated, but those aren't complex.
Migrating `registry.fedoraproject.org <https://registry.fedoraproject.org/>`_ to
`quay.io <https://quay.io/>`_ doesn't seem to be initiative worthy. Especially with
changes happening in Fedora 39. There needs to be changes done to all three build
pipelines we investigated, but those aren't complex.
Recommendations
---------------
It will be best to first set everything on staging and then continue with production.
Only redirect the Fedora 39+ workflow and continue with step 4 after Fedora 38 will
be EOL.
Only redirect the Fedora 39+ workflow and continue with step 4 after Fedora 38 will be
EOL.
Proposed Roadmap
----------------
* Step 1 - Create namespaces corresponding to candidate-registry and registry on `quay.io <https://quay.io/>`_ (Optional: Create staging namespaces as well)
* Step 2 - Modify configurations and scripts (see corresponding investigation document for what needs
to be changed)
* Step 3 - Redirect proxies to `quay.io <https://quay.io/>`_
* Step 4 - Decommision `candidate-registry <https://candidate-registry.fedoraproject.org/>`_ and `candidate-registry <https://candidate-registry.fedoraproject.org/>`_
- Step 1 - Create namespaces corresponding to candidate-registry and registry on
`quay.io <https://quay.io/>`_ (Optional: Create staging namespaces as well)
- Step 2 - Modify configurations and scripts (see corresponding investigation document
for what needs to be changed)
- Step 3 - Redirect proxies to `quay.io <https://quay.io/>`_
- Step 4 - Decommision `candidate-registry
<https://candidate-registry.fedoraproject.org/>`_ and `candidate-registry
<https://candidate-registry.fedoraproject.org/>`_
Estimate of work
----------------

76
docs/registry_to_quay/toolbox.rst
View file

@ -3,22 +3,23 @@
Fedora Toolbx images
====================
This document contains information about current workflow used to build and publish Toolbox images.
It also describes changes needed to make it work with `quay.io <https://quay.io>`_.
This document contains information about current workflow used to build and publish
Toolbox images. It also describes changes needed to make it work with `quay.io
<https://quay.io>`_.
Current workflow
----------------
The toolbx images are currently built using OSBS. The OSBS submits builds to
`candidate-registry.fedoraproject.org <https://candidate-registry.fedoraproject.org/>`_ using
dockercfg and user can create an update in Bodhi. After testing period is finished or the update
gets enough karma it is moved by bodhi to
`registry.fedoraproject_org <https://registry.fedoraproject.org/>`_.
`candidate-registry.fedoraproject.org <https://candidate-registry.fedoraproject.org/>`_
using dockercfg and user can create an update in Bodhi. After testing period is finished
or the update gets enough karma it is moved by bodhi to `registry.fedoraproject_org
<https://registry.fedoraproject.org/>`_.
This will change in F39 where the images will be built by
`koji-flatpak <https://pagure.io/koji-flatpak>`_ plugin. Releng script will then submit the builds
to both `registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ and
`quay.io <https://quay.io>`_.
This will change in F39 where the images will be built by `koji-flatpak
<https://pagure.io/koji-flatpak>`_ plugin. Releng script will then submit the builds to
both `registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ and `quay.io
<https://quay.io>`_.
.. image:: ../_static/toolbox_workflow_registry.drawio.png
:target: ../_images/toolbox_workflow_registry.drawio.png
@ -31,36 +32,40 @@ work with current setup. It will go through pipeline step by step and describe w
needs to be changed in case of toolbx workflow.
OSBS
^^^^
~~~~
OSBS is using Openshift local container registry, which is configured by
`dockercfg <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
file and pointing to candidate-registry. There is dockercfg for
`production <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_ and `staging <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-staging-secret.j2>`_.
This dockercfg is used as a secret in OSBS and it's deployed in
`setup-worker-namespace <https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/osbs/setup-worker-namespace.yml>`_ playbook.
To make this work with `quay.io <https://quay.io>`_ we need to replace the dockercfg with new
one pointing to `quay.io <https://quay.io>`_.
OSBS is using Openshift local container registry, which is configured by `dockercfg
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
file and pointing to candidate-registry. There is dockercfg for `production
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
and `staging
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-staging-secret.j2>`_.
This dockercfg is used as a secret in OSBS and it's deployed in `setup-worker-namespace
<https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/osbs/setup-worker-namespace.yml>`_
playbook. To make this work with `quay.io <https://quay.io>`_ we need to replace the
dockercfg with new one pointing to `quay.io <https://quay.io>`_.
.. note::
OSBS could have trouble reaching to `quay.io <https://quay.io>`_, this needs to be solved for this to work.
OSBS could have trouble reaching to `quay.io <https://quay.io>`_, this needs to be
solved for this to work.
Releng script
^^^^^^^^^^^^^
~~~~~~~~~~~~~
In the new workflow introduced in Fedora 39+ is a Release Engineering script that pushes the image
to registry. This script could be found in
`releng repository <https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh>`_
and it's already pushing images to `quay.io <https://quay.io>`_.
In the new workflow introduced in Fedora 39+ is a Release Engineering script that pushes
the image to registry. This script could be found in `releng repository
<https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh>`_ and
it's already pushing images to `quay.io <https://quay.io>`_.
Bodhi
^^^^^
~~~~~
Bodhi is pushing the container builds from
`candidate-registry.fedoraproject.org <https://candidate-registry.fedoraproject.org/>`_ to
`registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ and this is configured in
`production.ini template <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2>`_
Bodhi is pushing the container builds from `candidate-registry.fedoraproject.org
<https://candidate-registry.fedoraproject.org/>`_ to `registry.fedoraproject_org
<https://registry.fedoraproject.org/>`_ and this is configured in `production.ini
template
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2>`_
in ansible.
To make it work with quay.io we just need to change `container.destination_registry` and
@ -68,11 +73,12 @@ To make it work with quay.io we just need to change `container.destination_regis
.. note::
Bodhi wouldn't be part of the toolbx workflow in Fedora 39 and later. So no need to change
anything here.
Bodhi wouldn't be part of the toolbx workflow in Fedora 39 and later. So no need to
change anything here.
Toolbx
^^^^^^
~~~~~~
Toolbx has the `registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ set in code.
This needs to be changed before we completely switch to `quay.io <https://quay.io>`_.
Toolbx has the `registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ set
in code. This needs to be changed before we completely switch to `quay.io
<https://quay.io>`_.