Infrastructure/infra-docs-fpo
1
0
Fork 0
Code Issues 100 Pull requests 6 Projects Releases Packages Wiki Activity Actions
infra-docs-fpo/modules/release_guide/pages/layered_image_build_service.adoc
Tomas Hrcka 14c92455cb Initial import 
Add missing pages

Fix the nav for release guide

Cleanup branching sop

Add release version substitutions

Signed-off-by: Tomas Hrcka <thrcka@redhat.com>
2023-02-28 12:32:13 +01:00

263 lines
12 KiB
Text
Raw Blame History

== Fedora Layered Image Build System
The Fedora Layered Image Build System aims to provide an new type of
official binary artifact produced by Fedora. Currently, we produce two
main types of artifacts: RPMs, and images. The RPMs are created in
https://fedoraproject.org/wiki/Koji[Koji] from specfiles in dist-git.
The images come in different formats, but have in common creation in
Koji from kickstart files — this includes the official Fedora Docker
Base Image. This change introduces a new type of image, a
https://github.com/docker/docker/[Docker] Layered Image, which is
created from a Dockerfile and builds on top of that base image.
=== Layered Image Build Service Architecture
....
+------------------------------+
| Future Items to Integrate |
+------------------------------+
| +--------------------------+ |
| |PDC Integration | |
| +--------------------------+ |
| |New Hotness | |
| +--------------------------+ |
| |Other??? | |
| +--------------------------+ |
| Magical Future |
| |
+------------------------------+
+------------------+
|Fedora |
|Users/Contributors|
+--------+---------+
^
| +----------------+
| | Fedora |
+-------+-----------+ | Layered Image |
| Fedora Production | | Maintainers |
| Docker Registry | | |
+-------------------+ +-------+--------+
^ |
| V
| +-------------------+--+
+-----------+------------+ | |
| RelEng Automation | | +------------------+ |
| (Release) | | |Dockerfile | |
+-----------+------------+ | +------------------+ |
^ | |App "init" scripts| |
| | +------------------+ |
| | |Docs | |
+---------+------------+ | +------------------+ |
| Taskotron | | DistGit |
| Automated QA | | |
+---------+-------+----+ +-----------+----------+
^ ^ |
| | |
| | |
| | |
| +-------------------+ |
| | |
[docker images] | |
| | |
| [fedmsg] |
+---------------+-----------+ | |
| | | +---------------+
| +----------------------+ | | |
| |Intermediate Docker | | +------+-------------------+ |
| |Images for OSBS Builds| | | | |
| +----------------------+ | | +----------------------+ | |
| |Layered Images | | | |containerbuild plugin | | |
| +----------------------+ | | +----------------------+ | |
| |docker|distribution | | | |rpmbuilds | | |
| +----------------------+ | | +----------------------+ | |
| Registry | | koji | |
| | | | |
+-----+----+----------------+ +-----+---+----+-----------+ |
| ^ | ^ ^ |
| | | | | |
| | +---------------------------+ | | |
| | | [osbs-client api] | | |
| | | +---------------------------+ | |
| | | | | |
| | | | | |
V | V | | |
+--+----+---+---+---+ | V
| | +-------------------+---+
| +--------------+ | |fedpkg container-build +
| |OpenShift v3 | | +-----------------------+
| +--------------+ |
| |Atomic Reactor| |
| +--------------+ |
| OSBS |
| |
+-------------------+
[--------------------- Robosignatory ------------------------------------]
[ Every time an image is tagged or changes names, Robosignatory signs it ]
[ ]
[ NOTE: It's injection points in the diagram are ommitted for brevity ]
[------------------------------------------------------------------------]
....
=== Layered Image Build System Components
The main aspects of the Layered Image Build System are:
* Koji
** koji-containerbuild plugin
* OpenShift Origin v3
* Atomic Reactor
* osbs-client tools
* A docker registry
** docker-distribution
* Taskotron
* fedmsg
* RelEng Automation
The build system is setup such that Fedora Layered Image maintainers
will submit a build to Koji via the `fedpkg container-build` command a
`containers` namespace within
https://fedoraproject.org/wiki/Infrastructure/VersionControl/dist-git[DistGit].
This will trigger the build to be scheduled in
https://www.openshift.org/[OpenShift] via
https://github.com/projectatomic/osbs-client[osbs-client] tooling, this
will create a custom
https://docs.openshift.org/latest/dev_guide/builds.html[OpenShift Build]
which will use the pre-made buildroot Docker image that we have created.
The https://github.com/projectatomic/atomic-reactor[Atomic Reactor]
(`atomic-reactor`) utility will run within the buildroot and prep the
build container where the actual build action will execute, it will also
maintain uploading the
https://fedoraproject.org/wiki/Koji/ContentGenerators[Content Generator]
metadata back to https://fedoraproject.org/wiki/Koji[Koji] and upload
the built image to the candidate docker registry. This will run on a
host with iptables rules restricting access to the docker bridge, this
is how we will further limit the access of the buildroot to the outside
world verifying that all sources of information come from Fedora.
Completed layered image builds are hosted in a candidate docker registry
which is then used to pull the image and perform tests with
https://taskotron.fedoraproject.org/[Taskotron]. The taskotron tests are
triggered by a http://www.fedmsg.com/en/latest/[fedmsg] message that is
emitted from https://fedoraproject.org/wiki/Koji[Koji] once the build is
complete. Once the test is complete, taskotron will send fedmsg which is
then caught by the [.title-ref]#RelEng Automation# Engine that will run
the Automatic Release tasks in order to push the layered image into a
stable docker registry in the production space for end users to consume.
Note that every time the layered image tagged to a new registry,
ultimately changing it's name,
https://pagure.io/robosignatory[Robosignatory] will automatically sign
the new image. This will also occur as part of the Automated Release
process as the image will be tagged from the candidate docker registry
into the production docker registry in order to "graduate" the image to
stable.
==== Koji
https://fedoraproject.org/wiki/Koji[Koji] is the Fedora Build System.
==== koji-containerbuild plugin
The
https://github.com/release-engineering/koji-containerbuild[koji-containerbuild]
plugin integrates Koji and OSBS so that builds can be scheduled by koji
and integrated into the build system with imports of metadata, logs,
build data, and build artifacts.
==== OpenShift Origin v3
https://www.openshift.org/[OpenShift Origin v3] is an open source
Container Platform, built on top of http://kubernetes.io/[kubernetes]
and https://github.com/docker/docker/[Docker]. This provides many
aspects of the system needed including a build pipeline for Docker
images with custom build profiles, image streams, and triggers based on
events within the system.
==== Atomic Reactor
https://github.com/projectatomic/atomic-reactor[Atomic Reactor] is an
utility which allows for the building of containers from within other
containers providing hooks to trigger automation of builds as well as
plugins providing automatic integration many other utilities and
services.
==== osbs-client tools
https://github.com/projectatomic/osbs-client[osbs-client] tools allow
for users to access the build functionality of
https://www.openshift.org/[OpenShift Origin v3] using a simple set of
command line utilities.
==== docker-registry
A https://docs.docker.com/registry/[docker-registry] is a stateless,
highly scalable server side application that stores and lets you
distribute Docker images.
There are many different implementations of docker-registries, two main
ones are supported by the Fedora Layered Image Build System.
===== docker-distribution
The https://github.com/docker/distribution/[docker-distribution]
registry is considered the Docker upstream "v2 registry" such that it
was used by upstream to implement the new version 2 specification of the
docker-registry.
===== Fedora Production Registry
Implementation details of this are still unknown at the time of this
writing and will be updated at a later date. For the current status and
implementation notes please visit the
https://fedoraproject.org/wiki/Changes/FedoraDockerRegistry[FedoraDockerRegistry]
page.
==== Taskotron
https://taskotron.fedoraproject.org/[Taskotron] is an automated task
execution framework, written on top of http://buildbot.net/[buildbot]
that currently executes many Fedora automated QA tasks and we will be
adding the Layered Image automated QA tasks. The tests themselves will
be held in DistGit and maintained by the Layered Image maintainers.
==== RelEng Automation
https://pagure.io/releng-automation[RelEng Automation] is an ongoing
effort to automate as much of the RelEng process as possible by using
http://ansible.com/[Ansible] and being driven by
http://www.fedmsg.com/en/latest/[fedmsg] via
https://github.com/maxamillion/loopabull[Loopabull] to execute Ansible
Playbooks based on fedmsg events.
==== Robosignatory
https://pagure.io/robosignatory[Robosignatory] is a fedmsg consumer that
automatically signs artifacts and will be used to automatically sign
docker layered images for verification by client tools as well as end
users.
==== Future Integrations
In the future various other components of the
https://fedoraproject.org/wiki/Infrastructure[Fedora Infrastructure]
will likely be incorporated.
===== PDC
https://pdc.fedoraproject.org/[PDC] is Fedora's implementation of
https://github.com/product-definition-center/product-definition-center[Product
Definition Center] which allows Fedora to maintain a database of each
Compose and all of it's contents in a way that can be queried and used
to make decisions in a programatic way.
===== The New Hotness
https://github.com/fedora-infra/the-new-hotness[The New Hotness] is a
http://www.fedmsg.com/en/latest/[fedmsg] consumer that listens to
release-monitoring.org and files bugzilla bugs in response (to notify
packagers that they can update their packages).
Reference in a new issue View git blame Copy permalink