== Fedora Layered Image Build System The Fedora Layered Image Build System aims to provide an new type of official binary artifact produced by Fedora. Currently, we produce two main types of artifacts: RPMs, and images. The RPMs are created in https://fedoraproject.org/wiki/Koji[Koji] from specfiles in dist-git. The images come in different formats, but have in common creation in Koji from kickstart files — this includes the official Fedora Docker Base Image. This change introduces a new type of image, a https://github.com/docker/docker/[Docker] Layered Image, which is created from a Dockerfile and builds on top of that base image. === Layered Image Build Service Architecture .... +------------------------------+ | Future Items to Integrate | +------------------------------+ | +--------------------------+ | | |PDC Integration | | | +--------------------------+ | | |New Hotness | | | +--------------------------+ | | |Other??? | | | +--------------------------+ | | Magical Future | | | +------------------------------+ +------------------+ |Fedora | |Users/Contributors| +--------+---------+ ^ | +----------------+ | | Fedora | +-------+-----------+ | Layered Image | | Fedora Production | | Maintainers | | Docker Registry | | | +-------------------+ +-------+--------+ ^ | | V | +-------------------+--+ +-----------+------------+ | | | RelEng Automation | | +------------------+ | | (Release) | | |Dockerfile | | +-----------+------------+ | +------------------+ | ^ | |App "init" scripts| | | | +------------------+ | | | |Docs | | +---------+------------+ | +------------------+ | | Taskotron | | DistGit | | Automated QA | | | +---------+-------+----+ +-----------+----------+ ^ ^ | | | | | | | | | | | +-------------------+ | | | | [docker images] | | | | | | [fedmsg] | +---------------+-----------+ | | | | | +---------------+ | +----------------------+ | | | | |Intermediate Docker | | +------+-------------------+ | | |Images for OSBS Builds| | | | | | +----------------------+ | | +----------------------+ | | | |Layered Images | | | |containerbuild plugin | | | | +----------------------+ | | +----------------------+ | | | |docker|distribution | | | |rpmbuilds | | | | +----------------------+ | | +----------------------+ | | | Registry | | koji | | | | | | | +-----+----+----------------+ +-----+---+----+-----------+ | | ^ | ^ ^ | | | | | | | | | +---------------------------+ | | | | | | [osbs-client api] | | | | | | +---------------------------+ | | | | | | | | | | | | | | V | V | | | +--+----+---+---+---+ | V | | +-------------------+---+ | +--------------+ | |fedpkg container-build + | |OpenShift v3 | | +-----------------------+ | +--------------+ | | |Atomic Reactor| | | +--------------+ | | OSBS | | | +-------------------+ [--------------------- Robosignatory ------------------------------------] [ Every time an image is tagged or changes names, Robosignatory signs it ] [ ] [ NOTE: It's injection points in the diagram are ommitted for brevity ] [------------------------------------------------------------------------] .... === Layered Image Build System Components The main aspects of the Layered Image Build System are: * Koji ** koji-containerbuild plugin * OpenShift Origin v3 * Atomic Reactor * osbs-client tools * A docker registry ** docker-distribution * Taskotron * fedmsg * RelEng Automation The build system is setup such that Fedora Layered Image maintainers will submit a build to Koji via the `fedpkg container-build` command a `containers` namespace within https://fedoraproject.org/wiki/Infrastructure/VersionControl/dist-git[DistGit]. This will trigger the build to be scheduled in https://www.openshift.org/[OpenShift] via https://github.com/projectatomic/osbs-client[osbs-client] tooling, this will create a custom https://docs.openshift.org/latest/dev_guide/builds.html[OpenShift Build] which will use the pre-made buildroot Docker image that we have created. The https://github.com/projectatomic/atomic-reactor[Atomic Reactor] (`atomic-reactor`) utility will run within the buildroot and prep the build container where the actual build action will execute, it will also maintain uploading the https://fedoraproject.org/wiki/Koji/ContentGenerators[Content Generator] metadata back to https://fedoraproject.org/wiki/Koji[Koji] and upload the built image to the candidate docker registry. This will run on a host with iptables rules restricting access to the docker bridge, this is how we will further limit the access of the buildroot to the outside world verifying that all sources of information come from Fedora. Completed layered image builds are hosted in a candidate docker registry which is then used to pull the image and perform tests with https://taskotron.fedoraproject.org/[Taskotron]. The taskotron tests are triggered by a http://www.fedmsg.com/en/latest/[fedmsg] message that is emitted from https://fedoraproject.org/wiki/Koji[Koji] once the build is complete. Once the test is complete, taskotron will send fedmsg which is then caught by the [.title-ref]#RelEng Automation# Engine that will run the Automatic Release tasks in order to push the layered image into a stable docker registry in the production space for end users to consume. Note that every time the layered image tagged to a new registry, ultimately changing it's name, https://pagure.io/robosignatory[Robosignatory] will automatically sign the new image. This will also occur as part of the Automated Release process as the image will be tagged from the candidate docker registry into the production docker registry in order to "graduate" the image to stable. ==== Koji https://fedoraproject.org/wiki/Koji[Koji] is the Fedora Build System. ==== koji-containerbuild plugin The https://github.com/release-engineering/koji-containerbuild[koji-containerbuild] plugin integrates Koji and OSBS so that builds can be scheduled by koji and integrated into the build system with imports of metadata, logs, build data, and build artifacts. ==== OpenShift Origin v3 https://www.openshift.org/[OpenShift Origin v3] is an open source Container Platform, built on top of http://kubernetes.io/[kubernetes] and https://github.com/docker/docker/[Docker]. This provides many aspects of the system needed including a build pipeline for Docker images with custom build profiles, image streams, and triggers based on events within the system. ==== Atomic Reactor https://github.com/projectatomic/atomic-reactor[Atomic Reactor] is an utility which allows for the building of containers from within other containers providing hooks to trigger automation of builds as well as plugins providing automatic integration many other utilities and services. ==== osbs-client tools https://github.com/projectatomic/osbs-client[osbs-client] tools allow for users to access the build functionality of https://www.openshift.org/[OpenShift Origin v3] using a simple set of command line utilities. ==== docker-registry A https://docs.docker.com/registry/[docker-registry] is a stateless, highly scalable server side application that stores and lets you distribute Docker images. There are many different implementations of docker-registries, two main ones are supported by the Fedora Layered Image Build System. ===== docker-distribution The https://github.com/docker/distribution/[docker-distribution] registry is considered the Docker upstream "v2 registry" such that it was used by upstream to implement the new version 2 specification of the docker-registry. ===== Fedora Production Registry Implementation details of this are still unknown at the time of this writing and will be updated at a later date. For the current status and implementation notes please visit the https://fedoraproject.org/wiki/Changes/FedoraDockerRegistry[FedoraDockerRegistry] page. ==== Taskotron https://taskotron.fedoraproject.org/[Taskotron] is an automated task execution framework, written on top of http://buildbot.net/[buildbot] that currently executes many Fedora automated QA tasks and we will be adding the Layered Image automated QA tasks. The tests themselves will be held in DistGit and maintained by the Layered Image maintainers. ==== RelEng Automation https://pagure.io/releng-automation[RelEng Automation] is an ongoing effort to automate as much of the RelEng process as possible by using http://ansible.com/[Ansible] and being driven by http://www.fedmsg.com/en/latest/[fedmsg] via https://github.com/maxamillion/loopabull[Loopabull] to execute Ansible Playbooks based on fedmsg events. ==== Robosignatory https://pagure.io/robosignatory[Robosignatory] is a fedmsg consumer that automatically signs artifacts and will be used to automatically sign docker layered images for verification by client tools as well as end users. ==== Future Integrations In the future various other components of the https://fedoraproject.org/wiki/Infrastructure[Fedora Infrastructure] will likely be incorporated. ===== PDC https://pdc.fedoraproject.org/[PDC] is Fedora's implementation of https://github.com/product-definition-center/product-definition-center[Product Definition Center] which allows Fedora to maintain a database of each Compose and all of it's contents in a way that can be queried and used to make decisions in a programatic way. ===== The New Hotness https://github.com/fedora-infra/the-new-hotness[The New Hotness] is a http://www.fedmsg.com/en/latest/[fedmsg] consumer that listens to release-monitoring.org and files bugzilla bugs in response (to notify packagers that they can update their packages).