Tomas Hrcka
14c92455cb
Add missing pages Fix the nav for release guide Cleanup branching sop Add release version substitutions Signed-off-by: Tomas Hrcka <thrcka@redhat.com>
114 lines
4.9 KiB
Text
114 lines
4.9 KiB
Text
== Fedora RelEng Workflow Automation
|
|
|
|
The Fedora RelEng Workflow Automation is a means to allow RelEng to
|
|
define a pattern by which Release Engineering work is automated in an
|
|
uniform fashion. The automation technology of choice is
|
|
https://ansible.com/[ansible] and the "workflow engine" is powered by
|
|
https://github.com/maxamillion/loopabull[loopabull], which is an event
|
|
loop that allows us to pass the information contained within a
|
|
http://www.fedmsg.com/en/latest/[fedmsg] and insert it into
|
|
https://ansible.com/[ansible]
|
|
https://docs.ansible.com/ansible/playbooks.html[playbooks]. This will
|
|
effectively create an event driven workflow that can take action
|
|
conditionally based on the contents of arbitrary
|
|
http://www.fedmsg.com/en/latest/[fedmsg] data.
|
|
|
|
Background on the topic can be found in the
|
|
https://fedoraproject.org/wiki/Changes/ReleaseEngineeringAutomationWorkflowEngine[Release
|
|
Engineering Automation Workflow Engine] Change proposal, as well as in
|
|
the https://pagure.io/releng-automation[releng-automation] pagure
|
|
repository.
|
|
|
|
=== RelEng Workflow Automation Architecture
|
|
|
|
By using http://www.fedmsg.com/en/latest/[fedmsg] as the source of
|
|
information feeding the event loop, we will configure
|
|
https://github.com/maxamillion/loopabull[loopabull] to listen for
|
|
specific
|
|
https://fedora-fedmsg.readthedocs.io/en/latest/topics.html[fedmsg
|
|
topics] which will correspond with https://ansible.com/[ansible]
|
|
https://docs.ansible.com/ansible/playbooks.html[playbooks]. When one of
|
|
the appropriate
|
|
https://fedora-fedmsg.readthedocs.io/en/latest/topics.html[fedmsg
|
|
topics] is encountered across the message bus, it's message payload is
|
|
then injected into the corresponding playbook as an extra set of
|
|
variables. A member of the Fedora Release Engineering Team can at that
|
|
point use this as a means to perform whatever arbitrary action or series
|
|
of actions they can otherwise perform with https://ansible.com/[ansible]
|
|
(including what we can enable via custom
|
|
https://docs.ansible.com/ansible/modules.html[modules]) based on the
|
|
input of the message payload.
|
|
|
|
The general overview of the architecture is below as well as a
|
|
description of how it works:
|
|
|
|
....
|
|
+------------+
|
|
| fedmsg |
|
|
| |
|
|
+---+--------+
|
|
| ^
|
|
| |
|
|
| |
|
|
| |
|
|
| |
|
|
| |
|
|
V |
|
|
+------------------+-----------------+
|
|
| |
|
|
| Release Engineering |
|
|
| Workflow Automation Engine |
|
|
| |
|
|
| - RabbitMQ |
|
|
| - fedmsg-rabbitmq-serializer |
|
|
| - loopabull |
|
|
| |
|
|
+----------------+-------------------+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
V
|
|
+-----------------------+
|
|
| |
|
|
| composer/bodhi/etc |
|
|
| |
|
|
+-----------------------+
|
|
....
|
|
|
|
The flow of data will begin with an event somewhere in the
|
|
https://fedoraproject.org/wiki/Infrastructure[Fedora Infrastructure]
|
|
that sends a http://www.fedmsg.com/en/latest/[fedmsg] across the message
|
|
bus, then the messages will be taken in and serialized in to a
|
|
https://www.rabbitmq.com/[rabbitmq] worker queue using
|
|
https://pagure.io/fedmsg-rabbitmq-serializer[fedmsg-rabbitmq-serializer].
|
|
Then https://github.com/maxamillion/loopabull[loopabull] will be
|
|
listening to the rabbitmq worker queue for tasks to come in. Once a
|
|
message is recieved, it is processed and once it is either no-op'd or a
|
|
corresponding ansible playbook is run to completion, the message will be
|
|
`ack`'d and cleared from the worker queue. This will allow for us to
|
|
scale loopabull instances independently from the message queue as well
|
|
as ensure that work is not lost because of a downed or busy loopabull
|
|
instance. Also, as a point of note, the loopabull service instances will
|
|
be scaled using https://freedesktop.org/wiki/Software/systemd/[systemd]
|
|
https://fedoramagazine.org/systemd-template-unit-files/[unit templates].
|
|
|
|
Once a playbook has been triggered, it will run tasks on remote systems
|
|
on behalf of a loopabull automation user. These users can be privileged
|
|
if need be, however the scope of their privilege is based on the purpose
|
|
they serve. These user accounts are provisioned by the
|
|
https://fedoraproject.org/wiki/Infrastructure[Fedora Infrastructure]
|
|
Team based on the requirements of the
|
|
`RelEng Task Automation User Request Standard Operating
|
|
Procedure (SOP) <sop_requesting_task_automation_user>` document and
|
|
tasks are subject to code and security audit.
|
|
|
|
=== Fedora Lib RelEng
|
|
|
|
https://pagure.io/flr[Fedora Lib RelEng] (flr), is a library and set of
|
|
command line tools to expose the library that aims to provide re-usable
|
|
code for common tasks that need to be done in Release Engineering.
|
|
Combining this set of command line tools when necessary with the Release
|
|
Engineering Automation pipeline allows for easy separation of
|
|
permissions and responsibilities via sudo permissions on remote hosts.
|
|
This is explained in more detail on the project's pagure page.
|