Small Fedora-messaging triggered programs that we want to keep running around
7fe0937190
Previously, ToddlerBase and derived classes were used directly, but none of the methods is marked as a class or static method. Amazingly enough, it worked regardless. It's useful to have a constructor, though, so do things conventionally. As we can't just monkey-patch the classes then, use the shared `toddler` fixture which creates the object for the test methods that need it. Signed-off-by: Nils Philippsen <nils@redhat.com> |
||
|---|---|---|
| tests | ||
| toddlers | ||
| .gitignore | ||
| .zuul.yaml | ||
| bindep.txt | ||
| Dockerfile | ||
| LICENSE | ||
| README.rst | ||
| requirements.txt | ||
| setup.cfg | ||
| test-requirements.txt | ||
| toddlers.toml.example | ||
| tox.ini | ||
Toddlers
========
Toddlers is a simple application aimed at running tasks upon fedora-messaging
notifications.
In other words, it's a bunch of small programs that keeps running around.
How does it work?
-----------------
It is a `fedora-messaging <https://fedora-messaging.readthedocs.io/>`_ consumer
that listens for a number of topics defined by each toddler.
The runner will then asks each toddler if they are interested by the topic of
the message it just received and if so, it will send the full message to the
toddler to be processed.
How can I add a toddler?
------------------------
Toddlers need to be a subclass of ``toddlers.base.ToddlerBase`` (see the file
``toddlers/base.py``), it needs to have the 2 required property and the 2
required methods and needs to be in the folder: ``toddlers/plugins/`` from which
it will be automatically loaded.
What happens if a toddler trips and fails to process a message?
---------------------------------------------------------------
If a toddler fails to process a message (ie: it raises an exception)
the message is put back in the queue and will be processed later.
Depending on the order in which the toddler are run, it is possible that a
toddler sees multiple times the same message, the code should thus take this
into account and check if it needs to do something before doing it.
If none of the toddlers raised an exception the message is considered as
processed.
How does it scale?
------------------
Toddlers consumes queues that are created for it in rabbitmq, so you can start
as many processes as you like, rabbitmq will distribute the message in the queue
in a round robin fashion to each consumers, allowing you to easily scale up or
down depending on your load and needs.
How to run it locally?
----------------------
Follow these simples steps:
* Install the dependencies:
::
dnf install python3-requests python3-koji fedora-messaging
* Clone locally this repository and go into it:
::
git clone https://pagure.io/fedora-infra/toddlers.git
cd toddlers
* Adjust the example configuration file so it creates a temporary queue just for
you
::
sed -e "s/[0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}/$(uuidgen)/g" \
toddlers.toml.example > toddlers.toml
* Run the consumer:
::
PYTHONPATH=. fedora-messaging --conf=toddlers.toml consume
.. warning:: If you are adding a toddler to be run in the Fedora infrastructure
and this toddler needs to listen to a new topic, you will have to add it
to the ansible playbook for toddlers, the dynamic loading of the topics that
toddlers has works for local development but will not work in production.
The ansible playbook is at: https://pagure.io/fedora-infra/ansible/blob/master/f/playbooks/openshift-apps/toddlers.yml
How to run the tests?
---------------------
* Simply install:
::
dnf install python3-tox
* And run the tests:
::
tox