Toddlers
========

Toddlers is a simple application aimed at running tasks upon fedora-messaging
notifications.

In other words, it's a bunch of small programs that keeps running around.


How does it work?
-----------------

It is a `fedora-messaging <https://fedora-messaging.readthedocs.io/>`_ consumer
that listens for a number of topics defined by each toddler.

The runner will then asks each toddler if they are interested by the topic of
the message it just received and if so, it will send the full message to the
toddler to be processed.


How can I add a toddler?
------------------------

Toddlers need to be a subclass of ``toddlers.base.ToddlerBase`` (see the file
``toddlers/base.py``), it needs to have the 2 required property and the 2
required methods and needs to be in the folder: ``toddlers/plugins/`` from which
it will be automatically loaded.


What happens if a toddler trips and fails to process a message?
---------------------------------------------------------------

If a toddler fails to process a message (ie: it raises an exception)
the message is put back in the queue and will be processed later.
Depending on the order in which the toddler are run, it is possible that a
toddler sees multiple times the same message, the code should thus take this
into account and check if it needs to do something before doing it.

If none of the toddlers raised an exception the message is considered as
processed.


How does it scale?
------------------

Toddlers consumes queues that are created for it in rabbitmq, so you can start
as many processes as you like, rabbitmq will distribute the message in the queue
in a round robin fashion to each consumers, allowing you to easily scale up or
down depending on your load and needs.


How to run it locally?
----------------------

Follow these simples steps:

* Install the dependencies:

::

    dnf install python3-requests python3-koji fedora-messaging

* Clone locally this repository and go into it:

::

    git clone https://pagure.io/fedora-infra/toddlers.git
    cd toddlers

* Adjust the example configuration file so it creates a temporary queue just for
  you

::

    sed -e "s/[0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}/$(uuidgen)/g" \
    toddlers.toml.example > toddlers.toml

* Run the consumer:

::

    PYTHONPATH=. fedora-messaging --conf=toddlers.toml consume

* If you want to, you can run a single toddler (in case it has a `__name__ == "__main__"`):

Example:

::

    python3 -m toddlers.path.to.toddler

    python3 -m toddlers.plugins.pdc_import_compose


.. warning:: If you are adding a toddler to be run in the Fedora infrastructure
   and this toddler needs to listen to a new topic, you will have to add it
   to the ansible playbook for toddlers, the dynamic loading of the topics that
   toddlers has works for local development but will not work in production.
   The ansible playbook is at: https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/openshift-apps/toddlers.yml


How to run the tests?
---------------------

* Simply install:

::

    dnf install python3-tox cairo-devel cairo-gobject-devel gobject-introspection-devel libmodulemd1 python3.10-devel krb5-devel

* And run the tests:

::

    tox

How to test a single toddler locally
-------------------------------------

The best way to test a toddler locally is to replay a message that triggers the toddler.
Fedora-messaging has a replay and reconsume commands since version 3.7

- Make sure you have fedora-messaging 3.7 or older and upgrade if necessary:
  (All poetry commands have to be ran from inside the toddlers directory.)

::
    poetry install

- Search for suitable message of desired topic in datagrepper:
  [Install httpie](https://httpie.io/docs/cli/installation)

::

    http get https://apps.fedoraproject.org./datagrepper/v2/search \
    topic==<topic> delta==<delta>

Example command for search of messages in topic org.fedoraproject.prod.ipa.group_remove_member.v1 sent in the last day:

::

    http get https://apps.fedoraproject.org/datagrepper/v2/search \
    topic==org.fedoraproject.prod.ipa.group_remove_member.v1 delta==86400

- Get the message ID and supply it to the fedora-messaging reconsume command, with local config:

::

    poetry run fedora-messaging reconsume --conf toddlers.toml \
    --callback toddlers.plugins.<toddler_name>:<ToddlerClass> <message ID>

Example command:

::

    poetry run fedora-messaging reconsume --conf toddlers.toml \
    --callback toddlers.plugins.cleaning_packager_groups:CleanPackagerGroups \
    e7c3f150-3695-4502-b7ca-4ced7fe14a21


How to test with tiny-stage
----------------------------

The toddler cleaning_packager_groups uses `ipalib` freeipa library,
which makes local testing sligthly complex.
It's a good idea to use [tiny-stage](https://github.com/fedora-infra/tiny-stage).
Here's how to set it up:

Clone the tiny-stage repo and run following VMs:

::

    vagrant up tinystage ipa auth tiny-stage datagrepper src

Run the toddlers vagrant:

::

    vagrant up

Point Vagrant's host name to toddlers.dev.tinystage.test:

::

    toddlers.vm.host_name = "toddlers.dev.tinystage.test"

Make changes in toddlers.toml for toddlers VM to see the tiny-stage environment:

::

    amqp_url = "amqp://fedoramessages:fedoramessages@tinystage.tinystage.test"

Redirect urls to point to tiny-stage environment:

::

    fas_url = "https://fasjson.tinystage.test/fasjson/"
    fas_username = "williamyoung"
    fas_password = "password"

    dist_git_url = "http://src.tinystage.test"
    dist_git_token = ""

Tiny-stage webUIs are accesible at the following urls:

::

    webui for noggin:
    https://auth.tinystage.test/noggin/
    webui for distgit:
    http://src.tinystage.test/
    webui for ipa:
    https://ipa.tinystage.test/ipa/ui/
    webui for datagrepper:
    http://datagrepper.tinystage.test:5000/

Create an API token on distgit and fill it in the previous step.

Make the fedora-messages consumer listen to messages from the toddlers vagrant:

::

    poetry run fedora-messaging --conf toddlers.toml consume --callback toddlers.plugins.
    cleaning_packager_groups:CleanPackagerGroups

Trigger the toddler by making changes in the webUIs.


Developing with Vagrant
-----------------------

For your convenience, a Vagrantfile is provided, which will create a virtual machine with all the needed packages to start developing Toddlers.

First, make sure you have Vagrant installed on your machine:

::

    dnf install ansible libvirt vagrant-libvirt vagrant-sshfs vagrant-hostmanager

Then go to the local directory where you cloned Toddlers repository and run:

::

    $ vagrant up

Wait for the VM to be set up. After that run:

::

    $ vagrant ssh

to login the VM.