= Anitya Infrastructure SOP

Anitya is used by Fedora to track upstream project releases and maps
them to downstream distribution packages, including (but not limited to)
Fedora.

Anitya staging instance: https://stg.release-monitoring.org

Anitya production instance: https://release-monitoring.org

Anitya project page: https://github.com/fedora-infra/anitya

== Contact Information

Owner::
  Fedora Infrastructure Team
Contact::
  #fedora-admin, #fedora-apps
Persons::
  zlopez
Location::
  iad2.fedoraproject.org
Servers::
  * *Production* - os-master01.iad2.fedoraproject.org
  * *Staging* - os-master01.stg.iad2.fedoraproject.org
Purpose::
  Map upstream releases to Fedora packages.

== Hosts

The current deployment is made up of release-monitoring OpenShift
namespace.

=== release-monitoring

This OpenShift namespace runs following pods:

* The apache/mod_wsgi application for release-monitoring.org
* A libraries.io SSE client
* A service checking for new releases

This OpenShift project relies on:

* A postgres db server running in OpenShift
* Lots of external third-party services. The anitya webapp can scrape
pypi, rubygems.org, sourceforge and many others on command.
* Lots of external third-party services. The check service makes all
kinds of requests out to the Internet that can fail in various ways.
* Fedora messaging RabbitMQ hub for publishing messages

Things that rely on this host:

* `hotness-sop` is a fedora messaging consumer running in Fedora Infra
in OpenShift. It listens for Anitya messages from here and performs
actions on koji and bugzilla.

== Releasing

The release process is described in
https://anitya.readthedocs.io/en/latest/contributing.html#release-guide[Anitya
documentation].

=== Deploying

Staging deployment of Anitya is deployed in OpenShift on
os-master01.stg.iad2.fedoraproject.org.

To deploy staging instance of Anitya you need to push changes to staging
branch on https://github.com/fedora-infra/anitya[Anitya GitHub]. GitHub
webhook will then automatically deploy a new version of Anitya on
staging.

Production deployment of Anitya is deployed in OpenShift on
os-master01.iad2.fedoraproject.org.

To deploy production instance of Anitya you need to push changes to
production branch on https://github.com/fedora-infra/anitya[Anitya
GitHub]. GitHub webhook will then automatically deploy a new version of
Anitya on production.

==== Configuration

To deploy the new configuration, you need
xref:sshaccess.adoc[ssh
access] to batcave01.iad2.fedoraproject.org and
xref:ansible.adoc[permissions
to run the Ansible playbook].

All the following commands should be run from batcave01.

First, ensure there are no configuration changes required for the new
update. If there are, update the Ansible anitya role(s) and optionally
run the playbook:

....
$ sudo rbac-playbook openshift-apps/release-monitoring.yml
....

The configuration changes could be limited to staging only using:

....
$ sudo rbac-playbook openshift-apps/release-monitoring.yml -l staging
....

This is recommended for testing new configuration changes.

==== Upgrading

===== Staging

To deploy new version of Anitya you need to push changes to staging
branch on https://github.com/fedora-infra/anitya[Anitya GitHub]. GitHub
webhook will then automatically deploy a new version of Anitya on
staging.

===== Production

To deploy new version of Anitya you need to push changes to production
branch on https://github.com/fedora-infra/anitya[Anitya GitHub]. GitHub
webhook will then automatically deploy a new version of Anitya on
production.

Congratulations! The new version should now be deployed.

== Administrating release-monitoring.org

Anitya web application offers some functionality to administer itself.

User admin status is tracked in Anitya database. Admin users can grant
or revoke admin priviledges to users in the
https://release-monitoring.org/users[users tab].

Admin users have additional functionality available in web interface. In
particular, admins can view flagged projects, remove projects and remove
package mappings etc.

For more information see
https://anitya.readthedocs.io/en/stable/admin-user-guide.html[Admin user
guide] in Anitya documentation.

=== Flags

Anitya lets users flag projects for administrator attention. This is
accessible to administrators in the
https://release-monitoring.org/flags[flags tab].

== Monitoring

To monitor the activity of Anitya you can connect to Fedora infra
OpenShift and look at the state of pods.

For staging look at the [.title-ref]#release-monitoring# namespace in
https://os.stg.fedoraproject.org/console/project/release-monitoring/overview[staging
OpenShift instance].

For production look at the [.title-ref]#release-monitoring# namespace in
https://os.fedoraproject.org/console/project/release-monitoring/overview[production
OpenShift instance].

== Troubleshooting

This section contains various issues encountered during deployment or
configuration changes and possible solutions.

=== Cronjob is crashing

*Issue:* Cronjob pod is crashing on start, even after configuration
change that should fix the behavior.

*Solution:* Restart the cronjob. This could be done by OPS.

*Explanation:* Every time the cronjob is executed after crash it is
trying to actually reuse the pod with bad configuration instead of
creating a new one with new configuration.

=== Database migration is taking too long

*Issue:* Database migration is taking few hours to complete.

*Solution:* Stop every pod and cronjob before migration.

*Explanation:* When creating new index or doing some other complex
operation on database, the migration script needs exclusive access to
the database.

=== Old version is deployed instead the new one

*Issue:* The pod is deployed with old version of Anitya, but it says
that it was triggered by correct commit.

*Solution:* Set [.title-ref]#dockerStrategy# in buildconfig.yml to
noCache.

*Explanation:* The OpenShift is by default caching the layers of docker
containers, so if there is no change in Dockerfile it will just use the
cached version and don't run the commands again.
