196 lines
6 KiB
Text
196 lines
6 KiB
Text
= Anitya Infrastructure SOP
|
|
|
|
Anitya is used by Fedora to track upstream project releases and maps
|
|
them to downstream distribution packages, including (but not limited to)
|
|
Fedora.
|
|
|
|
Anitya staging instance: https://stg.release-monitoring.org
|
|
|
|
Anitya production instance: https://release-monitoring.org
|
|
|
|
Anitya project page: https://github.com/fedora-infra/anitya
|
|
|
|
== Contact Information
|
|
|
|
Owner::
|
|
Fedora Infrastructure Team
|
|
Contact::
|
|
#fedora-admin, #fedora-apps
|
|
Persons::
|
|
zlopez
|
|
Location::
|
|
rdu3.fedoraproject.org
|
|
Servers::
|
|
* *Production* - os-master01.rdu3.fedoraproject.org
|
|
* *Staging* - os-master01.stg.rdu3.fedoraproject.org
|
|
Purpose::
|
|
Map upstream releases to Fedora packages.
|
|
|
|
== Hosts
|
|
|
|
The current deployment is made up of release-monitoring OpenShift
|
|
namespace.
|
|
|
|
=== release-monitoring
|
|
|
|
This OpenShift namespace runs following pods:
|
|
|
|
* The apache/mod_wsgi application for release-monitoring.org
|
|
* A libraries.io SSE client
|
|
* A service checking for new releases
|
|
|
|
This OpenShift project relies on:
|
|
|
|
* A postgres db server running in OpenShift
|
|
* Lots of external third-party services. The anitya webapp can scrape
|
|
pypi, rubygems.org, sourceforge and many others on command.
|
|
* Lots of external third-party services. The check service makes all
|
|
kinds of requests out to the Internet that can fail in various ways.
|
|
* Fedora messaging RabbitMQ hub for publishing messages
|
|
|
|
Things that rely on this host:
|
|
|
|
* `hotness-sop` is a fedora messaging consumer running in Fedora Infra
|
|
in OpenShift. It listens for Anitya messages from here and performs
|
|
actions on koji and bugzilla.
|
|
|
|
== Releasing
|
|
|
|
The release process is described in
|
|
https://anitya.readthedocs.io/en/latest/contributing.html#release-guide[Anitya
|
|
documentation].
|
|
|
|
=== Deploying
|
|
|
|
Staging deployment of Anitya is deployed in OpenShift on
|
|
os-master01.stg.rdu3.fedoraproject.org.
|
|
|
|
To deploy staging instance of Anitya you need to push changes to staging
|
|
branch on https://github.com/fedora-infra/anitya[Anitya GitHub]. GitHub
|
|
webhook will then automatically deploy a new version of Anitya on
|
|
staging.
|
|
|
|
Production deployment of Anitya is deployed in OpenShift on
|
|
os-master01.rdu3.fedoraproject.org.
|
|
|
|
To deploy production instance of Anitya you need to push changes to
|
|
production branch on https://github.com/fedora-infra/anitya[Anitya
|
|
GitHub]. GitHub webhook will then automatically deploy a new version of
|
|
Anitya on production.
|
|
|
|
==== Configuration
|
|
|
|
To deploy the new configuration, you need
|
|
xref:sshaccess.adoc[ssh
|
|
access] to batcave01.rdu3.fedoraproject.org and
|
|
xref:ansible.adoc[permissions
|
|
to run the Ansible playbook].
|
|
|
|
All the following commands should be run from batcave01.
|
|
|
|
First, ensure there are no configuration changes required for the new
|
|
update. If there are, update the Ansible anitya role(s) and optionally
|
|
run the playbook:
|
|
|
|
....
|
|
$ sudo rbac-playbook openshift-apps/release-monitoring.yml
|
|
....
|
|
|
|
The configuration changes could be limited to staging only using:
|
|
|
|
....
|
|
$ sudo rbac-playbook openshift-apps/release-monitoring.yml -l staging
|
|
....
|
|
|
|
This is recommended for testing new configuration changes.
|
|
|
|
==== Upgrading
|
|
|
|
===== Staging
|
|
|
|
To deploy new version of Anitya you need to push changes to staging
|
|
branch on https://github.com/fedora-infra/anitya[Anitya GitHub]. GitHub
|
|
webhook will then automatically deploy a new version of Anitya on
|
|
staging.
|
|
|
|
===== Production
|
|
|
|
To deploy new version of Anitya you need to push changes to production
|
|
branch on https://github.com/fedora-infra/anitya[Anitya GitHub]. GitHub
|
|
webhook will then automatically deploy a new version of Anitya on
|
|
production.
|
|
|
|
Congratulations! The new version should now be deployed.
|
|
|
|
== Administrating release-monitoring.org
|
|
|
|
Anitya web application offers some functionality to administer itself.
|
|
|
|
User admin status is tracked in Anitya database. Admin users can grant
|
|
or revoke admin priviledges to users in the
|
|
https://release-monitoring.org/users[users tab].
|
|
|
|
Admin users have additional functionality available in web interface. In
|
|
particular, admins can view flagged projects, remove projects and remove
|
|
package mappings etc.
|
|
|
|
For more information see
|
|
https://anitya.readthedocs.io/en/stable/admin-user-guide.html[Admin user
|
|
guide] in Anitya documentation.
|
|
|
|
=== Flags
|
|
|
|
Anitya lets users flag projects for administrator attention. This is
|
|
accessible to administrators in the
|
|
https://release-monitoring.org/flags[flags tab].
|
|
|
|
== Monitoring
|
|
|
|
To monitor the activity of Anitya you can connect to Fedora infra
|
|
OpenShift and look at the state of pods.
|
|
|
|
For staging look at the [.title-ref]#release-monitoring# namespace in
|
|
https://os.stg.fedoraproject.org/console/project/release-monitoring/overview[staging
|
|
OpenShift instance].
|
|
|
|
For production look at the [.title-ref]#release-monitoring# namespace in
|
|
https://os.fedoraproject.org/console/project/release-monitoring/overview[production
|
|
OpenShift instance].
|
|
|
|
== Troubleshooting
|
|
|
|
This section contains various issues encountered during deployment or
|
|
configuration changes and possible solutions.
|
|
|
|
=== Cronjob is crashing
|
|
|
|
*Issue:* Cronjob pod is crashing on start, even after configuration
|
|
change that should fix the behavior.
|
|
|
|
*Solution:* Restart the cronjob. This could be done by OPS.
|
|
|
|
*Explanation:* Every time the cronjob is executed after crash it is
|
|
trying to actually reuse the pod with bad configuration instead of
|
|
creating a new one with new configuration.
|
|
|
|
=== Database migration is taking too long
|
|
|
|
*Issue:* Database migration is taking few hours to complete.
|
|
|
|
*Solution:* Stop every pod and cronjob before migration.
|
|
|
|
*Explanation:* When creating new index or doing some other complex
|
|
operation on database, the migration script needs exclusive access to
|
|
the database.
|
|
|
|
=== Old version is deployed instead the new one
|
|
|
|
*Issue:* The pod is deployed with old version of Anitya, but it says
|
|
that it was triggered by correct commit.
|
|
|
|
*Solution:* Set [.title-ref]#dockerStrategy# in buildconfig.yml to
|
|
noCache.
|
|
|
|
*Explanation:* The OpenShift is by default caching the layers of docker
|
|
containers, so if there is no change in Dockerfile it will just use the
|
|
cached version and don't run the commands again.
|