Added the infra SOPs ported to asciidoc.
This commit is contained in:
Adam Saleh
parent
8a7f111a12
commit
a0301e30f1
148 changed files with 18575 additions and 17 deletions
210
modules/sysadmin_guide/pages/anitya.adoc
Normal file
210
modules/sysadmin_guide/pages/anitya.adoc
Normal file
|
|
@ -0,0 +1,210 @@
|
|||
= Anitya Infrastructure SOP
|
||||
|
||||
Anitya is used by Fedora to track upstream project releases and maps
|
||||
them to downstream distribution packages, including (but not limited to)
|
||||
Fedora.
|
||||
|
||||
Anitya staging instance: https://stg.release-monitoring.org
|
||||
|
||||
Anitya production instance: https://release-monitoring.org
|
||||
|
||||
Anitya project page: https://github.com/fedora-infra/anitya
|
||||
|
||||
== Contact Information
|
||||
|
||||
Owner::
|
||||
Fedora Infrastructure Team
|
||||
Contact::
|
||||
#fedora-admin, #fedora-apps
|
||||
Persons::
|
||||
zlopez
|
||||
Location::
|
||||
iad2.fedoraproject.org
|
||||
Servers::
|
||||
Production
|
||||
+
|
||||
* os-master01.iad2.fedoraproject.org
|
||||
+
|
||||
Staging
|
||||
+
|
||||
* os-master01.stg.iad2.fedoraproject.org
|
||||
Purpose::
|
||||
Map upstream releases to Fedora packages.
|
||||
|
||||
== Hosts
|
||||
|
||||
The current deployment is made up of release-monitoring OpenShift
|
||||
namespace.
|
||||
|
||||
=== release-monitoring
|
||||
|
||||
This OpenShift namespace runs following pods:
|
||||
|
||||
* The apache/mod_wsgi application for release-monitoring.org
|
||||
* A libraries.io SSE client
|
||||
* A service checking for new releases
|
||||
|
||||
This OpenShift project relies on:
|
||||
|
||||
* A postgres db server running in OpenShift
|
||||
* Lots of external third-party services. The anitya webapp can scrape
|
||||
pypi, rubygems.org, sourceforge and many others on command.
|
||||
* Lots of external third-party services. The check service makes all
|
||||
kinds of requests out to the Internet that can fail in various ways.
|
||||
* Fedora messaging RabbitMQ hub for publishing messages
|
||||
|
||||
Things that rely on this host:
|
||||
|
||||
* `hotness-sop` is a fedora messaging consumer running in Fedora Infra
|
||||
in OpenShift. It listens for Anitya messages from here and performs
|
||||
actions on koji and bugzilla.
|
||||
|
||||
== Releasing
|
||||
|
||||
The release process is described in
|
||||
https://anitya.readthedocs.io/en/latest/contributing.html#release-guide[Anitya
|
||||
documentation].
|
||||
|
||||
=== Deploying
|
||||
|
||||
Staging deployment of Anitya is deployed in OpenShift on
|
||||
os-master01.stg.iad2.fedoraproject.org.
|
||||
|
||||
To deploy staging instance of Anitya you need to push changes to staging
|
||||
branch on https://github.com/fedora-infra/anitya[Anitya GitHub]. GitHub
|
||||
webhook will then automatically deploy a new version of Anitya on
|
||||
staging.
|
||||
|
||||
Production deployment of Anitya is deployed in OpenShift on
|
||||
os-master01.iad2.fedoraproject.org.
|
||||
|
||||
To deploy production instance of Anitya you need to push changes to
|
||||
production branch on https://github.com/fedora-infra/anitya[Anitya
|
||||
GitHub]. GitHub webhook will then automatically deploy a new version of
|
||||
Anitya on production.
|
||||
|
||||
==== Configuration
|
||||
|
||||
To deploy the new configuration, you need
|
||||
https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/sshaccess.html[ssh
|
||||
access] to batcave01.iad2.fedoraproject.org and
|
||||
https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/ansible.html[permissions
|
||||
to run the Ansible playbook].
|
||||
|
||||
All the following commands should be run from batcave01.
|
||||
|
||||
First, ensure there are no configuration changes required for the new
|
||||
update. If there are, update the Ansible anitya role(s) and optionally
|
||||
run the playbook:
|
||||
|
||||
....
|
||||
$ sudo rbac-playbook openshift-apps/release-monitoring.yml
|
||||
....
|
||||
|
||||
The configuration changes could be limited to staging only using:
|
||||
|
||||
....
|
||||
$ sudo rbac-playbook openshift-apps/release-monitoring.yml -l staging
|
||||
....
|
||||
|
||||
This is recommended for testing new configuration changes.
|
||||
|
||||
==== Upgrading
|
||||
|
||||
===== Staging
|
||||
|
||||
To deploy new version of Anitya you need to push changes to staging
|
||||
branch on https://github.com/fedora-infra/anitya[Anitya GitHub]. GitHub
|
||||
webhook will then automatically deploy a new version of Anitya on
|
||||
staging.
|
||||
|
||||
===== Production
|
||||
|
||||
To deploy new version of Anitya you need to push changes to production
|
||||
branch on https://github.com/fedora-infra/anitya[Anitya GitHub]. GitHub
|
||||
webhook will then automatically deploy a new version of Anitya on
|
||||
production.
|
||||
|
||||
Congratulations! The new version should now be deployed.
|
||||
|
||||
== Administrating release-monitoring.org
|
||||
|
||||
Anitya web application offers some functionality to administer itself.
|
||||
|
||||
User admin status is tracked in Anitya database. Admin users can grant
|
||||
or revoke admin priviledges to users in the
|
||||
https://release-monitoring.org/users[users tab].
|
||||
|
||||
Admin users have additional functionality available in web interface. In
|
||||
particular, admins can view flagged projects, remove projects and remove
|
||||
package mappings etc.
|
||||
|
||||
For more information see
|
||||
https://anitya.readthedocs.io/en/stable/admin-user-guide.html[Admin user
|
||||
guide] in Anitya documentation.
|
||||
|
||||
=== Flags
|
||||
|
||||
Anitya lets users flag projects for administrator attention. This is
|
||||
accessible to administrators in the
|
||||
https://release-monitoring.org/flags[flags tab].
|
||||
|
||||
== Monitoring
|
||||
|
||||
To monitor the activity of Anitya you can connect to Fedora infra
|
||||
OpenShift and look at the state of pods.
|
||||
|
||||
For staging look at the [.title-ref]#release-monitoring# namespace in
|
||||
https://os.stg.fedoraproject.org/console/project/release-monitoring/overview[staging
|
||||
OpenShift instance].
|
||||
|
||||
For production look at the [.title-ref]#release-monitoring# namespace in
|
||||
https://os.fedoraproject.org/console/project/release-monitoring/overview[production
|
||||
OpenShift instance].
|
||||
|
||||
== Troubleshooting
|
||||
|
||||
This section contains various issues encountered during deployment or
|
||||
configuration changes and possible solutions.
|
||||
|
||||
=== Fedmsg messages aren't sent
|
||||
|
||||
*Issue:* Fedmsg messages aren't sent.
|
||||
|
||||
*Solution:* Set USER environment variable in pod.
|
||||
|
||||
*Explanation:* Fedmsg is using USER env variable as a username inside
|
||||
messages. Without USER env set it just crashes and didn't send anything.
|
||||
|
||||
=== Cronjob is crashing
|
||||
|
||||
*Issue:* Cronjob pod is crashing on start, even after configuration
|
||||
change that should fix the behavior.
|
||||
|
||||
*Solution:* Restart the cronjob. This could be done by OPS.
|
||||
|
||||
*Explanation:* Every time the cronjob is executed after crash it is
|
||||
trying to actually reuse the pod with bad configuration instead of
|
||||
creating a new one with new configuration.
|
||||
|
||||
=== Database migration is taking too long
|
||||
|
||||
*Issue:* Database migration is taking few hours to complete.
|
||||
|
||||
*Solution:* Stop every pod and cronjob before migration.
|
||||
|
||||
*Explanation:* When creating new index or doing some other complex
|
||||
operation on database, the migration script needs exclusive access to
|
||||
the database.
|
||||
|
||||
=== Old version is deployed instead the new one
|
||||
|
||||
*Issue:* The pod is deployed with old version of Anitya, but it says
|
||||
that it was triggered by correct commit.
|
||||
|
||||
*Solution:* Set [.title-ref]#dockerStrategy# in buildconfig.yml to
|
||||
noCache.
|
||||
|
||||
*Explanation:* The OpenShift is by default caching the layers of docker
|
||||
containers, so if there is no change in Dockerfile it will just use the
|
||||
cached version and don't run the commands again.
|
||||
Loading…
Add table
Add a link
Reference in a new issue