Added the infra SOPs ported to asciidoc.
This commit is contained in:
Adam Saleh
parent
8a7f111a12
commit
a0301e30f1
148 changed files with 18575 additions and 17 deletions
208
modules/sysadmin_guide/pages/koschei.adoc
Normal file
208
modules/sysadmin_guide/pages/koschei.adoc
Normal file
|
|
@ -0,0 +1,208 @@
|
|||
= Koschei SOP
|
||||
|
||||
Koschei is a continuous integration system for RPM packages. Koschei
|
||||
runs package scratch builds after dependency change or after time elapse
|
||||
and reports package buildability status to interested parties.
|
||||
|
||||
Production instance: https://apps.fedoraproject.org/koschei Staging
|
||||
instance: https://apps.stg.fedoraproject.org/koschei
|
||||
|
||||
== Contact Information
|
||||
|
||||
Owner::
|
||||
mizdebsk, msimacek
|
||||
Contact::
|
||||
#fedora-admin
|
||||
Location::
|
||||
Fedora Cloud
|
||||
Purpose::
|
||||
continuous integration system
|
||||
|
||||
== Deployment
|
||||
|
||||
Koschei deployment is managed by two Ansible playbooks:
|
||||
|
||||
....
|
||||
sudo rbac-playbook groups/koschei-backend.yml
|
||||
sudo rbac-playbook groups/koschei-web.yml
|
||||
....
|
||||
|
||||
== Description
|
||||
|
||||
Koschei is deployed on two separate machines - `koschei-backend` and
|
||||
`koschei-web`
|
||||
|
||||
Frontend (`koschei-web`) is a Flask WSGi application running with httpd.
|
||||
It displays information to users and allows editing package groups and
|
||||
changing priorities.
|
||||
|
||||
Backend (`koschei-backend`) consists of multiple services:
|
||||
|
||||
* `koschei-watcher` - listens to fedmsg events for complete builds and
|
||||
changes build states in the database
|
||||
* `koschei-repo-resolver` - resolves package dependencies in given repo
|
||||
using hawkey and compares them with previous iteration to get a
|
||||
dependency diff. It resolves all packages in the newest repo available
|
||||
in Koji. The output is a base for scheduling new builds
|
||||
* `koschei-build-resolver` - resolves complete builds in the repo in
|
||||
which they were done in Koji. Produces the dependency differences
|
||||
visible in the frontend
|
||||
* `koschei-scheduler` - schedules new builds based on multiple criteria:
|
||||
** dependency priority - dependency changes since last build valued by
|
||||
their distance in the dependency graph
|
||||
** manual and static priorities - set manually in the frontend. Manual
|
||||
priority is reset after each build, static priority persists
|
||||
** time priority - time elapsed since the last build
|
||||
* `koschei-polling` - polls the same types of events as koschei-watcher
|
||||
without reliance on fedmsg. Additionaly takes care of package list
|
||||
synchronization and other regularly executed tasks
|
||||
|
||||
== Configuration
|
||||
|
||||
Koschei configuration is in `/etc/koschei/config-backend.cfg` and
|
||||
`/etc/koschei/config-frontend.cfg`, and is merged with the default
|
||||
configuration in `/usr/share/koschei/config.cfg` (the ones in `/etc`
|
||||
overrides the defaults in `/usr`). Note the merge is recursive. The
|
||||
configuration contains all configurable items for all Koschei services
|
||||
and the frontend. The alterations to configuration that aren't temporary
|
||||
should be done through ansible playbook. Configuration changes have no
|
||||
effect on already running services -- they need to be restarted, which
|
||||
happens automatically when using the playbook.
|
||||
|
||||
== Disk usage
|
||||
|
||||
Koschei doesn't keep on disk anything that couldn't be recreated easily
|
||||
- all important data is stored in PostgreSQL database, configuration is
|
||||
managed by Ansible, code installed by RPM and so on.
|
||||
|
||||
To speed up operation and reduce load on external servers, Koschei
|
||||
caches some data obtained from services it integrates with. Most
|
||||
notably, YUM repositories downloaded from Koji are kept in
|
||||
`/var/cache/koschei/repodata`. Each repository takes about 100 MB of
|
||||
disk space. Maximal number of repositories kept at time is controlled by
|
||||
`cache_l2_capacity` parameter in `config-backend.cfg`
|
||||
(`config-backend.cfg.j2` in Ansible). If repodata cache starts to
|
||||
consume too much disk space, that value can be decreased - after
|
||||
restart, `koschei-*-resolver` will remove least recently used cache
|
||||
entries to respect configured cache capacity.
|
||||
|
||||
== Database
|
||||
|
||||
Koschei needs to connect to a PostgreSQL database, other database
|
||||
systems are not supported. Database connection is specified in the
|
||||
configuration under the `database_config` key that can contain the
|
||||
following keys: `username, password, host, port, database`.
|
||||
|
||||
After an update of koschei, the database needs to be migrated to new
|
||||
schema. This happens automatically when using the upgrade playbook.
|
||||
Alternatively, it can be executed manulally using:
|
||||
|
||||
....
|
||||
koschei-admin alembic upgrade head
|
||||
....
|
||||
|
||||
The backend services need to be stopped during the migration.
|
||||
|
||||
== Managing koschei services
|
||||
|
||||
Koschei services are systemd units managed through `systemctl`. They can
|
||||
be started and stopped independently in any order. The frontend is run
|
||||
using httpd.
|
||||
|
||||
== Suspespending koschei operation
|
||||
|
||||
For stopping builds from being scheduled, stopping the
|
||||
`koschei-scheduler` service is enough. For planned Koji outages, it's
|
||||
recommended to stop `koschei-scheduler`. It is not necessary, as koschei
|
||||
can recover from Koji errors and network errors automatically, but when
|
||||
Koji builders are stopped, it may cause unexpected build failures that
|
||||
would be reported to users. Other services can be left running as they
|
||||
automatically restart themselves on Koji and network errors.
|
||||
|
||||
== Limiting Koji usage
|
||||
|
||||
Koschei is by default limited to 30 concurrently running builds. This
|
||||
limit can be changed in the configuration under `koji_config.max_builds`
|
||||
key. There's also Koji load monitoring, that prevents builds from being
|
||||
scheduled when Koji load is higher that certain threshold. That should
|
||||
prevent scheduling builds during mass rebuilds, so it's not necessary to
|
||||
stop scheduling during those.
|
||||
|
||||
== Fedmsg notifications
|
||||
|
||||
Koschei optionally supports sending fedmsg notifications for package
|
||||
state changes. The fedmsg dispatch can be turned on and off in the
|
||||
configuration (key `fedmsg-publisher.enabled`). Koschei doesn't supply
|
||||
configuration for fedmsg, it lets the library to load it's own (in
|
||||
`/etc/fedmsg.d/`).
|
||||
|
||||
== Setting admin announcement
|
||||
|
||||
Koschei can display announcement in web UI. This is mostly useful to
|
||||
inform users about outages or other problems.
|
||||
|
||||
To set announcement, run as koschei user:
|
||||
|
||||
....
|
||||
koschei-admin set-notice "Koschei operation is currently suspended due to scheduled Koji outage"
|
||||
....
|
||||
|
||||
or:
|
||||
|
||||
....
|
||||
koschei-admin set-notice "Sumbitting scratch builds by Koschei is currently disabled due to Fedora 23 mass rebuild"
|
||||
....
|
||||
|
||||
To clear announcement, run as koschei user:
|
||||
|
||||
....
|
||||
koschei-admin clear-notice
|
||||
....
|
||||
|
||||
== Adding package groups
|
||||
|
||||
Packages can be added to one or more group.
|
||||
|
||||
To add new group named "mynewgroup", run as koschei user:
|
||||
|
||||
....
|
||||
koschei-admin add-group mynewgroup
|
||||
....
|
||||
|
||||
To add new group named "mynewgroup" and populate it with some packages,
|
||||
run as koschei user:
|
||||
|
||||
....
|
||||
koschei-admin add-group mynewgroup pkg1 pkg2 pkg3
|
||||
....
|
||||
|
||||
== Set package static priority
|
||||
|
||||
Some packages are more or less important and can have higher or lower
|
||||
priority. Any user can change manual priority, which is reset after
|
||||
package is rebuilt. Admins can additionally set static priority, which
|
||||
is not affected by package rebuilds.
|
||||
|
||||
To set static priority of package "foo" to value "100", run as koschei
|
||||
user:
|
||||
|
||||
....
|
||||
koschei-admin --collection f27 set-priority --static foo 100
|
||||
....
|
||||
|
||||
== Branching a new Fedora release
|
||||
|
||||
After branching occurs and Koji build targets have been created, Koschei
|
||||
should be updated to reflect the new state. There is a special admin
|
||||
command for this purpose, which takes care of copying the configuration
|
||||
and also last builds from the history.
|
||||
|
||||
To branch the collection from Fedora 27 to Fedora 28, use the following:
|
||||
|
||||
....
|
||||
koschei-admin branch-collection f27 f28 -d 'Fedora 27' -t f28 --bugzilla-version 27
|
||||
....
|
||||
|
||||
Then you can optionally verify that the collection configuration is
|
||||
correct by visiting https://apps.fedoraproject.org/koschei/collections
|
||||
and examining the configuration of the newly branched collection.
|
||||
Loading…
Add table
Add a link
Reference in a new issue