3e16c0b7d8
I recently fixed some issue with disk space on koji builder and found out that this guide is outdated. So here is update to make it up to date. Signed-off-by: Michal Konečný <mkonecny@redhat.com>
87 lines
2.6 KiB
Text
87 lines
2.6 KiB
Text
= Koji Infrastructure SOP
|
|
|
|
== Contents
|
|
|
|
* <<_contact_information>>
|
|
* <<_description>>
|
|
* <<_add_packages_into_buildroot>>
|
|
* <<_troubleshooting_and_resolution>>
|
|
** <<_kojid_wont_start_or_some_builders_wont_connect>>
|
|
** <<_disk_space_issues>>
|
|
** <<_checking_builders_are_all_checking_in_correctly>>
|
|
|
|
== Contact Information
|
|
|
|
Owner::
|
|
Fedora Infrastructure Team
|
|
Contact::
|
|
#fedora-admin, sysadmin-build group
|
|
Persons::
|
|
mbonnet, dgilmore, f13, notting, mmcgrath, SmootherFrOgZ
|
|
Servers::
|
|
* koji.fedoraproject.org
|
|
* buildsys.fedoraproject.org
|
|
* xenbuilder[1-4]
|
|
* hammer1, ppc[1-4]
|
|
Purpose::
|
|
Build packages for Fedora.
|
|
|
|
== Description
|
|
|
|
Users submit builds to _koji.fedoraproject.org_ or
|
|
_buildsys.fedoraproject.org_. From there it gets passed on to the
|
|
builders.
|
|
|
|
== Add packages into Buildroot
|
|
|
|
Some contributors may have the need to build packages against fresh
|
|
built packages which are not into buildroot yet. Koji has override tags
|
|
as a Inheritance to the build tag in order to include them into
|
|
buildroot which can be set by:
|
|
|
|
....
|
|
koji tag-pkg dist-$release-override <package_nvr>
|
|
....
|
|
|
|
== Troubleshooting and Resolution
|
|
|
|
=== kojid won't start or some builders won't connect
|
|
|
|
In the event that some items are able to connect to koji while some are
|
|
not, please make sure that the database is not filled up on connections.
|
|
This is common if koji crashes and the db connections aren't properly
|
|
cleared. Upon restart many of the connections are full so koji cannot
|
|
reconnect. Clearing old connections is easy, guess about how long it the
|
|
new koji has been up and pick a number of minutes larger then that and
|
|
kill those queries. From _db-koji01_ as _postgres_ run:
|
|
|
|
....
|
|
echo "select procpid from pg_stat_activity where usename='koji' and now() - query_start \
|
|
>= '00:40:00' order by query_start;" | psql koji | grep "^ " | xargs kill
|
|
....
|
|
|
|
=== Disk Space Issues
|
|
|
|
The builders use a lot of temporary storage. Failed builds and old mock buildroots should be automatic cleaned, but in case it doesn't, remove /var/lib/mock/* and restart kojid on the affected builder:
|
|
|
|
....
|
|
systemctl restart kojid
|
|
....
|
|
|
|
[IMPORTANT]
|
|
====
|
|
aarch64 buildhw's have a lot of space taken up in /var/log/libvirt/qemu/nvram/ which can all be deleted to free up space.
|
|
====
|
|
|
|
=== Checking builders are all checking in correctly
|
|
|
|
To check builders, list all builders and grep by the time of last update. If the builders are checking in correctly the time of last update should be close to your current date/time, so use a command like the following example:
|
|
|
|
....
|
|
koji list-hosts --enabled | grep -v '04 Dec 2022 12:1'
|
|
....
|
|
|
|
[IMPORTANT]
|
|
====
|
|
Kojira process should only run on koji02. Never on koji01.
|
|
====
|