koji_builder / osbuild: adjustments for osbuild api scripting

Right now we run a script on all builders once a minute to update the
api/auth ip's for osbuild. This has a number of problems:
* Sometimes osbuild jobs land on s390x builders that have no internet
  access and hang or fail.
* Sometimes the update script hangs or takes a long time to run because
  the builder is heavily loaded with builds, resulting in locking emails
  to sysadmin-main folks.

So, in this commit we:
* make a new koji channel called 'osbuild' with all the buildhw-x86's in
  it. They are usually not too overloaded and there are 16 of them so it
  should be available all the time.
* Leave the cron job on all builders for now in case, but make them only
  update once a day since they won't be getting jobs. If this works out
  we can remove it entirely there.
* Make the buildhw-x86s only update every 5min. This opens a larger
  window for it being wrong, but it's still pretty small and should
  reduce the number of emails for stalled processes we get.

See https://pagure.io/fedora-infrastructure/issue/10982

Signed-off-by: Kevin Fenzi <kevin@scrye.com>

roles/koji_builder/files/osbuildapi-update.cron

@@ -1 +0,0 @@
-* * * * * root /usr/local/bin/lock-wrapper osbuildapi "/usr/local/bin/osbuildapi-update.sh" 2>&1 | /usr/local/bin/nag-once osbuildapi-update.sh 1d 2>&1

roles/koji_builder/tasks/main.yml

@@ -366,7 +366,7 @@
   - koji_builder/osbuildapi
 
 - name: install cron job to run osbuild api ip update script every minute.
-  copy: src=osbuildapi-update.cron dest=/etc/cron.d/osbuildapi-update.cron mode=644
+  template: src=osbuildapi-update.cron dest=/etc/cron.d/osbuildapi-update.cron mode=644
   tags:
   - koji_builder
   - koji_builder/osbuildapi

roles/koji_builder/templates/osbuildapi-update.cron

@@ -0,0 +1,5 @@
+{% if inventory_hostname.startswith('buildhw-x86') }
+*/5 * * * * root /usr/local/bin/lock-wrapper osbuildapi "/usr/local/bin/osbuildapi-update.sh" 2>&1 | /usr/local/bin/nag-once osbuildapi-update.sh 1d 2>&1
+{% else }
+23 17 * * * root /usr/local/bin/lock-wrapper osbuildapi "/usr/local/bin/osbuildapi-update.sh" 2>&1 | /usr/local/bin/nag-once osbuildapi-update.sh 1d 2>&1
+{% endif %}

roles/koji_hub/templates/hub.conf.j2

@@ -119,6 +119,7 @@ tag =
     all :: allow
 
 channel =
+    method osbuildImage :: use osbuild
     method newRepo distRepo buildSRPMFromSCM :: use createrepo
     method buildContainer :: use container
     has req_channel && has_perm customchannel :: req