module-build-service (mbs): retire service

With the EOL of Fedora 38 yesterday, we are no longer building any
modules and can retire our module build service.

Note that toddlers needs to be adjusted still, that will happen after
this.

Thanks for all the modules!

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This commit is contained in:
Kevin Fenzi 2024-05-22 13:38:53 -07:00
parent a7bdb31bfc
commit d366194a22
60 changed files with 3 additions and 1654 deletions

View file

@ -181,19 +181,6 @@
- name: restart chronyd
service: name=chronyd state=restarted
- name: restart mbs poller
systemd:
name: mbs-poller
state: restarted
when: not mbs_frontend
- name: restart mbs workers
systemd:
name: "mbs-worker@{{ item }}"
state: restarted
with_sequence: start=0 end={{ mbs_num_workers - 1 }}
when: not mbs_frontend
- name: restart kojira
systemd:
name: kojira

View file

@ -53,7 +53,6 @@ ipa_client_shell_groups:
- sysadmin-fedimg
- sysadmin-koschei
- sysadmin-libravatar
- sysadmin-mbs
- sysadmin-messaging
- sysadmin-noc
- sysadmin-odcs

View file

@ -1,12 +0,0 @@
---
ipa_client_shell_groups:
- sysadmin-noc
- sysadmin-releng
- sysadmin-mbs
- sysadmin-veteran
ipa_client_sudo_groups:
- sysadmin-releng
- sysadmin-mbs
ipa_host_group: mbs
ipa_host_group_desc: Modular Build Service hosts
primary_auth_source: ipa

View file

@ -1,36 +0,0 @@
---
csi_primary_contact: Modularity WG - modularity-wg-members@fedoraproject.org
csi_purpose: Run the module-build-service fedmsg-hub backend (the scheduler)
csi_relationship: |
The fedmsg-hub process running here is responsible for scheduling all rpm
builds in koji in response to requests submitted to the MBS API on the
mbs-frontend nodes.
NOTE - this system has a KRB service principal with elevated koji privileges.
# For the MOTD
csi_security_category: High
# These are consumed by a task in roles/fedmsg/base/main.yml
fedmsg_certs:
- can_send:
- mbs.module.state.change
- mbs.component.state.change
group: fedmsg
owner: root
service: mbs
# These people get told when something goes wrong.
fedmsg_error_recipients:
- ralph@fedoraproject.org
- jkaluza@fedoraproject.org
- fivaldi@fedoraproject.org
# Wait a little bit longer than usual.. I'm not seeing messages from mbs backend
fedmsg_post_init_sleep: 1.5
lvm_size: 20000
mbs_broker_url: "amqps://mbs-private-queue{{ env_suffix }}@rabbitmq{{ env_suffix }}.fedoraproject.org//mbs-private-queue"
mbs_frontend: false
mbs_num_workers: 3
mbs_systemd_wait_for_rabbitmq: true
mem_size: 16384
num_cpus: 2
# for systems that do not match the above - specify the same parameter in
# the host_vars/$hostname file
tcp_ports: [3000, 3001, 3002, 3003, 3004, 3005, 3006, 3007]

View file

@ -1,34 +0,0 @@
---
csi_primary_contact: Modularity WG - modularity-wg-members@fedoraproject.org
csi_purpose: Run the module-build-service fedmsg-hub backend (the scheduler)
csi_relationship: |
The fedmsg-hub process running here is responsible for scheduling all rpm
builds in koji in response to requests submitted to the MBS API on the
mbs-frontend nodes.
NOTE - this system has a KRB service principal with elevated koji privileges.
# For the MOTD
csi_security_category: High
# These are consumed by a task in roles/fedmsg/base/main.yml
fedmsg_certs:
- can_send:
- mbs.module.state.change
- mbs.component.state.change
group: fedmsg
owner: root
service: mbs
# These people get told when something goes wrong.
fedmsg_error_recipients:
- ralph@fedoraproject.org
- jkaluza@fedoraproject.org
- fivaldi@fedoraproject.org
lvm_size: 20000
mbs_broker_url: "amqps://mbs-private-queue{{ env_suffix }}@rabbitmq{{ env_suffix }}.fedoraproject.org//mbs-private-queue"
mbs_frontend: false
mbs_num_workers: 3
mbs_systemd_wait_for_rabbitmq: true
mem_size: 4096
num_cpus: 1
# for systems that do not match the above - specify the same parameter in
# the host_vars/$hostname file
tcp_ports: [3000, 3001, 3002, 3003, 3004, 3005, 3006, 3007]

View file

@ -1,39 +0,0 @@
---
csi_primary_contact: Modularity WG - modularity-wg-members@fedoraproject.org
csi_purpose: Run the module-build-service frontend API.
csi_relationship: |
The apache/mod_wsgi app is the only thing really running here
This host relies on db01 for its database of activity (what module builds
are in flight?)
It has no special credentials itself. When a module build it submitted, it
makes a note in the DB and publishes a fedmsg message. The mbs backend
nodes do all the work of talking to koji.
# For the MOTD
csi_security_category: Moderate
# Neeed for rsync from log01 for logs.
custom_rules: ['-A INPUT -p tcp -m tcp -s 10.3.163.39 --dport 873 -j ACCEPT', '-A INPUT -p tcp -m tcp -s 192.168.1.59 --dport 873 -j ACCEPT']
# These are consumed by a task in roles/fedmsg/base/main.yml
fedmsg_certs:
- can_send:
- mbs.module.state.change
# Only the backend sends this message..
#- mbs.component.state.change
group: fedmsg
owner: fedmsg
service: mbs
lvm_size: 20000
mbs_broker_url: ""
mbs_frontend: true
mbs_num_workers: 3
mem_size: 4096
num_cpus: 2
tcp_ports: [80]
# Definining these vars has a number of effects
# 1) mod_wsgi is configured to use the vars for its own setup
# 2) iptables opens enough ports for all threads for fedmsg
# 3) roles/fedmsg/base/ declares enough fedmsg endpoints for all threads
wsgi_fedmsg_service: mbs
wsgi_procs: 2
wsgi_threads: 2

View file

@ -1,39 +0,0 @@
---
csi_primary_contact: Modularity WG - modularity-wg-members@fedoraproject.org
csi_purpose: Run the module-build-service frontend API.
csi_relationship: |
The apache/mod_wsgi app is the only thing really running here
This host relies on db01 for its database of activity (what module builds
are in flight?)
It has no special credentials itself. When a module build it submitted, it
makes a note in the DB and publishes a fedmsg message. The mbs backend
nodes do all the work of talking to koji.
# For the MOTD
csi_security_category: Moderate
# Neeed for rsync from log01 for logs.
custom_rules: ['-A INPUT -p tcp -m tcp -s 10.3.163.39 --dport 873 -j ACCEPT', '-A INPUT -p tcp -m tcp -s 192.168.1.59 --dport 873 -j ACCEPT']
# These are consumed by a task in roles/fedmsg/base/main.yml
fedmsg_certs:
- can_send:
- mbs.module.state.change
# Only the backend sends this message..
#- mbs.component.state.change
group: fedmsg
owner: fedmsg
service: mbs
lvm_size: 20000
mbs_broker_url: ""
mbs_frontend: true
mbs_num_workers: 3
mem_size: 4096
num_cpus: 1
tcp_ports: [80]
# Definining these vars has a number of effects
# 1) mod_wsgi is configured to use the vars for its own setup
# 2) iptables opens enough ports for all threads for fedmsg
# 3) roles/fedmsg/base/ declares enough fedmsg endpoints for all threads
wsgi_fedmsg_service: mbs
wsgi_procs: 2
wsgi_threads: 2

View file

@ -1,11 +0,0 @@
---
ipa_client_shell_groups:
- sysadmin-noc
- sysadmin-releng
- sysadmin-mbs
- sysadmin-veteran
ipa_client_sudo_groups:
- sysadmin-releng
- sysadmin-mbs
ipa_host_group: mbs
ipa_host_group_desc: Modular Build Service hosts

View file

@ -16,12 +16,10 @@ fedmsg_certs:
owner: root
service: pdc
ipa_client_shell_groups:
- sysadmin-mbs
- sysadmin-noc
- sysadmin-releng
- sysadmin-veteran
ipa_client_sudo_groups:
- sysadmin-mbs
- sysadmin-releng
ipa_host_group: pdc-web
ipa_host_group_desc: Product Definition Center web app

View file

@ -16,12 +16,10 @@ fedmsg_certs:
owner: root
service: pdc
ipa_client_shell_groups:
- sysadmin-mbs
- sysadmin-noc
- sysadmin-releng
- sysadmin-veteran
ipa_client_sudo_groups:
- sysadmin-mbs
- sysadmin-releng
ipa_host_group: pdc-web
ipa_host_group_desc: Product Definition Center web app

View file

@ -14,7 +14,6 @@ databases:
- kerneltest
- koschei
- mailman
- mbs
- mirrormanager2
- notifications
- odcs
@ -38,7 +37,6 @@ dbs_to_backup:
- kerneltest
- koschei
- mailman
- mbs
- mirrormanager2
- notifications
- odcs

View file

@ -1,8 +0,0 @@
---
datacenter: iad2
eth0_ipv4_gw: 10.3.169.254
eth0_ipv4_ip: 10.3.169.108
ks_repo: http://10.3.163.35/repo/rhel/RHEL7-x86_64/
ks_url: http://10.3.163.35/repo/rhel/ks/kvm-rhel-7-iad2
vmhost: bvmhost-x86-03.iad2.fedoraproject.org
volgroup: /dev/vg_guests

View file

@ -1,8 +0,0 @@
---
datacenter: iad2
eth0_ipv4_gw: 10.3.167.254
eth0_ipv4_ip: 10.3.167.30
ks_repo: http://10.3.163.35/repo/rhel/RHEL7-x86_64/
ks_url: http://10.3.163.35/repo/rhel/ks/kvm-rhel-7-iad2
vmhost: bvmhost-x86-01.stg.iad2.fedoraproject.org
volgroup: /dev/vg_guests

View file

@ -1,8 +0,0 @@
---
datacenter: iad2
eth0_ipv4_gw: 10.3.169.254
eth0_ipv4_ip: 10.3.169.109
ks_repo: http://10.3.163.35/repo/rhel/RHEL7-x86_64/
ks_url: http://10.3.163.35/repo/rhel/ks/kvm-rhel-7-iad2
vmhost: bvmhost-x86-04.iad2.fedoraproject.org
volgroup: /dev/vg_guests

View file

@ -1,8 +0,0 @@
---
datacenter: iad2
eth0_ipv4_gw: 10.3.167.254
eth0_ipv4_ip: 10.3.167.31
ks_repo: http://10.3.163.35/repo/rhel/RHEL7-x86_64/
ks_url: http://10.3.163.35/repo/rhel/ks/kvm-rhel-7-iad2
vmhost: bvmhost-x86-03.stg.iad2.fedoraproject.org
volgroup: /dev/vg_guests

View file

@ -112,26 +112,6 @@ mailman01.iad2.fedoraproject.org
[mailman_stg]
mailman01.stg.iad2.fedoraproject.org
[mbs_frontend]
mbs-frontend01.iad2.fedoraproject.org
[mbs_frontend_stg]
mbs-frontend01.stg.iad2.fedoraproject.org
[mbs_backend]
mbs-backend01.iad2.fedoraproject.org
[mbs_backend_stg]
mbs-backend01.stg.iad2.fedoraproject.org
[mbs:children]
mbs_frontend
mbs_backend
[mbs_stg:children]
mbs_frontend_stg
mbs_backend_stg
[bodhi_backend]
# This one handles the mashing/releng stuff
bodhi-backend01.iad2.fedoraproject.org
@ -636,8 +616,6 @@ ipa03.stg.iad2.fedoraproject.org
ipsilon01.stg.iad2.fedoraproject.org
koji01.stg.iad2.fedoraproject.org
mailman01.stg.iad2.fedoraproject.org
mbs-backend01.stg.iad2.fedoraproject.org
mbs-frontend01.stg.iad2.fedoraproject.org
memcached01.stg.iad2.fedoraproject.org
mm-backend01.stg.iad2.fedoraproject.org
mm-crawler01.stg.iad2.fedoraproject.org
@ -756,13 +734,11 @@ wiki02.iad2.fedoraproject.org
[fedmsg_hubs:children]
busgateway
fedimg
mbs_backend
pkgs
[fedmsg_hubs_stg:children]
busgateway_stg
fedimg_stg
mbs_backend_stg
pkgs_stg
[fedmsg_ircs:children]
@ -1116,7 +1092,6 @@ koji
kojipkgs
logging
mailman
mbs
memcached
mm
nagios_iad2
@ -1153,7 +1128,6 @@ github2fedmsg_stg
ipa_stg
ipsilon_stg
koji_stg
mbs_stg
memcached_stg
mm_stg
oci_registry_stg

View file

@ -41,7 +41,6 @@
- import_playbook: /srv/web/infra/ansible/playbooks/groups/mailman.yml
- import_playbook: /srv/web/infra/ansible/playbooks/groups/maintainer-test.yml
- import_playbook: /srv/web/infra/ansible/playbooks/groups/mariadb-server.yml
- import_playbook: /srv/web/infra/ansible/playbooks/groups/mbs.yml
- import_playbook: /srv/web/infra/ansible/playbooks/groups/memcached.yml
- import_playbook: /srv/web/infra/ansible/playbooks/groups/mirrormanager.yml
- import_playbook: /srv/web/infra/ansible/playbooks/groups/nfs-servers.yml

View file

@ -1,122 +0,0 @@
- import_playbook: "/srv/web/infra/ansible/playbooks/include/virt-create.yml"
vars:
myhosts: "mbs:mbs_stg"
- name: make the box be real
hosts: mbs:mbs_stg
user: root
gather_facts: True
vars_files:
- /srv/web/infra/ansible/vars/global.yml
- "/srv/private/ansible/vars.yml"
- /srv/web/infra/ansible/vars/{{ ansible_distribution }}.yml
pre_tasks:
- import_tasks: "{{ tasks_path }}/yumrepos.yml"
roles:
- base
- rkhunter
- nagios_client
- zabbix/zabbix_agent
- hosts
# openvpn on the prod frontend nodes
- {role: openvpn/client, when: "'mbs_frontend' in group_names and datacenter == 'iad2'"}
- ipa/client
- rsyncd
- sudo
- collectd/base
tasks:
- import_tasks: "{{ tasks_path }}/motd.yml"
handlers:
- import_tasks: "{{ handlers_path }}/restart_services.yml"
- name: Set up apache on the frontend MBS API app
hosts: mbs_frontend:mbs_frontend_stg
user: root
gather_facts: True
vars_files:
- /srv/web/infra/ansible/vars/global.yml
- "/srv/private/ansible/vars.yml"
- "{{ vars_path }}/{{ ansible_distribution }}.yml"
roles:
- mod_wsgi
handlers:
- import_tasks: "{{ handlers_path }}/restart_services.yml"
- name: set up fedmsg configuration and common mbs files
hosts: mbs:mbs_stg
user: root
gather_facts: True
vars_files:
- /srv/web/infra/ansible/vars/global.yml
- "/srv/private/ansible/vars.yml"
- /srv/web/infra/ansible/vars/{{ ansible_distribution }}.yml
roles:
- fedmsg/base
- mbs/common
handlers:
- import_tasks: "{{ handlers_path }}/restart_services.yml"
- name: deploy the frontend MBS API app
hosts: mbs_frontend:mbs_frontend_stg
user: root
gather_facts: True
vars_files:
- /srv/web/infra/ansible/vars/global.yml
- "/srv/private/ansible/vars.yml"
- "{{ vars_path }}/{{ ansible_distribution }}.yml"
roles:
- mbs/frontend
post_tasks:
# Shouldn't be necessary after this change makes it out
# https://src.fedoraproject.org/rpms/module-build-service/c/d19515a7c053aa90cddccd5e10a5615b773a7bd2
- name: Make sure fedmsg-hub isn't running on the frontend.
service:
name: fedmsg-hub
state: stopped
enabled: false
tags:
- mbs
- mbs/frontend
handlers:
- import_tasks: "{{ handlers_path }}/restart_services.yml"
- name: deploy the backend MBS scheduler daemon
hosts: mbs_backend:mbs_backend_stg
user: root
gather_facts: True
vars_files:
- /srv/web/infra/ansible/vars/global.yml
- "/srv/private/ansible/vars.yml"
- "{{ vars_path }}/{{ ansible_distribution }}.yml"
roles:
- role: keytab/service
service: mbs
owner_user: fedmsg
host: "mbs{{env_suffix}}.fedoraproject.org"
- role: fedmsg/hub
tags: fedmsg/hub
- role: collectd/fedmsg-service
process: fedmsg-hub
# Amazingly, there isn't need for a mbs/backend role. The fedmsg/hub role
# along with mbs/common is enough.
#- mbs/backend
handlers:
- import_tasks: "{{ handlers_path }}/restart_services.yml"

View file

@ -653,11 +653,6 @@
proxyurl: http://localhost:10051
when: env == "staging"
- role: httpd/reverseproxy
website: mbs.fedoraproject.org
destname: mbs
proxyurl: http://localhost:10063
- role: httpd/reverseproxy
website: koji.fedoraproject.org
destname: koji

View file

@ -924,12 +924,6 @@
tags: zabbix
when: env == "staging"
- role: httpd/website
site_name: mbs.fedoraproject.org
sslonly: true
server_aliases: [mbs.stg.fedoraproject.org]
cert_name: "{{wildcard_cert_name}}"
- role: httpd/website
site_name: odcs.fedoraproject.org
sslonly: true

View file

@ -1,5 +1,5 @@
- name: Uninstall IPA client
hosts: bodhi_backend_stg:bugzilla2fedmsg_stg:github2fedmsg_stg:ipsilon_stg:mbs_stg:buildvm_stg:buildvm_ppc64le_stg:buildvm_aarch64_stg:buildvm_armv7_stg:buildvm_s390x_stg
hosts: bodhi_backend_stg:bugzilla2fedmsg_stg:github2fedmsg_stg:ipsilon_stg:buildvm_stg:buildvm_ppc64le_stg:buildvm_aarch64_stg:buildvm_armv7_stg:buildvm_s390x_stg
user: root
vars_files:
- /srv/web/infra/ansible/vars/global.yml
@ -15,7 +15,6 @@
- import_playbook: "/srv/web/infra/ansible/playbooks/groups/bugzilla2fedmsg.yml"
- import_playbook: "/srv/web/infra/ansible/playbooks/groups/github2fedmsg.yml"
- import_playbook: "/srv/web/infra/ansible/playbooks/groups/ipsilon.yml"
- import_playbook: "/srv/web/infra/ansible/playbooks/groups/mbs.yml"
- import_playbook: "/srv/web/infra/ansible/playbooks/groups/buildvm.yml"

View file

@ -48,8 +48,6 @@
arches: s390x
# Users allowed to use content generators, only in staging
- cg_users:
- user_name: mbs/mbs.stg.fedoraproject.org
cg_name: module-build-service
- user_name: obudai
cg_name: osbuild

View file

@ -103,7 +103,7 @@ insert into host_channels (host_id, channel_id, creator_id) values (
-- Add some people to be admins, only in staging. Feel free to grow this list..
{% for username in ['modularity', 'mizdebsk', 'psabata', 'jkaluza', 'fivaldi', 'mprahl', 'mbs/mbs.stg.fedoraproject.org'] %}
{% for username in ['mizdebsk', 'psabata', 'jkaluza', 'fivaldi'] %}
select now() as time, 'adding staging admin {{username}}' as msg;
insert into users (name, usertype, status) values ('{{username}}', 0, 0) on conflict do nothing;
insert into user_perms (user_id, perm_id, active, creator_id) values (

View file

@ -1,158 +0,0 @@
- name: push packages out to frontend
hosts: mbs_frontend:mbs_frontend_stg
user: root
vars_files:
- /srv/web/infra/ansible/vars/global.yml
- "/srv/private/ansible/vars.yml"
- /srv/web/infra/ansible/vars/{{ ansible_distribution }}.yml
vars:
testing: False
handlers:
- import_tasks: "{{ handlers_path }}/restart_services.yml"
tasks:
- name: clean all metadata {%if testing%}(with infrastructure-testing on){%endif%}
command: yum clean all {%if testing%} --enablerepo=infrastructure-tags-stg {%endif%}
check_mode: no
- name: update mbs packages from main repo
package:
name:
- module-build-service
- python2-solv
state: latest
when: not testing
- name: update mbs packages from testing repo
yum:
name:
- module-build-service
- python2-solv
state: latest
enablerepo: infrastructure-tags-stg
when: testing
- name: push packages out to backend
hosts: mbs_backend:mbs_backend_stg
user: root
vars_files:
- /srv/web/infra/ansible/vars/global.yml
- "/srv/private/ansible/vars.yml"
- /srv/web/infra/ansible/vars/{{ ansible_distribution }}.yml
vars:
testing: False
handlers:
- import_tasks: "{{ handlers_path }}/restart_services.yml"
tasks:
- name: clean all metadata {%if testing%}(with infrastructure-testing on){%endif%}
command: yum clean all {%if testing%} --enablerepo=infrastructure-tags-stg {%endif%}
check_mode: no
- name: update mbs packages from main repo
package:
name:
- module-build-service
- python2-solv
state: latest
when: not testing
- name: update mbs packages from testing repo
yum:
name:
- module-build-service
- python2-solv
state: latest
enablerepo: infrastructure-tags-stg
when: testing
- name: verify the frontend and stop it
hosts: mbs_frontend:mbs_frontend_stg
user: root
vars_files:
- /srv/web/infra/ansible/vars/global.yml
- "/srv/private/ansible/vars.yml"
- /srv/web/infra/ansible/vars/{{ ansible_distribution }}.yml
handlers:
- import_tasks: "{{ handlers_path }}/restart_services.yml"
vars:
mbs_import_default_modules: False
pre_tasks:
- name: tell nagios to shush w.r.t. the frontend
nagios: action=downtime minutes=15 service=host host={{ inventory_hostname_short }}{{ env_suffix }}
delegate_to: noc01.iad2.fedoraproject.org
ignore_errors: true
roles:
- mbs/common
- mbs/frontend
post_tasks:
- service: name="httpd" state=stopped
- name: verify the backend, stop it, and then upgrade the db
hosts: mbs_backend:mbs_backend_stg
user: root
vars_files:
- /srv/web/infra/ansible/vars/global.yml
- "/srv/private/ansible/vars.yml"
- /srv/web/infra/ansible/vars/{{ ansible_distribution }}.yml
handlers:
- import_tasks: "{{ handlers_path }}/restart_services.yml"
vars:
mbs_import_default_modules: False
pre_tasks:
- name: tell nagios to shush w.r.t. the backend
nagios: action=downtime minutes=15 service=host host={{ inventory_hostname_short }}{{ env_suffix }}
delegate_to: noc01.iad2.fedoraproject.org
ignore_errors: true
roles:
- mbs/common
#- mbs/backend
tasks:
- name: Stop the mbs backend
service: name="fedmsg-hub" state=stopped
- name: Upgrade the database
command: mbs-upgradedb
ignore_errors: true
- name: And... start the backend again
service: name="fedmsg-hub" state=started
- name: Import the default-modules
command: /usr/bin/mbs-manager import_module /etc/module-build-service/default-modules/{{ item | basename }}
with_fileglob:
- "{{ playbook_dir }}/../../../roles/mbs/common/files/default-modules.{{ env }}/*.yaml"
post_tasks:
- name: tell nagios to unshush w.r.t. the backend
nagios: action=unsilence service=host host={{ inventory_hostname_short }}{{ env_suffix }}
delegate_to: noc01.iad2.fedoraproject.org
ignore_errors: true
- name: restart the frontend
hosts: mbs_frontend:mbs_frontend_stg
user: root
vars_files:
- /srv/web/infra/ansible/vars/global.yml
- "/srv/private/ansible/vars.yml"
- /srv/web/infra/ansible/vars/{{ ansible_distribution }}.yml
handlers:
- import_tasks: "{{ handlers_path }}/restart_services.yml"
tasks:
- service: name="httpd" state=started
# Shouldn't be necessary after this change makes it out
# https://src.fedoraproject.org/rpms/module-build-service/c/d19515a7c053aa90cddccd5e10a5615b773a7bd2
- name: Make sure fedmsg-hub isn't running on the frontend.
service:
name: fedmsg-hub
state: stopped
enabled: false
post_tasks:
- name: tell nagios to unshush w.r.t. the frontend
nagios: action=unsilence service=host host={{ inventory_hostname_short }}{{ env_suffix }}
delegate_to: noc01.iad2.fedoraproject.org
ignore_errors: true

View file

@ -36,16 +36,6 @@
# Setup for fedora-messaging
- role: rabbit/queue
username: "mts{{ env_suffix }}"
queue_name: "mts{{ env_suffix }}"
routing_keys:
- "org.fedoraproject.*.mbs.module.state.change"
thresholds:
warning: 10
critical: 100
sent_topics: ^org\.fedoraproject\.{{ env_short }}\.build\.tag\..*
# cacert, certificate and private key for fedora-messaging
- role: openshift/secret-file

View file

@ -96,7 +96,6 @@
- endpoints.py
- endpoints-anitya.py
- endpoints-fedbadges.py
- endpoints-mbs-backend.py
- endpoints-hotness.py
- endpoints-mailman.py
- endpoints-fedimg.py

View file

@ -1,25 +0,0 @@
{% if datacenter == 'iad2' %}
{% if env == 'staging' %}
suffix = 'stg.iad2.fedoraproject.org'
{% else %}
suffix = 'iad2.fedoraproject.org'
vpn_suffix = 'vpn.fedoraproject.org'
{% endif %}
{% else %}
{% if env == 'staging' %}
suffix = 'stg.fedoraproject.org'
{% else %}
suffix = 'fedoraproject.org'
vpn_suffix = 'vpn.fedoraproject.org'
{% endif %}
{% endif %}
config = dict(
endpoints={
"mbs.mbs-backend01": [
"tcp://mbs-backend01.%s:30%0.2i" % (suffix, i)
for i in range(8)
],
},
)

View file

@ -351,18 +351,6 @@ backend kojipkgs-backend
option httpchk GET /
{% endif %}
frontend mbs-frontend
bind 0.0.0.0:10063
default_backend mbs-backend
backend mbs-backend
balance hdr(appserver)
server mbs-frontend01 mbs-frontend01:80 check inter 20s rise 2 fall 3
{% if env == "production" %}
server mbs-frontend02 mbs-frontend02:80 check inter 20s rise 2 fall 3
{% endif %}
option httpchk GET /module-build-service/1/component-builds/
frontend odcs-frontend
bind 0.0.0.0:10066
default_backend odcs-backend

View file

@ -13,7 +13,7 @@ global enabled=allow
[provider_config]
global enabled=openid,saml2,openidc
openidc enabled extensions=fedora-account,mbs,beaker,waiverdb,odcs,wiki,src,kerneltest
openidc enabled extensions=fedora-account,beaker,waiverdb,odcs,wiki,src,kerneltest
{% if env == 'staging' %}
openidc subject salt={{ ipsilon_stg_openidc_subject_salt }}

View file

@ -97,8 +97,6 @@ Plugins = osbuild koji-fedoramessaging runroot_hub hub_containerbuild tag2distre
tag =
# We don't want to allow any draft builds to be tagged yet
is_draft :: deny
user mbs/mbs.fedoraproject.org && tag module-* && package kernel shim grub2 pesign fwupd fwupd-efi :: allow
user mbs/mbs.fedoraproject.org && fromtag module-* && package kernel shim grub2 pesign fwupd fwupd-efi:: allow
user bodhi && tag *-override && package kernel shim grub2 pesign fwupd fwupd-efi :: allow
has_perm autosign && fromtag *-pending && package kernel shim grub2 pesign fwupd fwupd-efi :: allow
has_perm autosign && fromtag *-candidate && package kernel shim grub2 pesign fwupd fwupd-efi :: allow

View file

@ -1,4 +0,0 @@
---
mbs_broker_url: ""
mbs_systemd_wait_for_rabbitmq: false
mbs_celery_max_worker_tasks: 50

View file

@ -1,28 +0,0 @@
data:
description: ELN base
license:
module: [MIT]
name: platform
profiles:
buildroot:
rpms: [bash, bzip2, coreutils, cpio, diffutils, fedora-release, findutils, gawk,
glibc-minimal-langpack, grep, gzip, info, make, patch, redhat-rpm-config,
rpm-build, sed, shadow-utils, tar, unzip, util-linux, which, xz]
srpm-buildroot:
rpms: [bash, fedora-release, fedpkg-minimal, glibc-minimal-langpack, gnupg2,
redhat-rpm-config, rpm-build, shadow-utils]
stream: eln
summary: ELN base
context: 00000000
version: 1
xmd:
mbs:
buildrequires: {}
commit: eln
requires: {}
koji_tag: module-eln-build
mse: TRUE
default_modules_scm_url: https://pagure.io/releng/fedora-module-defaults.git
document: modulemd
version: 1

View file

@ -1,27 +0,0 @@
data:
description: Fedora 39 traditional base
license:
module: [MIT]
name: platform
profiles:
buildroot:
rpms: [bash, bzip2, coreutils, cpio, diffutils, fedora-release, findutils, gawk,
glibc-minimal-langpack, grep, gzip, info, make, patch, redhat-rpm-config,
rpm-build, sed, shadow-utils, tar, unzip, util-linux, which, xz]
srpm-buildroot:
rpms: [bash, fedora-release, fedpkg-minimal, glibc-minimal-langpack, gnupg2,
redhat-rpm-config, rpm-build, shadow-utils]
stream: f39
summary: Fedora 39 traditional base
context: 00000000
version: 1
xmd:
mbs:
buildrequires: {}
commit: f39
requires: {}
koji_tag: module-f39-build
mse: TRUE
virtual_streams: [fedora]
document: modulemd
version: 1

View file

@ -1,25 +0,0 @@
document: modulemd
version: 1
data:
name: platform
stream: el8_playground
version: 1
context: 00000000
summary: EPEL 8 playground base
description: EPEL 8 playground base
license:
module: [MIT]
profiles:
buildroot:
rpms: [bash, bzip2, coreutils, cpio, diffutils, epel-release, epel-rpm-macros, fedpkg-minimal, findutils, gawk, gcc, gcc-c++, grep, gzip, info, make, patch, redhat-release, redhat-release-everything, redhat-release-server, redhat-rpm-config, rpm-build, sed, shadow-utils, tar, unzip, util-linux, util-linux-ng, which, xz]
srpm-buildroot:
rpms: [bash, epel-release, epel-rpm-macros, fedpkg-minimal, git, gnupg, make, redhat-release, redhat-release-everything, redhat-release-server, redhat-rpm-config, rpm-build, shadow-utils]
xmd:
mbs:
buildrequires: {}
commit: el8
requires: {}
koji_tag: module-el8-playground-build
mse: TRUE
default_modules_scm_url: https://pagure.io/modularity/fedora-stg-module-defaults.git
use_default_modules: TRUE

View file

@ -1,28 +0,0 @@
data:
description: ELN base
license:
module: [MIT]
name: platform
profiles:
buildroot:
rpms: [bash, bzip2, coreutils, cpio, diffutils, fedora-release, findutils, gawk,
glibc-minimal-langpack, grep, gzip, info, make, patch, redhat-rpm-config,
rpm-build, sed, shadow-utils, tar, unzip, util-linux, which, xz]
srpm-buildroot:
rpms: [bash, fedora-release, fedpkg-minimal, glibc-minimal-langpack, gnupg2,
redhat-rpm-config, rpm-build, shadow-utils]
stream: eln
summary: ELN base
context: 00000000
version: 1
xmd:
mbs:
buildrequires: {}
commit: eln
requires: {}
koji_tag: module-eln-build
mse: TRUE
default_modules_scm_url: https://pagure.io/modularity/fedora-stg-module-defaults.git
document: modulemd
version: 1

View file

@ -1,28 +0,0 @@
data:
description: Fedora 29 traditional base
license:
module: [MIT]
name: platform
profiles:
buildroot:
rpms: [bash, bzip2, coreutils, cpio, diffutils, fedora-release, findutils, gawk,
grep, gzip, info, make, patch, redhat-rpm-config, rpm-build, sed, shadow-utils,
tar, unzip, util-linux, which, xz]
srpm-buildroot:
rpms: [bash, fedora-release, fedpkg-minimal, gnupg2, redhat-rpm-config, rpm-build,
shadow-utils]
stream: f29
summary: Fedora 29 traditional base
context: 00000000
version: 5
xmd:
mbs:
buildrequires: {}
commit: f29
requires: {}
koji_tag: module-f29-build
mse: TRUE
virtual_streams: [fedora]
document: modulemd
version: 1

View file

@ -1,28 +0,0 @@
data:
description: Fedora 30 traditional base
license:
module: [MIT]
name: platform
profiles:
buildroot:
rpms: [bash, bzip2, coreutils, cpio, diffutils, fedora-release, findutils, gawk,
grep, gzip, info, make, patch, redhat-rpm-config, rpm-build, sed, shadow-utils,
tar, unzip, util-linux, which, xz]
srpm-buildroot:
rpms: [bash, fedora-release, fedpkg-minimal, gnupg2, redhat-rpm-config, rpm-build,
shadow-utils]
stream: f30
summary: Fedora 30 traditional base
context: 00000000
version: 5
xmd:
mbs:
buildrequires: {}
commit: f30
requires: {}
koji_tag: module-f30-build
mse: TRUE
virtual_streams: [fedora]
document: modulemd
version: 1

View file

@ -1,30 +0,0 @@
data:
description: Fedora 31 traditional base
license:
module: [MIT]
name: platform
profiles:
buildroot:
rpms: [bash, bzip2, coreutils, cpio, diffutils, fedora-release, findutils, gawk,
glibc-minimal-langpack, grep, gzip, info, make, patch, redhat-rpm-config,
rpm-build, sed, shadow-utils, tar, unzip, util-linux, which, xz]
srpm-buildroot:
rpms: [bash, fedora-release, fedpkg-minimal, glibc-minimal-langpack, gnupg2,
redhat-rpm-config, rpm-build, shadow-utils]
stream: f31
summary: Fedora 31 traditional base
context: 00000000
version: 1
xmd:
mbs:
buildrequires: {}
commit: f31
requires: {}
koji_tag: module-f31-build
mse: TRUE
virtual_streams: [fedora]
default_modules_scm_url: https://pagure.io/modularity/fedora-stg-module-defaults.git
use_default_modules: TRUE
document: modulemd
version: 1

View file

@ -1,7 +0,0 @@
{
"fedora": {
"host": "https://pdc.fedoraproject.org/rest_api/v1/",
"develop": true,
"insecure": false
}
}

View file

@ -1,7 +0,0 @@
{
"fedora": {
"host": "https://pdc.stg.fedoraproject.org/rest_api/v1/",
"develop": true,
"insecure": false
}
}

View file

@ -1,268 +0,0 @@
---
# Common configuration for the Module Build Service (MBS) pieces
- name: install needed packages
package:
state: present
name:
- module-build-service
- python-psycopg2
- libsemanage-python
- python-memcached
- python2-distro
notify:
- restart apache
- restart fedmsg-hub
# - restart mbs poller
# - restart mbs workers
tags:
- mbs
- mbs/common
- name: kill development configs
file: path=/etc/fedmsg.d/{{ item }} state=absent
with_items:
- module_build_service.py
- mbs-logging.py
notify:
- restart apache
- restart fedmsg-hub
tags:
- mbs
- mbs/common
- name: copy app configuration
template: >
src=config.py dest=/etc/module-build-service/config.py
owner=root group=fedmsg mode=0640
notify:
- restart apache
- restart fedmsg-hub
# - restart mbs poller
# - restart mbs workers
tags:
- mbs
- mbs/common
- name: copy koji configuration
template: >
src=koji.conf dest=/etc/module-build-service/koji.conf
owner=root group=fedmsg mode=0644
notify:
- restart fedmsg-hub
tags:
- mbs
- mbs/common
- name: copy fedmsg configuration
template: >
src=mbs-fedmsg.py dest=/etc/fedmsg.d/mbs-fedmsg.py
owner=root group=fedmsg mode=0644
notify:
- restart apache
- restart fedmsg-hub
tags:
- mbs
- mbs/common
- name: copy client secrets
template: >
src=client_secrets.json.{{env}} dest=/etc/module-build-service/client_secrets.json
owner=root group=fedmsg mode=0640
when: inventory_hostname.startswith('mbs-frontend')
notify:
- restart apache
tags:
- mbs
- mbs/common
- name: create /var/cache/fedmsg/ directory for krb ccache
file:
path: /var/cache/fedmsg/
state: directory
owner: fedmsg
group: fedmsg
mode: 0750
tags:
- mbs
- mbs/common
- name: create /etc/pdc.d directory
file:
path: /etc/pdc.d
state: directory
owner: root
group: root
mode: 0775
- name: copy pdc client config file
copy: >
src=fedora.json.{{env}} dest=/etc/pdc.d/fedora.json
owner=root group=root mode=0644
notify:
- restart apache
- restart fedmsg-hub
tags:
- mbs
- mbs/common
- name: Configure MBS virtual host in RabbitMQ
block:
- name: copy the MBS rabbitmq private queue crt
copy:
src: "{{private}}/files/rabbitmq/{{env}}/pki/issued/mbs-private-queue{{env_suffix}}.crt"
dest: /etc/module-build-service/mbs-private-queue{{env_suffix}}.crt
owner: root
group: fedmsg
mode: 0640
tags:
- mbs
- mbs/common
- name: copy the MBS rabbitmq private queue key
copy:
src: "{{private}}/files/rabbitmq/{{env}}/pki/private/mbs-private-queue{{env_suffix}}.key"
dest: /etc/module-build-service/mbs-private-queue{{env_suffix}}.key
owner: root
group: fedmsg
mode: 0640
tags:
- mbs
- mbs/common
- name: copy the MBS rabbitmq CA cert
copy:
src: "{{private}}/files/rabbitmq/{{env}}/pki/ca.crt"
dest: /etc/module-build-service/ca.crt
owner: root
group: fedmsg
mode: 0640
tags:
- mbs
- mbs/common
- name: Configure the MBS virtual host
run_once: true
delegate_to: "rabbitmq01{{ env_suffix }}.iad2.fedoraproject.org"
rabbitmq_vhost:
name: /mbs
state: present
tags:
- rabbitmq_cluster
- config
- mbs
- mbs/common
- name: Configure the HA policy for the MBS queues
run_once: true
delegate_to: "rabbitmq01{{ env_suffix }}.iad2.fedoraproject.org"
rabbitmq_policy:
name: HA
apply_to: queues
pattern: .*
tags:
ha-mode: all
ha-sync-mode: automatic # Auto sync queues to new cluster members
ha-sync-batch-size: 10000 # Larger is faster, but must finish in 1 net_ticktime
vhost: /mbs
tags:
- rabbitmq_cluster
- config
- mbs
- mbs/common
- name: Grant the mbs user access to the MBS vhost
run_once: true
delegate_to: "rabbitmq01{{ env_suffix }}.iad2.fedoraproject.org"
rabbitmq_user:
user: "mbs{{ env_suffix }}"
vhost: /mbs
configure_priv: .*
read_priv: .*
write_priv: .*
tags:
- rabbitmq_cluster
- config
- mbs
- mbs/common
when: not mbs_frontend
- name: Configure the MBS workers and poller
block:
- name: Add the systemd service files
template:
src: "{{ item.src }}"
dest: "{{ item.dest }}"
owner: root
group: root
mode: "0644"
with_items:
- src: mbs-worker@.service.j2
dest: /etc/systemd/system/mbs-worker@.service
- src: mbs-poller.service.j2
dest: /etc/systemd/system/mbs-poller.service
notify:
- restart mbs poller
- restart mbs workers
tags:
- mbs
- mbs/common
- name: Enable the MBS workers
systemd:
name: "mbs-worker@{{ item }}"
daemon_reload: yes
enabled: yes
state: started
with_sequence: start=0 end={{ mbs_num_workers - 1 }}
tags:
- mbs
- mbs/common
- name: Populate the service facts to detect if there are MBS workers to disable
service_facts: {}
tags:
- mbs
- mbs/common
- name: Disable any extra MBS workers
systemd:
name: "mbs-worker@{{ worker_num }}"
enabled: no
state: stopped
with_items: "{{ ansible_facts.services | select('match', 'mbs-worker@\\d+.service') | list }}"
vars:
worker_num: "{{ item | regex_search('\\d+') }}"
when: (worker_num | int) >= mbs_num_workers
tags:
- mbs
- mbs/common
when: not mbs_frontend
- name: create /etc/module-build-service/default-modules directory
file:
path: /etc/module-build-service/default-modules
state: directory
owner: root
group: root
mode: 0775
tags:
- mbs
- mbs/common
- name: copy default modules to /etc/module-build-service/default-modules
copy: src={{ item }} dest=/etc/module-build-service/default-modules
with_fileglob:
- default-modules.{{ env }}/*.yaml
tags:
- mbs
- mbs/common
- name: import default-modules
command: /usr/bin/mbs-manager import_module /etc/module-build-service/default-modules/{{ item | basename }}
with_fileglob:
- default-modules.{{ env }}/*.yaml
when: mbs_import_default_modules | default(True)
tags:
- mbs
- mbs/common

View file

@ -1,11 +0,0 @@
{
"web": {
"auth_uri": "https://id.fedoraproject.org/openidc/Authorization",
"client_id": "mbs-prod",
"client_secret": "{{ mbs_prod_oidc_client_secret }}",
"redirect_uris": [],
"token_uri": "https://id.fedoraproject.org/openidc/Token",
"token_introspection_uri": "https://id.fedoraproject.org/openidc/TokenInfo",
"userinfo_uri": "https://id.fedoraproject.org/openidc/UserInfo"
}
}

View file

@ -1,11 +0,0 @@
{
"web": {
"auth_uri": "https://id.stg.fedoraproject.org/openidc/Authorization",
"client_id": "mbs-stg",
"client_secret": "{{ mbs_stg_oidc_client_secret }}",
"redirect_uris": [],
"token_uri": "https://id.stg.fedoraproject.org/openidc/Token",
"token_introspection_uri": "https://id.stg.fedoraproject.org/openidc/TokenInfo",
"userinfo_uri": "https://id.stg.fedoraproject.org/openidc/UserInfo"
}
}

View file

@ -1,262 +0,0 @@
from os import path
import ssl
# FIXME: workaround for this moment till confdir, dbdir (installdir etc.) are
# declared properly somewhere/somehow
confdir = path.abspath(path.dirname(__file__))
# use parent dir as dbdir else fallback to current dir
dbdir = path.abspath(path.join(confdir, '..')) if confdir.endswith('conf') \
else confdir
class BaseConfiguration(object):
DEBUG = False
# Make this random (used to generate session keys)
SECRET_KEY = '74d9e9f9cd40e66fc6c4c2e9987dce48df3ce98542529fd0'
SQLALCHEMY_DATABASE_URI = 'sqlite:///{0}'.format(path.join(
dbdir, 'module_build_service.db'))
SQLALCHEMY_TRACK_MODIFICATIONS = True
# Where we should run when running "manage.py runssl" directly.
HOST = '0.0.0.0'
PORT = 5000
CELERY_BROKER_URL = '{{ mbs_broker_url }}'
CELERY_BROKER_TRANSPORT_OPTIONS = {
"ssl":
{
'certfile': "/etc/module-build-service/mbs-private-queue{{env_suffix}}.crt",
'keyfile': "/etc/module-build-service/mbs-private-queue{{env_suffix}}.key",
'ca_certs': "/etc/module-build-service/ca.crt",
'cert_reqs': ssl.CERT_REQUIRED,
'ssl_version': ssl.PROTOCOL_TLSv1_2,
},
}
CELERY_BROKER_LOGIN_METHOD = "EXTERNAL"
# Global network-related values, in seconds
NET_TIMEOUT = 120
NET_RETRY_INTERVAL = 30
SYSTEM = 'koji'
MESSAGING = 'fedmsg' # or amq
MESSAGING_TOPIC_PREFIX = ['org.fedoraproject.prod']
KOJI_CONFIG = '/etc/module-build-service/koji.conf'
KOJI_PROFILE = 'koji'
ARCHES = ['i686', 'x86_64']
KOJI_PROXYUSER = True
KOJI_REPOSITORY_URL = 'https://kojipkgs.stg.fedoraproject.org/repos'
COPR_CONFIG = '/etc/module-build-service/copr.conf'
PDC_URL = 'http://modularity.fedorainfracloud.org:8080/rest_api/v1'
PDC_INSECURE = True
PDC_DEVELOP = True
SCMURLS = ["git+https://src.fedoraproject.org/modules/"]
RAWHIDE_BRANCH = 'rawhide'
# How often should we resort to polling, in seconds
# Set to zero to disable polling
POLLING_INTERVAL = 3600
RPMS_DEFAULT_REPOSITORY = 'git+https://src.fedoraproject.org/rpms/'
RPMS_ALLOW_REPOSITORY = False
RPMS_DEFAULT_CACHE = 'https://src.fedoraproject.org/repo/pkgs/'
RPMS_ALLOW_CACHE = False
MODULES_DEFAULT_REPOSITORY = 'git+https://src.fedoraproject.org/modules/'
MODULES_ALLOW_REPOSITORY = False
# Available backends are: console, file, journal.
LOG_BACKEND = 'journal'
# Path to log file when LOG_BACKEND is set to "file".
LOG_FILE = 'module_build_service.log'
# Available log levels are: debug, info, warn, error.
LOG_LEVEL = 'info'
# Settings for Kerberos
KRB_KEYTAB = None
KRB_PRINCIPAL = None
KRB_CCACHE = None
# Number of celery workers
NUM_WORKERS = {{ mbs_num_workers }}
# AMQ prefixed variables are required only while using 'amq' as messaging backend
# Addresses to listen to
AMQ_RECV_ADDRESSES = ['amqps://messaging.mydomain.com/Consumer.m8y.VirtualTopic.eng.koji',
'amqps://messaging.mydomain.com/Consumer.m8y.VirtualTopic.eng.module_build_service']
# Address for sending messages
AMQ_DEST_ADDRESS = 'amqps://messaging.mydomain.com/Consumer.m8y.VirtualTopic.eng.module_build_service'
AMQ_CERT_FILE = '/etc/module_build_service/msg-m8y-client.crt'
AMQ_PRIVATE_KEY_FILE = '/etc/module_build_service/msg-m8y-client.key'
AMQ_TRUSTED_CERT_FILE = '/etc/module_build_service/Root-CA.crt'
class ProdConfiguration(BaseConfiguration):
DEBUG = False # Don't turn this on.
# These groups are allowed to submit builds.
ALLOWED_GROUPS = [
# https://pagure.io/fesco/issue/1763
'packager',
]
# These groups are allowed to cancel the builds of other users.
ADMIN_GROUPS = [
'factory2',
'releng-team',
]
REBUILD_STRATEGY = 'only-changed'
REBUILD_STRATEGY_ALLOW_OVERRIDE = True
{% if env == 'staging' %}
SECRET_KEY = '{{ mbs_stg_secret_key }}'
SQLALCHEMY_DATABASE_URI = 'postgresql://mbs:{{mbs_stg_db_password}}@db-mbs/mbs'
{% else %}
SECRET_KEY = '{{ mbs_prod_secret_key }}'
SQLALCHEMY_DATABASE_URI = 'postgresql://mbs:{{mbs_prod_db_password}}@db-mbs/mbs'
{% endif %}
{% if env == 'staging' %}
KRB_PRINCIPAL = 'mbs/mbs.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG'
{% else %}
KRB_PRINCIPAL = 'mbs/mbs.fedoraproject.org@FEDORAPROJECT.ORG'
{% endif %}
KRB_KEYTAB = '/etc/krb5.mbs_mbs{{env_suffix}}.fedoraproject.org.keytab'
KRB_CCACHE = '/var/cache/fedmsg/mbs-krb5cc'
# https://pagure.io/fm-orchestrator/issue/334
KOJI_PROXYUSER = False
LOG_LEVEL = 'debug'
LOG_BACKEND = 'console'
# Our per-build logs for the koji-content generator go here.
# CG imports are controlled by KOJI_ENABLE_CONTENT_GENERATOR
BUILD_LOGS_DIR = '/var/tmp'
# Yes, use tls.
PDC_INSECURE = False
# No, don't try to obtain a token (we just read. we don't write.)
PDC_DEVELOP = True
KOJI_CONFIG = path.join(confdir, 'koji.conf')
{% if env == 'staging' %}
KOJI_PROFILE = 'staging'
ARCHES = ['aarch64', 'ppc64le', 's390x', 'x86_64']
BASE_MODULE_ARCHES = {
'platform:f31': ['aarch64', 'x86_64'],
}
KOJI_REPOSITORY_URL = 'https://kojipkgs.stg.fedoraproject.org/repos'
MESSAGING_TOPIC_PREFIX = ['org.fedoraproject.stg']
PDC_URL = 'https://pdc.stg.fedoraproject.org/rest_api/v1'
SCMURLS = ['git+https://src.stg.fedoraproject.org/modules/',
'https://src.stg.fedoraproject.org/modules/',
'https://src.stg.fedoraproject.org/git/modules/',
'git+https://src.stg.fedoraproject.org/flatpaks/',
'https://src.stg.fedoraproject.org/flatpaks/',
'https://src.stg.fedoraproject.org/git/flatpaks/']
RPMS_DEFAULT_REPOSITORY = 'git+https://src.stg.fedoraproject.org/rpms/'
RPMS_DEFAULT_CACHE = 'https://src.stg.fedoraproject.org/repo/pkgs/'
MODULES_DEFAULT_REPOSITORY = 'git+https://src.stg.fedoraproject.org/modules/'
{% else %}
KOJI_PROFILE = 'production'
ARCHES = ['aarch64', 'i686', 'ppc64le', 'x86_64', 's390x']
BASE_MODULE_ARCHES = {
# Fedora >= 37 removes armv7hl, Fedora < 37 still have it
# https://fedoraproject.org/wiki/Changes/RetireARMv7
'platform:f35': ['aarch64', 'armv7hl', 'i686', 'ppc64le', 'x86_64', 's390x'],
'platform:f36': ['aarch64', 'armv7hl', 'i686', 'ppc64le', 'x86_64', 's390x'],
'platform:el8' : ['aarch64', 'ppc64le', 'x86_64', 's390x']
}
KOJI_REPOSITORY_URL = 'https://kojipkgs.fedoraproject.org/repos'
MESSAGING_TOPIC_PREFIX = ['org.fedoraproject.prod']
PDC_URL = 'https://pdc.fedoraproject.org/rest_api/v1'
SCMURLS = ['git+https://src.fedoraproject.org/modules/',
'https://src.fedoraproject.org/modules/',
'https://src.fedoraproject.org/git/modules/',
'git+https://src.fedoraproject.org/flatpaks/',
'https://src.fedoraproject.org/flatpaks/',
'https://src.fedoraproject.org/git/flatpaks/']
{% endif %}
RESOLVER = "db"
# Made possible by https://pagure.io/releng/issue/6799
KOJI_ENABLE_CONTENT_GENERATOR = True
# See https://pagure.io/releng/issue/7012
BASE_MODULE_NAMES = set(['platform', 'bootstrap'])
KOJI_CG_BUILD_TAG_TEMPLATE = "{}-modular-updates-candidate"
KOJI_CG_DEFAULT_BUILD_TAG = "modular-updates-candidate"
# This is a whitelist of prefixes of koji tags we're allowed to manipulate
KOJI_TAG_PREFIXES = [
# This is our standard prefix. All module tags should start with this.
'module',
# Our very first manually bootstrapped tag has this name.
'f26-modularity',
# Scratch module builds have this prefix
'scrmod',
]
# Extra options set for newly created Koji tags
KOJI_TAG_EXTRA_OPTS = {
"mock.package_manager": "dnf",
# This is needed to include all the Koji builds (and therefore
# all the packages) from all inherited tags into this tag.
# See https://pagure.io/koji/issue/588 and
# https://pagure.io/fm-orchestrator/issue/660 for background.
"repo_include_all": True,
# Has been requested by Fedora infra in
# https://pagure.io/fedora-infrastructure/issue/7620.
# Disables systemd-nspawn for chroot.
"mock.new_chroot": 0,
# Works around fail-safe mechanism added in DNF 4.2.7
# https://pagure.io/fedora-infrastructure/issue/8410
"mock.yum.module_hotfixes": 1,
}
# If this is too long, we could change it to 'fm_' some day.
DEFAULT_DIST_TAG_PREFIX = 'module_'
# Delete module-* targets one hour after build
KOJI_TARGET_DELETE_TIME = 3600
# These aren't really secret.
OIDC_CLIENT_SECRETS = path.join(confdir, 'client_secrets.json')
OIDC_REQUIRED_SCOPE = 'https://mbs.fedoraproject.org/oidc/submit-build'
# yes, we want everyone to authenticate
NO_AUTH = False # Obviously.
# Don't let people submit yaml directly. it has to come from dist-git
YAML_SUBMIT_ALLOWED = False
# Relative Koji task priority (0 means default priority of 20).
KOJI_BUILD_PRIORITY = 0
# Check branch EOL before building. Block EOL modules from building.
# https://pagure.io/fm-orchestrator/issue/960
# Because of https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/KGXBMTUR72FHQEG7IBHDPPX276QHSD2I/#MFT5SAWPKMCNLKAWEJFCFIVX5GJ7RBSP
# we decided to hold on to this and ask the maintainers to create a
# releng ticket to retire their modules.
CHECK_FOR_EOL = False
# Koji Content Generator "-devel" modules aren't used in Fedora, so we can just disable them
KOJI_CG_DEVEL_MODULE = False
MODULES_ALLOW_SCRATCH = True
# By default, MBS allows buildrequiring only modules built against
# compatible version of platform base module. By compatible, we mean
# less or equal minor number of "stream_version". For example, when building module
# against platform:f30, it wouldn't be possible to buildrequire a module
# built against platform:f29. This is not intended behaviour in Fedora
# and therefore we want to turn this feature off.
ALLOW_ONLY_COMPATIBLE_BASE_MODULES = False

View file

@ -1,15 +0,0 @@
[production]
server = https://koji.fedoraproject.org/kojihub
weburl = https://koji.fedoraproject.org/koji
topurl = https://kojipkgs.fedoraproject.org/
authtype = kerberos
krb_rdns = false
use_fast_upload = true
[staging]
server = https://koji.stg.fedoraproject.org/kojihub
weburl = https://koji.stg.fedoraproject.org/koji
topurl = https://kojipkgs.stg.fedoraproject.org/
authtype = kerberos
krb_rdns = false
use_fast_upload = true

View file

@ -1,7 +0,0 @@
import socket
config = {
# So that the MBS can find it's cert in /etc/fedmsg.d/ssl.py
'cert_prefix': 'mbs',
'name': 'mbs.%s' % socket.gethostname().split('.', 1)[0],
}

View file

@ -1,14 +0,0 @@
[Unit]
Description=MBS Poller
After=network.target{{ ' rabbitmq-server.service' if mbs_systemd_wait_for_rabbitmq else '' }}
[Service]
Type=simple
# Always restart the service, even if it exits cleanly
Restart=always
RestartSec=1
User=fedmsg
ExecStart=/usr/bin/celery beat -A module_build_service.scheduler.celery_app --loglevel=info --pidfile /var/run/fedmsg/mbs-scheduler.pid -s /var/run/fedmsg/mbs-scheduler.db
[Install]
WantedBy=multi-user.target

View file

@ -1,14 +0,0 @@
[Unit]
Description=MBS worker %I
After=network.target{{ ' rabbitmq-server.service' if mbs_systemd_wait_for_rabbitmq else '' }}
[Service]
Type=simple
# Always restart the service, even if it exits cleanly
Restart=always
RestartSec=1
User=fedmsg
ExecStart=/usr/bin/celery worker -n mbs-worker-%I -Q mbs-default,mbs-%I -A module_build_service.scheduler.celery_app --loglevel=info --max-tasks-per-child={{ mbs_celery_max_worker_tasks }}
[Install]
WantedBy=multi-user.target

View file

@ -1,5 +0,0 @@
config = {
# The frontend should have these turned off in perpetuity.
'mbsconsumer': False,
'mbspoller': False,
}

View file

@ -1,6 +0,0 @@
#-*- coding: utf-8 -*-
import logging
logging.basicConfig(level='DEBUG')
from module_build_service import app as application

View file

@ -1,68 +0,0 @@
---
# Configuration for the Module Build Service (MBS) frontend webapp.
- name: disable the scheduler on the frontend
copy: >
src={{ item }} dest=/etc/fedmsg.d/{{ item }}
owner=fedmsg group=fedmsg mode=0644
with_items:
- mbs-scheduler.py
notify:
- restart apache
tags:
- mbs
- mbs/frontend
- name: Make sure fedmsg-hub isn't running on the frontend.
service:
name: fedmsg-hub
state: stopped
enabled: false
tags:
- mbs
- mbs/frontend
- name: copy mbs httpd config
template: >
src=mbs.conf dest=/etc/httpd/conf.d/mbs.conf
owner=apache group=apache mode=0644
notify:
- restart apache
tags:
- mbs
- mbs/frontend
- file: path=/usr/share/mbs/ state=directory
tags:
- mbs
- mbs/frontend
- name: copy custom wsgi file
copy: src=mbs.wsgi dest=/usr/share/mbs/mbs.wsgi mode=0644
notify:
- restart apache
tags:
- mbs
- mbs/frontend
- name: ensure selinux lets httpd talk to postgres, memcached, and mail
seboolean: name={{item}} state=yes persistent=yes
with_items:
- httpd_can_network_connect_db
- httpd_can_network_memcache
- httpd_can_network_connect
- httpd_can_sendmail
tags:
- mbs
- mbs/frontend
- selinux
- name: make httpd logs world readable
file:
name: /var/log/httpd
state: directory
mode: 0755
tags:
- mbs
- mbs/frontend

View file

@ -1,18 +0,0 @@
WSGIDaemonProcess mbs user=fedmsg group=fedmsg maximum-requests=1000 display-name=mbs processes={{ wsgi_procs }} threads={{ wsgi_threads }}
WSGISocketPrefix run/wsgi
WSGIRestrictStdout On
WSGIRestrictSignal Off
WSGIPythonOptimize 1
# For our Authorization bearer token header
WSGIPassAuthorization On
WSGIScriptAlias / /usr/share/mbs/mbs.wsgi
<Location />
WSGIProcessGroup mbs
Require all granted
</Location>
RewriteEngine on
RewriteRule ^(|/+)$ /module-build-service/1/module-builds/ [L,R=302]

View file

@ -13,7 +13,6 @@ command[check_fedmsg_cp_notifs_backend]={{libdir}}/nagios/plugins/check_fedmsg_p
command[check_fedmsg_cp_fedimg_backend]={{libdir}}/nagios/plugins/check_fedmsg_producers_consumers.py fedmsg-hub FedimgConsumer MonitoringProducer
command[check_fedmsg_cp_hotness_backend]={{libdir}}/nagios/plugins/check_fedmsg_producers_consumers.py fedmsg-hub BugzillaTicketFiler MonitoringProducer
command[check_fedmsg_cp_packages_backend]={{libdir}}/nagios/plugins/check_fedmsg_producers_consumers.py fedmsg-hub CacheInvalidator MonitoringProducer
command[check_fedmsg_cp_mbs_backend]={{libdir}}/nagios/plugins/check_fedmsg_producers_consumers.py fedmsg-hub MBSConsumer MonitoringProducer
command[check_fedmsg_cexceptions_busgateway_relay]={{libdir}}/nagios/plugins/check_fedmsg_consumer_exceptions.py fedmsg-relay RelayConsumer 1 10
{% if (ansible_distribution == 'RedHat' and ansible_distribution_major_version|int < 8) or (ansible_distribution_major_version|int < 30 and ansible_distribution == 'Fedora') %}
@ -29,7 +28,6 @@ command[check_fedmsg_cexceptions_notifs_backend]={{libdir}}/nagios/plugins/check
command[check_fedmsg_cexceptions_fedimg_backend]={{libdir}}/nagios/plugins/check_fedmsg_consumer_exceptions.py fedmsg-hub FedimgConsumer 1 10
command[check_fedmsg_cexceptions_hotness_backend]={{libdir}}/nagios/plugins/check_fedmsg_consumer_exceptions.py fedmsg-hub BugzillaTicketFiler 1 10
command[check_fedmsg_cexceptions_packages_backend]={{libdir}}/nagios/plugins/check_fedmsg_consumer_exceptions.py fedmsg-hub CacheInvalidator 1 10
command[check_fedmsg_cexceptions_mbs_backend]={{libdir}}/nagios/plugins/check_fedmsg_consumer_exceptions.py fedmsg-hub MBSConsumer 1 10
command[check_fedmsg_cbacklog_busgateway_relay]={{libdir}}/nagios/plugins/check_fedmsg_consumer_backlog.py fedmsg-relay RelayConsumer 10 50
{% if (ansible_distribution == 'RedHat' and ansible_distribution_major_version|int < 8) or (ansible_distribution_major_version|int < 30 and ansible_distribution == 'Fedora') %}
@ -46,7 +44,6 @@ command[check_fedmsg_cbacklog_bugzilla2fedmsg]={{libdir}}/nagios/plugins/check_f
command[check_fedmsg_cbacklog_fedimg_backend]={{libdir}}/nagios/plugins/check_fedmsg_consumer_backlog.py fedmsg-hub FedimgConsumer 2000 5000
command[check_fedmsg_cbacklog_hotness_backend]={{libdir}}/nagios/plugins/check_fedmsg_consumer_backlog.py fedmsg-hub BugzillaTicketFiler 1000 5000
command[check_fedmsg_cbacklog_packages_backend]={{libdir}}/nagios/plugins/check_fedmsg_consumer_backlog.py fedmsg-hub CacheInvalidator 30000 40000
command[check_fedmsg_cbacklog_mbs_backend]={{libdir}}/nagios/plugins/check_fedmsg_consumer_backlog.py fedmsg-hub MBSConsumer 10000 20000
command[check_fedmsg_fmn_digest_last_ran]={{libdir}}/nagios/plugins/check_fedmsg_producer_last_ran.py fedmsg-hub DigestProducer 90 600
command[check_fedmsg_fmn_confirm_last_ran]={{libdir}}/nagios/plugins/check_fedmsg_producer_last_ran.py fedmsg-hub ConfirmationProducer 90 600

View file

@ -243,14 +243,6 @@ define service {
# use defaulttemplate
#}
define service {
host_name mbs-backend01.iad2.fedoraproject.org
service_description Check fedmsg consumers and producers hub
check_command check_by_nrpe!check_fedmsg_cp_mbs_backend
use defaulttemplate
}
# BEGIN exceptions counter
define service {
host_name busgateway01.iad2.fedoraproject.org
@ -288,15 +280,6 @@ define service {
# use defaulttemplate
#}
define service {
host_name mbs-backend01.iad2.fedoraproject.org
service_description Check fedmsg-hub consumers exceptions
check_command check_by_nrpe!check_fedmsg_cexceptions_mbs_backend
use defaulttemplate
}
# BEGIN backlog checking
define service {
host_name busgateway01.iad2.fedoraproject.org
@ -333,10 +316,3 @@ define service {
# check_command check_by_nrpe!check_fedmsg_cbacklog_packages_backend
# use defaulttemplate
#}
define service {
host_name mbs-backend01.iad2.fedoraproject.org
service_description Check fedmsg-hub consumers backlog
check_command check_by_nrpe!check_fedmsg_cbacklog_mbs_backend
use defaulttemplate
}

View file

@ -115,14 +115,6 @@ define service {
use websitetemplate
}
define service {
hostgroup_name proxies
service_description http-mbs
check_command check_website_ssl!mbs.fedoraproject.org!/module-build-service/1/component-builds/!items
max_check_attempts 8
use websitetemplate
}
define service {
hostgroup_name proxies
service_description http-odcs

View file

@ -391,7 +391,6 @@ command[check_fedmsg_cp_notifs_backend]=/usr/lib64/nagios/plugins/check_fedmsg_p
command[check_fedmsg_cp_fedimg_backend]=/usr/lib64/nagios/plugins/check_fedmsg_producers_consumers.py fedmsg-hub FedimgConsumer MonitoringProducer
command[check_fedmsg_cp_hotness_backend]=/usr/lib64/nagios/plugins/check_fedmsg_producers_consumers.py fedmsg-hub BugzillaTicketFiler MonitoringProducer
command[check_fedmsg_cp_packages_backend]=/usr/lib64/nagios/plugins/check_fedmsg_producers_consumers.py fedmsg-hub CacheInvalidator MonitoringProducer
command[check_fedmsg_cp_mbs_backend]=/usr/lib64/nagios/plugins/check_fedmsg_producers_consumers.py fedmsg-hub MBSConsumer MonitoringProducer
command[check_fedmsg_cexceptions_busgateway_relay]=/usr/lib64/nagios/plugins/check_fedmsg_consumer_exceptions.py fedmsg-relay RelayConsumer 1 10
command[check_fedmsg_cexceptions_busgateway_gateway]=/usr/lib64/nagios/plugins/check_fedmsg_consumer_exceptions.py fedmsg-gateway GatewayConsumer 1 10
@ -402,7 +401,6 @@ command[check_fedmsg_cexceptions_notifs_backend]=/usr/lib64/nagios/plugins/check
command[check_fedmsg_cexceptions_fedimg_backend]=/usr/lib64/nagios/plugins/check_fedmsg_consumer_exceptions.py fedmsg-hub FedimgConsumer 1 10
command[check_fedmsg_cexceptions_hotness_backend]=/usr/lib64/nagios/plugins/check_fedmsg_consumer_exceptions.py fedmsg-hub BugzillaTicketFiler 1 10
command[check_fedmsg_cexceptions_packages_backend]=/usr/lib64/nagios/plugins/check_fedmsg_consumer_exceptions.py fedmsg-hub CacheInvalidator 1 10
command[check_fedmsg_cexceptions_mbs_backend]=/usr/lib64/nagios/plugins/check_fedmsg_consumer_exceptions.py fedmsg-hub MBSConsumer 1 10
command[check_fedmsg_cbacklog_busgateway_relay]=/usr/lib64/nagios/plugins/check_fedmsg_consumer_backlog.py fedmsg-relay RelayConsumer 10 50
command[check_fedmsg_cbacklog_busgateway_gateway]=/usr/lib64/nagios/plugins/check_fedmsg_consumer_backlog.py fedmsg-gateway GatewayConsumer 10 50
@ -413,7 +411,6 @@ command[check_fedmsg_cbacklog_notifs_backend]=/usr/lib64/nagios/plugins/check_fe
command[check_fedmsg_cbacklog_fedimg_backend]=/usr/lib64/nagios/plugins/check_fedmsg_consumer_backlog.py fedmsg-hub FedimgConsumer 2000 5000
command[check_fedmsg_cbacklog_hotness_backend]=/usr/lib64/nagios/plugins/check_fedmsg_consumer_backlog.py fedmsg-hub BugzillaTicketFiler 100 500
command[check_fedmsg_cbacklog_packages_backend_hub]=/usr/lib64/nagios/plugins/check_fedmsg_consumer_backlog.py fedmsg-hub CacheInvalidator 30000 40000
command[check_fedmsg_cbacklog_mbs_backend_hub]=/usr/lib64/nagios/plugins/check_fedmsg_consumer_backlog.py fedmsg-hub MBSConsumer 1000 2000
command[check_fedmsg_fmn_digest_last_ran]=/usr/lib64/nagios/plugins/check_fedmsg_producer_last_ran.py fedmsg-hub DigestProducer 90 600
command[check_fedmsg_fmn_confirm_last_ran]=/usr/lib64/nagios/plugins/check_fedmsg_producer_last_ran.py fedmsg-hub ConfirmationProducer 30 300

View file

@ -254,10 +254,6 @@
configure_priv: "^$"
read_priv: "^$"
write_priv: "^$"
- vhost: /mbs-private-queue
configure_priv: "^$"
read_priv: "^$"
write_priv: "^$"
- vhost: /centos-odcs
configure_priv: "^$"
read_priv: "^$"
@ -286,10 +282,6 @@
configure_priv: "^$"
read_priv: "^$"
write_priv: "^$"
- vhost: /mbs-private-queue
configure_priv: "^$"
read_priv: "^$"
write_priv: "^$"
- vhost: /centos-odcs
configure_priv: "^$"
read_priv: "^$"
@ -495,9 +487,3 @@
tags:
- rabbitmq_cluster
- config
# VirtualHost /mbs-private-queue
- import_tasks: vhost-mbs-private-queue.yml
tags:
- rabbitmq_cluster
- config

View file

@ -1,92 +0,0 @@
- name: Configure the mbs-private-queue virtual host
run_once: true
delegate_to: "rabbitmq01{{ env_suffix }}.iad2.fedoraproject.org"
rabbitmq_vhost:
name: /mbs-private-queue
state: present
tags:
- mbs-private-queue
- name: Configure the HA policy for the mbs-private-queue queues
run_once: true
delegate_to: "rabbitmq01{{ env_suffix }}.iad2.fedoraproject.org"
rabbitmq_policy:
name: HA
apply_to: queues
pattern: .*
tags:
ha-mode: all
ha-sync-mode: automatic # Auto sync queues to new cluster members
ha-sync-batch-size: 10000 # Larger is faster, but must finish in 1 net_ticktime
vhost: /mbs-private-queue
tags:
- mbs-private-queue
- name: Add a policy to limit queues to 1GB and remove after a month of no use
run_once: true
delegate_to: "rabbitmq01{{ env_suffix }}.iad2.fedoraproject.org"
rabbitmq_policy:
apply_to: queues
name: pubsub_sweeper
state: present
pattern: ".*"
tags:
# Unused queues are killed after 1000 * 60 * 60 * 31 milliseconds (~a month)
expires: 111600000
# Queues can use at most 1GB of storage
max-length-bytes: 1073741824
vhost: /mbs-private-queue
tags:
- mbs-private-queue
- name: Create the mbs-private-queue user for the mbs-private-queue vhost (prod)
run_once: true
delegate_to: "rabbitmq01{{ env_suffix }}.iad2.fedoraproject.org"
rabbitmq_user:
user: mbs-private-queue
password: "{{ (env == 'production')|ternary(rabbitmq_mbs_private_queue_admin_password_production, rabbitmq_mbs_private_queue_admin_password_staging) }}"
vhost: /mbs-private-queue
configure_priv: .*
read_priv: .*
write_priv: .*
tags:
- mbs-private-queue
- name: Dump the admin password in a file for administrative operations
run_once: true
delegate_to: "rabbitmq01{{ env_suffix }}.iad2.fedoraproject.org"
copy:
dest: /root/.mbs-private-queue-rabbitmqpass
content: "{{ (env == 'production')|ternary(rabbitmq_mbs_private_queue_admin_password_production, rabbitmq_mbs_private_queue_admin_password_staging) }}"
mode: 0600
owner: root
group: root
tags:
- mbs-private-queue
- name: Grant the admin user access to the mbs-private-queue vhost
run_once: true
delegate_to: "rabbitmq01{{ env_suffix }}.iad2.fedoraproject.org"
rabbitmq_user:
user: admin
vhost: /mbs-private-queue
configure_priv: .*
read_priv: .*
write_priv: .*
tags: administrator
tags:
- mbs-private-queue
- name: Create a user for mbs-private-queue access
run_once: true
delegate_to: "rabbitmq01{{ env_suffix }}.iad2.fedoraproject.org"
rabbitmq_user:
user: "mbs-private-queue{{ env_suffix }}"
vhost: /mbs-private-queue
configure_priv: .*
write_priv: .*
read_priv: .*
state: present
tags:
- mbs-private-queue

View file

@ -82,7 +82,6 @@ handlers = ["console"]
# The keys here need to be the same in the sigul bridge
[consumer_config.koji_instances.primary]
url = "https://koji{{ env_suffix }}.fedoraproject.org/kojihub"
mbs_user = "mbs/mbs{{ env_suffix }}.fedoraproject.org"
[consumer_config.koji_instances.primary.options]
# Only ssl and kerberos are supported at the moment

View file

@ -12,7 +12,6 @@ scp db01.iad2.fedoraproject.org:/backups/bodhi2-$(date +%F).dump.xz /srv/web/inf
scp db01.iad2.fedoraproject.org:/backups/pdc-$(date +%F).dump.xz /srv/web/infra/db-dumps/pdc.dump.xz
scp db01.iad2.fedoraproject.org:/backups/anitya-public-$(date +%F).dump.xz /srv/web/infra/db-dumps/anitya.dump.xz
scp db01.iad2.fedoraproject.org:/backups/mailman-$(date +%F).dump.xz /srv/web/infra/db-dumps/mailman.dump.xz
scp db01.iad2.fedoraproject.org:/backups/mbs-$(date +%F).dump.xz /srv/web/infra/db-dumps/mbs.dump.xz
scp db01.iad2.fedoraproject.org:/backups/odcs-$(date +%F).dump.xz /srv/web/infra/db-dumps/odcs.dump.xz
scp db01.iad2.fedoraproject.org:/backups/hyperkitty-$(date +%F).dump.xz /srv/web/infra/db-dumps/hyperkitty.dump.xz
scp db01.iad2.fedoraproject.org:/backups/resultsdb-$(date +%F).dump.xz /srv/web/infra/db-dumps/resultsdb.dump.xz