= System Administrator Guide

Welcome to The Fedora Infrastructure system administration guide.

== Getting Started

If you haven't already, you should complete the general
xref:developer_guide:getting-started.adoc[Getting Started] guide.
Once you've completed that, you're ready to get
involved in the
https://accounts.fedoraproject.org/group/fi-apprentice/[Fedora
Infrastructure Apprentice] group.

=== Fedora Infrastructure Apprentice

The
https://accounts.fedoraproject.org/group/fi-apprentice/[Fedora
Infrastructure Apprentice] group in the Fedora Account System grants
read-only access to many Fedora infrastructure machines. This group is
used for new folks to look around at the infrastructure setup, check
machines and processes and see where they might like to contribute
moving forward. This also allows apprentices to examine and gather info
on problems, then propose solutions.

[NOTE]
====
This group will be pruned often of inactive folks who miss the monthly
email check-in on the
https://lists.fedoraproject.org/admin/lists/infrastructure.lists.fedoraproject.org/[infrastructure
mailing list]. There's nothing personal in this and you're welcome to
re-join later when you have more time, we just want to make sure the
group only has active members.
====

Members of the https://accounts.fedoraproject.org/group/fi-apprentice/[Fedora
Infrastructure Apprentice] group have ssh/shell access to many machines,
but no sudo rights or ability to commit to the
https://pagure.io/fedora-infra/ansible/[Ansible repository] (but they do
have read-only access). Apprentice can, however, contribute to the
infrastructure documentation by making a pull request to the
https://pagure.io/infra-docs/[infra-docs] repository. Access is via the
bastion.fedoraproject.org machine and from there to each machine. See
the xref:sshaccess.adoc[ssh-sop] for instructions on how to set up SSH.
You can see a list of hosts that allow apprentice access by using:

....
$  ./scripts/hosts_with_var_set -i inventory/ -o ipa_client_shell_groups=fi-apprentice
....

from a checkout of the https://pagure.io/fedora-infra/ansible/[Ansible
repository]. The Ansible repository is hosted on pagure.io at
`https://pagure.io/fedora-infra/ansible.git`.

=== Selecting a Ticket

Start by checking out the
https://pagure.io/fedora-infrastructure/issues?status=Open&tags=easyfix[easyfix
tickets]. Tickets marked with this tag are a good place for apprentices
to learn how things are setup, and also contribute a fix.

Since apprentices do not have commit access to the
https://pagure.io/fedora-infra/ansible/[Ansible repository], you should
make your change, produce a patch with `git diff`, and attach it to the
infrastructure ticket you are working on. It will then be reviewed.

== Standard Operating Procedures

Below is a table of contents containing all the standard operating
procedures for Fedora Infrastructure applications. For information on
how to write a new standard operating procedure, consult the guide on
xref:developer_guide:sops.adoc[Developing Standard Operating Procedures].

* xref:accountdeletion.adoc[Account Deletion SOP]
* xref:fedmsg-new-message-type.adoc[Adding a new fedmsg message type]
* xref:anitya.adoc[Anitya Infrastructure SOP]
* xref:ansible.adoc[Ansible]
* xref:apps-fp-o.adoc[apps.fedoraproject.org]
* xref:aws-access.adoc[Amazon Web Services Access]
* xref:mirrormanager-S3-EC2-netblocks.adoc[Amazon Web Services Mirrors]
* xref:bastion-hosts-info.adoc[Bastion Hosts]
* xref:blockerbugs.adoc[Blockerbugs Infrastructure]
* xref:bodhi.adoc[Bodhi Infrastructure - Releng]
* xref:bodhi-deploy.adoc[Bodhi Infrastructure - Deployment]
* xref:bugzilla2fedmsg.adoc[bugzilla2fedmsg]
* xref:collectd.adoc[Collectd]
* xref:compose-tracker.adoc[Compose Tracker]
* xref:registry.adoc[Container registry]
* xref:contenthosting.adoc[Content Hosting Infrastructure]
* xref:copr.adoc[Copr]
* xref:coreos-cincinnati.adoc[CoreOS Cincinnati]
* xref:database.adoc[Database Infrastructure]
* xref:datanommer.adoc[Datanommer]
* xref:debuginfod.adoc[Debuginfod Service]
* xref:departing-admin.adoc[Departing admin]
* xref:dnf-counting.adoc[DNF Counting]
* xref:dns.adoc[DNS repository for fedoraproject]
* xref:docs.fedoraproject.org.adoc[Docs]
* xref:externally-hosted-services.adoc[Externally Hosted Services]
* xref:fas-openid.adoc[FAS-OpenID]
* xref:fedmsg-certs.adoc[fedmsg (Fedora Messaging) Certs, Keys, and CA]
* xref:fedmsg-gateway.adoc[fedmsg-gateway]
* xref:fedmsg-introduction.adoc[fedmsg introduction and basics]
* xref:fedmsg-relay.adoc[fedmsg-relay]
* xref:fedocal.adoc[Fedocal]
* xref:fedora-releases.adoc[Fedora Release Infrastructure]
* xref:gather-easyfix.adoc[Fedora gather easyfix]
* xref:status-fedora.adoc[Fedora Status Service]
* xref:gdpr_delete.adoc[GDPR Delete]
* xref:gdpr_sar.adoc[GDPR SAR]
* xref:geoip-city-wsgi.adoc[geoip-city-wsgi]
* xref:github.adoc[Using github for Infra Projects]
* xref:github2fedmsg.adoc[github2fedmsg]
* xref:greenwave.adoc[Greenwave]
* xref:guestdisk.adoc[Guest Disk Resize]
* xref:guestedit.adoc[Guest Editing]
* xref:guest_migrate.adoc[Migrate Guest VMs]
* xref:haproxy.adoc[Haproxy Infrastructure]
* xref:hotfix.adoc[HOTFIXES]
* xref:tickets.adoc[How to handle new tickets in fedora-infrastructure]
* xref:infra-git-repo.adoc[Infrastructure Git Repos]
* xref:infra-hostrename.adoc[Infrastructure Host Rename]
* xref:infra_handover.adoc[Initiative Handover]
* xref:infra-raidmismatch.adoc[Infrastructure Raid Mismatch Count]
* xref:infra-repo.adoc[Infrastructure Yum Repo]
* xref:infra-retiremachine.adoc[Infrastructure retire machine]
* xref:ipsilon.adoc[Ipsilon Infrastructure]
* xref:iscsi.adoc[iSCSI]
* xref:kerneltest-harness.adoc[Kerneltest-harness]
* xref:kickstarts.adoc[Kickstart Infrastructure]
* xref:koji-archive.adoc[Koji Archive]
* xref:virt-image.adoc[Kpartx Notes]
* xref:koji.adoc[Koji Infrastructure]
* xref:koschei.adoc[Koschei]
* xref:layered-image-buildsys.adoc[Layered Image Build System]
* xref:virt-notes.adoc[Libvirt Notes]
* xref:syslog.adoc[Log Infrastructure]
* xref:publictest-dev-stg-production.adoc[Machine Classes]
* xref:mailman.adoc[Mailman Infrastructure]
* xref:massupgrade.adoc[Mass Upgrade Infrastructure]
* xref:mastermirror.adoc[Master Mirror Infrastructure]
* xref:mbs.adoc[Module Build Service Infra]
* xref:memcached.adoc[Memcached Infrastructure]
* xref:message-tagging-service.adoc[Message Tagging Service]
* xref:mini_initiatives.adoc[Mini initiative Process]
* xref:mirrorhiding.adoc[Mirror Hiding Infrastructure]
* xref:mirrormanager.adoc[MirrorManager Infrastructure]
* xref:mote.adoc[mote]
* xref:nagios.adoc[Nagios]
* xref:netapp.adoc[Netapp Infrastructure]
* xref:nonhumanaccounts.adoc[Non-human Accounts Infrastructure]
* xref:ocp4:sops.adoc[Openshift SOPs]
* xref:odcs.adoc[On Demand Compose Service]
* xref:openqa.adoc[OpenQA Infrastructure]
* xref:openvpn.adoc[OpenVPN]
* xref:outage.adoc[Outage Infrastructure]
* xref:packagereview.adoc[Package Review]
* xref:pagure.adoc[Pagure Infrastructure]
* xref:pdc.adoc[PDC]
* xref:pesign-upgrade.adoc[Pesign upgrades/reboots]
* xref:planetsubgroup.adoc[Planet Subgroup Infrastructure]
* xref:rabbitmq.adoc[RabbitMQ]
* xref:rdiff-backup.adoc[rdiff-backup]
* xref:requestforresources.adoc[Request for resources]
* xref:resultsdb.adoc[ResultsDB]
* xref:retrace.adoc[Retrace]
* xref:scmadmin.adoc[SCM Admin]
* xref:selinux.adoc[SELinux Infrastructure]
* xref:koji-builder-setup.adoc[Setup Koji Builder]
* xref:sigul-upgrade.adoc[Sigul servers upgrades/reboots]
* xref:sshaccess.adoc[SSH Access Infrastructure]
* xref:sshknownhosts.adoc[SSH known hosts Infrastructure]
* xref:ssl-certificates.adoc[SSL Certificates]
* xref:staging.adoc[Staging]
* xref:hotness.adoc[The New Hotness]
* xref:2-factor.adoc[Two factor auth]
* xref:unbound.adoc[Unbound Notes]
* xref:new-virtual-hosts.adoc[Virtual Host Addition]
* xref:voting.adoc[Voting Infrastructure]
* xref:waiverdb.adoc[WaiverDB]
* xref:fedorawebsites.adoc[Websites Release]
* xref:fedmsg-websocket.adoc[WebSocket]
* xref:wcidff.adoc[What Can I Do For Fedora]
* xref:wiki.adoc[Wiki Infrastructure]
* xref:zodbot.adoc[Zodbot Infrastructure]

== HOWTOs

In this section is list of guides for common tasks that are done in Fedora Infrastructure.

* xref:howtos:access_rabbitmq_ui.adoc[How to access the rabbitmq administrative UI]
* xref:howtos:archive-old-fedora.adoc[How to Archive Old Fedora Releases]
* xref:howtos:add_a_package_to_infra_tag.adoc[How to add a package to an infra tag]
* xref:howtos:add_external_hardware_to_vpn.adoc[Add external servers to vpn]
* xref:howtos:build_against_infra_tags.adoc[How to build against an infra tag in koji]
* xref:howtos:check_robosignatory_production_logs.adoc[How to check robosignatory productions logs]
* xref:howtos:clean_2f_tokens.adoc[How to remove 2 factor authentication tokens in IPA]
* xref:howtos:clean_monitoring_sidetags.adoc[How to clean up the side-tags created by the monitor-gating project]
* xref:howtos:create_keytab.adoc[How to create a keytab for an user]
* xref:howtos:create_new_mailing_list.adoc[Creating a new mailing list]
* xref:howtos:creating_groups_distgit.adoc[How to create a group in dist-git]
* xref:howtos:delete_mailman_thread.adoc[How to delete a thread in mailman]
* xref:howtos:destroy_a_virt_instance.adoc[How to destroy a virt instance]
* xref:howtos:discourse_spam.adoc[How to deal with spam posts on discourse]
* xref:howtos:pagure_spam.adoc[How to deal with spam posts on pagure]
* xref:howtos:fedora_messaging_certificates.adoc[How to create TLS certificates for fedora-messaging]
* xref:howtos:fix_robosignatory.adoc[How to check/fix robosignatory]
* xref:howtos:free_space_in_openshift.adoc[How to free some space in OpenShift]
* xref:howtos:generate_openvpn_keys.adoc[How to generate private key and certificate for OpenVPN client]
* xref:howtos:get_logs_pod_openshift.adoc[How to get logs of a pod in OpenShift]
* xref:howtos:give_groups_dist_git.adoc[How to give a group from someone to someone else in dist-git]
* xref:howtos:groups_in_fedora.adoc[Groups in Fedora]
* xref:howtos:make_mailman_user_admin.adoc[Make mailman user an admin]
* xref:howtos:rebuild_osbs_buildroot.adoc[How to rebuild OSBS buildroot image]
* xref:howtos:refresh_osbs_odcs_oicd_token.adoc[How to refresh the ODCS OIDC token used by OSBS]
* xref:howtos:remove_meeting_minutes_meetbot.adoc[How to remove meeting minutes from meetbot]
* xref:howtos:remove_monitoring_rabbitmq_queue.adoc[How to remove the monitoring of a rabbitmq queue]
* xref:howtos:remove_branch_distgit.adoc[How to remove a git branch in a dist-git repository]
* xref:howtos:remove_fedora_user_at_launch_aws.adoc[How to add allow root ssh login and remove fedora user]
* xref:howtos:remove_user_from_watchlist_pagure.adoc[How to remove someone from a watch list on Pagure]
* xref:howtos:restart_sigul_bridge.adoc[How to restart the sigul bridge]
* xref:howtos:scale_up_or_down_deployment.adoc[How to scale up/down a deployment in OpenShift]
* xref:howtos:share_tmux_session.adoc[How to share a tmux session accross users]
* xref:howtos:unblock_bodhi_rawhide_updates.adoc[How to unblock Bodhi rawhide updates]
* xref:howtos:update_watch_dist_git.adoc[How to udpate the watch status of someone in dist-git]
