= System Administrator Guide

Welcome to The Fedora Infrastructure system administration guide.

== Getting Started

If you haven't already, you should complete the general
xref:developer_guide:getting-started.adoc[Getting Started] guide.
Once you've completed that, you're ready to get
involved in the
https://accounts.fedoraproject.org/group/fi-apprentice/[Fedora
Infrastructure Apprentice] group.

=== Fedora Infrastructure Apprentice

The
https://accounts.fedoraproject.org/group/fi-apprentice/[Fedora
Infrastructure Apprentice] group in the Fedora Account System grants
read-only access to many Fedora infrastructure machines. This group is
used for new folks to look around at the infrastructure setup, check
machines and processes and see where they might like to contribute
moving forward. This also allows apprentices to examine and gather info
on problems, then propose solutions.

[NOTE]
====
This group will be pruned often of inactive folks who miss the monthly
email check-in on the
https://lists.fedoraproject.org/admin/lists/infrastructure.lists.fedoraproject.org/[infrastructure
mailing list]. There's nothing personal in this and you're welcome to
re-join later when you have more time, we just want to make sure the
group only has active members.
====

Members of the https://accounts.fedoraproject.org/group/fi-apprentice/[Fedora
Infrastructure Apprentice] group have ssh/shell access to many machines,
but no sudo rights or ability to commit to the
https://pagure.io/fedora-infra/ansible/[Ansible repository] (but they do
have read-only access). Apprentice can, however, contribute to the
infrastructure documentation by making a pull request to the
https://pagure.io/infra-docs/[infra-docs] repository. Access is via the
bastion.fedoraproject.org machine and from there to each machine. See
the xref:sshaccess.adoc[ssh-sop] for instructions on how to set up SSH.
You can see a list of hosts that allow apprentice access by using:

....
$  ./scripts/hosts_with_var_set -i inventory/ -o ipa_client_shell_groups=fi-apprentice
....

from a checkout of the https://pagure.io/fedora-infra/ansible/[Ansible
repository]. The Ansible repository is hosted on pagure.io at
`https://pagure.io/fedora-infra/ansible.git`.

=== Selecting a Ticket

Start by checking out the
https://pagure.io/fedora-infrastructure/issues?status=Open&tags=easyfix[easyfix
tickets]. Tickets marked with this tag are a good place for apprentices
to learn how things are setup, and also contribute a fix.

Since apprentices do not have commit access to the
https://pagure.io/fedora-infra/ansible/[Ansible repository], you should
make your change, produce a patch with `git diff`, and attach it to the
infrastructure ticket you are working on. It will then be reviewed.

== Standard Operating Procedures

Below is a table of contents containing all the standard operating
procedures for Fedora Infrastructure applications. For information on
how to write a new standard operating procedure, consult the guide on
xref:developer_guide:sops.adoc[Developing Standard Operating Procedures].

* xref:2-factor.adoc[Two factor auth]
* xref:accountdeletion.adoc[Account Deletion SOP]
* xref:anitya.adoc[Anitya Infrastructure SOP]
* xref:ansible.adoc[Ansible]
* xref:apps-fp-o.adoc[apps.fedoraproject.org]
* xref:aws-access.adoc[Amazon Web Services Access]
* xref:bastion-hosts-info.adoc[Bastion Hosts]
* xref:blockerbugs.adoc[Blockerbugs Infrastructure]
* xref:bodhi-deploy.adoc[Bodhi Infrastructure - Deployment]
* xref:bodhi.adoc[Bodhi Infrastructure - Releng]
* xref:bugzilla2fedmsg.adoc[bugzilla2fedmsg]
* xref:cloud-image-uploader.adoc[Cloud Image Uploader]
* xref:collectd.adoc[Collectd]
* xref:compose-tracker.adoc[Compose Tracker]
* xref:contenthosting.adoc[Content Hosting Infrastructure]
* xref:copr.adoc[Copr]
* xref:coreos-cincinnati.adoc[CoreOS Cincinnati]
* xref:database.adoc[Database Infrastructure]
* xref:datanommer.adoc[Datanommer]
* xref:debuginfod.adoc[Debuginfod Service]
* xref:departing-admin.adoc[Departing admin]
* xref:dnf-counting.adoc[DNF Counting]
* xref:dns.adoc[DNS repository for fedoraproject]
* xref:docs.fedoraproject.org.adoc[Docs]
* xref:externally-hosted-services.adoc[Externally Hosted Services]
* xref:failedharddrive.adoc[Replacing Failed Hard Drives]
* xref:fas-openid.adoc[FAS-OpenID]
* xref:fedmsg-certs.adoc[fedmsg (Fedora Messaging) Certs, Keys, and CA]
* xref:fedocal.adoc[Fedocal]
* xref:fedora-releases.adoc[Fedora Release Infrastructure]
* xref:fedorawebsites.adoc[Websites Release]
* xref:gather-easyfix.adoc[Fedora gather easyfix]
* xref:gdpr_delete.adoc[GDPR Delete]
* xref:gdpr_sar.adoc[GDPR SAR]
* xref:geoip-city-wsgi.adoc[geoip-city-wsgi]
* xref:github.adoc[Using github for Infra Projects]
* xref:greenwave.adoc[Greenwave]
* xref:guest_migrate.adoc[Migrate Guest VMs]
* xref:guestdisk.adoc[Guest Disk Resize]
* xref:guestedit.adoc[Guest Editing]
* xref:haproxy.adoc[Haproxy Infrastructure]
* xref:hardware_troubleshooting_power.adoc[Hardware Troubleshoot Power Issue]
* xref:hotfix.adoc[HOTFIXES]
* xref:hotness.adoc[The New Hotness]
* xref:infra-git-repo.adoc[Infrastructure Git Repos]
* xref:infra-hostrename.adoc[Infrastructure Host Rename]
* xref:infra-raidmismatch.adoc[Infrastructure Raid Mismatch Count]
* xref:infra-repo.adoc[Infrastructure DNF Repo]
* xref:infra-retiremachine.adoc[Infrastructure retire machine]
* xref:infra_handover.adoc[Initiative Handover]
* xref:ipa.adoc[IPA infrastructure]
* xref:ipsilon.adoc[Ipsilon Infrastructure]
* xref:iscsi.adoc[iSCSI]
* xref:kerneltest-harness.adoc[Kerneltest-harness]
* xref:kickstarts.adoc[Kickstart Infrastructure]
* xref:koji-archive.adoc[Koji Archive]
* xref:koji-builder-setup.adoc[Setup Koji Builder]
* xref:koji.adoc[Koji Infrastructure]
* xref:koschei.adoc[Koschei]
* xref:layered-image-buildsys.adoc[Layered Image Build System]
* xref:mailman.adoc[Mailman Infrastructure]
* xref:massupgrade.adoc[Mass Upgrade Infrastructure]
* xref:mastermirror.adoc[Master Mirror Infrastructure]
* xref:memcached.adoc[Memcached Infrastructure]
* xref:message-tagging-service.adoc[Message Tagging Service]
* xref:mini_initiatives.adoc[Mini initiative Process]
* xref:mirrorhiding.adoc[Mirror Hiding Infrastructure]
* xref:mirrormanager-S3-EC2-netblocks.adoc[Amazon Web Services Mirrors]
* xref:mirrormanager.adoc[MirrorManager Infrastructure]
* xref:mote.adoc[mote]
* xref:nagios.adoc[Nagios]
* xref:netapp.adoc[Netapp Infrastructure]
* xref:new-virtual-hosts.adoc[Virtual Host Addition]
* xref:nonhumanaccounts.adoc[Non-human Accounts Infrastructure]
* xref:openshift_sops.adoc[Openshift SOPs]
* xref:openqa.adoc[OpenQA Infrastructure]
* xref:openvpn.adoc[OpenVPN]
* xref:outage.adoc[Outage Infrastructure]
* xref:packagereview.adoc[Package Review]
* xref:pagure.adoc[Pagure Infrastructure]
* xref:pesign-upgrade.adoc[Pesign upgrades/reboots]
* xref:planetsubgroup.adoc[Planet Subgroup Infrastructure]
* xref:publictest-dev-stg-production.adoc[Machine Classes]
* xref:rabbitmq.adoc[RabbitMQ]
* xref:rdiff-backup.adoc[rdiff-backup]
* xref:registry.adoc[Container registry]
* xref:requestforresources.adoc[Request for resources]
* xref:resultsdb.adoc[ResultsDB]
* xref:retrace.adoc[Retrace]
* xref:scmadmin.adoc[SCM Admin]
* xref:selinux.adoc[SELinux Infrastructure]
* xref:sigul-upgrade.adoc[Sigul servers upgrades/reboots]
* xref:sshaccess.adoc[SSH Access Infrastructure]
* xref:sshknownhosts.adoc[SSH known hosts Infrastructure]
* xref:ssl-certificates.adoc[SSL Certificates]
* xref:staging.adoc[Staging]
* xref:status-fedora.adoc[Fedora Status Service]
* xref:syslog.adoc[Log Infrastructure]
* xref:tickets.adoc[How to handle new tickets in fedora-infrastructure]
* xref:unbound.adoc[Unbound Notes]
* xref:virt-image.adoc[Kpartx Notes]
* xref:virt-notes.adoc[Libvirt Notes]
* xref:voting.adoc[Voting Infrastructure]
* xref:waiverdb.adoc[WaiverDB]
* xref:wcidff.adoc[What Can I Do For Fedora]
* xref:wiki.adoc[Wiki Infrastructure]
* xref:zabbix.adoc[Zabbix Infrastructure]
* xref:zodbot.adoc[Zodbot Infrastructure]

== HOWTOs

In this section is list of guides for common tasks that are done in Fedora Infrastructure.

* xref:howtos:access_rabbitmq_ui.adoc[How to access the rabbitmq administrative UI]
* xref:howtos:archive_list.adoc[How to Archive mailing list]
* xref:howtos:archive-old-fedora.adoc[How to Archive Old Fedora Releases]
* xref:howtos:add_external_hardware_to_vpn.adoc[Add external servers to vpn]
* xref:howtos:check_robosignatory_production_logs.adoc[How to check robosignatory productions logs]
* xref:howtos:clean_2f_tokens.adoc[How to remove 2 factor authentication tokens in IPA]
* xref:howtos:clean_monitoring_sidetags.adoc[How to clean up the side-tags created by the monitor-gating project]
* xref:howtos:create_keytab.adoc[How to create a keytab for an user]
* xref:howtos:create_new_mailing_list.adoc[Creating a new mailing list]
* xref:howtos:creating_groups_distgit.adoc[How to create a group in dist-git]
* xref:howtos:delete_mailman_thread.adoc[How to delete a thread in mailman]
* xref:howtos:destroy_a_virt_instance.adoc[How to destroy a virt instance]
* xref:howtos:discourse_spam.adoc[How to deal with spam posts on discourse]
* xref:howtos:pagure_spam.adoc[How to deal with spam posts on pagure]
* xref:howtos:fedora_messaging_certificates.adoc[How to create TLS certificates for fedora-messaging]
* xref:howtos:fix_robosignatory.adoc[How to check/fix robosignatory]
* xref:howtos:free_space_in_openshift.adoc[How to free some space in OpenShift]
* xref:howtos:generate_openvpn_keys.adoc[How to generate private key and certificate for OpenVPN client]
* xref:howtos:get_logs_pod_openshift.adoc[How to get logs of a pod in OpenShift]
* xref:howtos:give_groups_dist_git.adoc[How to give a group from someone to someone else in dist-git]
* xref:howtos:groups_in_fedora.adoc[Groups in Fedora]
* xref:howtos:make_mailman_user_admin.adoc[Make mailman user an admin]
* xref:howtos:rebuild_osbs_buildroot.adoc[How to rebuild OSBS buildroot image]
* xref:howtos:refresh_osbs_odcs_oicd_token.adoc[How to refresh the ODCS OIDC token used by OSBS]
* xref:howtos:remove_meeting_minutes_meetbot.adoc[How to remove meeting minutes from meetbot]
* xref:howtos:remove_monitoring_rabbitmq_queue.adoc[How to remove the monitoring of a rabbitmq queue]
* xref:howtos:remove_branch_distgit.adoc[How to remove a git branch in a dist-git repository]
* xref:howtos:remove_fedora_user_at_launch_aws.adoc[How to add allow root ssh login and remove fedora user]
* xref:howtos:remove_user_from_watchlist_pagure.adoc[How to remove someone from a watch list on Pagure]
* xref:howtos:restart_sigul_bridge.adoc[How to restart the sigul bridge]
* xref:howtos:restart_datacenter_server.adoc[How to restart server in datacenter]
* xref:howtos:scale_up_or_down_deployment.adoc[How to scale up/down a deployment in OpenShift]
* xref:howtos:share_tmux_session.adoc[How to share a tmux session accross users]
* xref:howtos:unblock_bodhi_rawhide_updates.adoc[How to unblock Bodhi rawhide updates]
* xref:howtos:update_watch_dist_git.adoc[How to update the watch status of someone in dist-git]
* xref:howtos:idrac-firmware-update.adoc[How to perform firmware upgrades on iDRAC management controllers?]
