= System Administrator Guide Welcome to The Fedora Infrastructure system administration guide. == Getting Started If you haven't already, you should complete the general xref:developer_guide:getting-started.adoc[Getting Started] guide. Once you've completed that, you're ready to get involved in the https://accounts.fedoraproject.org/group/fi-apprentice/[Fedora Infrastructure Apprentice] group. === Fedora Infrastructure Apprentice The https://accounts.fedoraproject.org/group/fi-apprentice/[Fedora Infrastructure Apprentice] group in the Fedora Account System grants read-only access to many Fedora infrastructure machines. This group is used for new folks to look around at the infrastructure setup, check machines and processes and see where they might like to contribute moving forward. This also allows apprentices to examine and gather info on problems, then propose solutions. [NOTE] ==== This group will be pruned often of inactive folks who miss the monthly email check-in on the https://lists.fedoraproject.org/admin/lists/infrastructure.lists.fedoraproject.org/[infrastructure mailing list]. There's nothing personal in this and you're welcome to re-join later when you have more time, we just want to make sure the group only has active members. ==== Members of the https://accounts.fedoraproject.org/group/fi-apprentice/[Fedora Infrastructure Apprentice] group have ssh/shell access to many machines, but no sudo rights or ability to commit to the https://pagure.io/fedora-infra/ansible/[Ansible repository] (but they do have read-only access). Apprentice can, however, contribute to the infrastructure documentation by making a pull request to the https://pagure.io/infra-docs/[infra-docs] repository. Access is via the bastion.fedoraproject.org machine and from there to each machine. See the xref:sshaccess.adoc[ssh-sop] for instructions on how to set up SSH. You can see a list of hosts that allow apprentice access by using: .... $ ./scripts/hosts_with_var_set -i inventory/ -o ipa_client_shell_groups=fi-apprentice .... from a checkout of the https://pagure.io/fedora-infra/ansible/[Ansible repository]. The Ansible repository is hosted on pagure.io at `https://pagure.io/fedora-infra/ansible.git`. === Selecting a Ticket Start by checking out the https://pagure.io/fedora-infrastructure/issues?status=Open&tags=easyfix[easyfix tickets]. Tickets marked with this tag are a good place for apprentices to learn how things are setup, and also contribute a fix. Since apprentices do not have commit access to the https://pagure.io/fedora-infra/ansible/[Ansible repository], you should make your change, produce a patch with `git diff`, and attach it to the infrastructure ticket you are working on. It will then be reviewed. == Standard Operating Procedures Below is a table of contents containing all the standard operating procedures for Fedora Infrastructure applications. For information on how to write a new standard operating procedure, consult the guide on xref:developer_guide:sops.adoc[Developing Standard Operating Procedures]. * xref:2-factor.adoc[Two factor auth] * xref:accountdeletion.adoc[Account Deletion SOP] * xref:anitya.adoc[Anitya Infrastructure SOP] * xref:ansible.adoc[Ansible] * xref:apps-fp-o.adoc[apps.fedoraproject.org] * xref:aws-access.adoc[Amazon Web Services Access] * xref:bastion-hosts-info.adoc[Bastion Hosts] * xref:blockerbugs.adoc[Blockerbugs Infrastructure] * xref:bodhi-deploy.adoc[Bodhi Infrastructure - Deployment] * xref:bodhi.adoc[Bodhi Infrastructure - Releng] * xref:bugzilla2fedmsg.adoc[Bugzilla 2 Fedmsg] * xref:bugzilla2fedmsg.adoc[bugzilla2fedmsg] * xref:collectd.adoc[Collectd] * xref:compose-tracker.adoc[Compose Tracker] * xref:contenthosting.adoc[Content Hosting Infrastructure] * xref:copr.adoc[Copr] * xref:coreos-cincinnati.adoc[CoreOS Cincinnati] * xref:database.adoc[Database Infrastructure] * xref:datanommer.adoc[Datanommer] * xref:debuginfod.adoc[Debuginfod Service] * xref:departing-admin.adoc[Departing admin] * xref:dnf-counting.adoc[DNF Counting] * xref:dns.adoc[DNS repository for fedoraproject] * xref:docs.fedoraproject.org.adoc[Docs] * xref:externally-hosted-services.adoc[Externally Hosted Services] * xref:failedharddrive.adoc[Replacing Failed Hard Drives] * xref:fas-openid.adoc[FAS-OpenID] * xref:fedmsg-certs.adoc[fedmsg (Fedora Messaging) Certs, Keys, and CA] * xref:fedmsg-gateway.adoc[fedmsg-gateway] * xref:fedmsg-introduction.adoc[fedmsg introduction and basics] * xref:fedmsg-new-message-type.adoc[Adding a new fedmsg message type] * xref:fedmsg-relay.adoc[fedmsg-relay] * xref:fedmsg-websocket.adoc[WebSocket] * xref:fedocal.adoc[Fedocal] * xref:fedora-releases.adoc[Fedora Release Infrastructure] * xref:fedorawebsites.adoc[Websites Release] * xref:gather-easyfix.adoc[Fedora gather easyfix] * xref:gdpr_delete.adoc[GDPR Delete] * xref:gdpr_sar.adoc[GDPR SAR] * xref:geoip-city-wsgi.adoc[geoip-city-wsgi] * xref:github.adoc[Using github for Infra Projects] * xref:github2fedmsg.adoc[github2fedmsg] * xref:greenwave.adoc[Greenwave] * xref:guest_migrate.adoc[Migrate Guest VMs] * xref:guestdisk.adoc[Guest Disk Resize] * xref:guestedit.adoc[Guest Editing] * xref:haproxy.adoc[Haproxy Infrastructure] * xref:hardware_troubleshooting_power.adoc[Hardware Troubleshoot Power Issue] * xref:hotfix.adoc[HOTFIXES] * xref:hotness.adoc[The New Hotness] * xref:infra-git-repo.adoc[Infrastructure Git Repos] * xref:infra-hostrename.adoc[Infrastructure Host Rename] * xref:infra-raidmismatch.adoc[Infrastructure Raid Mismatch Count] * xref:infra-repo.adoc[Infrastructure DNF Repo] * xref:infra-retiremachine.adoc[Infrastructure retire machine] * xref:infra_handover.adoc[Initiative Handover] * xref:ipa.adoc[IPA infrastructure] * xref:ipsilon.adoc[Ipsilon Infrastructure] * xref:iscsi.adoc[iSCSI] * xref:kerneltest-harness.adoc[Kerneltest-harness] * xref:kickstarts.adoc[Kickstart Infrastructure] * xref:koji-archive.adoc[Koji Archive] * xref:koji-builder-setup.adoc[Setup Koji Builder] * xref:koji.adoc[Koji Infrastructure] * xref:koschei.adoc[Koschei] * xref:layered-image-buildsys.adoc[Layered Image Build System] * xref:mailman.adoc[Mailman Infrastructure] * xref:massupgrade.adoc[Mass Upgrade Infrastructure] * xref:mastermirror.adoc[Master Mirror Infrastructure] * xref:mbs.adoc[Module Build Service Infra] * xref:memcached.adoc[Memcached Infrastructure] * xref:message-tagging-service.adoc[Message Tagging Service] * xref:mini_initiatives.adoc[Mini initiative Process] * xref:mirrorhiding.adoc[Mirror Hiding Infrastructure] * xref:mirrormanager-S3-EC2-netblocks.adoc[Amazon Web Services Mirrors] * xref:mirrormanager.adoc[MirrorManager Infrastructure] * xref:mote.adoc[mote] * xref:nagios.adoc[Nagios] * xref:netapp.adoc[Netapp Infrastructure] * xref:new-virtual-hosts.adoc[Virtual Host Addition] * xref:nonhumanaccounts.adoc[Non-human Accounts Infrastructure] * xref:ocp4:sops.adoc[Openshift SOPs] * xref:odcs.adoc[On Demand Compose Service] * xref:openqa.adoc[OpenQA Infrastructure] * xref:openvpn.adoc[OpenVPN] * xref:outage.adoc[Outage Infrastructure] * xref:packagereview.adoc[Package Review] * xref:pagure.adoc[Pagure Infrastructure] * xref:pdc.adoc[PDC] * xref:pesign-upgrade.adoc[Pesign upgrades/reboots] * xref:planetsubgroup.adoc[Planet Subgroup Infrastructure] * xref:publictest-dev-stg-production.adoc[Machine Classes] * xref:rabbitmq.adoc[RabbitMQ] * xref:rdiff-backup.adoc[rdiff-backup] * xref:registry.adoc[Container registry] * xref:requestforresources.adoc[Request for resources] * xref:resultsdb.adoc[ResultsDB] * xref:retrace.adoc[Retrace] * xref:scmadmin.adoc[SCM Admin] * xref:selinux.adoc[SELinux Infrastructure] * xref:sigul-upgrade.adoc[Sigul servers upgrades/reboots] * xref:sshaccess.adoc[SSH Access Infrastructure] * xref:sshknownhosts.adoc[SSH known hosts Infrastructure] * xref:ssl-certificates.adoc[SSL Certificates] * xref:staging.adoc[Staging] * xref:status-fedora.adoc[Fedora Status Service] * xref:syslog.adoc[Log Infrastructure] * xref:tickets.adoc[How to handle new tickets in fedora-infrastructure] * xref:unbound.adoc[Unbound Notes] * xref:virt-image.adoc[Kpartx Notes] * xref:virt-notes.adoc[Libvirt Notes] * xref:voting.adoc[Voting Infrastructure] * xref:waiverdb.adoc[WaiverDB] * xref:wcidff.adoc[What Can I Do For Fedora] * xref:wiki.adoc[Wiki Infrastructure] * xref:zabbix.adoc[Zabbix Infrastructure] * xref:zodbot.adoc[Zodbot Infrastructure] == HOWTOs In this section is list of guides for common tasks that are done in Fedora Infrastructure. * xref:howtos:access_rabbitmq_ui.adoc[How to access the rabbitmq administrative UI] * xref:howtos:archive_list.adoc[How to Archive mailing list] * xref:howtos:archive-old-fedora.adoc[How to Archive Old Fedora Releases] * xref:howtos:add_external_hardware_to_vpn.adoc[Add external servers to vpn] * xref:howtos:check_robosignatory_production_logs.adoc[How to check robosignatory productions logs] * xref:howtos:clean_2f_tokens.adoc[How to remove 2 factor authentication tokens in IPA] * xref:howtos:clean_monitoring_sidetags.adoc[How to clean up the side-tags created by the monitor-gating project] * xref:howtos:create_keytab.adoc[How to create a keytab for an user] * xref:howtos:create_new_mailing_list.adoc[Creating a new mailing list] * xref:howtos:creating_groups_distgit.adoc[How to create a group in dist-git] * xref:howtos:delete_mailman_thread.adoc[How to delete a thread in mailman] * xref:howtos:destroy_a_virt_instance.adoc[How to destroy a virt instance] * xref:howtos:discourse_spam.adoc[How to deal with spam posts on discourse] * xref:howtos:pagure_spam.adoc[How to deal with spam posts on pagure] * xref:howtos:fedora_messaging_certificates.adoc[How to create TLS certificates for fedora-messaging] * xref:howtos:fix_robosignatory.adoc[How to check/fix robosignatory] * xref:howtos:free_space_in_openshift.adoc[How to free some space in OpenShift] * xref:howtos:generate_openvpn_keys.adoc[How to generate private key and certificate for OpenVPN client] * xref:howtos:get_logs_pod_openshift.adoc[How to get logs of a pod in OpenShift] * xref:howtos:give_groups_dist_git.adoc[How to give a group from someone to someone else in dist-git] * xref:howtos:groups_in_fedora.adoc[Groups in Fedora] * xref:howtos:make_mailman_user_admin.adoc[Make mailman user an admin] * xref:howtos:rebuild_osbs_buildroot.adoc[How to rebuild OSBS buildroot image] * xref:howtos:refresh_osbs_odcs_oicd_token.adoc[How to refresh the ODCS OIDC token used by OSBS] * xref:howtos:remove_meeting_minutes_meetbot.adoc[How to remove meeting minutes from meetbot] * xref:howtos:remove_monitoring_rabbitmq_queue.adoc[How to remove the monitoring of a rabbitmq queue] * xref:howtos:remove_branch_distgit.adoc[How to remove a git branch in a dist-git repository] * xref:howtos:remove_fedora_user_at_launch_aws.adoc[How to add allow root ssh login and remove fedora user] * xref:howtos:remove_user_from_watchlist_pagure.adoc[How to remove someone from a watch list on Pagure] * xref:howtos:restart_sigul_bridge.adoc[How to restart the sigul bridge] * xref:howtos:restart_datacenter_server.adoc[How to restart server in datacenter] * xref:howtos:scale_up_or_down_deployment.adoc[How to scale up/down a deployment in OpenShift] * xref:howtos:share_tmux_session.adoc[How to share a tmux session accross users] * xref:howtos:unblock_bodhi_rawhide_updates.adoc[How to unblock Bodhi rawhide updates] * xref:howtos:update_watch_dist_git.adoc[How to udpate the watch status of someone in dist-git]