Added the infra SOPs ported to asciidoc.
This commit is contained in:
Adam Saleh
parent
8a7f111a12
commit
a0301e30f1
148 changed files with 18575 additions and 17 deletions
252
modules/sysadmin_guide/pages/ansible.adoc
Normal file
252
modules/sysadmin_guide/pages/ansible.adoc
Normal file
|
|
@ -0,0 +1,252 @@
|
|||
= Ansible infrastructure SOP/Information.
|
||||
|
||||
== Background
|
||||
|
||||
Fedora infrastructure used to use func and puppet for system change
|
||||
management. We are now using ansible for all system change mangement and
|
||||
ad-hoc tasks.
|
||||
|
||||
== Overview
|
||||
|
||||
Ansible runs from batcave01 or backup01. These hosts run a ssh-agent
|
||||
that has unlocked the ansible root ssh private key. (This is unlocked
|
||||
manually by a human with the passphrase each reboot, the passphrase
|
||||
itself is not stored anywhere on the machines). Using 'sudo -i',
|
||||
sysadmin-main members can use this agent to access any machines with the
|
||||
ansible root ssh public key setup, either with 'ansible' for one-off
|
||||
commands or 'ansible-playbook' to run playbooks.
|
||||
|
||||
Playbooks are idempotent (or should be). Meaning you should be able to
|
||||
re-run the same playbook over and over and it should get to a state
|
||||
where 0 items are changing.
|
||||
|
||||
Additionally (see below) there is a rbac wrapper that allows members of
|
||||
some other groups to run playbooks against specific hosts.
|
||||
|
||||
=== GIT repositories
|
||||
|
||||
There are 2 git repositories associated with Ansible:
|
||||
|
||||
* The Fedora Infrastructure Ansible repository and replicas.
|
||||
+
|
||||
[CAUTION]
|
||||
.Caution
|
||||
====
|
||||
This is a public repository. Never commit private data to this repo.
|
||||
====
|
||||
+
|
||||
image:ansible-repositories.png[image]
|
||||
+
|
||||
This repository exists as several copies or replicas:
|
||||
** The "upstream" repository on Pagure.
|
||||
+
|
||||
https://pagure.io/fedora-infra/ansible
|
||||
+
|
||||
This repository is the public facing place where people can contribute
|
||||
(e.g. pull requests) as well as the authoritative source. Members of the
|
||||
`sysadmin` FAS group or the `fedora-infra` Pagure group have commit
|
||||
access to this repository.
|
||||
+
|
||||
To contribute changes, fork the repository on Pagure and submit a Pull
|
||||
Request. Someone from the aforementioned groups can then review and
|
||||
merge them.
|
||||
+
|
||||
It is recommended that you configure git to use `pull --rebase` by
|
||||
default by running `git config --bool pull.rebase true` in your ansible
|
||||
clone directory. This configuration prevents unneeded merges which can
|
||||
occur if someone else pushes changes to the remote repository while you
|
||||
are working on your own local changes.
|
||||
** Two bare mirrors on [.title-ref]#batcave01#, `/srv/git/ansible.git`
|
||||
and `/srv/git/mirrors/ansible.git`
|
||||
+
|
||||
[CAUTION]
|
||||
.Caution
|
||||
====
|
||||
These are public repositories. Never commit private data to these
|
||||
repositories. Don't commit or push to these repos directly, unless
|
||||
Pagure is unavailable.
|
||||
====
|
||||
+
|
||||
The `mirror_pagure_ansible` service on [.title-ref]#batcave01# receives
|
||||
bus messages about changes in the repository on Pagure, fetches these
|
||||
into `/srv/git/mirrors/ansible.git` and pushes from there to
|
||||
`/srv/git/ansible.git`. When this happens, various actions are triggered
|
||||
via git hooks:
|
||||
*** The working copy at `/srv/web/infra/ansible` is updated.
|
||||
*** A mail about the changes is sent to [.title-ref]#sysadmin-members#.
|
||||
*** The changes are announced on the message bus, which in turn triggers
|
||||
announcements on IRC.
|
||||
+
|
||||
You can check out the repo locally on [.title-ref]#batcave01# with:
|
||||
+
|
||||
....
|
||||
git clone /srv/git/ansible.git
|
||||
....
|
||||
+
|
||||
If the Ansible repository on Pagure is unavailable, members of the
|
||||
[.title-ref]#sysadmin# group may commit directly, provided this
|
||||
procedure is followed:
|
||||
[arabic]
|
||||
. The synchronization service is stopped and disabled:
|
||||
+
|
||||
....
|
||||
sudo systemctl disable --now mirror_pagure_ansible.service
|
||||
....
|
||||
. Changes are applied to the repository on [.title-ref]#batcave01#.
|
||||
. After Pagure is available again, the changes are pushed to the
|
||||
repository there.
|
||||
. The synchronization service is enabled and started:
|
||||
+
|
||||
....
|
||||
sudo systemctl enable --now mirror_pagure_ansible.service
|
||||
....
|
||||
** `/srv/web/infra/ansible` on [.title-ref]#batcave01#, the working copy
|
||||
from which playbooks are run.
|
||||
+
|
||||
[CAUTION]
|
||||
.Caution
|
||||
====
|
||||
This is a public repository. Never commit private data to this repo.
|
||||
Don't commit or push to this repo directly, unless Pagure is
|
||||
unavailable.
|
||||
====
|
||||
+
|
||||
You can access it also via a cgit web interface at:
|
||||
https://pagure.io/fedora-infra/ansible/
|
||||
+
|
||||
[verse]
|
||||
--
|
||||
|
||||
--
|
||||
* `/srv/git/ansible-private` on [.title-ref]#batcave01#.
|
||||
+
|
||||
[CAUTION]
|
||||
.Caution
|
||||
====
|
||||
This is a private repository for passwords and other sensitive data. It
|
||||
is not available in cgit, nor should it be cloned or copied remotely.
|
||||
====
|
||||
+
|
||||
This repository is only accessible to members of 'sysadmin-main'.
|
||||
|
||||
=== Cron job/scheduled runs
|
||||
|
||||
With use of run_ansible-playbook_cron.py that is run daily via cron we
|
||||
walk through playbooks and run them with [.title-ref]#--check --diff#
|
||||
params to perform a dry-run.
|
||||
|
||||
This way we make sure all the playbooks are idempotent and there is no
|
||||
unexpected changes on servers (or playbooks).
|
||||
|
||||
=== Logging
|
||||
|
||||
We have in place a callback plugin that stores history for any
|
||||
ansible-playbook runs and then sends a report each day to
|
||||
sysadmin-logs-members with any CHANGED or FAILED actions. Additionally,
|
||||
there's a fedmsg plugin that reports start and end of ansible playbook
|
||||
runs to the fedmsg bus. Ansible also logs to syslog verbose reporting of
|
||||
when and what commands and playbooks were run.
|
||||
|
||||
=== role based access control for playbooks
|
||||
|
||||
There's a wrapper script on batcave01 called 'rbac-playbook' that allows
|
||||
non sysadmin-main members to run specific playbooks against specific
|
||||
groups of hosts. This is part of the ansible_utils package. The upstream
|
||||
for ansible_utils is: https://bitbucket.org/tflink/ansible_utils
|
||||
|
||||
To add a new group:
|
||||
|
||||
[arabic]
|
||||
. add the playbook name and sysadmin group to the rbac-playbook
|
||||
(ansible-private repo)
|
||||
. add that sysadmin group to sudoers on batcave01 (also in
|
||||
ansible-private repo)
|
||||
|
||||
To use the wrapper:
|
||||
|
||||
....
|
||||
sudo rbac-playbook playbook.yml
|
||||
....
|
||||
|
||||
== Directory setup
|
||||
|
||||
=== Inventory
|
||||
|
||||
The inventory directory tells ansible all the hosts that are managed by
|
||||
it and the groups they are in. All files in this dir are concatenated
|
||||
together, so you can split out groups/hosts into separate files for
|
||||
readability. They are in ini file format.
|
||||
|
||||
Additionally under the inventory directory are host_vars and group_vars
|
||||
subdirectories. These are files named for the host or group and
|
||||
containing variables to set for that host or group. You should strive to
|
||||
set variables in the highest level possible, and precedence is in:
|
||||
global, group, host order.
|
||||
|
||||
=== Vars
|
||||
|
||||
This directory contains global variables as well as OS specific
|
||||
variables. Note that in order to use the OS specific ones you must have
|
||||
'gather_facts' as 'True' or ansible will not have the facts it needs to
|
||||
determine the OS.
|
||||
|
||||
=== Roles
|
||||
|
||||
Roles are a collection of tasks/files/templates that can be used on any
|
||||
host or group of hosts that all share that role. In other words, roles
|
||||
should be used except in cases where configuration only applies to a
|
||||
single host. Roles can be reused between hosts and groups and are more
|
||||
portable/flexable than tasks or specific plays.
|
||||
|
||||
=== Scripts
|
||||
|
||||
In the ansible git repo under scripts are a number of utilty scripts for
|
||||
sysadmins.
|
||||
|
||||
=== Playbooks
|
||||
|
||||
In the ansible git repo there's a directory for playbooks. The top level
|
||||
contains utility playbooks for sysadmins. These playbooks perform
|
||||
one-off functions or gather information. Under this directory are hosts
|
||||
and groups playbooks. These playbooks are for specific hosts and groups
|
||||
of hosts, from provision to fully configured. You should only use a host
|
||||
playbook in cases where there will never be more than one of that thing.
|
||||
|
||||
=== Tasks
|
||||
|
||||
This directory contains one-off tasks that are used in playbooks. Some
|
||||
of these should be migrated to roles (we had this setup before roles
|
||||
existed in ansible). Those that are truely only used on one host/group
|
||||
could stay as isolated tasks.
|
||||
|
||||
=== Syntax
|
||||
|
||||
Ansible now warns about depreciated syntax. Please fix any cases you see
|
||||
related to depreciation warnings.
|
||||
|
||||
Templates use the jinja2 syntax.
|
||||
|
||||
== Libvirt virtuals
|
||||
|
||||
* TODO: add steps to make new libvirt virtuals in staging and production
|
||||
* TODO: merge in new-hosts.txt
|
||||
|
||||
== Cloud Instances
|
||||
|
||||
* TODO: add how to make new cloud instances
|
||||
* TODO: merge in from ansible README file.
|
||||
|
||||
== rdiff-backups
|
||||
|
||||
see:
|
||||
https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/rdiff-backup.html
|
||||
|
||||
== Additional Reading/Resources
|
||||
|
||||
Upstream docs:::
|
||||
https://docs.ansible.com/
|
||||
Example repo with all kinds of examples:::
|
||||
* https://github.com/ansible/ansible-examples
|
||||
* https://gist.github.com/marktheunissen/2979474
|
||||
Jinja2 docs:::
|
||||
http://jinja.pocoo.org/docs/
|
||||
Loading…
Add table
Add a link
Reference in a new issue