= rdiff-backup SOP

== Contact Information

Owner::
  Fedora Infrastructure Team
Contact::
  #fedora-admin
Servers::
  backup03 and others
Purpose::
  backups of critical data

== Description

We are now running a rdiff-backup of all our critical data on a daily
basis. This allows us to keep incremental changes over time as well has
have a recent copy in case of disaster recovery.

The backups are run from _backup03_ every day at 22:10UTC as root. All
config is in ansible.

The cron job checks out the ansible repo from git, then runs
ansible-playbook with the rdiff-backup playbook. This playbook looks at
variables to decide which machines and partitions to backup.

* First, machines in the _backup_clients_ group in inventory are operated
on. If a host is not in that group it is not backed up via rdiff-backup.
* Next, any machines in the _backup_clients_ group will have their `/etc`
and `/home` directories backed up by the server running rdiff-backup and
using the rdiff-backup ssh key to access the client.
* Next, if any of the hosts in _backup_clients_ have a variable set for
_host_backup_targets_, those directories will also be backed up in the
same manner as above with the rdiff-backup ssh key.

For each backup an email will be sent to _sysadmin-backup-members_ with a
summary.

Backups are stored on a netapp volume, so in addition to the
incrementals that rdiff-backup provides there are netapp snapshots. This
netapp volume is mounted on `/fedora_backups` and is running dedup on the
netapp side.

== Rebooting backup03

When backup03 is rebooted, you must restart the ssh-agent and reload the
rdiff-backup ssh key into that agent so backups can take place.

....
sudo -i
ssh-agent -s > sshagent
source sshgent
ssh-add .ssh/rdiff-backup-key
....

== Adding a new host to backups

[arabic]
. add the host to the backup_clients inventory group in ansible.
. If you wish to backup more than `/etc` and `/home`, add a variable to
  `inventory/group_vars/<app_name>` like: `host_backup_targets: ['/srv']`
  See https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/pagure#_81
  for example.
. On the client to be backed up, install rdiff-backup.
.  On the client to be backed up, install the rdiff-backup ssh public key
to `/root/.ssh/authorized_keys` It should be restricted from:
+
....
from="10.5.126.161,192.168.1.64" 
....
+
and command can be restricted to:
+
....
command="rdiff-backup --server --restrict-update-only"
....

== Restoring from backups

rdiff backup keeps a copy of the most recent version of files on disk,
so if you wish to restore the last backup copy, simply rsync from
_backup03_. If you wish an older incremental, see rdiff-backup man page
for how to specify the exact time.

== Retention

Backups are currently kept forever, but likely down the road we will
look at pruning them some to match available space.

== Public_key:

....
ssh-dss
AAAAB3NzaC1kc3MAAACBAJr3xqn/hHIXeth+NuXPu9P91FG9jozF3Q1JaGmg6szo770rrmhiSsxso/Ibm2mObqQLCyfm/qSOQRynv6tL3tQVHA6EEx0PNacnBcOV7UowR5kd4AYv82K1vQhof3YTxOMmNIOrdy6deDqIf4sLz1TDHvEDwjrxtFf8ugyZWNbTAAAAFQCS5puRZF4gpNbaWxe6gLzm3rBeewAAAIBcEd6pRatE2Qc/dW0YwwudTEaOCUnHmtYs2PHKbOPds0+Woe1aWH38NiE+CmklcUpyRsGEf3O0l5vm3VrVlnfuHpgt/a/pbzxm0U6DGm2AebtqEmaCX3CIuYzKhG5wmXqJ/z+Hc5MDj2mn2TchHqsk1O8VZM+1Ml6zX3Hl4vvBsQAAAIALDt5NFv6GLuid8eik/nn8NORd9FJPDBJxgVqHNIm08RMC6aI++fqwkBhVPFKBra5utrMKQmnKs/sOWycLYTqqcSMPdWSkdWYjBCSJ/QNpyN4laCmPWLgb3I+2zORgR0EjeV2e/46geS0MWLmeEsFwztpSj4Tv4e18L8Dsp2uB2Q==
root@backup03-rdiff-backup
....