= Virtual Host Addition SOP

You should be able to follow these steps in order to create a new set of
hosts in infrastructure.

== Walkthrough

=== Get a DNS repo checkout on batcave01

....
git clone /srv/git/dns
cd dns
....

An example always helps, so you can use git grep for something that has
been recently added to the data center/network that you want:

....
git grep badges-web01
  built/126.5.10.in-addr.arpa:69       IN        PTR      badges-web01.stg.iad2.fedoraproject.org.
  [...lots of other stuff in built/ ignore these as they'll be generated later...]
  master/126.5.10.in-addr.arpa:69       IN        PTR      badges-web01.stg.iad2.fedoraproject.org.
  master/126.5.10.in-addr.arpa:101      IN        PTR      badges-web01.iad2.fedoraproject.org.
  master/126.5.10.in-addr.arpa:102      IN        PTR      badges-web02.iad2.fedoraproject.org.
  master/168.192.in-addr.arpa:109.1   IN      PTR     badges-web01.vpn.fedoraproject.org
  master/168.192.in-addr.arpa:110.1   IN      PTR     badges-web02.vpn.fedoraproject.org
  master/iad2.fedoraproject.org:badges-web01.stg   IN      A       10.5.126.69
  master/iad2.fedoraproject.org:badges-web01        IN  A       10.5.126.101
  master/iad2.fedoraproject.org:badges-web02        IN  A       10.5.126.102
  master/vpn.fedoraproject.org:badges-web01    IN A         192.168.1.109
  master/vpn.fedoraproject.org:badges-web02    IN A         192.168.1.110
....

So those are the files we need to edit. In the above example, two of
those files are for the host on the IAD network. The other two are for
the host to be able to talk over the VPN. Although the VPN is not always
needed, the common case is that the host will need it. (If any clients
_need to connect to it via the proxy servers_ or it is not hosted in
IAD2 it will need a VPN connection). An common exception is here the
staging environment: since we only have one proxy server in staging and
it is in IAD2, a VPN connection is not typically needed for staging
hosts.

Edit the zone file for the reverse lookup first (the *in-addr.arpa file)
and find ips to use. The ips will be listed with a domain name of
"unused." If you're configuring a web application server, you probably
want two hosts for stg and at least two for production. Two in
production means that we don't need downtime for reboots and updates.
Two in stg means that we'll be less likely to encounter problems related
to having multiple web application servers when we take a change tested
in stg into production:

....
-105      IN        PTR      unused.
-106      IN        PTR      unused.
-107      IN        PTR      unused.
-108      IN        PTR      unused.
+105      IN        PTR      elections01.stg.iad2.fedoraproject.org.
+106      IN        PTR      elections02.stg.iad2.fedoraproject.org.
+107      IN        PTR      elections01.iad2.fedoraproject.org.
+108      IN        PTR      elections02.iad2.fedoraproject.org.
....

Edit the forward domain (iad2.fedoraproject.org in our example) next:

....
elections01.stg IN      A       10.5.126.105
elections02.stg IN      A       10.5.126.106
elections01     IN      A       10.5.126.107
elections02     IN      A       10.5.126.108
....

Repeat these two steps if you need to make them available on the VPN.
Note: if your stg hosts are in IAD2, you don't need to configure VPN for
them as all our stg proxy servers are in IAD2.

Also remember to update the Serial at the top of all zone files.

Once the files are edited, you need to run a script to build the zones.
But first, commit the changes you just made to the "source":

....
git add .
git commit -a -m 'Added staging and production elections hosts.'
....

Once that is committed, you need to run a script to build the zones and
then push them to the dns servers.:

....
sudo ./do-domains # This builds the files
git add .
git commit -a -m 'done build'
sudo git push

$ sudo -i ansible ns\* -a '/usr/local/bin/update-dns' # This tells the dns servers to load the new files
....

== Make certs

[WARNING]
====
If you already had a clone of private, make VERY sure to do a
git pull first! It's quite likely somebody else added a new host without
you noticing it, and you cannot merge the keys repos manually.
(seriously, don't: the index and serial files just wouldn't match up
with the certificate, and you would revoke the wrong certificate upon
revocation).
====

When doing 2 factor auth for sudo, the hosts that we connect from need
to have valid SSL Certs. These are currently stored in the private repo:

....
git clone /srv/git/ansible-private && chmod 0700 ansible-private
cd ansible-private/files/2fa-certs
. ./vars
./build-and-sign-key $FQDN  # ex: elections01.stg.iad2.fedoraproject.org
....

The `$FQDN` should be the iad2 domain name if it's in iad2, vpn if not in
iad2, and if it has no vpn and is not in iad2 we should add it to the
vpn.:

....
git add .
git commit -a
git push
....

[NOTE]
====
Make sure to re-run vars from the vpn repo. If you forget to do
that, You will just (try to) generate a second pair of 2fa certs, since
the ./vars script create an environment var to the root key directory,
which is different.
====

Servers that are on the VPN also need certs for that. These are also
stored in the private repo:

....
cd ansible-private/files/vpn/openvpn
. ./vars
./build-and-sign-key $FQDN  # ex: elections01.iad2.fedoraproject.org
./build-and-sign-key $FQDN  # ex: elections02.iad2.fedoraproject.org
....

The `$FQDN` should be the iad2 domain name if it's in iad2, and just
fedoraproject.org if it's not in IAD2 (note that there is never .vpn in
the FQDN in the openvpn keys). Now commit and push.:

....
git add .
git commit -a
git push
....

== ansible

....
git clone https://pagure.io/fedora-infra/ansible.git
cd ansible
....

To see an example:

....
git grep badges-web01 (example)
find . -name badges-web01\*
find . -name badges-web'\'*'
....

=== inventory

The ansible inventory file lists all the hosts that ansible knows about
and also allows you to create sets of hosts that you can refer to via a
group name. For a typical web application server set of hosts we'd
create things like this:

....
[elections]
elections01.iad2.fedoraproject.org
elections02.iad2.fedoraproject.org

[elections-stg]
elections01.stg.iad2.fedoraproject.org
elections02.stg.iad2.fedoraproject.org

[... find the staging group and add there: ...]

[staging]
db-fas01.stg.iad2.fedoraproject.org
elections01.stg.iad2.fedoraproject.org
electionst02.stg.iad2.fedoraproject.org
....

The hosts should use their fully qualified domain names here. The rules
are slightly different than for 2fa certs. If the host is in IAD2, use
the .iad2.fedoraproject.org domain name. If they aren't in IAD2, then
they usually just have .fedoraproject.org as their domain name. (If in
doubt about a not-in-IAD2 host, just ask).

=== VPN config

If the machine is in VPN, create a file in ansible at
`roles/openvpn/server/files/ccd/$FQDN` with contents like:

....
ifconfig-push 192.168.1.X 192.168.0.X
....

Where X is the last octet of the DNS IP address assigned to the host, so
for example for _elections01.iad2.fedoraproject.org_ that would be:

....
ifconfig-push 192.168.1.44 192.168.0.44
....

=== host_vars and group_vars

ansible consults files in
https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars[inventory/group_vars]
and https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/host_vars[inventory/host_vars]
to set parameters that can be used in templates and playbooks. You may
need to edit these

It's usually easy to copy the `host_vars` and `group_vars` from an existing
host that's similar to the one you are working on and then modify a few
names to make it work. For instance, for a web application server:

....
cd ~/ansible/inventory/group_vars
cp badges-web elections
....

Change the following:

....
- fas_client_groups: sysadmin-noc,sysadmin-badges
+ fas_client_groups: sysadmin-noc,sysadmin-web
....

(You can change disk size, mem_size, number of cpus, and ports too if
you need them).

Some things will definitely need to be defined differently for each host
in a group -- notably, ip_address. You should use the ip_address you
claimed in the dns repo:

....
cd ~/ansible/inventory/host_vars
cp badges-web01.stg.iad2.fedoraproject.org elections01.stg.iad2.fedoraproject.org
<edit appropriately>
....

The host will need vmhost declaration. There is a script in
`ansible/scripts/vhost-info` that will report how much free memory and
how many free cpus each vmhost has. You can use that to inform your
decision. By convention, staging hosts go on virthost12.

[NOTE]
====
The `vhost-info` needs to be executed as `root` user on `batcave01`
....
python3.11 /srv/web/infra/ansible/scripts/vhost-info --host <ansible list of hosts or groups>
# for example virthost
....
====

Each vmhost has a different volume group. To figure out what volume
group that is, execute the following command on the virthost.:

....
vgdisplay
....

You mant want to run "lsblk" to check that the volume group you expect
is the one actually used for virtual guests.

[NOTE]
====
19:16:01 <nirik> 3. add ./inventory/host_vars/FQDN host_vars for the new host.

19:16:56 <nirik> that will have in it ip addresses, dns
resolv.conf, ks url/repo, volume group to make the host lv in, etc etc.

19:17:10 <nirik> 4. add any needed vars to inventory/group_vars/ for
the group

19:17:33 <nirik> this has memory size, lvm size, cpus, etc |

19:17:45 <nirik> 5. add tasks/virt_instance_create.yml task to top of
group/host playbook

19:18:10 <nirik> 6. run the playbook and it will
go to the virthost you set, create the lv, guest, install it, wait for
it to come up, then continue configuring it.
====

mailman.yml::
  copy it from another file.

....
./ans-vhost-freemem --hosts=virtost\*
....

group vars

* vmhost (of the host that will host the VM)
* kickstart info (url of the kickstart itself and the repo)
* datacenter (although most likely won't change)

The host playbook is rather basic

* Change the name
* Most things won't change much

....
ansible-playbook /srv/web/infra/ansible/infra/ansible/playbooks/groups/mailman.yml
....

== Adding a new proxy or webserver

When adding a new web server other files must be edited by hand
currently until templates replace them. These files cover getting httpd
logs from the server onto log01 so that log analysis can be done.

....
roles/base/files/syncHttpLogs.sh
roles/epylog/files/merged/modules.d/rsyncd.conf
roles/hosts/files/staging-hosts
roles/mediawiki123/templates/LocalSettings.php.fp.j2
....

There are also nagios files which will need to be edited but that should
be done following the nagios document.

== References

* The making a new instance section of:
http://meetbot.fedoraproject.org/meetbot/fedora-meeting-1/2013-07-17/infrastructure-ansible-meetup.2013-07-17-19.00.html
