340 lines
10 KiB
Text
340 lines
10 KiB
Text
= Virtual Host Addition SOP
|
|
|
|
You should be able to follow these steps in order to create a new set of
|
|
hosts in infrastructure.
|
|
|
|
== Walkthrough
|
|
|
|
=== Get a DNS repo checkout on batcave01
|
|
|
|
....
|
|
git clone /srv/git/dns
|
|
cd dns
|
|
....
|
|
|
|
An example always helps, so you can use git grep for something that has
|
|
been recently added to the data center/network that you want:
|
|
|
|
....
|
|
git grep badges-web01
|
|
built/126.5.10.in-addr.arpa:69 IN PTR badges-web01.stg.rdu3.fedoraproject.org.
|
|
[...lots of other stuff in built/ ignore these as they'll be generated later...]
|
|
master/126.5.10.in-addr.arpa:69 IN PTR badges-web01.stg.rdu3.fedoraproject.org.
|
|
master/126.5.10.in-addr.arpa:101 IN PTR badges-web01.rdu3.fedoraproject.org.
|
|
master/126.5.10.in-addr.arpa:102 IN PTR badges-web02.rdu3.fedoraproject.org.
|
|
master/168.192.in-addr.arpa:109.1 IN PTR badges-web01.vpn.fedoraproject.org
|
|
master/168.192.in-addr.arpa:110.1 IN PTR badges-web02.vpn.fedoraproject.org
|
|
master/rdu3.fedoraproject.org:badges-web01.stg IN A 10.5.126.69
|
|
master/rdu3.fedoraproject.org:badges-web01 IN A 10.5.126.101
|
|
master/rdu3.fedoraproject.org:badges-web02 IN A 10.5.126.102
|
|
master/vpn.fedoraproject.org:badges-web01 IN A 192.168.1.109
|
|
master/vpn.fedoraproject.org:badges-web02 IN A 192.168.1.110
|
|
....
|
|
|
|
So those are the files we need to edit. In the above example, two of
|
|
those files are for the host on the IAD network. The other two are for
|
|
the host to be able to talk over the VPN. Although the VPN is not always
|
|
needed, the common case is that the host will need it. (If any clients
|
|
_need to connect to it via the proxy servers_ or it is not hosted in
|
|
RDU3 it will need a VPN connection). An common exception is here the
|
|
staging environment: since we only have one proxy server in staging and
|
|
it is in RDU3, a VPN connection is not typically needed for staging
|
|
hosts.
|
|
|
|
Edit the zone file for the reverse lookup first (the *in-addr.arpa file)
|
|
and find ips to use. The ips will be listed with a domain name of
|
|
"unused." If you're configuring a web application server, you probably
|
|
want two hosts for stg and at least two for production. Two in
|
|
production means that we don't need downtime for reboots and updates.
|
|
Two in stg means that we'll be less likely to encounter problems related
|
|
to having multiple web application servers when we take a change tested
|
|
in stg into production:
|
|
|
|
....
|
|
-105 IN PTR unused.
|
|
-106 IN PTR unused.
|
|
-107 IN PTR unused.
|
|
-108 IN PTR unused.
|
|
+105 IN PTR elections01.stg.rdu3.fedoraproject.org.
|
|
+106 IN PTR elections02.stg.rdu3.fedoraproject.org.
|
|
+107 IN PTR elections01.rdu3.fedoraproject.org.
|
|
+108 IN PTR elections02.rdu3.fedoraproject.org.
|
|
....
|
|
|
|
Edit the forward domain (rdu3.fedoraproject.org in our example) next:
|
|
|
|
....
|
|
elections01.stg IN A 10.5.126.105
|
|
elections02.stg IN A 10.5.126.106
|
|
elections01 IN A 10.5.126.107
|
|
elections02 IN A 10.5.126.108
|
|
....
|
|
|
|
Repeat these two steps if you need to make them available on the VPN.
|
|
Note: if your stg hosts are in RDU3, you don't need to configure VPN for
|
|
them as all our stg proxy servers are in RDU3.
|
|
|
|
Also remember to update the Serial at the top of all zone files.
|
|
|
|
Once the files are edited, you need to run a script to build the zones.
|
|
But first, commit the changes you just made to the "source":
|
|
|
|
....
|
|
git add .
|
|
git commit -a -m 'Added staging and production elections hosts.'
|
|
....
|
|
|
|
Once that is committed, you need to run a script to build the zones and
|
|
then push them to the dns servers.:
|
|
|
|
....
|
|
./do-domains # This builds the files
|
|
git add .
|
|
git commit -a -m 'done build'
|
|
git push
|
|
|
|
$ sudo -i ansible ns\* -a '/usr/local/bin/update-dns' # This tells the dns servers to load the new files
|
|
....
|
|
|
|
== Make certs
|
|
|
|
[WARNING]
|
|
====
|
|
If you already had a clone of private, make VERY sure to do a
|
|
git pull first! It's quite likely somebody else added a new host without
|
|
you noticing it, and you cannot merge the keys repos manually.
|
|
(seriously, don't: the index and serial files just wouldn't match up
|
|
with the certificate, and you would revoke the wrong certificate upon
|
|
revocation).
|
|
====
|
|
|
|
When doing 2 factor auth for sudo, the hosts that we connect from need
|
|
to have valid SSL Certs. These are currently stored in the private repo:
|
|
|
|
....
|
|
git clone /srv/git/ansible-private && chmod 0700 ansible-private
|
|
cd ansible-private/files/2fa-certs
|
|
. ./vars
|
|
./build-and-sign-key $FQDN # ex: elections01.stg.rdu3.fedoraproject.org
|
|
....
|
|
|
|
The `$FQDN` should be the rdu3 domain name if it's in rdu3, vpn if not in
|
|
rdu3, and if it has no vpn and is not in rdu3 we should add it to the
|
|
vpn.:
|
|
|
|
....
|
|
git add .
|
|
git commit -a
|
|
git push
|
|
....
|
|
|
|
[NOTE]
|
|
====
|
|
Make sure to re-run vars from the vpn repo. If you forget to do
|
|
that, You will just (try to) generate a second pair of 2fa certs, since
|
|
the ./vars script create an environment var to the root key directory,
|
|
which is different.
|
|
====
|
|
|
|
Servers that are on the VPN also need certs for that. These are also
|
|
stored in the private repo:
|
|
|
|
....
|
|
cd ansible-private/files/vpn/
|
|
./addhost.sh $FQDN # ex: zabbix01.rdu3.fedoraproject.org
|
|
....
|
|
|
|
The `$FQDN` should be the rdu3 domain name if it's in rdu3, and just
|
|
fedoraproject.org if it's not in RDU3 (note that there is never .vpn in
|
|
the FQDN in the openvpn keys). Now commit and push.:
|
|
|
|
....
|
|
git add .
|
|
git commit -a
|
|
git push
|
|
....
|
|
|
|
== ansible
|
|
|
|
....
|
|
git clone https://pagure.io/fedora-infra/ansible.git
|
|
cd ansible
|
|
....
|
|
|
|
To see an example:
|
|
|
|
....
|
|
git grep badges-web01 (example)
|
|
find . -name badges-web01\*
|
|
find . -name badges-web'\'*'
|
|
....
|
|
|
|
=== inventory
|
|
|
|
The ansible inventory file lists all the hosts that ansible knows about
|
|
and also allows you to create sets of hosts that you can refer to via a
|
|
group name. For a typical web application server set of hosts we'd
|
|
create things like this:
|
|
|
|
....
|
|
[elections]
|
|
elections01.rdu3.fedoraproject.org
|
|
elections02.rdu3.fedoraproject.org
|
|
|
|
[elections-stg]
|
|
elections01.stg.rdu3.fedoraproject.org
|
|
elections02.stg.rdu3.fedoraproject.org
|
|
|
|
[... find the staging group and add there: ...]
|
|
|
|
[staging]
|
|
db-fas01.stg.rdu3.fedoraproject.org
|
|
elections01.stg.rdu3.fedoraproject.org
|
|
electionst02.stg.rdu3.fedoraproject.org
|
|
....
|
|
|
|
The hosts should use their fully qualified domain names here. The rules
|
|
are slightly different than for 2fa certs. If the host is in RDU3, use
|
|
the .rdu3.fedoraproject.org domain name. If they aren't in RDU3, then
|
|
they usually just have .fedoraproject.org as their domain name. (If in
|
|
doubt about a not-in-RDU3 host, just ask).
|
|
|
|
=== VPN config
|
|
|
|
If the machine is in VPN, create a file in ansible at
|
|
`roles/openvpn/server/files/ccd/$FQDN` with contents like:
|
|
|
|
....
|
|
ifconfig-push 192.168.1.X 192.168.0.X
|
|
....
|
|
|
|
Where X is the last octet of the DNS IP address assigned to the host, so
|
|
for example for _elections01.rdu3.fedoraproject.org_ that would be:
|
|
|
|
....
|
|
ifconfig-push 192.168.1.44 192.168.0.44
|
|
....
|
|
|
|
=== host_vars and group_vars
|
|
|
|
ansible consults files in
|
|
https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars[inventory/group_vars]
|
|
and https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/host_vars[inventory/host_vars]
|
|
to set parameters that can be used in templates and playbooks. You may
|
|
need to edit these
|
|
|
|
It's usually easy to copy the `host_vars` and `group_vars` from an existing
|
|
host that's similar to the one you are working on and then modify a few
|
|
names to make it work. For instance, for a web application server:
|
|
|
|
....
|
|
cd ~/ansible/inventory/group_vars
|
|
cp badges-web elections
|
|
....
|
|
|
|
Change the following:
|
|
|
|
....
|
|
- fas_client_groups: sysadmin-noc,sysadmin-badges
|
|
+ fas_client_groups: sysadmin-noc,sysadmin-web
|
|
....
|
|
|
|
(You can change disk size, mem_size, number of cpus, and ports too if
|
|
you need them).
|
|
|
|
Some things will definitely need to be defined differently for each host
|
|
in a group -- notably, ip_address. You should use the ip_address you
|
|
claimed in the dns repo:
|
|
|
|
....
|
|
cd ~/ansible/inventory/host_vars
|
|
cp badges-web01.stg.rdu3.fedoraproject.org elections01.stg.rdu3.fedoraproject.org
|
|
<edit appropriately>
|
|
....
|
|
|
|
The host will need vmhost declaration. There is a script in
|
|
`ansible/scripts/vhost-info` that will report how much free memory and
|
|
how many free cpus each vmhost has. You can use that to inform your
|
|
decision. By convention, staging hosts go on virthost12.
|
|
|
|
[NOTE]
|
|
====
|
|
The `vhost-info` needs to be executed as `root` user on `batcave01`
|
|
....
|
|
python3.11 /srv/web/infra/ansible/scripts/vhost-info --host <ansible list of hosts or groups>
|
|
# for example virthost
|
|
....
|
|
====
|
|
|
|
Each vmhost has a different volume group. To figure out what volume
|
|
group that is, execute the following command on the virthost.:
|
|
|
|
....
|
|
vgdisplay
|
|
....
|
|
|
|
You mant want to run "lsblk" to check that the volume group you expect
|
|
is the one actually used for virtual guests.
|
|
|
|
[NOTE]
|
|
====
|
|
19:16:01 <nirik> 3. add ./inventory/host_vars/FQDN host_vars for the new host.
|
|
|
|
19:16:56 <nirik> that will have in it ip addresses, dns
|
|
resolv.conf, ks url/repo, volume group to make the host lv in, etc etc.
|
|
|
|
19:17:10 <nirik> 4. add any needed vars to inventory/group_vars/ for
|
|
the group
|
|
|
|
19:17:33 <nirik> this has memory size, lvm size, cpus, etc |
|
|
|
|
19:17:45 <nirik> 5. add tasks/virt_instance_create.yml task to top of
|
|
group/host playbook
|
|
|
|
19:18:10 <nirik> 6. run the playbook and it will
|
|
go to the virthost you set, create the lv, guest, install it, wait for
|
|
it to come up, then continue configuring it.
|
|
====
|
|
|
|
mailman.yml::
|
|
copy it from another file.
|
|
|
|
....
|
|
./ans-vhost-freemem --hosts=virtost\*
|
|
....
|
|
|
|
group vars
|
|
|
|
* vmhost (of the host that will host the VM)
|
|
* kickstart info (url of the kickstart itself and the repo)
|
|
* datacenter (although most likely won't change)
|
|
|
|
The host playbook is rather basic
|
|
|
|
* Change the name
|
|
* Most things won't change much
|
|
|
|
....
|
|
ansible-playbook /srv/web/infra/ansible/infra/ansible/playbooks/groups/mailman.yml
|
|
....
|
|
|
|
== Adding a new proxy or webserver
|
|
|
|
When adding a new web server other files must be edited by hand
|
|
currently until templates replace them. These files cover getting httpd
|
|
logs from the server onto log01 so that log analysis can be done.
|
|
|
|
....
|
|
roles/base/files/syncHttpLogs.sh
|
|
roles/epylog/files/merged/modules.d/rsyncd.conf
|
|
roles/hosts/files/staging-hosts
|
|
roles/mediawiki123/templates/LocalSettings.php.fp.j2
|
|
....
|
|
|
|
There are also nagios files which will need to be edited but that should
|
|
be done following the nagios document.
|
|
|
|
== References
|
|
|
|
* The making a new instance section of:
|
|
http://meetbot.fedoraproject.org/meetbot/fedora-meeting-1/2013-07-17/infrastructure-ansible-meetup.2013-07-17-19.00.html
|