Infrastructure/infra-docs-fpo
1
0
Fork 0
Code Issues 100 Pull requests 6 Projects Releases Packages Wiki Activity Actions

Review new-hosts SOP

Browse source 
Signed-off-by: Michal Konečný <mkonecny@redhat.com>
This commit is contained in:
Michal Konečný 2021-09-07 14:44:51 +02:00
parent a3b51f1193
commit 6eb29f495c
2 changed files with 68 additions and 54 deletions
Download patch file Download diff file Expand all files Collapse all files

2
modules/sysadmin_guide/nav.adoc
View file

@ -76,7 +76,7 @@
** xref:mote.adoc[mote - SOP]
** xref:nagios.adoc[Fedora Infrastructure Nagios - SOP]
** xref:netapp.adoc[Netapp Infrastructure - SOP]
** xref:new-hosts.adoc[new-hosts - SOP in review ]
** xref:new-hosts.adoc[DNS Host Addition - SOP]
** xref:nonhumanaccounts.adoc[nonhumanaccounts - SOP in review ]
** xref:nuancier.adoc[nuancier - SOP in review ]
** xref:odcs.adoc[odcs - SOP in review ]

120
modules/sysadmin_guide/pages/new-hosts.adoc
View file

@ -17,28 +17,28 @@ been recently added to the data center/network that you want:
....
git grep badges-web01
built/126.5.10.in-addr.arpa:69 IN PTR badges-web01.stg.phx2.fedoraproject.org.
built/126.5.10.in-addr.arpa:69 IN PTR badges-web01.stg.iad2.fedoraproject.org.
[...lots of other stuff in built/ ignore these as they'll be generated later...]
master/126.5.10.in-addr.arpa:69 IN PTR badges-web01.stg.phx2.fedoraproject.org.
master/126.5.10.in-addr.arpa:101 IN PTR badges-web01.phx2.fedoraproject.org.
master/126.5.10.in-addr.arpa:102 IN PTR badges-web02.phx2.fedoraproject.org.
master/126.5.10.in-addr.arpa:69 IN PTR badges-web01.stg.iad2.fedoraproject.org.
master/126.5.10.in-addr.arpa:101 IN PTR badges-web01.iad2.fedoraproject.org.
master/126.5.10.in-addr.arpa:102 IN PTR badges-web02.iad2.fedoraproject.org.
master/168.192.in-addr.arpa:109.1 IN PTR badges-web01.vpn.fedoraproject.org
master/168.192.in-addr.arpa:110.1 IN PTR badges-web02.vpn.fedoraproject.org
master/phx2.fedoraproject.org:badges-web01.stg IN A 10.5.126.69
master/phx2.fedoraproject.org:badges-web01 IN A 10.5.126.101
master/phx2.fedoraproject.org:badges-web02 IN A 10.5.126.102
master/iad2.fedoraproject.org:badges-web01.stg IN A 10.5.126.69
master/iad2.fedoraproject.org:badges-web01 IN A 10.5.126.101
master/iad2.fedoraproject.org:badges-web02 IN A 10.5.126.102
master/vpn.fedoraproject.org:badges-web01 IN A 192.168.1.109
master/vpn.fedoraproject.org:badges-web02 IN A 192.168.1.110
....
So those are the files we need to edit. In the above example, two of
those files are for the host on the PHX network. The other two are for
those files are for the host on the IAD network. The other two are for
the host to be able to talk over the VPN. Although the VPN is not always
needed, the common case is that the host will need it. (If any clients
_need to connect to it via the proxy servers_ or it is not hosted in
PHX2 it will need a VPN connection). An common exception is here the
IAD2 it will need a VPN connection). An common exception is here the
staging environment: since we only have one proxy server in staging and
it is in PHX2, a VPN connection is not typically needed for staging
it is in IAD2, a VPN connection is not typically needed for staging
hosts.
Edit the zone file for the reverse lookup first (the *in-addr.arpa file)
@ -55,13 +55,13 @@ in stg into production:
-106 IN PTR unused.
-107 IN PTR unused.
-108 IN PTR unused.
+105 IN PTR elections01.stg.phx2.fedoraproject.org.
+106 IN PTR elections02.stg.phx2.fedoraproject.org.
+107 IN PTR elections01.phx2.fedoraproject.org.
+108 IN PTR elections02.phx2.fedoraproject.org.
+105 IN PTR elections01.stg.iad2.fedoraproject.org.
+106 IN PTR elections02.stg.iad2.fedoraproject.org.
+107 IN PTR elections01.iad2.fedoraproject.org.
+108 IN PTR elections02.iad2.fedoraproject.org.
....
Edit the forward domain (phx2.fedoraproject.org in our example) next:
Edit the forward domain (iad2.fedoraproject.org in our example) next:
....
elections01.stg IN A 10.5.126.105
@ -71,8 +71,8 @@ elections02 IN A 10.5.126.108
....
Repeat these two steps if you need to make them available on the VPN.
Note: if your stg hosts are in PHX2, you don't need to configure VPM for
them as all our stg proxy servers are in PHX2.
Note: if your stg hosts are in IAD2, you don't need to configure VPN for
them as all our stg proxy servers are in IAD2.
Also remember to update the Serial at the top of all zone files.
@ -98,12 +98,15 @@ $ sudo -i ansible ns\* -a '/usr/local/bin/update-dns' # This tells the dns serve
== Make certs
WARNING: If you already had a clone of private, make VERY sure to do a
[WARNING]
====
If you already had a clone of private, make VERY sure to do a
git pull first! It's quite likely somebody else added a new host without
you noticing it, and you cannot merge the keys repos manually.
(seriously, don't: the index and serial files just wouldn't match up
with the certificate, and you would revoke the wrong certificate upon
revocation).
====
When doing 2 factor auth for sudo, the hosts that we connect from need
to have valid SSL Certs. These are currently stored in the private repo:
@ -112,11 +115,11 @@ to have valid SSL Certs. These are currently stored in the private repo:
git clone /srv/git/ansible-private && chmod 0700 ansible-private
cd ansible-private/files/2fa-certs
. ./vars
./build-and-sign-key $FQDN # ex: elections01.stg.phx2.fedoraproject.org
./build-and-sign-key $FQDN # ex: elections01.stg.iad2.fedoraproject.org
....
The $FQDN should be the phx2 domain name if it's in phx2, vpn if not in
phx2, and if it has no vpn and is not in phx2 we should add it to the
The `$FQDN` should be the iad2 domain name if it's in iad2, vpn if not in
iad2, and if it has no vpn and is not in iad2 we should add it to the
vpn.:
....
@ -125,10 +128,13 @@ git commit -a
git push
....
NOTE: Make sure to re-run vars from the vpn repo. If you forget to do
[NOTE]
====
Make sure to re-run vars from the vpn repo. If you forget to do
that, You will just (try to) generate a second pair of 2fa certs, since
the ./vars script create an environment var to the root key directory,
which is different.
====
Servers that are on the VPN also need certs for that. These are also
stored in the private repo:
@ -136,12 +142,12 @@ stored in the private repo:
....
cd ansible-private/files/vpn/openvpn
. ./vars
./build-and-sign-key $FQDN # ex: elections01.phx2.fedoraproject.org
./build-and-sign-key $FQDN # ex: elections02.phx2.fedoraproject.org
./build-and-sign-key $FQDN # ex: elections01.iad2.fedoraproject.org
./build-and-sign-key $FQDN # ex: elections02.iad2.fedoraproject.org
....
The $FQDN should be the phx2 domain name if it's in phx2, and just
fedoraproject.org if it's not in PHX2 (note that there is never .vpn in
The `$FQDN` should be the iad2 domain name if it's in iad2, and just
fedoraproject.org if it's not in IAD2 (note that there is never .vpn in
the FQDN in the openvpn keys). Now commit and push.:
....
@ -174,42 +180,42 @@ create things like this:
....
[elections]
elections01.phx2.fedoraproject.org
elections02.phx2.fedoraproject.org
elections01.iad2.fedoraproject.org
elections02.iad2.fedoraproject.org
[elections-stg]
elections01.stg.phx2.fedoraproject.org
elections02.stg.phx2.fedoraproject.org
elections01.stg.iad2.fedoraproject.org
elections02.stg.iad2.fedoraproject.org
[... find the staging group and add there: ...]
[staging]
db-fas01.stg.phx2.fedoraproject.org
elections01.stg.phx2.fedoraproject.org
electionst02.stg.phx2.fedoraproject.org
db-fas01.stg.iad2.fedoraproject.org
elections01.stg.iad2.fedoraproject.org
electionst02.stg.iad2.fedoraproject.org
....
The hosts should use their fully qualified domain names here. The rules
are slightly different than for 2fa certs. If the host is in PHX2, use
the .phx2.fedoraproject.org domain name. If they aren't in PHX2, then
are slightly different than for 2fa certs. If the host is in IAD2, use
the .iad2.fedoraproject.org domain name. If they aren't in IAD2, then
they usually just have .fedoraproject.org as their domain name. (If in
doubt about a not-in-PHX2 host, just ask).
doubt about a not-in-IAD2 host, just ask).
=== VPN config
If the machine is in VPN, create a file in ansible at
roles/openvpn/server/files/ccd/$FQDN with contents like:
`roles/openvpn/server/files/ccd/$FQDN` with contents like:
____
....
ifconfig-push 192.168.1.X 192.168.0.X
____
....
Where X is the last octet of the DNS IP address assigned to the host, so
for example for elections01.phx2.fedoraproject.org that would be:
for example for _elections01.iad2.fedoraproject.org_ that would be:
____
....
ifconfig-push 192.168.1.44 192.168.0.44
____
....
== Work in progress
@ -217,11 +223,13 @@ From here to the end of file is still being worked on
=== host_vars and group_vars
ansible consults files in inventory/group_vars and inventory/host_vars
ansible consults files in
https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars[inventory/group_vars]
and https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/host_vars[inventory/host_vars]
to set parameters that can be used in templates and playbooks. You may
need to edit these
It's usually easy to copy the host_vars and group_vars from an existing
It's usually easy to copy the `host_vars` and `group_vars` from an existing
host that's similar to the one you are working on and then modify a few
names to make it work. For instance, for a web application server:
@ -246,7 +254,7 @@ claimed in the dns repo:
....
cd ~/ansible/inventory/host_vars
cp badges-web01.stg.phx2.fedoraproject.org elections01.stg.phx2.fedoraproject.org
cp badges-web01.stg.iad2.fedoraproject.org elections01.stg.iad2.fedoraproject.org
<edit appropriately>
....
@ -266,15 +274,21 @@ You mant want to run "lsblk" to check that the volume group you expect
is the one actually used for virtual guests.
[NOTE]
.Note
====
| 19:16:01 <nirik> 3. add ./inventory/host_vars/FQDN host_vars for the
new host. | 19:16:56 <nirik> that will have in it ip addresses, dns
19:16:01 <nirik> 3. add ./inventory/host_vars/FQDN host_vars for the new host.
19:16:56 <nirik> that will have in it ip addresses, dns
resolv.conf, ks url/repo, volume group to make the host lv in, etc etc.
| 19:17:10 <nirik> 4. add any needed vars to inventory/group_vars/ for
the group | 19:17:33 <nirik> this has memory size, lvm size, cpus, etc |
19:17:10 <nirik> 4. add any needed vars to inventory/group_vars/ for
the group
19:17:33 <nirik> this has memory size, lvm size, cpus, etc |
19:17:45 <nirik> 5. add tasks/virt_instance_create.yml task to top of
group/host playbook | 19:18:10 <nirik> 6. run the playbook and it will
group/host playbook
19:18:10 <nirik> 6. run the playbook and it will
go to the virthost you set, create the lv, guest, install it, wait for
it to come up, then continue configuring it.
====
@ -307,12 +321,12 @@ When adding a new web server other files must be edited by hand
currently until templates replace them. These files cover getting httpd
logs from the server onto log01 so that log analysis can be done.
____
....
roles/base/files/syncHttpLogs.sh
roles/epylog/files/merged/modules.d/rsyncd.conf
roles/hosts/files/staging-hosts
roles/mediawiki123/templates/LocalSettings.php.fp.j2
____
....
There are also nagios files which will need to be edited but that should
be done following the nagios document.