Added the infra SOPs ported to asciidoc.

modules/sysadmin_guide/pages/syslog.adoc

@@ -0,0 +1,169 @@
+= Log Infrastructure SOP
+
+Logs are centrally referred to our loghost and managed from there by
+rsyslog to create several log outputs.
+
+Epylog provides twice-daily log reports of activities on our systems. It
+runs on our central loghost and generates reports on all systems
+centrally logging.
+
+== Contact Information
+
+Owner:::
+  Fedora Infrastructure Team
+Contact:::
+  #fedora-admin, sysadmin-main
+Location:::
+  Phoenix
+Servers:::
+  log01.phx2.fedoraproject.org
+Purpose:::
+  Provides our central logs and reporting
+
+== Essential data/locations:
+
+* Logs compiled using rsyslog on log01 into a single set of logs for all
+systems:
++
+....
+/var/log/merged/
+....
++
+These logs are rotated every day and kept for only 2 days. This set of
+logs is only used for immediate analysis and more trivial 'tailing' of
+the log file to watch for events.
+* Logs for each system separately in `/var/log/hosts`
++
+These logs are maintained forever, practically, or for as long as we
+possibly can. They are broken out into a `$hostname/$YEAR/$MON/$DAY`
+directory structure so we can locate a specific day's log immediately.
+* Log reports generated by epylog: Log reports generated by epylog are
+outputted to /srv/web/epylog/merged
++
+The reports are accessible via a web browser from
+https://admin.fedoraproject.org/epylog/merged/
++
+This path requires a username and a password to access. To add your
+username and password you must first join the sysadmin-logs group then
+login to `log01.phx2.fedoraproject.org` and run this command:
++
+....
+htpasswd -m /srv/web/epylog/.htpasswd $your_username
+....
++
+when prompted for a password please input a password which is NOT YOUR
+FEDORA ACCOUNT SYSTEM PASSWORD.
+
+[IMPORTANT]
+.Important
+====
+Let's say that again to be sure you got it:
+
+DO _link:[NOT] HAVE THIS BE THE SAME AS YOUR FAS PASSWORD
+====
+
+== Configs:
+
+Epylog configs are controlled by ansible - please see the ansible epylog
+module for more details. Specifically the files in
+`roles/epylog/files/merged/`
+
+=== Generating a one-off epylog report:
+
+If you wish to generate a specific log report you will need to run the
+following command on log01:
+
+....
+sudo /usr/sbin/epylog -c /etc/epylog/merged/epylog.conf --last 5h
+....
+
+You can replace '5h' with other time measurements to control the amount
+of time you want to view from the merged logs. This will mail a report
+notification to all the people in the sysadmin-logs group.
+
+=== Audit logs, centrally:
+
+We've taken the audit logs and enabled our rsyslogd on the hosts to
+relay the audit log contents to our central log server.
+
+Here's how we did that:
+
+[arabic]
+. modify the selinux policy so that rsyslogd can read the file(s) in
+`/var/log/audit/audit.log`
+
+____
+BEGIN Selinux policy module:
+
+....
+module audit_via_syslog 1.0;
+
+require {
+    type syslogd_t;
+    type auditd_log_t;
+    class dir { search };
+    class file { getattr read open };
+
+}
+
+#============= syslogd_t ==============
+allow syslogd_t auditd_log_t:dir search;
+allow syslogd_t auditd_log_t:file { getattr read open };
+....
+
+END selinux policy module
+____
+
+[arabic, start=2]
+. add config to rsyslog on the clients to repeatedly send all changes to
+their audit.log file to the central syslog server as local6:
++
+....
+# monitor auditd log and send out over local6 to central loghost
+$ModLoad imfile.so
+
+# auditd audit.log
+$InputFileName /var/log/audit/audit.log
+$InputFileTag tag_audit_log:
+$InputFileStateFile audit_log
+$InputFileSeverity info
+$InputFileFacility local6
+$InputRunFileMonitor
+....
+
+____
+then modify your emitter to the syslog server to send local6.* there
+____
+
+[arabic, start=3]
+. on the syslog server - setup log destinations for:
+* merged audit logs of all hosts explicitly drop any non-AVC audit
+message here) magic exclude line is:
++
+....
+:msg, !contains, "type=AVC" ~
+....
++
+that line must be directly above the log entry you want to filter and it
+has a cascade effect on everything below it unless you disable the
+filter
++
+____
+* per-host audit logs - this is everything from audit.log
+____
+. On the syslog server - we can run audit2allow/audit2why on the audit
+logs sent there by doing this:
++
+....
+grep 'hostname' /var/log/merged/audit.log | sed 's/^.*tag_audit_log: //' | audit2allow
+....
++
+the sed is to remove the log prefix garbage from syslog transferring the
+msg
+
+== Future:
+
+* additional log reports for errors from http processes or servers
+* SEC Simple Event Coordinator to report, immediately, on events from a
+log stream - available in fedora/epel.
+* New report modules within epylog