= Log Infrastructure SOP

Logs are centrally referred to our loghost and managed from there by
rsyslog to create several log outputs.

Epylog provides twice-daily log reports of activities on our systems. It
runs on our central loghost and generates reports on all systems
centrally logging.

== Contact Information

Owner:::
  Fedora Infrastructure Team
Contact:::
  #fedora-admin, sysadmin-main
Location:::
  Phoenix
Servers:::
  log01.phx2.fedoraproject.org
Purpose:::
  Provides our central logs and reporting

== Essential data/locations:

* Logs compiled using rsyslog on log01 into a single set of logs for all
systems:
+
....
/var/log/merged/
....
+
These logs are rotated every day and kept for only 2 days. This set of
logs is only used for immediate analysis and more trivial 'tailing' of
the log file to watch for events.
* Logs for each system separately in `/var/log/hosts`
+
These logs are maintained forever, practically, or for as long as we
possibly can. They are broken out into a `$hostname/$YEAR/$MON/$DAY`
directory structure so we can locate a specific day's log immediately.
* Log reports generated by epylog: Log reports generated by epylog are
outputted to /srv/web/epylog/merged
+
The reports are accessible via a web browser from
https://admin.fedoraproject.org/epylog/merged/
+
This path requires a username and a password to access. To add your
username and password you must first join the sysadmin-logs group then
login to `log01.phx2.fedoraproject.org` and run this command:
+
....
htpasswd -m /srv/web/epylog/.htpasswd $your_username
....
+
when prompted for a password please input a password which is NOT YOUR
FEDORA ACCOUNT SYSTEM PASSWORD.

[IMPORTANT]
.Important
====
Let's say that again to be sure you got it:

DO _link:[NOT] HAVE THIS BE THE SAME AS YOUR FAS PASSWORD
====

== Configs:

Epylog configs are controlled by ansible - please see the ansible epylog
module for more details. Specifically the files in
`roles/epylog/files/merged/`

=== Generating a one-off epylog report:

If you wish to generate a specific log report you will need to run the
following command on log01:

....
sudo /usr/sbin/epylog -c /etc/epylog/merged/epylog.conf --last 5h
....

You can replace '5h' with other time measurements to control the amount
of time you want to view from the merged logs. This will mail a report
notification to all the people in the sysadmin-logs group.

=== Audit logs, centrally:

We've taken the audit logs and enabled our rsyslogd on the hosts to
relay the audit log contents to our central log server.

Here's how we did that:

[arabic]
. modify the selinux policy so that rsyslogd can read the file(s) in
`/var/log/audit/audit.log`

____
BEGIN Selinux policy module:

....
module audit_via_syslog 1.0;

require {
    type syslogd_t;
    type auditd_log_t;
    class dir { search };
    class file { getattr read open };

}

#============= syslogd_t ==============
allow syslogd_t auditd_log_t:dir search;
allow syslogd_t auditd_log_t:file { getattr read open };
....

END selinux policy module
____

[arabic, start=2]
. add config to rsyslog on the clients to repeatedly send all changes to
their audit.log file to the central syslog server as local6:
+
....
# monitor auditd log and send out over local6 to central loghost
$ModLoad imfile.so

# auditd audit.log
$InputFileName /var/log/audit/audit.log
$InputFileTag tag_audit_log:
$InputFileStateFile audit_log
$InputFileSeverity info
$InputFileFacility local6
$InputRunFileMonitor
....

____
then modify your emitter to the syslog server to send local6.* there
____

[arabic, start=3]
. on the syslog server - setup log destinations for:
* merged audit logs of all hosts explicitly drop any non-AVC audit
message here) magic exclude line is:
+
....
:msg, !contains, "type=AVC" ~
....
+
that line must be directly above the log entry you want to filter and it
has a cascade effect on everything below it unless you disable the
filter
+
____
* per-host audit logs - this is everything from audit.log
____
. On the syslog server - we can run audit2allow/audit2why on the audit
logs sent there by doing this:
+
....
grep 'hostname' /var/log/merged/audit.log | sed 's/^.*tag_audit_log: //' | audit2allow
....
+
the sed is to remove the log prefix garbage from syslog transferring the
msg

== Future:

* additional log reports for errors from http processes or servers
* SEC Simple Event Coordinator to report, immediately, on events from a
log stream - available in fedora/epel.
* New report modules within epylog