infra-docs-fpo/modules/sysadmin_guide/pages/syslog.adoc

= Log Infrastructure SOP

Logs are centrally referred to our loghost and managed from there by
rsyslog to create several log outputs.

Epylog provides twice-daily log reports of activities on our systems. It
runs on our central loghost and generates reports on all systems
centrally logging.

== Contact Information

Owner:::
  Fedora Infrastructure Team
Contact:::
  #fedora-admin, sysadmin-main
Servers:::
  log01.iad2.fedoraproject.org
Purpose:::
  Provides our central logs and reporting

== Essential data/locations:

* Logs compiled using rsyslog on log01 into a single set of logs for all
systems:
+
....
/var/log/merged/
....
+
These logs are rotated every day and kept for only 2 days. This set of
logs is only used for immediate analysis and more trivial 'tailing' of
the log file to watch for events.
* Logs for each system separately in `/var/log/hosts`
+
These logs are maintained forever, practically, or for as long as we
possibly can. They are broken out into a `$hostname/$YEAR/$MON/$DAY`
directory structure so we can locate a specific day's log immediately.
* Log reports generated by epylog: Log reports generated by epylog are
outputted to `/srv/web/epylog/merged`
+
This path requires a username and a password to access. To add your
username and password you must first join the sysadmin-logs group then
login to `log01.iad2.fedoraproject.org` and run this command:
+
....
htpasswd -m /srv/web/epylog/.htpasswd $your_username
....
+
when prompted for a password please input a password which is NOT YOUR
FEDORA ACCOUNT SYSTEM PASSWORD.

[IMPORTANT]
====
Let's say that again to be sure you got it:

DO *NOT* HAVE THIS BE THE SAME AS YOUR FAS PASSWORD
====

== Configs

Epylog configs are controlled by ansible - please see the ansible epylog
module for more details. Specifically the files in
`roles/epylog/files/merged/`

=== Generating a one-off epylog report

If you wish to generate a specific log report you will need to run the
following command on log01:

....
sudo /usr/sbin/epylog -c /etc/epylog/merged/epylog.conf --last 5h
....

You can replace '5h' with other time measurements to control the amount
of time you want to view from the merged logs. This will mail a report
notification to all the people in the sysadmin-logs group.

=== Audit logs, centrally

We've taken the audit logs and enabled our rsyslogd on the hosts to
relay the audit log contents to our central log server.

Here's how we did that:

[arabic]
. modify the selinux policy so that rsyslogd can read the file(s) in
`/var/log/audit/audit.log`

____
BEGIN Selinux policy module:

....
module audit_via_syslog 1.0;

require {
    type syslogd_t;
    type auditd_log_t;
    class dir { search };
    class file { getattr read open };

}

#============= syslogd_t ==============
allow syslogd_t auditd_log_t:dir search;
allow syslogd_t auditd_log_t:file { getattr read open };
....

END selinux policy module
____

. add config to rsyslog on the clients to repeatedly send all changes to
their audit.log file to the central syslog server as local6:
+
....
# monitor auditd log and send out over local6 to central loghost
$ModLoad imfile.so

# auditd audit.log
$InputFileName /var/log/audit/audit.log
$InputFileTag tag_audit_log:
$InputFileStateFile audit_log
$InputFileSeverity info
$InputFileFacility local6
$InputRunFileMonitor
....

____
then modify your emitter to the syslog server to send local6.* there
____

. on the syslog server - setup log destinations for:
* merged audit logs of all hosts explicitly drop any non-AVC audit
message here) magic exclude line is:
+
....
:msg, !contains, "type=AVC" ~
....
+
that line must be directly above the log entry you want to filter and it
has a cascade effect on everything below it unless you disable the
filter
+
____
* per-host audit logs - this is everything from audit.log
____
. On the syslog server - we can run audit2allow/audit2why on the audit
logs sent there by doing this:
+
....
grep 'hostname' /var/log/merged/audit.log | sed 's/^.*tag_audit_log: //' | audit2allow
....
+
the sed is to remove the log prefix garbage from syslog transferring the
msg

== Future

* additional log reports for errors from http processes or servers
* SEC Simple Event Coordinator to report, immediately, on events from a
log stream - available in fedora/epel.
* New report modules within epylog