169 lines
4.4 KiB
Text
169 lines
4.4 KiB
Text
= Log Infrastructure SOP
|
|
|
|
Logs are centrally referred to our loghost and managed from there by
|
|
rsyslog to create several log outputs.
|
|
|
|
Epylog provides twice-daily log reports of activities on our systems. It
|
|
runs on our central loghost and generates reports on all systems
|
|
centrally logging.
|
|
|
|
== Contact Information
|
|
|
|
Owner:::
|
|
Fedora Infrastructure Team
|
|
Contact:::
|
|
#fedora-admin, sysadmin-main
|
|
Location:::
|
|
Phoenix
|
|
Servers:::
|
|
log01.phx2.fedoraproject.org
|
|
Purpose:::
|
|
Provides our central logs and reporting
|
|
|
|
== Essential data/locations:
|
|
|
|
* Logs compiled using rsyslog on log01 into a single set of logs for all
|
|
systems:
|
|
+
|
|
....
|
|
/var/log/merged/
|
|
....
|
|
+
|
|
These logs are rotated every day and kept for only 2 days. This set of
|
|
logs is only used for immediate analysis and more trivial 'tailing' of
|
|
the log file to watch for events.
|
|
* Logs for each system separately in `/var/log/hosts`
|
|
+
|
|
These logs are maintained forever, practically, or for as long as we
|
|
possibly can. They are broken out into a `$hostname/$YEAR/$MON/$DAY`
|
|
directory structure so we can locate a specific day's log immediately.
|
|
* Log reports generated by epylog: Log reports generated by epylog are
|
|
outputted to /srv/web/epylog/merged
|
|
+
|
|
The reports are accessible via a web browser from
|
|
https://admin.fedoraproject.org/epylog/merged/
|
|
+
|
|
This path requires a username and a password to access. To add your
|
|
username and password you must first join the sysadmin-logs group then
|
|
login to `log01.phx2.fedoraproject.org` and run this command:
|
|
+
|
|
....
|
|
htpasswd -m /srv/web/epylog/.htpasswd $your_username
|
|
....
|
|
+
|
|
when prompted for a password please input a password which is NOT YOUR
|
|
FEDORA ACCOUNT SYSTEM PASSWORD.
|
|
|
|
[IMPORTANT]
|
|
.Important
|
|
====
|
|
Let's say that again to be sure you got it:
|
|
|
|
DO _link:[NOT] HAVE THIS BE THE SAME AS YOUR FAS PASSWORD
|
|
====
|
|
|
|
== Configs:
|
|
|
|
Epylog configs are controlled by ansible - please see the ansible epylog
|
|
module for more details. Specifically the files in
|
|
`roles/epylog/files/merged/`
|
|
|
|
=== Generating a one-off epylog report:
|
|
|
|
If you wish to generate a specific log report you will need to run the
|
|
following command on log01:
|
|
|
|
....
|
|
sudo /usr/sbin/epylog -c /etc/epylog/merged/epylog.conf --last 5h
|
|
....
|
|
|
|
You can replace '5h' with other time measurements to control the amount
|
|
of time you want to view from the merged logs. This will mail a report
|
|
notification to all the people in the sysadmin-logs group.
|
|
|
|
=== Audit logs, centrally:
|
|
|
|
We've taken the audit logs and enabled our rsyslogd on the hosts to
|
|
relay the audit log contents to our central log server.
|
|
|
|
Here's how we did that:
|
|
|
|
[arabic]
|
|
. modify the selinux policy so that rsyslogd can read the file(s) in
|
|
`/var/log/audit/audit.log`
|
|
|
|
____
|
|
BEGIN Selinux policy module:
|
|
|
|
....
|
|
module audit_via_syslog 1.0;
|
|
|
|
require {
|
|
type syslogd_t;
|
|
type auditd_log_t;
|
|
class dir { search };
|
|
class file { getattr read open };
|
|
|
|
}
|
|
|
|
#============= syslogd_t ==============
|
|
allow syslogd_t auditd_log_t:dir search;
|
|
allow syslogd_t auditd_log_t:file { getattr read open };
|
|
....
|
|
|
|
END selinux policy module
|
|
____
|
|
|
|
[arabic, start=2]
|
|
. add config to rsyslog on the clients to repeatedly send all changes to
|
|
their audit.log file to the central syslog server as local6:
|
|
+
|
|
....
|
|
# monitor auditd log and send out over local6 to central loghost
|
|
$ModLoad imfile.so
|
|
|
|
# auditd audit.log
|
|
$InputFileName /var/log/audit/audit.log
|
|
$InputFileTag tag_audit_log:
|
|
$InputFileStateFile audit_log
|
|
$InputFileSeverity info
|
|
$InputFileFacility local6
|
|
$InputRunFileMonitor
|
|
....
|
|
|
|
____
|
|
then modify your emitter to the syslog server to send local6.* there
|
|
____
|
|
|
|
[arabic, start=3]
|
|
. on the syslog server - setup log destinations for:
|
|
* merged audit logs of all hosts explicitly drop any non-AVC audit
|
|
message here) magic exclude line is:
|
|
+
|
|
....
|
|
:msg, !contains, "type=AVC" ~
|
|
....
|
|
+
|
|
that line must be directly above the log entry you want to filter and it
|
|
has a cascade effect on everything below it unless you disable the
|
|
filter
|
|
+
|
|
____
|
|
* per-host audit logs - this is everything from audit.log
|
|
____
|
|
. On the syslog server - we can run audit2allow/audit2why on the audit
|
|
logs sent there by doing this:
|
|
+
|
|
....
|
|
grep 'hostname' /var/log/merged/audit.log | sed 's/^.*tag_audit_log: //' | audit2allow
|
|
....
|
|
+
|
|
the sed is to remove the log prefix garbage from syslog transferring the
|
|
msg
|
|
|
|
== Future:
|
|
|
|
* additional log reports for errors from http processes or servers
|
|
* SEC Simple Event Coordinator to report, immediately, on events from a
|
|
log stream - available in fedora/epel.
|
|
* New report modules within epylog
|