= retrace SOP

____
Retrace server - provides complete tracebacks for unhandled crashes and
show aggregated information for developers.
____

== Contact Information

Owner:::
  Fedora QA Devel, Fedora Infrastructure Team, ABRT team
Contact:::
  #abrt, #fedora-admin, #fedora-noc
Servers:::
  retrace*, faf*
Purpose:::
  Provides complete tracebacks for unhandled crashes and show aggregated
  information for developers.

== Description

The physical server (retrace.fedoraproject.org) runs two main services:
retrace-server and FAF.

=== Retrace-server

The upstream for retrace server lives at:

____
https://github.com/abrt/retrace-server
____

When a user has the ABRT client installed and a process crashes with an
unhandled exception (e.g., traceback or core dump), the user can send a
request to retrace-server. The server will install the same set of
packages plus debuginfo, and will return a traceback to the user that
includes function names instead of plain pointers. This information is
useful for debugging.

The upstream retrace-server allows users to upload coredumps through a
web interface, but the Fedora instance disables this feature.

=== FAF

When a user decides to report a crash, data is sent to FAF. ABRT can
also be configured to send microreports automatically, if desired.

FAF can aggregate similar reports into one entity (called a Problem).
FAF provides a nice web interface for developers, allowing them to see
crashes of their packages. It lives at:

____
https://retrace.fedoraproject.org/faf/
____

== Playbook

The playbook is split into several roles. There are two main roles

____
* abrt/faf
* abrt/retrace
____

These roles are copied from upstream. You should never update it
directly. The new version can be fetched from upstream using:

____
# cd ansible/abrt # rm -rf faf retrace # ansible-galaxy install -f -r
requirements.yml --ignore-errors -p ./
____

You should review the new differences, and commit and push.

Then there are some roles which are local for our instance:

____
* abrt/faf-local - This is run _before_ abrt/faf.
* abrt/retrace-local - This is run _after_ abrt/retrace.
* abrt/retrace-local-pre - This is run _before_ abrt/retrace.
____

== Services

FAF and retrace-server are web applications; only httpd is required.

== Cron

FAF and retrace-server each have cron tasks. They are _not_ installed
under /etc/cron* but are installed as user cron jobs for the 'faf' and
'retrace' users.

You can list those crons using:

____
* sudo -u faf crontab -l
* sudo -u retrace crontab -l
____

All cronjobs should be Ansible managed. Just make sure if you delete
some cron from Ansible that it does not remain on the server (not always
possible with state=absent).

== Directories

* /srv/ssd - fast disk, used for PostgreSQL storage
* /srv - big fat disk, used for storing packages. Mainly:
** /srv/faf/lob
** /srv/retrace
* {blank}
+
/srv/faf/db-backup/ - Daily backups of DB. No rotating yet. Needs to
be::
  manually deleted occasionally.
* {blank}
+
/srv/faf/lob/InvalidUReport/ - Invalid reports, can be pretty big.::
  No automatic removal too. Need to be purged manually occasionally.

== Front-page

The main web page is handled by the [.title-ref]#abrt-server-info-page#
package, which can be controlled using:

____
/usr/lib/python2.7/site-packages/abrt-server-info-page/config.py
____

== DB

Only FAF uses a database. We use our own instance of PostgreSQL. You can
connect to it using:

____
sudo -u faf psql faf
____