To make changing the gunicorn configuration more easily let's move
configuration values from systemd service to separate configuration
file.
The file will live in /etc/mailman3/gunicorn.conf.py.
This will unify all the handlers to use first uppercase letter for
ansible-lint to stop complaining.
I went through all `notify:` occurrences and fixed them by running
```
set TEXT "text_to_replace"; set REPLACEMENT "replacement_text"; git grep
-rlz "$TEXT" . | xargs -0 sed -i "s/$TEXT/$REPLACEMENT/g"
```
Then I went through all the changes and removed the ones that wasn't
expected to be changed.
Fixes https://pagure.io/fedora-infrastructure/issue/12391
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
The config is now provided by the mailman3 package and it's the same
provided by the ansible role.
And we have two of them in the role, probably a oversight.
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
The whoosh engine doesn't seem to be maintained anymore and when generating the
index it got stuck when the size was about 20 GB. Xapian on the other hand looks
more stable and much quicker when generating the index.
Unfortunately the xapian-haystack package is not available in Fedora/EPEL yet
and waiting for review to be finished.
Link to review bug: https://bugzilla.redhat.com/show_bug.cgi?id=2313507
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
This script is added to implement
https://pagure.io/fedora-infrastructure/issue/11427
The script was already executed on staging environment and finished without
issue. It could be executed multiple times and only affects list that don't have
the settings set yet. I will document this as another step to take when creating
a new list.
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
The API didn't work and failed with 500, because the REST framework didn't know
how to render the response. This fixes the issue.
https://pagure.io/fedora-infrastructure/issue/12134
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
The index generated by the hyperkitty-hourly script generated unusually big
index (around 1TB).
This script should help with generating the search index from scratch, after
that we can leave it to hyperkitty-hourly job again.
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
The database log is really big even on info level, let' set it manually to
warning to save space and don't generate GB of logs.
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
There was a typo in logging section of mailman configuration, which caused that
only default log files were created.
Also there is separate logging for gunicorn now, let's use that.
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
Mailman has huge memory spikes after migration, those are caused by visiting
`/archives/search` URL. This is probably because of the search index still being
regenerated.
This patch will disable the search from both web UI and as a HTTP URL endpoint.
Fixes https://pagure.io/fedora-infrastructure/issue/12043
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
To be able to enable timer it needs to have WantedBy in [Install] section. Let's
add a hyperkitty.target to that section.
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
This commit is setting both mailman and hyperkitty to devmode and doesn't allow
them to sent mails anywhere.
I tested that from web UI and using `mail` command on the machine.
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
Set django to have 4 workers and have a retry longer than the timeout.
Disable local memcached and use the memcached01 instance.
Enabled debug logging on gunicorn to see what it's doing.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Looked at logs of servers being hit by the 'non-responsive' bots and
the following were hit heavily every day multiple times a day:
100006 nagios.fedoraproject.org-access.log
102150 koschei.fedoraproject.org-access.log
162296 lists.fedoraproject.org-access.log
495776 fedoraproject.org-access.log
850471 dl.fedoraproject.org-access.log
Added bloks to dl.fedoraproject to try and lower its hit rate. Others
need review from people who know their internals more.
Signed-off-by: Stephen Smoogen <ssmoogen@redhat.com>
I went through the last couple of logs afer the first round of 'turn
off the spiders' went out. I looked at the areas which the /robots.txt
disregard and then looked for the bots which ignored it and still
looked up stuff in 'accounts'. This may cut down CPU spikes as these
are looking at dynamic data which can 'blow' things up.
It might be good to add similar tooling to pagure and src since they
seem to be hit a lot in the logs also.
Signed-off-by: Stephen Smoogen <ssmoogen@redhat.com>
Use the recommended CPU*2+1 amount of workers and limit max requests done. After
that the gunicorn reloads itself and free up all the memory.
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
It seems that I forgot to actually create the certificates for new mailman
fedora archiver. Let's fix that.
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
