From 83948e50e379b02feafffca2aacc35d96c8c189b Mon Sep 17 00:00:00 2001 From: Patrick Uiterwijk Date: Thu, 23 Jul 2015 22:11:25 +0000 Subject: [PATCH] Separate Pagure main and Pagure docs RCA of issue sent on IRC: It's a very interresting edge case and related to my previous diag. In short: both the pagure main app and pagure docs app were using the same process pool (WSGIProcessDaemon). As soon as they would both be loaded in the same thread, they would both load the FFI (C wrapper) code, and only the latest process to load it would still have valid type references, the other would start sending wrong references, which causes it to error out (correctly), because it doesn't know the types it got. So basically, the fix I just applied is put pagure docs into its own WSGI daemon process, that keeps them nicely separated. the reason that this didn't hit in staging and why it also worked *sometimes* in production is that it would only crash if: 1. both pagure main app and docs app were loaded in the thread that's used for the current request 2. pagure docs app was loaded last in the current thread, overriding the types for pagure main app, and 3. we have 4 processes with 4 threads each, so each request gets into one of 16 threads, making the staging not likely to hit the previous two conditions, but prod has so many requests it's likely to hit 1 and 2 --- roles/pagure/frontend/templates/0_pagure.conf | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/roles/pagure/frontend/templates/0_pagure.conf b/roles/pagure/frontend/templates/0_pagure.conf index bf6c667c36..ea81a92db9 100644 --- a/roles/pagure/frontend/templates/0_pagure.conf +++ b/roles/pagure/frontend/templates/0_pagure.conf @@ -4,6 +4,7 @@ WSGIRestrictSignal Off WSGIPythonOptimize 1 WSGIPassAuthorization On WSGIDaemonProcess pagure user=git group=git maximum-requests=1000 display-name=pagure processes=4 threads=4 inactivity-timeout=300 +WSGIDaemonProcess paguredocs user=git group=git maximum-requests=1000 display-name=paguredocs processes=4 threads=4 inactivity-timeout=300 {% if env == 'pagure-staging' %} @@ -48,7 +49,7 @@ WSGIDaemonProcess pagure user=git group=git maximum-requests=1000 display-name=p Alias /static /usr/lib/python2.7/site-packages/pagure/static/ - WSGIProcessGroup pagure + WSGIProcessGroup paguredocs # Apache 2.4 Require all granted