106 lines
3.4 KiB
Text
106 lines
3.4 KiB
Text
= fedmsg-gateway SOP
|
|
|
|
Outgoing raw ZeroMQ message stream.
|
|
|
|
[NOTE]
|
|
.Note
|
|
====
|
|
see also: fedmsg-websocket
|
|
====
|
|
|
|
== Contact Information
|
|
|
|
Owner:::
|
|
Messaging SIG, Fedora Infrastructure Team
|
|
Contact:::
|
|
#fedora-apps, #fedora-admin, #fedora-noc
|
|
Servers:::
|
|
busgateway01, proxy0*
|
|
Purpose:::
|
|
Expose raw ZeroMQ messages outside the FI environment.
|
|
|
|
== Description
|
|
|
|
Users outside of Fedora Infrastructure can listen to the production
|
|
message bus by connecting to specific addresses. This is required for
|
|
local users to run their own hubs and message processors ("Consumers").
|
|
It is also required for user-facing tools like fedmsg-notify to work.
|
|
|
|
The specific public endpoints are:
|
|
|
|
production::
|
|
tcp://hub.fedoraproject.org:9940
|
|
staging::
|
|
tcp://stg.fedoraproject.org:9940
|
|
|
|
fedmsg-gateway, the daemon running on busgateway01, is listening to the
|
|
FI production fedmsg bus and will relay every message that it receives
|
|
out to a special ZMQ pub endpoint bound to port 9940. haproxy mediates
|
|
connections to the fedmsg-gateway daemon.
|
|
|
|
== Connection Flow
|
|
|
|
Clients connect through haproxy on proxy0*:9940 are redirected to
|
|
busgateway0*:9940. This can be found in the haproxy.cfg entry for
|
|
`listen fedmsg-raw-zmq 0.0.0.0:9940`.
|
|
|
|
This is different than the apache reverse proxy pass setup we have for
|
|
the app0* and packages0* machines. _That_ flow looks something like
|
|
this:
|
|
|
|
....
|
|
Client -> apache(proxy01) -> haproxy(proxy01) -> apache(app01)
|
|
....
|
|
|
|
The flow for the raw zmq stream provided by fedmsg-gateway looks
|
|
something like this:
|
|
|
|
....
|
|
Client -> haproxy(proxy01) -> fedmsg-gateway(busgateway01)
|
|
....
|
|
|
|
haproxy is listening on a public port.
|
|
|
|
At the time of this writing, haproxy does not actually load balance
|
|
zeromq session requests across multiple busgateway0* machines, but there
|
|
is nothing stopping us from adding them. New hosts can be added in
|
|
ansible and pressed from busgateway01's template. Add them to the
|
|
fedmsg-raw-zmq listen in haproxy's config and it should Just Work.
|
|
|
|
== Increasing the Maximum Number of Concurrent Connections
|
|
|
|
HTTP requests are typically very short (a few seconds at most). This
|
|
means that the number of concurrent tcp connections we require for most
|
|
of our services is quite low (1024 is overkill). ZeroMQ tcp connections,
|
|
on the other hand, are expected to live for quite a long time.
|
|
Consequently we needed to scale up the number of possible concurrent tcp
|
|
connections.
|
|
|
|
All of this is in ansible and should be handled for us automatically if
|
|
we bring up new nodes.
|
|
|
|
* The pam_limits user limit for the fedmsg user was increased from 1024
|
|
to 160000 on busgateway01.
|
|
* The pam_limits user limit for the haproxy user was increased from 1024
|
|
to 160000 on the proxy0* machines.
|
|
* The zeromq High Water Mark (HWM) was increased to 160000 on
|
|
busgateway01.
|
|
* The maximum number of connections allowed was increased in
|
|
haproxy.cfg.
|
|
|
|
== Nagios
|
|
|
|
New nagios checks were added for this that check to see if the number of
|
|
concurrent connections through haproxy is approaching the maximum number
|
|
allowed.
|
|
|
|
You can check these numbers by hand by inspecting the haproxy web
|
|
interface: https://admin.fedoraproject.org/haproxy/proxy1#fedmsg-raw-zmq
|
|
|
|
Look at the "Sessions" section. "Cur" is the current number of sessions
|
|
versus "Max", the maximum number seen at the same time and "Limit", the
|
|
maximum number of concurrent connections allowed.
|
|
|
|
== RHIT
|
|
|
|
We had RHIT open up port 9940 special to proxy01.phx2 for this.
|