= fedmsg-gateway SOP

Outgoing raw ZeroMQ message stream.

[NOTE]
====
See also: <<fedmsg-websocket.adoc#>>
====

== Contact Information

Owner:::
  Messaging SIG, Fedora Infrastructure Team
Contact:::
  #fedora-apps, #fedora-admin, #fedora-noc
Servers:::
  busgateway01, proxy0*
Purpose:::
  Expose raw ZeroMQ messages outside the FI environment.

== Description

Users outside of Fedora Infrastructure can listen to the production
message bus by connecting to specific addresses. This is required for
local users to run their own hubs and message processors ("Consumers").

The specific public endpoints are:

production::
  tcp://hub.fedoraproject.org:9940
staging::
  tcp://stg.fedoraproject.org:9940

_fedmsg-gateway_, the daemon running on _busgateway01_, is listening to the
FI production fedmsg bus and will relay every message that it receives
out to a special ZMQ pub endpoint bound to port 9940. haproxy mediates
connections to the _fedmsg-gateway_ daemon.

== Connection Flow

Clients connect through haproxy on `proxy0*:9940` are redirected to
`busgateway0*:9940`. This can be found in the `haproxy.cfg` entry for
`listen fedmsg-raw-zmq 0.0.0.0:9940`.

This is different than the apache reverse proxy pass setup we have for
the _app0*_ and _packages0*_ machines. _That_ flow looks something like
this:

....
Client -> apache(proxy01) -> haproxy(proxy01) -> apache(app01)
....

The flow for the raw zmq stream provided by _fedmsg-gateway_ looks
something like this:

....
Client -> haproxy(proxy01) -> fedmsg-gateway(busgateway01)
....

_haproxy_ is listening on a public port.

At the time of this writing, _haproxy_ does not actually load balance
zeromq session requests across multiple _busgateway0*_ machines, but there
is nothing stopping us from adding them. New hosts can be added in
ansible and pressed from _busgateway01_'s template. Add them to the
fedmsg-raw-zmq listen in _haproxy_'s config and it should Just Work.

== Increasing the Maximum Number of Concurrent Connections

HTTP requests are typically very short (a few seconds at most). This
means that the number of concurrent tcp connections we require for most
of our services is quite low (1024 is overkill). ZeroMQ tcp connections,
on the other hand, are expected to live for quite a long time.
Consequently we needed to scale up the number of possible concurrent tcp
connections.

All of this is in ansible and should be handled for us automatically if
we bring up new nodes.

* The pam_limits user limit for the fedmsg user was increased from 1024
to 160000 on _busgateway01_.
* The pam_limits user limit for the haproxy user was increased from 1024
to 160000 on the _proxy0*_ machines.
* The zeromq High Water Mark (HWM) was increased to 160000 on
_busgateway01_.
* The maximum number of connections allowed was increased in
`haproxy.cfg`.

== Nagios

New nagios checks were added for this that check to see if the number of
concurrent connections through haproxy is approaching the maximum number
allowed.

You can check these numbers by hand by inspecting the _haproxy_ web
interface: https://admin.fedoraproject.org/haproxy/proxy1#fedmsg-raw-zmq

Look at the "Sessions" section. "Cur" is the current number of sessions
versus "Max", the maximum number seen at the same time and "Limit", the
maximum number of concurrent connections allowed.

== RHIT

We had RHIT open up port 9940 special to _proxy01.iad2_ for this.