= Haproxy Infrastructure SOP

haproxy is an application that does load balancing at the tcp layer or
at the http layer. It can do generic tcp balancing but it does
specialize in http balancing. Our proxy servers are still running apache
and that is what our users connect to. But instead of using
mod_proxy_balancer and ProxyPass balancer://, we do a ProxyPass to
http://localhost:10001/ or http://localhost:10002/. haproxy must
be told to listen to an individual port for each farm. All haproxy farms
are listed in /etc/haproxy/haproxy.cfg.

== Contents

* <<_contact_information>>
* <<_how_it_works>>
* <<_configuration_example>>
* <<_stats>>
* <<_advanced_usage>>

== Contact Information

Owner:::
  Fedora Infrastructure Team
Contact:::
  #fedora-admin, sysadmin-main, sysadmin-web group
Location:::
  Phoenix, Tummy, Telia
Servers:::
  proxy1, proxy2, proxy3, proxy4, proxy5
Purpose:::
  Provides load balancing from the proxy layer to our application layer.

== How it works

haproxy is a load balancer. If you're familiar, this section won't be
that interesting. haproxy in its normal usage acts just like a web
server. It listens on a port for requests. Unlike most webservers though
it then sends that request to one of our back end application servers
and sends the response back. This is referred to as reverse proxying. We
typically configure haproxy to send check to a specific url and look for
the response code. If this url isn't sent, it just does basic checks to
/. In most of our configurations we're using round robin balancing. IE,
request 1 goes to app1, request2 goes to app2, request 3 goes to app3
request 4 goes to app1, and the whole process repeats.

[WARNING]
====
These checks do add load to the app servers. As well as additional
connections. Be smart about which url you're checking as it gets checked
often. Also be sure to verify the application servers can handle your
new settings, monitor them closely for the hour or two after you make
changes.
====

== Configuration example

The below example is how our fedoraproject wiki could be configured.
Each application should have its own farm. Even though it may have an
identical configuration to another farm, this allows easy addition and
subtraction of specific nodes when we need them.:

....
listen  fpo-wiki 0.0.0.0:10001
balance roundrobin
server  app1 app1.fedora.iad2.redhat.com:80 check inter 2s rise 2 fall 5
server  app2 app2.fedora.iad2.redhat.com:80 check inter 2s rise 2 fall 5
server  app4 app4.fedora.iad2.redhat.com:80 backup check inter 2s rise 2 fall 5
option  httpchk GET /wiki/Infrastructure
....

* The first line "listen ...." Says to create a farm called _fpo-wiki_.
Listening on all IP's on port 10001. _fpo-wiki_ can be arbitrary but make
it something obvious. Aside from that the important bit is :10001.
Always make sure that when creating a new farm, its listening on a
unique port. In Fedora's case we're starting at 10001, and moving up by
one. Just check the config file for the lowest open port above 10001.
* The next line _balance roundrobin_ says to use round robin balancing.
* The server lines each add a new node to the balancer farm. In this
case the wiki is being served from app1, app2 and app4. If the wiki is
available at http://app1.fedora.iad2.redhat.com/wiki/ Then this
config would be used in conjunction with "RewriteRule ^/wiki/(.*)
http://localhost:10001/wiki/$1 [P,L]".
* _server_ means we're adding a new node to the farm
* _app1_ is the worker name, it is analagous to fpo-wiki but should::
  match shorthostname of the node to make it easy to follow.
* _app1.fedora.iad2.redhat.com:80_ is the hostname and port to be
contacted.
* _check_ means to check via bottom line "option httpchk GET
/wiki/Infrastructure" which will use /wiki/Infrastructure to verify the
wiki is working. If that URL fails, that entire node will be taken out
of the farm mix.
* _inter 2s_ means to check every 2 seconds. 2s is the same as 2000 in
this case.
* _rise 2_ means to not put this node back in the mix until it has had
two successful connections in a row. haproxy will continue to check
every 2 seconds whether a node is up or down
* _fall 5_ means to take a node out of the farm after 5 failures.
* _backup_ You'll notice that app4 has a _backup_ option. We don't
actually use this for the wiki but do for other farms. It basically
means to continue checking and treat this node like any other node but
don't send it any production traffic unless the other two nodes are
down.

All of these options can be tweaked so keep that in mind when changing
or building a new farm. There are other configuration options in this
file that are global. Please see the haproxy documentation for more
info:

....
/usr/share/doc/haproxy-1.3.14.6/haproxy-en.txt
....

== Stats

In order to view the stats for a farm please see the stats page. Each
proxy server has its own stats page since each one is running its own
haproxy server. To view the stats point your browser to
https://admin.fedoraproject.org/haproxy/shorthostname/ so proxy1 is at
https://admin.fedoraproject.org/haproxy/proxy1/ The trailing / is
important.

* https://admin.fedoraproject.org/haproxy/proxy1/
* https://admin.fedoraproject.org/haproxy/proxy2/
* https://admin.fedoraproject.org/haproxy/proxy3/
* https://admin.fedoraproject.org/haproxy/proxy4/
* https://admin.fedoraproject.org/haproxy/proxy5/

== Advanced Usage

haproxy has some more advanced usage that we've not needed to worry
about yet but is worth mentioning. For example, one could send users to
just one app server based on session id. If user A happened to hit app1
first and user B happened to hit app4 first. All subsequent requests for
user A would go to app1 and user B would go to app4. This is handy for
applications that cannot normally be balanced because of shared storage
needs or other locking issues. This won't solve all problems though and
can have negative affects for example when app1 goes down user A would
either lose their session, or be unable to work until app1 comes back
up. Please do some great testing before looking in to this option.
