Keep Connections Alive : Browser - Load Balancer - Web Server - mod jk - Tomcat : Part 2

 

This series is about how connections are kept alive between different components along the route.

 Browser - Load Balancer - Web Server - mod jk - Tomcat 

Part 1 : Browser - Load Balancer.

Part 2 : Load Balancer - Apache HTTP Server

Part 3 : Apache HTTP Server - Mod jk

Part 4 : Mod jk - Tomcat

Load Balancer -> Apache Web Server

How is the connection between web server and load balancer maintained? 

There is usually a seperate load balancer TCP profile for the connection between load balancer and the web servers. As described here, the default "Idle Timeout" is 5 minutes and default "Keep Alive Interval" is 30 minutes.

When a request comes in for a page that takes a long time to render, a connection is established between the load balancer and the web server and can be observed by the netstat command. 

> netstat -an | grep :81 |grep EST

tcp  0  0 ::ffff:111.111.111.11:81    ::ffff:111.111.111.12:55789  ESTABLISHED

After 5 minutes, netstat will show that the connection is gone, dropped by the load balancer even if the request is still being processed by downstream servers. So doesn't Apache web server send keep alive signals to the load balancer to maintain the connection, just as Google Chrome does? The answer is no.

Here is the source code for Apache web server when it makes a connection. It makes the following call to setup Keep Alive sockets.

stat = apr_socket_opt_set(s, APR_SO_KEEPALIVE, one); 

However, according to the linux documentation, here and here, using default values, linux's keepalive routines will wait for two hours (7200 secs) before sending the first keepalive probe, and then resend it every 75 seconds.

cat /proc/sys/net/ipv4/tcp_keepalive_time  -- > 7200
cat /proc/sys/net/ipv4/tcp_keepalive_intvl  -- > 75
cat /proc/sys/net/ipv4/tcp_keepalive_probes  -- > 9

These OS-level settings can be overwritten by programs, but Apache web server doesn't make specific calls to change them. This means that it will not send keep-alive signals to the load balancer during its ususally short life. 

The Answer 

So, if you need the connection between LB and web server to be maintained longer than default value 5 minutes, you need to do either of the following:

• Change the "Keep Alive Interval" setting on the load balancer TCP profile from 30 minutes (1800 secs) to 60 second.
• Change the tcp_keepalive_time parameter on Linux that hosts Apache from two hours (7200 secs) to less than "Idle Timeout", which is 5 minutes. 
• Change the "Idle Timeout" setting on the load balancer TCP profile from 5 minutes to longer

We used the first option, keeping the "keep alive interval" setting of the F5 server-side tcp profile in sync with the client-side tcp profile.

More Things to Consider

The connection between mod_jk and Tomcat Servers is discussed in part 4 of this topic.