Connection optimization between NLB and target groups

Recently called upon to investigate a slow performance issue between two application endpoints where one endpoint was at a DC and the other endpoint was at AWS US-East-1 behind an NLB. While going through some TCP dumps collected on servers hosted at the DC lot of SYN re-transmissions were observed for the NLB.

Image. 1. SYN re-transmissions

Below are the network topology and the endpoints where the SYN re-transmissions happened are highlighted.

Image. 2. Topology diagram

Since the NLB was configured as a TCP listener with the target group as Instance IDs while receiving requests from clients client source IP was persevered and pass through to the target group members which are the Nginx servers.

Since NLB is acting as a pass-through device and only doing packet forwarding focus was pointed to the Nginx servers and few packet captures were analyzed and noticed when F5 starts to create a socket with Nginx the reply contains an abnormal acknowledgment number which F5 is not aware of causing the F5 to send an RST and re-transmit the packet.

Image. 3. Abnormal acknowledgment

While digging deep in to why the Nginx server replies to an SYN request with an abnormal acknowledgment number we found out that all the sessions from the three clusters will connect to the Nginx with a single source IP A.B.C.D, a new SYN request can hit Nginx on an already established socket forcing it to send ACK with the acknowledgement number representing already established sessions data. As highlighted above this abnormal acknowledgment number is not what F5 expected (acknowledgment no should be Ack=1) hence it tears down the session with an RST and re-transmit the packet again.

The solution

Configuring the NLB not to preserve the customer source IP, NLB will originate connections with the Nginx server using its IP as the source IP with persistent connection maintain via connection pooling reducing the chance of landing a new session establishment request to an already open socket.

To configure the NLB, not to preserver the customer source IP:-

Target group members should be defined as IP addresses, not as Instance IDs.

After the change connection management

After the change, a persistent set of connections was maintained between the NLB and Nginx resulting in connection pooling.

Image. 4. Long duration sessions due to the connection pooling

Conclusion

To have optimized connection management between an NLB and target group members defining the targets based on IP addresses will be the best option as it will manage connections efficiently with the help of connection pooling.