NETWORK -- Performance Checklist -- SRE / DevOps / Performance

This is a Performance, Scalability and Availability checklist which can be used to check if NETWORK is the reason for the impact.

Types of Latency

• Name Resolution Latency
• Ping Latency
• Connection Latency
• First Byte Latency
• Round trip time
• Connection life span

To really understand network issue, an in-depth understanding on what really happens in a network connection establishment has to be known. I have written another article on that.

Checklist for Network Performance

1) NETSTAT INTERFACES -- netstat -i

Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships

--interfaces=iface , -I=iface , -i

• Display a table of all network interfaces, or the specified iface.

1. OK -- Packets Transferred successfully
2. ERR -- Packet Errors
3. DRP -- Packet Drops
4. OVR -- Packet Overruns

2) NETSTAT STATISTICS -- netstat -s

--statistics , -s

• Display summary statistics for each protocol.

Few Metrics to start with

• Number of packets received.
• Ratio of sent packets s retransmitted packets.
• Active and passive connections.
• Connections resets.

3) NETSTAT -a, --all

• Show both listening and non-listening (for TCP this means established connections) sockets.
• With the --interfaces option, show interfaces that are not marked.

4) NETSTAT --route , -r

• Display the kernel routing tables.

5) cat /proc/net/snmp

6) SAR -- Network Statistics

• The sar command writes to standard output the contents of selected cumulative activity counters in the operating system.
• The accounting system, based on the values in the count and interval parameters, writes information the specified number of times spaced at the specified intervals in seconds.
• If the interval parameter is set to zero, the sar command displays the average statistics for the time since the system was started. 

The system activity reporter, sar( 1), can be used to observe current activity and can be configured to archive and report historical statistics.

• -n DEV: network interface statistics
• -n EDEV: network interface errors
• -n IP: IP datagram statistics
• -n EIP: IP error statistics
• -n TCP: TCP statistics
• -n ETCP: TCP error statistics
• -n SOCK: socket usage

More information on the fields

• active/s – new active TCP connections (connect())
• passive/s – new passive TCP connections (listen())
• iseg/s – input segments (received)
• oseg/s – output segments (received
• rxpck/s — Total number of packets received per second.
• txpck/s — Total number of packets transmitted per second.
• rxkB/s — Total number of kilobytes received per second.
• txkB/s — Total number of kilobytes transmitted per second.
• rxcmp/s — Number of compressed packets received per second (for cslip etc.).
• txcmp/s — Number of compressed packets transmitted per second.
• rxmcst/s — Number of multicast packets received per second.

7) IFCONFIG

• Configure a network interface
• Ifconfig is used to configure the kernel-resident network interfaces.
• It is used at boot time to set up interfaces as necessary.
• After that, it is usually only needed when debugging or when system tuning is needed.

8) IP

• Show / manipulate routing, devices, policy routing and tunnels

9) NICSTAT

• prints out network statistics for all NICs, including packets, kilobytes per second, average packet sizes and more.
• There are multiple options to understand different metrics.

10) PING

• Uses the ICMP protocol's mandatory ECHO_REQUEST datagram to elicit an ICMP ECHO_RESPONSE from a host or gateway.
• ECHO_REQUEST datagrams (''pings'') have an IP and ICMP header, followed by a struct timeval and then an arbitrary number of ''pad'' bytes used to fill out the packet.

11) TRACEROUTE

• Traceroute tracks the route packets taken from an IP network on their way to a given host.
• It utilizes the IP protocol's time to live (TTL) field and attempts to elicit an ICMP TIME_EXCEEDED response from each gateway along the path to the host.
• The only required parameter is the name or IP address of the destination host .
• The optional packet_len is the total size of the probing packet (default 60 bytes for IPv4 and 80 for IPv6).
• The specified size can be ignored in some situations or increased up to a minimal value.

12) PCHAR

• pchar is a tool to characterize the bandwidth, latency, and loss of links along an end-to-end path through the Internet.

13) TCPDUMP

• Network packets can be captured and inspected using the tcpdump(8) utility.
• This can either print packet summaries on STDOUT, or write packet data to a file for later analysis.

Command Line options

• Version Checking — tcpdump -h
• Check for the interfaces — sudo tcpdump -D
• Capture packets on all interfaces — sudo tcpdump -i any -- Capture will stop only after an interrupt. Ctrl + C
• Specified number of packets — sudo tcpdump -i any -c 5 -- stops after specified number of packets
• Easy to work with IP addresses — sudo tcpdump -i any -c 5 -n

Fields in TCP dump

• timestamp
• Protocol
• Source IP.Port
• Destination IP.Port
• TCP Flags — First letter for all flags. ACK for .
• Sequence number — Relative sequence number
• Window size
• Packet length

14) WIRESHARK

• The Wireshark tool (formerly Ethereal) provides a graphical interface for packet capture and inspection.
• First step will always be to use the tcp.analysis.flags.

Various flags that are important to be verified are

1. TCP Windowing issue
2. TCP Window Full
3. TCP Window Update
4. TCP Zero Window
5. TCP Retransmissions
6. TCP Out-of-Orders 
7. TCP Dup ACKS
8. TCP Spurious Retransmission
9. TCP Keep-Alive
10. TCP previous segment not captured
11. TCP Fast Retransmission

• Analysis may be based on TCP Flags such as TCP Retransmission
• Inference from the TCP retransmission flag

Know if Network dropped the sync packet

• Sync packet reached Server, Server responded with SYN/ACK but this packet got dropped.
• Server busy and didn't respond to the Sync request.

It is important to have a dual side capture on both the client side and the server side. This will help understand id the SYN request sent was received and server didn’t respond.

15) DTRACE

• DTrace is a comprehensive dynamic tracing framework created by Sun Microsystems for troubleshooting kernel and application problems on production systems in real time.

16) IPERF

• iPerf3 is a tool for active measurements of the maximum achievable bandwidth on IP networks.
• It supports tuning of various parameters related to timing, buffers and protocols (TCP, UDP, SCTP with IPv4 and IPv6).
• For each test it reports the bandwidth, loss, and other parameters. 

17) TCPRETRANS

• show TCP retransmits, with address and other details.

18) TCPCONNECT

• Trace TCP connect()s.

19) TCPACCEPT

• Trace TCP accept()s.

20) TCPTRACER

• Trace TCP connections.

21) TCPLIFE

• Trace the lifespan of TCP sessions and summarize.

22) TCPCONNLAT

• Trace TCP active connection latency (connect).

23) PERF

24) TCPTOP

• Summarize TCP send/recv throughput by host.

25) SOCKET STATISTICS

• ss is used to dump socket statistics. It allows showing information similar to netstat.
• It can display more TCP and state informations than other tools.