Network Basics: What Is Link Aggregation & How Does It Work?
by Kevin Dooley for Auvik Networks
Link aggregation is a way of bundling a bunch of individual (Ethernet) links together so they act like a single logical link.
If you have a switch with a whole lot of Gigabit Ethernet ports, you can connect all of them to another device that also has a bunch of ports and balance the traffic among these links to improve performance.
Another important reason for using link aggregation is to provide fast and transparent recovery in case one of the individual links fails.
Individual packets are kept intact and sent from one device to the other over one of the links. In fact, the protocol usually tries to keep whole sessions on a single link. A packet from the next conversation could go over a different link.
The idea is to achieve improved performance by transmitting several packets simultaneously down different links. But standard Ethernet link aggregation never chops up the packet and sends the bits over different links.
The official IEEE standard for link aggregation used to be called 802.3ad, but is now 802.1AX, as I will explain later. However, several vendors have also developed their own proprietary variants.
Common link aggregation terminology
A lot of potentially confusing terms appear in any discussion of link aggregation. So let’s quickly review them before digging a bit further into the technology.
- A group of ports combined together is called a link aggregation group, or LAG. Different vendors have their own terms for the concept. A LAG can also be called a port-channel, a bond, or a team.
- The rule that defines which packets are sent along which link is called the scheduling algorithm.
- The active monitoring protocol that allows devices to include or remove individual links from the LAG is called Link Aggregation Control Protocol (LACP).
Scheduling algorithms
The first important thing to know is that all links in a LAG must be a type of Ethernet (10/100/1000/10G, etc) and they must all be identical.
LACP can’t balance traffic among two Gigabit Ethernet links and a 100Mbps Ethernet link, for example. If you try, the devices will refuse to include the different link in the LAG. They might even refuse to bring up the LAG.
Further, all the links must be configured the same way. You can’t have a mix of duplex settings or different VLAN configurations or queuing features.
You can put a maximum of eight individual links in a LAG group, although some devices will restrict you to a smaller number. That said, because of the simple way most of the scheduling algorithms work, you’ll generally get better, more even load balancing if you use an even number, and preferably a power of 2 such as 2, 4, or 8.
An important concept of link aggregation is that all the packets belonging to any individual session should go down the same single link. Otherwise you risk out-of-order packets, which causes serious problems for a lot of applications.
Most scheduling algorithms use some sort of simple hash function that looks at fields in the Layer 2 and/or Layer 3 headers. The most common hashes involve the source and destination MAC addresses, the source and destination IP addresses, or both sets of addresses.
Many devices give you the option of selecting the appropriate load balancing algorithm for your network.
It’s important to note that two devices connected by link aggregation don’t need to agree on the load balancing algorithm, and sometimes you might not want them to. The goal is to select an algorithm that randomizes your packets as much as possible. That way you can expect fairly even use of all the links, which will provide the best possible performance.
On a general switched Layer 2 network with a lot of devices talking across the aggregated link in arbitrary patterns among themselves, the simplest MAC address hashing algorithm works well. Even if most of the traffic involves devices talking to a single central server, the algorithm still works well because the randomness of the MAC addresses of the other devices ensures reasonably even load balancing.
However, if the link is basically just two devices talking directly to one another across an aggregated link, then a MAC-based load balancing algorithm means all the traffic uses just one of the links.
This is the case, for example, if you have two routers (or Layer 3 switches) or two firewalls, or one of each talking across the link. You might be communicating with the whole Internet, but if all the packets are going to the same firewall, that’s one MAC address. And if all the packets are coming from one core switch, that’s also one MAC address. So a hash based only on MAC addresses won’t give you any performance advantage in such cases. In situations like this, it’s useful to use IP addresses in the load balancing algorithm.
On Cisco switches, depending on the software version, the command will be some variation of “port-channel load-balance <algorithm>”. Hitting a question mark where I have put the word <algorithm> prompts the switch to give you a list of available options.
You can easily tell whether your load balancing algorithm is appropriate by looking at the link utilizations on each of the individual links in a bundle using the “show interface” command.
If you see that one link is consistently more heavily utilized, then it might be a good idea to change your algorithm.
Note that when you change the algorithm on a device, you only change how that device behaves when sending packets. If the traffic imbalance is in the inbound direct, representing received packets, then you need to adjust the device on the other end.
Link Aggregation Control Protocol
Most of the time you’ll use 803.2ad or 802.1AX, also called Link Aggregation Control Protocol or LACP.
There are also various proprietary link aggregation protocols. Before the standardization of LACP, Cisco developed an option called Port Aggregation Protocol (PAgP) on some Cisco switches. Other vendors have similar pre-standard protocols.
PAgP is a proprietary protocol with no significant advantages over the standard LACP protocol. It really shouldn’t be used unless you happen to be connecting to a very old Cisco device that doesn’t support LACP.
The big question you’ll have to answer when configuring a port-channel is whether to configure it to be active (or, equivalently, LACP on some devices) or merely on.
The active option means the device will actively monitor the state of the link and automatically remove any failed links from the bundle. This is obviously a very good idea because it gives you fault tolerance as well as load sharing. So why would anybody ever opt not to use it?
The short answer is compatibility. If the device on one side of the link decides one of the individual connections is bad, then the device on the other end should really agree. Otherwise one device will keep dumping packets down a link that the other device isn’t watching.
A lot of server implementations don’t seem to implement the standard properly, or they cut corners and don’t implement the active monitoring function at all. I usually make everything active and only change to on if I run into trouble with it.