Getting Started with DPDK on AWS EC2

The Data Plane Development Kit (DPDK) is a set of libraries and NIC drivers for fast packet processing, managed as an open source project under the Linux Foundation. It is the "go to" solution for situations where Linux or Windows kernel TCP/IP stacks are too slow and complex for high packet-per-second (pps) or high bandwidth packet processing. DPDK runs in userspace, and uses poll mode drivers instead of interrupt mode drivers. I'm interested in DPDK for use with SMPTE ST 2110-20 uncompressed video over IP, with bit rates > 1 Gbps and pps > 100,000.

The DPDK project was started to perform software defined networking (SDN) using an Intel microprocessor as the data plane processor. As a result, most of the DPDK examples you will see revolve around packet forwarding, hashes, packet classification, QOS, VLANs, etc. There also is a significant concentration on multi-core parallelism. The libraries are very extensive, and much of the packaged example code tries to exercise a range of functionality. Building DPDK and the examples can also be challenging, as there have been many versions available, and you should be cautious with following Googled instructions that may not reflect the most recent versions. And as with many complex open source projects, dependencies and different Linux distributions can be a real challenge.

The good news is that it is not all that hard to introduce yourself to DPDK on a controlled environment, EC2 instances on AWS. I have assembled a brief tutorial to get you up and running with DPDK in under 10 minutes on AWS using a Red Hat Enterprise Linux AMI, and the AWS Elastic Network Adapter (ENA).

Once you have built DPDK and looked at some examples, you may be ready to try your own code. I'm not going to say it is easy, but I have put together some minimal code for DPDK packet sending and receiving. It is not hard to launch two AWS EC2 instances and then send packets between them using this code.

How much faster is DPDK than the OS TCP/IP stack? I did a test on AWS EC2 c5.9xlarge instance running RHEL with 862 byte UDP packets. This size packet can be used by SMPTE ST 2110-20 RTP to carry 800 bytes of row data – 320 pixels * 20 bits per pixel, of 1/4 line of 720p 4:2:2 10-bit video. These tests all used a single thread on a single core. The c5.9xlarge is rated for 10 Gbps network bandwidth. I kept the transmitter and receiver instances on the same subnet inside a Cluster Placement Group which promises "...all nodes within the placement group can talk to all other nodes within the placement group at the full line rate of 10 Gbps flows and 25 aggregate without any slowing due to over-subscription."

A program written in C using kernel sockets of type AF_INET, SOCK_DGRAM can transmit that RTP flow at 3.8 Gbps maximum, or about 551k packets per second. A DPDK program can transmit 6.1 Gbps, or about 884k pps.

For receiving that flow, kernel sockets begin to drop significant packets around 3.9 Gbps (565k pps). In addition to the application logging dropped RTP sequence numbers, the kernel shows its UDP receive buffer errors using "netstat -s -u". Note that this is even with setting up large UDP buffers with "sysctl -w net.core.rmem_max=67108864" and "sysctl -w net.core.netdev_max_backlog=2000". A DPDK receiver will continue to receive with low packet loss all the way up to at least 6.1 Gbps.

I intend to do some more testing of DPDK on AWS, looking at different instance types and multi-threaded solutions. Also AWS has announced the Elastic Fabric Adapter that can use Amazon's Scalable Reliable Datagram (SRD) protocol. Although aimed at HPC, it promises reliable high-bandwidth, low-latency transport in AWS.