Observability and eBPF - Extended Berkeley Packet Filter

eBPF stands for “extended Berkeley Packet Filter.” This technology allows developers to write and run highly efficient and secure programs that analyze and modify data packets as they move through a system. eBPF programs can be used for various purposes, including network monitoring, security, and performance optimization.

eBPF origins in the Linux kernel that can run sandboxed programs in a privileged context, such as the operating system kernel. It is used to safely and efficiently extend the kernel's capabilities without requiring changing kernel source code or load kernel modules.

Historically, the operating system has been an ideal place to implement observability, security, and networking functionality due to the kernel’s privileged ability to oversee and control the entire system. At the same time, an operating system kernel is hard to evolve due to its central role and high requirement for stability and security. The rate of innovation at the operating system level has thus traditionally been lower compared to functionality implemented outside of the operating system.

eBPF changes this formula fundamentally. By allowing sandboxed programs to run within the operating system, application developers can run eBPF programs to add additional capabilities to the operating system at runtime. The operating system then guarantees safety and execution efficiency as if natively compiled with the aid of a Just-In-Time (JIT) compiler and verification engine. This has led to a wave of eBPF-based projects covering various use cases, including next-generation networking, observability, and security functionality.

Today, eBPF is used extensively to drive a wide variety of use cases: Providing high-performance networking and load-balancing in modern data centers and cloud-native environments, extracting fine-grained security observability data at low overhead, helping application developers trace applications, providing insights for performance troubleshooting, preventive application and container runtime security enforcement, and much more. The possibilities are endless, and the innovation that eBPF is unlocking has only just begun.

Programmability

Twenty years ago, web pages used to be almost exclusively written in the static markup language (HTML). A web page was basically a document with an application (browser) able to display it. Web pages today have become full-blown applications and web-based technology has replaced a vast majority of applications written in languages requiring compilation. Programmability with the introduction of JavaScript enabled this evolution. It unlocked a massive revolution resulting in browsers evolving into almost independent operating systems.

Programmers were no longer as bound to users running particular browser versions. Instead of convincing standards bodies that a new HTML tag was needed, the availability of the necessary building blocks decoupled the pace of innovation of the underlying browser from the application running on top. This is of course a bit oversimplified as HTML did evolve over time and contributed to the success but the evolution of HTML itself would not have been sufficient.

Key aspects that were vital in the introduction of JavaScript:

★ Safety: Untrusted code runs in the browser of the user. This was solved by sandboxing JavaScript programs and abstracting access to browser data.

★ Continuous Delivery: Evolution of program logic must be possible without requiring to constantly ship new browser versions. This was solved by providing the right low-level building blocks sufficient to build arbitrary logic.

★ Performance: Programmability must be provided with minimal overhead. This was solved with the introduction of a Just-in-Time (JIT) compiler. For all of the above, exact counter parts can be found in eBPF for the same reason.

eBPF's impact on the Linux Kernel

In order to understand the programmability impact of eBPF on the Linux kernel, it helps to have a high-level understanding of the architecture of the Linux kernel and how it interacts with applications and the hardware.

The main purpose of the Linux kernel is to abstract the hardware or virtual hardware and provide a consistent API (system calls) allowing for applications to run and share the resources. In order to achieve this, a wide set of subsystems and layers are maintained to distribute these responsibilities. Each subsystem typically allows for some level of configuration to account for different needs of users. If a desired behavior cannot be configured, a kernel change is required, historically, leaving two options:

Native Support

★ Change kernel source code and convince the Linux kernel community that the change is required.

★ Wait several years for the new kernel version to become a commodity.

Kernel Module

★ Write a kernel module

★ Fix it up regularly, as every kernel release may break it

★ Risk corrupting your Linux kernel due to lack of security boundaries

With eBPF, a new option is available that allows for reprogramming the behavior of the Linux kernel without requiring changes to kernel source code or loading a kernel module. In many ways, this is very similar to how JavaScript and other scripting languages unlocked the evolution of systems which had become difficult or expensive to change.

eBPF Use Cases

Security

Extending the basic capabilities of seeing and interpreting all system calls and providing packet and socket-level views of all networking operations enables the development of revolutionary approaches to system security.

Typically, entirely independent systems have handled different aspects of system call filtering, process context tracing, and network-level filtering. On the other hand, eBPF facilitates the combination of control and visibility over all aspects. This allows you to develop security systems that operate with more context and improved control.

Networking

Efficient and programmability make eBPF a good candidate for all networking solutions’ packet processing requirements. The programmability of eBPF provides a means of adding additional protocol parsers and smoothly programs any forwarding logic to address changing requirements without ever exiting the Linux kernel’s packet processing context. The effectiveness offered by the JIT compiler offers execution performance near that of natively compiled in-kernel code.

Tracing and Profiling

Attaching eBPF programs to trace points in addition to the kernel and user application probe points enables visibility into the runtime behavior of applications and the system.

Both views can be combined by providing introspection capabilities to the system and the application. This gives unique and powerful insights to troubleshoot system performance issues. Advanced statistical data structures let you effectively extract useful visibility data without exporting huge amounts of sampling data typical for similar systems.

Observability and Monitoring

Rather than relying on gauges and static counters exposed by the operating system, eBPF allows for the generation of visibility events and the collection and in-kernel aggregation of custom metrics based on a broad range of potential sources.

This increases the depth of visibility that might be attained and decreases the overall system overhead dramatically. This is achieved by collecting only the required visibility data and by producing histograms and similar data structures at the source of the event rather than depending on the export of samples.

We will talk more about that later on.

How eBPF Works

eBPF programs are used to access hardware and services from the Linux kernel area. These programs are used for debugging, tracing, firewalls, networking, and more.

Developed out of a need for improved Linux tracing tools, eBPF was influenced by dtrace, a dynamic tracing tool available mainly for BSD and Solaris operating systems. Unlike dtrace, Linux could not achieve a global overview of running systems. Rather, it was restricted to specific frameworks for library calls, functions, and system calls.

eBPF is an extension of its precursor, BPF. BPF is a tool used for writing packer-filtering code via an in-kernel VM. A group of engineers started to build on the BPF backend to offer a similar series of features as dtrace, which eventually evolved into eBPF. Although initially released in a limited capacity in 2014 with Linux 3.18, you need at least Linux 4.4 or above to use eBPF fully.

The diagram below is a simplified illustration of eBPF architecture. Before being loaded into the kernel, the eBPF program must pass a particular requirement. Verification includes executing the eBPF program in the virtual machine.

This permits the verifier, with 10,000+ lines of code, to carry out a set of checks. The verifier will go over the potential paths the eBPF program might take when executed in the kernel, to ensure the program runs to completion without any looping, which would result in a kernel lockup.

Additional checks—from program size, to valid register state, to out-of-bound jumps—should also be made. eBPF distinguishes itself from Linux Loadable Kernel Modules (LKM) by adding these additional safety controls.

If all checks are cleared, the eBPF program is loaded and compiled into the kernel at a location in a code path, and waits for the appropriate signal. When the signal is received in the form of an event, the eBPF program is loaded in the code path. Once initiated, the bytecode collects and executes information according to its instructions.

To summarize, the role of eBPF is to allow programmers to execute custom bytecode safely within the Linux kernel, without adding to or changing the kernel source code. Though it cannot replace LKMs altogether, eBPF programs introduce custom code that relates to protected hardware resources, with limited threat to the kernel.

eBPF Observability

eBPF observability encompasses the light yet thorough implementation of programs designed to monitor events in the kernel. This is primarily carried out in Linux kernels. However, significant progress has been made with Windows, as pointed out earlier in this post.

As an observability tool, eBPF stands out because it can execute programs to exfiltrate monitoring data within the kernel without altering the source code. 

Observability with eBPF is very secure, isolated, and non-obtrusive and can be exported to centralized platforms. It enhances observability by providing a great deal of visibility, context, and accuracy of infrastructure and network events.

Typically, eBPF programs are either written in Rust or C. The just-in-time (JIT) compiler compiles, verifies, and loads the code into the kernel. After the program is loaded into the kernel, you have to attach it to kernel functions or events.

The eBPF programs will then run every time the respective function or event is executed. eBPF programs can be attached to events like tracepoints, entry to and exit from functions, perf events (used to collect performance data), the Linux Security Module (LSM) interface, network interfaces, network sockets, and more.

★ Tracepoints – Tracepoints are lightweight hooks that can be used to call a function at runtime and are commonly used to account for tracing and performance in the kernel. eBPF programs can be attached to tracepoints to trace events like system calls.

★ Entry to and exit from functions – Operators can attach eBPF programs to entry and exit functions so that their custom kernel programs run in the event of these respective scenarios.

★ Perf events – As the name implies, this is a subsystem used to collect performance data. With eBPF, you can attach custom programs to where this data is collected.

★ LSI module interface – This is a powerful module interface used by tools like AppArmor and SELinux. Both AppArmor and SELinux are commonly used in Kubernetes security techniques to apply low-level system measures to prevent and mitigate the risks of running container workloads. Similarly, eBPF programs can be attached to the LSI module interface checkpoints for runtime security with dynamic policies.

★ Network interfaces (XDP) – The eXpress Data Path (XDP) allows operators to attach eBPF programs to network interfaces so that their custom programs are executed whenever a network packet is received.

Pros of eBPF for Observability

★ Unintrusive – As detailed earlier, eBPF programs run in the kernel in a sandboxed environment similar to that of a VM. This provides sufficient isolation from the kernel itself and other modules.

★ Secure – eBPF programs not only run in an isolated environment, but they also don’t require any altering of the kernel source code and have to go through a verification process by the JIT compiler before being loaded into the kernel. 

★ Centralized – eBPF programs provide finer granular details of events taking place in the kernel. This provides great insight into container and network activity and can be exported to centralized platforms for aggregation and visibility.

Cons of eBPF for Observability

★ Market Adoption & Novelty – There is a lot of excitement and buzz around the technology, and the support from top tech companies has bolstered its case in the cloud-native space. However, it’s still relatively new and requires relatively advanced domain knowledge. As such, market adoption hasn’t been rapid.

★ Linux Restrictions – Currently, eBPF is primarily used in Linux kernels. This presents a challenge for teams looking to adopt eBPF technology for Kubernetes and container workloads running with environments that aren’t Linux based. However, eBPF for Windows is under active development. 

Who should go for eBPF Observability?

The adoption of eBPF in cloud-native apps is accelerating. As a result, the two situations where eBPF is most commonly required are as follows:

★ When observability is required employing kernel tracing – eBPF is quicker and more precise in this case. Because eBPF applications are event-based and don’t use context switches, nothing happens without a specified trigger, and you won’t miss any occurrences.

★ Traditional security monitoring is inefficient – Kubernetes and other distributed and container-based environments are big users of eBPF. In such environments, eBPF can narrow the visibility gap because it provides visibility into HTTP traffic.

You may also find eBPF deployed for other security measures, including:

★Network performance monitoring

★Firewalls

★ And, device drivers

Best Practices for eBPF

★ Compiling Code – Previously, assembling and compiling code for eBPF programs was very restricted and error-prone because of a lack of sufficient optimal tooling. Nowadays, eBPF programs can be compiled with the use of LLVM Clang. 

★ Writing Programs – To assist with writing optimal kernel programs for tracing and manipulation, you can use the BFP Compiler Collection (BCC) toolkit.

How Middleware uses eBPF 

Middleware uses eBPF to provide in-depth data and insight into the infrastructure, container behavior, and network activity for better tracing and monitoring capabilities. With a simple installation command, Middleware allows us to execute an eBPF base installation to start collecting data from the system kernel. 

The program installed in the kernel will monitor all the network traffic and activity that flows through the system. All hops will be detected, along with source and destination IP data, in order to provide accurate and real-time service maps.

This is what enables Middleware’s comprehensive tracing and allows us to have a full scope of monitoring. The agent exporting data to the Middleware platform is able to recognize various network protocol requests such as HTTP, gRPC calls, SQL queries, Kafka Calls, and more.

If we’re running fronted, backend, and database applications, we will be able to view a service map for the network flow. In addition, we can review and visualize the queries made from the backend to the database and any issues or occurrences in the flow of network traffic between the respective services. 

The Middleware agent converts the collected data into OpenTelemtry (OTel) format enabling teams to consume the data with a wide variety of tools that support OTel specifications.

From the ePBF.io article "What is eBPF?", the Tigera article "eBPF Explained: Use Cases, Concepts, and Architecture" and the Middleware article "The Ultimate Guide to eBPF Observability."