How eBPF works

Some organizations that promote their solutions as ADR utilize eBPF technology to underpin their offerings. However, there are potential limitations to this approach. Unlike an eBPF approach, which operates at the kernel level, Contrast ADR is instrumented inside the application. This instrumentation approach enables Contrast ADR to detect and block known and zero-day attacks, leveraging deeper visibility and context-rich insights at the application level, and enabling runtime detection and prevention. In comparison, because eBPF-based approaches operate at the kernel level, these solutions cannot provide this level of protection.

What is eBPF?

Extended Berkeley Packet Filter (eBPF) is a transformative technology operating within the Linux kernel. It allows you to run sandboxed programs in the kernel space without requiring changes to the kernel source code or loading kernel modules. It extends the capabilities of classic Berkeley Packet Filter (cBPF), which was initially designed for network traffic filtering, making the kernel programmable in a safe and efficient way.

How does eBPF work?

Here's a breakdown of how eBPF works:

• Event-driven: eBPF programs don’t run continuously; rather, specific events trigger them. These events can occur within the kernel or user space, such as system calls, network activity (like receiving a packet), kernel function calls (kprobes), user-space function calls (uprobes), tracepoints and more.  
• Bytecode and verification: Programs are typically written in a restricted C subset and compiled into eBPF bytecode. Before execution, the eBPF verifier — a key safety mechanism — rigorously checks the bytecode. This verification process ensures the program is safe to run, preventing kernel crashes, infinite loops, invalid memory access and other harmful behaviors that could destabilize the system. This makes eBPF significantly safer than traditional kernel modules. 
• Just-in-time (JIT) compilation: For optimal performance, the verified bytecode is then translated into native machine code by a JIT compiler before execution.  
• Maps: eBPF programs can interact with kernel space and user space through eBPF maps, which are efficient key-value stores for sharing data and maintaining state.  
• Helper functions: Instead of directly calling arbitrary kernel functions, eBPF programs use a set of stable helper functions provided by the kernel, ensuring compatibility across different kernel versions.

What is eBPF observability?

eBPF observability refers to the use of Extended Berkeley Packet Filter (eBPF) technology to gather detailed, low-level data from the Linux kernel with minimal overhead. This allows for monitoring, troubleshooting and gaining insights into the behavior of systems and applications.  

Instead of relying on traditional methods like parsing logs or instrumenting applications with user-space agents, eBPF allows you to run sandboxed programs within the kernel itself. These programs can hook into a wide array of kernel events (such as system calls, network events, function entries/exits and tracepoints) to collect rich telemetry.

Benefits of eBPF

For solutions designed to provide observability, eBPF has a few advantages:

• Flexibility and programmability: for those teams that can allocate the expertise, eBPF allows developers to add custom logic and extend kernel functionality dynamically without recompiling or rebooting the kernel.  
• Observability at the kernel layer: It provides visibility into system calls, network activity and process interactions at the kernel level, enabling monitoring of the host itself.
• Enhanced safety and stability: eBPF programs run in a restricted virtual machine within the Linux kernel, operating in a sandboxed environment. Before execution, the code is checked by the eBPF Verifier, which is a critical safety mechanism that ensures the code is free from infinite loops, invalid memory access, and other operations that could crash or destabilize the kernel. This makes eBPF significantly safer than traditional kernel modules, where a bug can easily compromise the entire system. The design aims to limit the potential consequences of agent failure.
• High efficiency and performance: eBPF is designed to be very lightweight and efficient, performing tasks with minimal overhead. It utilizes a just-in-time (JIT) compiler to translate eBPF bytecode into native machine code, optimizing execution speed. This allows for real-time monitoring and data collection with low latency and negligible performance impact — often cited as less than 0.1% CPU usage for some applications. Furthermore, eBPF programs can be dynamically loaded and unloaded at runtime without requiring system reboots, which adds to its operational efficiency and unobtrusiveness.
• Language agnosticism: Because eBPF operates at the Linux kernel level, it is inherently independent of the programming languages (e.g., Java, Python, Go, Node.js) used to build the applications running on the system. A single eBPF-based solution can monitor applications written in any language without requiring language-specific agents. 

Downsides of eBPF

While eBPF offers some advantages for ADR, it's essential to acknowledge the potential downsides and challenges associated with its use in this context. Here are some key considerations:

• Limited application-layer visibility and context: A primary limitation of eBPF is its restricted insight into the internal workings, logic and data flows within applications. Operating at the kernel level — effectively as a shim between the application and the OS — eBPF monitors system calls, network activity and other kernel events. However, this means it generally:
  • Lacks deep application context: It collects raw system-level data but doesn't inherently understand application-specific vulnerabilities or semantics. For instance, it cannot typically see internal function calls, object states (e.g., ObjectInputStream.readObject() in Java), or precisely how data is processed within the application code.
  • Struggles with managed runtimes: Traditional kernel probes face difficulties reading the virtual memory managed by runtimes like Python, Java or Node.js, as these manage their own memory in user space.
  • May miss application-pure threats: While it might detect the system-level effects of an exploited vulnerability (like an RCE payload), eBPF often cannot identify the root cause or the vulnerability within the application code itself, leading to blindspots for threats like complex SQL injection, JNDI injections, SSRF or business logic flaws that don't immediately translate to anomalous system calls.
• Performance impact with user probes: While eBPF is generally designed for efficiency, implementing application-level visibility using user-space probes (uprobes) can lead to significant performance degradation. This is because user probes require an additional context switch for every event. This hurts performance, especially at scale and with intensive or I/O-heavy workloads. This can result in a 200% increase in overhead compared with kernel probe implementations in some cases. This performance impact can make user probes unsuitable for production environments.
• Not originally designed for security monitoring: eBPF was primarily intended as a networking and debugging tool. While it has many uses in improving computer security, eBPF observability tools weren’t designed to be used directly for security monitoring. There’s no guarantee that probes will fire as expected in a security context.
• Data truncation and limited instruction count: eBPF programs have a limited stack space (512 bytes), which can lead to data truncation for larger pieces of data like file paths (which can be up to 4,096 bytes). Workarounds exist but involve added complexity. Additionally, an eBPF program is limited to 4,096 instructions. While workarounds like using bounded loops or tail calls exist, the latter can lead to loss of program state.
• Time-of-check to time-of-use (TOCTOU) issues: Since eBPF programs can run concurrently without synchronization primitives, data races and TOCTOU issues are a serious concern. Mitigating these can be difficult or impossible when dealing with kernel code that uses locks inaccessible from eBPF.
• Reliability considerations for security monitoring: While eBPF's verifier is crucial for ensuring that eBPF programs themselves won't crash the kernel (providing operational safety), this guarantee does not automatically extend to the reliable or complete capture of all security events. eBPF was originally designed more for networking and debugging, and when applied to security monitoring, several potential pitfalls can arise.
• Event overload: A large volume of events can overwhelm an eBPF attach point, potentially leading to missed events, data loss (due to lack of space or overwriting), or data corruption. User-mode components need to handle incoming data from eBPF as untrusted and implement mechanisms to detect missing or corrupted data.
• Page faults: eBPF runs with page faults disabled. If memory accessed by an eBPF probe is paged out to disk, it cannot be accessed, which can be problematic for security monitoring tools that need to read specific memory locations. Workarounds involve carefully choosing hook points and handling potential errors.
• Limited blocking capabilities: eBPF works asynchronously and often cannot prevent exploitation directly. While mechanisms exist (like integrating with Linux security modules [LSMs]) to allow control and blocking of attacks, this requires specific implementation. OpenTelemetry leveraging eBPF alone does not offer deep response actions.
• Kernel dependency and compatibility: eBPF is a Linux-only technology. Furthermore, its features have evolved rapidly, meaning specific eBPF programs or tools may require relatively recent kernel versions (e.g., kernel 4.x or 5.x and newer) to function correctly or to access all capabilities. Ensuring compatibility across diverse Linux distributions and kernel versions within an organization can be a significant operational challenge, and legacy systems may lack necessary eBPF support.
• Complexity of development: While powerful, writing, debugging, and maintaining robust eBPF programs requires specialized knowledge of kernel internals, eBPF bytecode, verifier limitations and associated user-space tooling. Though libraries and frameworks aim to simplify development, programming with eBPF remains more complex than typical application programming.

eBPF observability use cases

eBPF has become a foundational technology for a wide range of applications, including:

• Networking: High-performance packet filtering, load balancing, traffic shaping, network monitoring and security. 
• Observability: System call tracing, performance monitoring, application profiling and creating custom metrics without the need for agents.
• Security: Intrusion detection and prevention, policy enforcement, and malware analysis by observing kernel-level events and network activity.  
• Tracing and profiling: Analyzing application and kernel behavior to identify bottlenecks and performance issues.  
• Container security and observability: Monitoring and securing containerized environments by attaching to cgroup hooks and observing system calls within containers.

For security use cases specifically, eBPF plays a significant role in modern security practices by enabling:

• Real-time threat detection: Monitoring system calls and network traffic at the kernel level allows for immediate detection of some suspicious activities like privilege escalations or container escapes.  
• Policy enforcement: Implementing and enforcing security policies directly in the kernel. This can apply to network traffic, process behavior and resource access. For blocking of unauthorized actions, eBPF is often integrated with Linux security modules (LSMs). 
• Enhanced visibility: Gaining deep insights into system behavior to identify anomalies and potential security risks.  
• Low overhead monitoring: Performing security monitoring efficiently within the kernel without significant performance impact.

eBPF vs instrumentation application security monitoring

eBPF-based ADR and Contrast ADR, while both aiming to improve application security, differ in their approach and the level of detail provided about their mechanisms. While both eBPF and Contrast ADR can fall under the umbrella of Application Detection and Response, eBPF focuses at the kernel level while Contrast ADR employs a more comprehensive approach, involving instrumentation, to achieve deeper application-level visibility, real-time prevention and context-rich insights. Contrast ADR is instrumented inside the application via an agent, while eBPF gleans information from monitoring system calls.

Here are some of the key distinctions:

• • Location and layer of operation: 
    • eBPF runs programs in a sandboxed environment within the Linux kernel. It primarily interacts with the operating system layer. 
    • Application instrumentation involves embedding code or agents directly within or alongside the application in user space. It operates at the application layer.
  • Visibility and context: 
    • eBPF provides visibility into kernel-level events such as system calls (process interactions, file I/O, network activity), network events and tracepoints. However, it generally lacks insight into the internal workings, logic and data flows within applications. It cannot typically see application-level function calls or objects inside managed runtimes. Nor can it easily read the virtual memory managed by these runtimes in user space. It collects raw system-level data but doesn't inherently provide context about application vulnerabilities.
    • Instrumentation offers visibility into the application logic, data flows and function execution. It can see how user data flows through the code and interacts with dangerous APIs. Distributed tracing, a form of instrumentation, maps service-to-service communications and traces requests through potentially hundreds of microservices. This allows it to provide rich context about vulnerabilities and attack paths directly from within the application.
  • Blocking and response:
    • eBPF often works asynchronously and cannot prevent exploitation directly without specific implementations like integrating with Linux security modules.
    • Instrumentation (application-level, such as ADR tools leveraging instrumentation) can enforce security policies and block attacks in real time from within the application context.
  • Coverage:
    • eBPF is effective for monitoring kernel-level threats but covers only a small fraction of common application/API vulnerabilities/attacks by itself. It may miss application-specific attacks like SQL injection and JNDI injection.
    • Instrumentation focuses on and covers a broad range of application-specific vulnerabilities and attack techniques.

While eBPF provides monitoring at the kernel/OS boundary, its inherent limitation is the lack of deep visibility and context into the application's internal logic. Application instrumentation, conversely, excels at seeing inside the application. The choice between eBPF-based approaches and instrumentation often depends on the specific security goals, the types of threats being addressed, performance considerations and the operational trade-offs an organization is willing to make.