Note
Please be aware that we currently won't merge 3rd party PRs because this repository is temporary. We are waiting for the decision of the OpenTelemetry technical commitee on the donation of this repository.
In case the donation gets accepted, this repository will move to the GitHub open-telemetry organization, which requires signing a different CLA. At that point we will start working on reviewing and merging 3rd party PRs.
This repository implements a whole-system, cross-language profiler for Linux via eBPF. The repository serves as a staging space in the process of donating the agent to OpenTelementry.
- Implements the experimental OTel profiling signal
- Very low CPU and memory overhead (1% CPU and 250MB memory are our upper limits in testing and the agent typically manages to stay way below that)
- Support for native C/C++ executables without the need for DWARF debug
information (by leveraging
.eh_framedata as described in US11604718B1) - Support profiling of system libraries without frame pointers and without debug symbols on the host.
- Support for mixed stacktraces between runtimes - stacktraces go from Kernel space through unmodified system libraries all the way into high-level languages.
- Support for native code (C/C++, Rust, Zig, Go, etc. without debug symbols on host)
- Support for a broad set of HLLs (Hotspot JVM, Python, Ruby, PHP, Node.JS, V8, Perl), .NET is in preparation.
- 100% non-intrusive: there's no need to load agents or libraries into the processes that are being profiled.
- No need for any reconfiguration, instrumentation or restarts of HLL interpreters and VMs: the agent supports unwinding each of the supported languages in the default configuration.
- ARM64 support for all unwinders except NodeJS.
- Support for native
inline frames, which provide insights into compiler optimizations and offer a higher precision of function call chains.
Note
If you simply wish to take the agent for a spin with minimal effort, you can also immediately jump to the "Visualizing data locally" section, launch devfiler and follow the download links for agent binaries within its "Add data" dialogue.
The agent can be built without affecting your environment by using the provided
make targets. You need to have docker installed, though.
Builds on amd64 and arm64 architectures are supported.
The first step is to build the Docker image that contains the build environment:
make docker-imageThen, you can build the agent:
make agentThe resulting binary will be in the current directory as otel-profiling-agent.
Alternatively, you can build without Docker. Please see the Dockerfile for required dependencies.
After installing the dependencies, just run make to build.
You can start the agent with the following command:
sudo ./otel-profiling-agent -collection-agent=127.0.0.1:11000 -disable-tlsThe agent comes with a functional but work-in-progress / evolving implementation of the recently released OTel profiling signal.
The agent loads the eBPF program and its maps, starts unwinding and reports captured traces to the backend.
We created a desktop application called "devfiler" that allows visualizing the
profiling agent's output locally, making it very convenient for development use.
devfiler spins up a local server that listens on 0.0.0.0:11000.
To run it, simply download and unpack the archive from the following URL:
https://upload.elastic.co/d/0891b6a006b1ee8224e638d2454b967f7c1ac596110b5c149ac7c98107655d9b
Authentication token: a217abfdd7c438e9
The archive contains a build for each of the following platforms:
- macOS (Intel)
- macOS (Apple Silicon)
- Linux AppImage (x86_64)
- Linux AppImage (aarch64)
Note
devfiler is currently in an experimental preview stage.
This build of devfiler is currently not signed with a globally trusted Apple
developer ID, but with a developer certificate. If you simply double-click the
application, you'll run into an error. Instead of opening it with a double
click, simply do a right-click on devfiler.app, then choose "Open". If you
go this route, you'll instead be presented with the option to run it anyway.
The AppImages in the archive should run on any Linux distribution with a reasonably modern glibc and libgl installation. To run the application, simply extract the archive and then do:
./devfiler-appimage-$(uname -m).AppImageThe host agent is a Go application that is deployed to all machines customers wish to profile. It collects, processes and pushes observed stack traces and related meta-information to a backend collector.
A file ID uniquely identifies an executable, kernel or script language source file.
File IDs for native applications are created by taking the SHA256 checksum of a file's head, tail, and size, then truncating the hash digest to 16 bytes (128 bits):
Input ← Concat(File[:4096], File[-4096:], BigEndianUInt64(Len(File)))
Digest ← SHA256(Input)
FileID ← Digest[:16]
File IDs for script and JIT languages are created in an interpreter-specific fashion.
File IDs for Linux kernels are calculated by taking the FNV128 hash of their GNU build ID.
Stack unwinding is the process of recovering the list of function calls that lead execution to the point in the program at which the profiler interrupted it.
How stacks are unwound varies depending on whether a thread is running native, JITed or interpreted code, but the basic idea is always the same: every language that supports arbitrarily nested function calls needs a way to keep track of which function it needs to return to after the current function completes. Our unwinder uses that same information to repeatedly determine the caller until we reach the thread's entry point.
In simplified pseudo-code:
pc ← interrupted_process.cpu.pc
sp ← interrupted_process.cpu.sp
while !is_entry_point(pc):
file_id, start_addr, interp_type ← file_id_at_pc(pc)
push_frame(interp_type, file_id, pc - start_addr)
unwinder ← unwinder_for_interp(interp_type)
pc, sp ← unwinder.next_frame(pc, sp)
Symbolization is the process of assigning source line information to the raw addresses extracted during stack unwinding.
For script and JIT languages that always have symbol information available on the customer machines, the host agent is responsible for symbolizing frames.
For native code the symbolization occurs in the backend. Stack frames are sent as file IDs and the offset within the file and the symbolization service is then responsible for assigning the correct function name, source file and lines in the background. Symbols for open-source software installed from OS package repos are pulled in from our global symbolization infrastructure and symbols for private executables can be manually uploaded by the customer.
The primary reason for doing native symbolization in the backend is that native executables in production will often be stripped. Asking the customer to deploy symbols to production would be both wasteful in terms of disk usage and also a major friction point in initial adoption.
We have two major representations for our stack traces.
The raw trace format produced by our BPF unwinders:
https://github.com/elastic/otel-profiling-agent/blob/0945fe6/host/host.go#L60-L66
The final format produced after additional processing in user-land:
https://github.com/elastic/otel-profiling-agent/blob/0945fe6/libpf/libpf.go#L458-L463
The two might look rather similar at first glance, but there are some important differences:
- the BPF variant uses truncated 64-bit file IDs to save precious kernel memory
- for interpreter frames the BPF variant uses the file ID and line number fields to store more or less arbitrary interpreter-specific data that is needed by the user-mode code to conduct symbolization
A third trace representation exists within our network protocol, but it essentially just a deduplicated, compressed representation of the user-land trace format.
In profiling it is common to see the same trace many times. Traces can be up to 128 entries long, and repeatedly symbolizing and sending the same traces over the network would be very wasteful. We use trace hashing to avoid this. Different hashing schemes are used for the BPF and user-mode trace representations. Multiple 64 bit hashes can end up being mapped to the same 128 bit hash, but not vice-versa.
BPF trace hash (64 bit):
H(kernel_stack_id, frames_user, PID)
User-land trace hash (128 bit)
H(frames_user_kernel)
The tracer is a central user-land component that loads and attaches our BPF programs to their corresponding BPF probes during startup and then continues to serve as the primary event pump for BPF <-> user-land communication. It further instantiates and owns other important subcomponents like the process manager.
The trace handler is responsible for converting traces from the BPF format to
the user-space format. It receives raw traces tracer, converts them
to the user-space format and then sends them on to the reporter.
The majority of the conversion logic happens via a call into the process
manager's ConvertTrace function.
Since converting and enriching BPF-format traces is not a cheap operation, the trace handler is also responsible for keeping a cache (mapping) of trace hashes: from 64bit BPF hash to the user-space 128bit hash.
The reporter receives traces and trace counts in the user-mode format from the trace handler, converts them to the gRPC representation and then sends them out to a backend collector.
It also receives additional meta-information (such as metrics and host metadata) which it also converts and sends out to a backend collector over gRPC.
The reporter does not offer strong guarantees regarding reliability of network operations and may drop data at any point, an "eventual consistency" model.
The process manager receives process creation/termination events from tracer and is responsible for making available any information to the BPF code that it needs to conduct unwinding. It maintains a map of the executables mapped into each process, loads stack unwinding deltas for native modules and creates interpreter handlers for each memory mapping that belongs to a supported language interpreter.
During trace conversion the process manager is further responsible for routing symbolization requests to the correct interpreter handlers.
Each interpreted or JITed language that we support has a corresponding type that implements the interpreter handler interface. It is responsible for:
- detecting the interpreter's version and structure layouts
- placing information that the corresponding BPF interpreter unwinder needs into BPF maps
- translating interpreter frames from the BPF format to the user-land format by symbolizing them
Unwinding the stack of native executables compiled without frame pointers requires stack deltas. These deltas are essentially a mapping from each PC in an executable to instructions describing how to find the caller and how to adjust the unwinder machine state in preparation of locating the next frame. Typically these instructions consist of a register that is used as a base address and an offset (delta) that needs to be added to it -- hence the name. The stack delta provider is responsible for analyzing executables and creating stack deltas for them.
For most native executables, we rely on the information present in .eh_frame.
.eh_frame was originally meant only for C++ exception unwinding, but it has
since been repurposed for stack unwinding in general. Even applications written
in many other native languages like C, Zig or Rust will typically come with
.eh_frame.
One important exception to this general pattern is Go. As of writing, Go
executables do not come with .eh_frame sections unless they are built with CGo
enabled. Even with CGo the .eh_frame section will only contain information for
a small subset of functions that are either written in C/C++ or part of the CGo
runtime. For Go executables we extract the stack delta information from the
Go-specific section called .gopclntab. In-depth documentation on the format is
available in a separate document).
The BPF portion of the host agent implements the actual stack unwinding. It uses
the eBPF virtual machine to execute our code directly in the Linux kernel. The
components are implemented in BPF C and live in the
otel-profiling-agent/support/ebpf directory.
BPF programs must adhere to various restrictions imposed by the verifier. Many of these limitations are significantly relaxed in newer kernel versions, but we still have to stick to the old limits because we wish to continue supporting older kernels.
The minimum supported Linux kernel versions are
- 4.19 for amd64/x86_64
- 5.5 for arm64/aarch64
The most notable limitations are the following two:
- 4096 instructions per program
A single BPF program can consist of a maximum of 4096 instructions, otherwise older kernels will refuse to load it. Since BPF does not allow for loops, they instead need to be unrolled. - 32 tail-calls
Linux allows BPF programs to do a tail-call to another BPF program. A tail call is essentially ajmpinto another BPF program, ending execution of the current handler and starting a new one. This allows us to circumvent the 4096 instruction limit a bit by doing a tail-call before we run into the limit. There's a maximum of 32 tail calls that a BPF program can do.
These limitations mean that we generally try to prepare as much work as possible
in user-land and then only do the minimal work necessary within BPF. We can only
use
Unwinding always begins in native_tracer_entry. This entry point for our
tracer starts by reading the register state of the thread that we just
interrupted and initializes the PerCPURecord structure. The per-CPU record
persists data between tail-calls of the same unwinder invocation. The unwinder's
current PC, SP etc. values are initialized from register values.
After the initial setup the entry point consults a BPF map that is maintained
by the user-land portion of the agent to determine which interpreter unwinder
is responsible for unwinding the code at PC. If a record for the memory
region is found, we then tail-call to the corresponding interpreter unwinder.
Each interpreter unwinder has their own BPF program. The interpreter unwinders typically have an unrolled main loop where they try to unwind as many frames for that interpreter as they can without going over the instruction limit. After each iteration the unwinders will typically check whether the current PC value still belongs to the current unwinder and tail-call to the right unwinder otherwise.
When an unwinder detects that we've reached the last frame in the trace,
unwinding is terminated with a tail call to unwind_stop. For most traces
this call will happen in the native unwinder, since even JITed languages
usually call through a few layers of native C/C++ code before entering the VM.
We detect the end of a trace by heuristically marking certain functions with
PROG_UNWIND_STOP in the BPF maps prepared by user-land. unwind_stop then
sends the completed BPF trace to user-land.
If any frame in the trace requires symbolization in user-mode, we additionally send a BPF event to request an expedited read from user-land. For all other traces user-land will simply read and then clear this map on a timer.
The BPF components are responsible for notifying user-land about new and exiting
processes. An event about a new process is produced when we first interrupt it
with the unwinders. Events about exiting processes are created with a
sched_process_exit probe. In both cases the BPF code sends a perf event to
notify user-land. We also re-report a PID if we detect execution in previously
unknown memory region to prompt re-scan of the mappings.
All collected information is reported to a backend collector via a push-based, stateless, one-way gRPC protocol.
All data to be transmitted is stored in bounded FIFO queues (ring buffers). Old data is overwritten when the queues fill up (e.g. due to a lagging or offline backend collector). There is no explicit reliability or redundancy (besides retries internal to gRPC) and the assumption is that data will be resent (eventually consistent).
The host agent contains an internal pipeline that incrementally processes the raw traces that are produced by the BPF unwinders, enriches them with additional information (e.g. symbols for interpreter frames and container info), deduplicates known traces and combines trace counts that occurred in the same update period.
The traces produced in BPF start out with the information shown in the following diagram.
Note: please read this if you wish to update the diagrams
The diagrams in this section were created via draw.io. The SVGs can be loaded into draw.io for editing. When you're done, make sure to export via File -> Export As -> SVG and then select a zoom level of 200%. If you simply save the diagram via CTRL+S, it won't fill the whole width of the documentation page. Also make sure that "Include a copy of my diagram" remains ticked to keep the diagram editable.
Our backend collector expects to receive trace information in a normalized and enriched format. This diagram below is relatively close to the data-structures that are actually sent over the network, minus the batching and domain-specific deduplication that we apply prior to sending it out.
The diagram below provides a detailed overview on how the various components of the host agent interact to transform raw traces into the network format. It is focused around our data structures and how data flows through them. Dotted lines represent indirect interaction with data structures, solid ones correspond to code flow. "UM" is short for "user mode".
The host agent code is tested with three test suites:
- Go unit tests
Functionality of individual functions and types is tested with regular Go unit tests. This works great for the user-land portion of the agent, but is unable to test any of the unwinding logic and BPF interaction. - coredump test suite
The coredump test suite (utils/coredump) we compile the whole BPF unwinder code into a user-mode executable, then use the information from a coredump to simulate a realistic environment to test the unwinder code in. The coredump suite essentially implements all required BPF helper functions in user-space, reading memory and thread contexts from the coredump. The resulting traces are then compared to a frame list in a JSON file, serving as regression tests. - BPF integration tests
A special build of the host agent with theintegrationtag is created that enables specialized test cases that actually load BPF tracers into the kernel. These test cases require root privileges and thus cannot be part of the regular unit test suite. The test cases focus on covering the interaction and communication of BPF with user-mode code, as well as testing that our BPF code passes the BPF verifier. Our CI builds the integration test executable once and then executes it on a wide range of different Linux kernel versions via qemu.
Probabilistic profiling allows you to reduce storage costs by collecting a representative sample of profiling data. This method decreases storage costs with a visibility trade-off, as not all Profiling Host Agents will have profile collection enabled at all times.
Profiling Events linearly correlate with the probabilistic profiling value. The lower the value, the fewer events are collected.
To configure probabilistic profiling, set the -probabilistic-threshold and -probabilistic-interval options.
Set the -probabilistic-threshold option to a unsigned integer between 1 and 99 to enable
probabilistic profiling. At every probabilistic interval, a random number between 0 and 99 is chosen.
If the probabilistic threshold that you've set is greater than this random number, the agent collects
profiles from this system for the duration of the interval. The default value is 100.
Set the -probabilistic-interval option to a time duration to define the time interval for which
probabilistic profiling is either enabled or disabled. The default value is 1 minute.
The following example shows how to configure the profiling agent with a threshold of 50 and an interval of 2 minutes and 30 seconds:
sudo ./otel-profiling-agent -probabilistic-threshold=50 -probabilistic-interval=2m30sThis project is licensed under the Apache License 2.0 (Apache-2.0). Apache License 2.0
The eBPF source code is licensed under the GPL 2.0 license. GPL 2.0
To display a summary of the dependencies' licenses:
make legalDetails can be found in the generated deps.profiling-agent.csv file.
At the time of writing this, the summary is
Count License
52 Apache-2.0
17 BSD-3-Clause
17 MIT
3 BSD-2-Clause
1 ISC
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent