netflix / bpftop

bpftop provides a dynamic real-time view of running eBPF programs. It displays the average runtime, events per second, and estimated total CPU % for each program.

bpftop

bpftop provides a dynamic real-time view of running eBPF programs. It displays the average runtime, events per second, and estimated total CPU % for each program. It also provides graphical views of these statistics over time. This tool minimizes overhead by enabling performance statistics only while it is active.

Installation

To download the latest x86_64 release of bpftop, use the following command:

curl -fLJ https://github.com/Netflix/bpftop/releases/latest/download/bpftop-x86_64-unknown-linux-gnu -o bpftop && chmod +x bpftop

or install via your distribution's package manager:

Fedora

You can install bpftop from the official repositories using dnf:

sudo dnf install bpftop

Arch Linux

You can install bpftop from the official repositories using pacman:

sudo pacman -S bpftop

Nix

You can install bpftop from the NixOS 24.05 stable channel:

nix-channel --add https://nixos.org/channels/nixos-24.05 nixpkgs
nix-channel --update
nix-env -iA nixpkgs.bpftop

Features

• Displays a list of all running eBPF programs on the host, including the ID, type, and name
• Shows the period and total average runtime for each eBPF program.
• Calculates the events per second and estimated CPU utilization for each eBPF program
• Provides a graphical view of the average runtime, events per second, and estimated CPU utilization over a 10-second time period
• Dynamically updates the list every second
• Enables the statistics-gathering function only while it is active

Prerequisites

• bpftop requires sudo privileges to run.
• The binary is dynamically linked to libz and libelf, so these libraries must be present on the systems where you intend to run bpftop.

Usage

Run the following command to start bpftop on your host:

sudo ./bpftop

Relate links

• Announcement blog post
• LWN.net
• The New Stack

How it works

bpftop uses the BPF_ENABLE_STATS BPF syscall command to enable global eBPF runtime statistics gathering, which is disabled by default to reduce performance overhead. It collects these statistics every second, calculating the average runtime, events per second, and estimated CPU utilization for each eBPF program within that sample period. This information is displayed in a top-like tabular format. Once bpftop terminates, it disables the statistics-gathering function by deleting the file descriptor returned by BPF_ENABLE_STATS.

Building from source

1. Install and setup cross
2. Run cross build --release for x86_64
3. Run cross build --target=aarch64-unknown-linux-gnu --release for Arm64

bpftop's Issues

XDP programs are accounted to xdp-dispatcher

Thanks for bpftop! I gave it a try and saw that XDP programs - which are managed by something called xdp-dispatcher - are not accounted indiviually, but rather to the xdp-dispatcher.
Is there a way to distinguish BPF statistics between individual XDP programs, or is this something that would have to be fixed in XDP?

Add Arm64 as target build platform

It would be really nice if bpftop were to build native Arm64 binaries in addition to x86. All of the eBPF tooling being measured and instrumented is available on Arm64, and as a Rust program, there should be no porting issues.

Some components in graph view do not use foreground color

The "Program Information" panel in the graph view does not use the terminal's foreground color. Also, the left axis of the graphs. Besides that, they appear to be fine with bright background.

The image below should indicate the problematic components.

Should leave alt screen in panic hook

bpftop/src/main.rs

Line 155 in e34c088

if let Err(err) = disable_raw_mode() {

Above this somewhere you should execute LeaveAlternateScreen to restore the buffer. Alternatively, define a struct that implements Drop to do that for you so you can bind it to a variable right after EnterAlternateScreen (or have it do both).

It’s a poor UX when TUIs don’t restore the user’s screen buffer, restore colors, etc.

Publish Debian package

Publish bpftop as a Debian package. This is an open item for anyone in the community with experience packaging Rust applications as Debian packges.

How to understand the cpu_time_percent(or 'Total CPU %') for multi-core CPUs?

Add scrollbar to tabular view

In the tabular view, when the number of BPF programs exceeds the terminal window's capacity, it’s not immediately clear that more items are available. To enhance usability, a scrollbar widget from Ratatui should be integrated to indicate the presence of additional items.

Aggregate statistics for all programs

It may be helpful to see the aggregate view of all performance statistics. It would let you answer questions like "How much CPU overhead are all eBPF programs adding to this host?".

Occasional freeze under load

Occasionally, bpftop freezes up when the system is under load. It can only be terminated by killing it.
Tested was an xdp program loaded to two 10G interfaces, with traffic of about 5 Gbit/s. Final load displayed by bpftop was 42%.

core.bpftop-freeze.gz

From the core file pkill -QUIT bpftop :

#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000559a818929b4 in std::sys::unix::futex::futex_wait ()
    at library/std/src/sys/unix/futex.rs:62
#2  std::sys::unix::locks::futex_mutex::Mutex::lock_contended ()
    at library/std/src/sys/unix/locks/futex_mutex.rs:56
#3  0x0000559a818a5cf2 in bpftop::app::App::selected_program ()
#4  0x0000559a81899036 in bpftop::render_graphs ()
#5  0x0000559a81896321 in bpftop::ui ()
#6  0x0000559a8189ec35 in ratatui::terminal::terminal::Terminal<B>::draw ()
#7  0x0000559a81895f76 in bpftop::main ()
#8  0x0000559a818a4133 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#9  0x0000559a818a186d in _ZN3std2rt10lang_start28_$u7b$$u7b$closure$u7d$$u7d$17h1d1f4f019d401e61E.llvm.4513734777494074926 ()
#10 0x0000559a818f3cf1 in core::ops::function::impls::{impl#2}::call_once<(), (dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)> () at library/core/src/ops/function.rs:284
#11 std::panicking::try::do_call<&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe), i32> ()
    at library/std/src/panicking.rs:552
#12 std::panicking::try<i32, &(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)> ()
    at library/std/src/panicking.rs:516
#13 std::panic::catch_unwind<&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe), i32> ()
    at library/std/src/panic.rs:142
#14 std::rt::lang_start_internal::{closure#2} () at library/std/src/rt.rs:148
#15 std::panicking::try::do_call<std::rt::lang_start_internal::{closure_env#2}, isize> () at library/std/src/panicking.rs:552
#16 std::panicking::try<isize, std::rt::lang_start_internal::{closure_env#2}>
    () at library/std/src/panicking.rs:516
#17 std::panic::catch_unwind<std::rt::lang_start_internal::{closure_env#2}, isize> () at library/std/src/panic.rs:142
#18 std::rt::lang_start_internal () at library/std/src/rt.rs:148
#19 0x0000559a8189b315 in main ()

Add option to unload ebpf programs

Can we add an arm build in release page?

Publish on crates.io

Please publish this program on crates.io
It would make installing and keeping the program up-to-date much easier for single-machine-deployments.

Panic when arrow up/down with no eBPF programs loaded

Grabbed the latest release per Installation instructions. Launched it when I didn't have any of my eBPF programs loaded and hit down arrow and it panicked:

thread 'main' panicked at src/app.rs:155:34:
                                            index out of bounds: the len is 0 but the index is 0
                                                                                                note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

libc version incompatibility

Hi, seems like a very useful tool, excited to try it out. Unfortunately I am getting the same libc error both when directly running latest executable or compiling from the source with cross.

• With latest executable
  curl -fLJ https://github.com/Netflix/bpftop/releases/latest/download/bpftop -o bpftop && chmod +x bpftop

sudo ./bpftop
./bpftop: /lib64/libc.so.6: version `GLIBC_2.29' not found (required by ./bpftop)
./bpftop: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by ./bpftop)
./bpftop: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by ./bpftop)
./bpftop: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./bpftop)
./bpftop: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./bpftop)

• With source build, I've downloaded cross tool with
  wget https://github.com/cross-rs/cross/releases/download/v0.2.5/cross-x86_64-unknown-linux-gnu.tar.gz

$ ./cross build --release
./cross: /lib64/libc.so.6: version `GLIBC_2.29' not found (required by ./cross)
./cross: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by ./cross)
./cross: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by ./cross)
./cross: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by ./cross)
./cross: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./cross)
./cross: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./cross)

• My kernel version is above 5.8

$ uname -r
5.10.192-183.736.amzn2.x86_64

• Os release

cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Not familiar with the rust ecosystem, I'd appreciate any help. Thanks.

Dependency on systemd

Running the program on a Linux distro without systemd gives this error:

$ sudo bpftop 
Error: No such file or directory (os error 2)

Tracing it with strace revals that I tries to access a socket located at /run/systemd/journal/socket, but it fails and the program shuts down:

$ sudo strace bpftop 2>&1 | tail -n 14
readlink("/proc/self/exe", "/usr/bin/bpftop", 256) = 15
sendto(3, "", 0, MSG_NOSIGNAL, {sa_family=AF_UNIX, sun_path="/run/systemd/journal/socket"}, 30) = -1 ENOENT (No such file or directory)
close(3)                                = 0
write(2, "Error: ", 7Error: )                  = 7
write(2, "No such file or directory", 25No such file or directory) = 25
write(2, " (os error ", 11 (os error )             = 11
write(2, "2", 12)                        = 1
write(2, ")", 1))                        = 1
write(2, "\n", 1
)                       = 1
sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=8192}, NULL) = 0
munmap(0x704292d3c000, 12288)           = 0
exit_group(1)                           = ?
+++ exited with 1 +++

Does this tool have a hard-dependency on systemd and its libraries, or it can still work without them?

Sortable columns

bpftop should let you sort the entries by column, in ascending or descending order. This would require us to track the selected program by its ID rather than its index in the items vec.

Search filter for BPF programs in tabular view

The app should incorporate functionality that enables users to filter BPF programs in the tabular view based on a search string. This filter should apply to the 'name' and 'type' of the BPF programs.

Publish RPM package to Fedora

I've added RPM packaging for bpftop and I'm currently going through the process to get it published in Fedora: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2281565

The crates that bpftop depends on must also be published as Fedora packages, and I'm currently working on the missing 2. The current blocker is finding a Fedora package sponsor.