iovisor / ubpf

Userspace eBPF VM

uBPF

Userspace eBPF VM

About

This project aims to create an Apache-licensed library for executing eBPF programs. The primary implementation of eBPF lives in the Linux kernel, but due to its GPL license it can't be used in many projects.

Linux documentation for the eBPF instruction set

Instruction set reference

API Documentation

This project includes an eBPF assembler, disassembler, interpreter (for all platforms), and JIT compiler (for x86-64 and Arm64 targets).

Checking Out

Before following any of the instructions below for building, testing, contributing, etc, please be sure to properly check out the source code which requires properly initializing submodules:

git submodule update --init --recursive

Preparing system for build

In order to prepare your system to successfully generate the build system using CMake, follow the platform-specific instructions below.

Windows

Building, compiling and testing on Windows requires an installation of Visual Studio (not VS Code -- the MSVC compiler is necessary!).

Note: There are free-to-use versions of Visual Studio for individual developers. These versions are known as the community version.

You can build, compile and test uBPF using VS Code but Visual Studio is still required.

The other requirement is that you have nuget.exe in your PATH. You can determine if your host meets this criteria by testing whether

> nuget.exe

produces output about how to execute the program. With nuget.exe installed, the cmake configuration system will download all the required developer libraries as it configures the build system.

macOS

First, make sure that you have the XCode Command Line Tools installed:

$ xcode-select --install

Installing the XCode Command Linux Tools will install Apple's version of the Clang compiler and other developer-support tools.

uBpf requires that your host have several support libraries installed. The easiest way to configure your host to meet these requirements,

1. Install homebrew
2. Install boost:

$ brew install boost

3. Install LLVM (and related tools):

$ brew install llvm cmake
$ brew install clang-format

Installing LLVM from Homebrew is optional for developing and using uBPF on macOS. It is required if you plan on compiling/creating eBPF programs by compiling LLVM and storing them in ELF files. If you do install LLVM from Homebrew, add -DUBPF_ALTERNATE_LLVM_PATH=/opt/homebrew/opt/llvm/bin to the cmake configuration command:

cmake -S . -B build -DUBPF_ENABLE_TESTS=true -DUBPF_ALTERNATE_LLVM_PATH=/opt/homebrew/opt/llvm/bin

Linux

./scripts/build-libbpf.sh

Building with CMake

A build system for compiling and testing ubpf is generated for Windows, Linux and macOS platforms using cmake:

cmake -S . -B build -DUBPF_ENABLE_TESTS=true
cmake --build build --config Debug

Running the tests

Linux and MacOS

cmake --build build --target test --

Windows

ctest --test-dir build

Contributing

We love contributions!

Preparing Code Contributions

We aim to maintain code coverage with every code change. The CI/CD pipeline will verify this invariant as part of the contribution process. However, you can calculate code coverage locally by

coveralls --gcov-options '\-lp' -i $PWD/vm/ubpf_vm.c -i $PWD/vm/ubpf_jit_x86_64.c -i $PWD/vm/ubpf_loader.c

We also aim to maintain a consistent code format. The pre-commit git hooks configured for the uBPF repository will guarantee that code changes match the format we expect. In order for those hooks to work effectively, you must have clang-format installed and available on your system.

Compiling C to eBPF

You'll need Clang 3.7.

clang-3.7 -O2 -target bpf -c prog.c -o prog.o

You can then pass the contents of prog.o to ubpf_load_elf, or to the stdin of the vm/test binary.

License

Copyright 2015, Big Switch Networks, Inc. Licensed under the Apache License, Version 2.0 <LICENSE.txt or http://www.apache.org/licenses/LICENSE-2.0>.

ubpf's Issues

uBPF ubpf_exec should separate errors returned from the return value of the eBPF program

uBPF ubpf_exec should separate errors returned from the return value of the eBPF program

Currently, ebpf_exec returns both errors from the eBPF program and return values from the eBPF program using the same 64bits. This makes it impossible to differentiate between failures in the eBPF program (divide by zero and out of bounds) from an eBPF program that returns -1.

ELF64_R_TYPE from clang is 10 not 2

Hi guys, thanks for pulling this repo together. Very interesting read. When reviewing the code I noticed that you expect relocation type 2. I think clang produces ELF code with relocation type 10 (R_BPF_64_32)
Thanks
Rolf

uBPF should support "bpf_tail_call" semantics

uBPF should support an extension function that has tail call semantics. If the tail call returns success, the program should unwind immediately, permitting the caller to then invoke the tail program. The proposal is to provide an API to mark an extension index as "unwind on success" semantic.

       long bpf_tail_call(void *ctx, struct bpf_map *prog_array_map, u32
       index)

              Description
                     This special helper is used to trigger a "tail
                     call", or in other words, to jump into another eBPF
                     program. The same stack frame is used (but values
                     on stack and in registers for the caller are not
                     accessible to the callee). This mechanism allows
                     for program chaining, either for raising the
                     maximum number of available eBPF instructions, or
                     to execute given programs in conditional blocks.
                     For security reasons, there is an upper limit to
                     the number of successive tail calls that can be
                     performed.

                     Upon call of this helper, the program attempts to
                     jump into a program referenced at index index in
                     prog_array_map, a special map of type
                     BPF_MAP_TYPE_PROG_ARRAY, and passes ctx, a pointer
                     to the context.

                     If the call succeeds, the kernel immediately runs
                     the first instruction of the new program. This is
                     not a function call, and it never returns to the
                     previous program. If the call fails, then the
                     helper has no effect, and the caller continues to
                     run its subsequent instructions. A call can fail if
                     the destination program for the jump does not exist
                     (i.e. index is superior to the number of entries in
                     prog_array_map), or if the maximum number of tail
                     calls has been reached for this chain of programs.
                     This limit is defined in the kernel by the macro
                     MAX_TAIL_CALL_CNT (not accessible to user space),
                     which is currently set to 32.

              Return 0 on success, or a negative error in case of
                     failure.

CMake without tests fails

E:\ubpf>cmake -S . -B build
-- Building for: Visual Studio 16 2019
-- Selecting Windows SDK version 10.0.22000.0 to target Windows 10.0.22621.
-- The C compiler identification is MSVC 19.29.30145.0
-- The CXX compiler identification is MSVC 19.29.30145.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Warning at vm/CMakeLists.txt:17 (message):
  ubpf - elf.h was not found, disabling ELF support


CMake Error at vm/CMakeLists.txt:89 (target_link_libraries):
  Cannot specify link libraries for target "ubpf_test" which is not built by
  this project.


-- Configuring incomplete, errors occurred!
See also "E:/ubpf/build/CMakeFiles/CMakeOutput.log".

Interest in verifier passes?

I'm interested in adding support to ubpf for verifier passes that are similar in purpose to what the kernel has. My primary question is: are these passes desired? If so, should they be optional, and should they be the default?

Secondarily, what kinds of passes should we start with? I've already developed a pass to detect unreachable instructions and a pass to detect loops which work decently well on simple examples; I'd be happy to send these upstream for review.

Code fails to compile against GCC 9.3.0 (version that ships with Alpine Linux)

bash-5.0# cc --version
cc (Alpine 9.3.0) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

cc -Wall -Werror -Iinc -O2 -g -Wunused-parameter -std=c99 -fPIC   -c -o ubpf_loader.o ubpf_loader.c
ubpf_loader.c: In function 'ubpf_load_elf':
ubpf_loader.c:133:30: error: invalid type argument of '->' (have 'struct section')
  133 |                 (&sections[i]->size > 0)) {

Looks like GCC is confused about order of operators here. Adding a () around it fixes it.

Divide by zero and modulo zero

As noted in the pending eBPF standard ISA documentation to be published by the eBPF Foundation, with draft at https://github.com/dthaler/ebpf-docs/blob/update/isa/kernel.org/instruction-set.rst#arithmetic-instructions:

Underflow and overflow are allowed during arithmetic operations, meaning the 64-bit or 32-bit value will wrap. If eBPF program execution would result in division by zero, the destination register is instead set to zero. If execution would result in module by zero, the destination register is instead set to the source value.

Above is the current behavior on Linux and the desire is that it be the standard, as discussed at Linux Plumbers Conference in the eBPF track this week.

elftools version 0.28 causes tests to fail during CI/CD

https://app.travis-ci.com/github/Alan-Jowett/ubpf/jobs/564087024

======================================================================
ERROR: test_elf.test_datafiles('elf/wrong-version.data',)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alanjo/.local/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/alanjo/ubpf/test_framework/test_elf.py", line 180, in check_datafile
    elf = generate_elf(data['pyelf'])
  File "/home/alanjo/ubpf/test_framework/test_elf.py", line 167, in generate_elf
    return serialize(parts)
  File "/home/alanjo/ubpf/test_framework/test_elf.py", line 157, in serialize
    data = serializer(part)
  File "/home/alanjo/.local/lib/python3.8/site-packages/elftools/construct/core.py", line 204, in build
    self.build_stream(obj, stream)
  File "/home/alanjo/.local/lib/python3.8/site-packages/elftools/construct/core.py", line 211, in build_stream
    self._build(obj, stream, Container())
  File "/home/alanjo/.local/lib/python3.8/site-packages/elftools/construct/core.py", line 666, in _build
    sc._build(subobj, stream, context)
  File "/home/alanjo/.local/lib/python3.8/site-packages/elftools/construct/core.py", line 1074, in _build
    self.subcon._build(obj, stream2, context)
  File "/home/alanjo/.local/lib/python3.8/site-packages/elftools/construct/core.py", line 664, in _build
    subobj = getattr(obj, sc.name)
AttributeError: 'Container' object has no attribute 'local'

----------------------------------------------------------------------

Document how to build uBPF with cmake

Document how to build uBPF with cmake

Add github CI/CD workflow to ubpf

It would make sense to add a CI/CD workflow to this project.

Ubpf doesn't compile or emit code correctly for Windows

Downstream project has adopted parts of ubpf for implementing eBPF on top of Windows. To make this work, we made the following changes:

1. Add option to just translate eBPF code -> x64 machine code and return the machine code (instead of embedding the resulting code in ubpf_vm).
2. Windows platform specific headers (to allow compiling parts of ubpf for Windows user mode or kernel mode).
3. Support either Windows or Linux ABI (different registers/calling conventions).
4. Switch ubpf_exec to return an error code instead of printf.

What is the simplest `prog.c` can be passed to `vm/test`?

Hello,

I'd like to play with this tool, however, I'm not quite sure how to get started with it.

Is there a simple example to get started with? Like a simple prog.c?

Thanks,
Alireza

Commit 089f627 leads to library linking errors due to header file missing <stdbool.h>

In 089f627 we added bounds_checks and introduced the bool type in ubpf.h which requires that <stdbool.h> be added to the include path. Since <stdbool.h> was already used internally this did not impact the compilation of libubpf.a. However this does cause a regression issue when linking to the library from a program that does currently include <stdbool.h>.

For example prior to commit 089f627 the following would compile using gcc:

#include <ubpf.h>                                                                                                                                                                                                                                                               
int main(void) { ubpf_create(); return 0; }

but post this commit this now fails with

/usr/local/include/ubpf.h:36:1: error: unknown type name 'bool'
   36 | bool toggle_bounds_check(struct ubpf_vm *vm, bool enable);
      | ^~~~
/usr/local/include/ubpf.h:36:46: error: unknown type name 'bool'
   36 | bool toggle_bounds_check(struct ubpf_vm *vm, bool enable);
      |                                              ^~~~

I think the correct way to solve this is to add <stdbool.h> to the external facing header file (ubpf.h) and optionally remove this from the internal files. I am happy to do a patch for this if required.

uBPF should support BPF_JMP32 instruction class

The eBPF spec defines the following instruction classes:

Three LSB bits store instruction class which is one of:

  Classic BPF classes:    eBPF classes:

  BPF_LD    0x00          BPF_LD    0x00
  BPF_LDX   0x01          BPF_LDX   0x01
  BPF_ST    0x02          BPF_ST    0x02
  BPF_STX   0x03          BPF_STX   0x03
  BPF_ALU   0x04          BPF_ALU   0x04
  BPF_JMP   0x05          BPF_JMP   0x05
  BPF_RET   0x06          BPF_JMP32 0x06
  BPF_MISC  0x07          BPF_ALU64 0x07

https://www.kernel.org/doc/Documentation/networking/filter.txt

BPF_JMP32 is missing.

Is there support for ARM64 JIT compiler ?

Wondering if there is any plan to support ARM64 JIT for ubpf ?

EBPF_OP_LDDW should be rejected if src != 0

https://github.com/dthaler/ebpf-docs/blob/update/isa/kernel.org/instruction-set.rst#bit-immediate-instructions

Other values for src are possible, but uBPF only supports 0 (immediate).

ubpf on Windows x64 doesn't handle extension functions with 5 parameters

Windows x64 ABI passes the 5th parameter on the stack and also assumes there are 4 registers worth of space as home register space.

To fix this uBPF needs to do the following:

1. Always push R5 to the stack.
2. Allocate space on the stack for R1, R2, R3, and R4 prior to the call.
3. Free space on stack after the call.

Running the test framework?

Is there instructions showing how to run the test framework?

Adopt clang formatting for repo

To make sure coding guidelines are followed, add checkin enforcement of clang formatting for repo.

Register R12 requires special encoding in modrm bytes - Avoid using it to simplify x64 code gen

Register R12 requires special encoding in modrm byte - Avoid using it to simplify x64 code gen.

0: 45 8b 3b mov r15d,DWORD PTR [r11]
0: 45 8b 3c 24 mov r15d,DWORD PTR [r12]

For reason not yet understood, a move operations with R12 as indirect source results in a 4 byte instruction.

Switch the register mapping to avoid R12.

ubpf doesn't handle relocation correctly

ubpf_loader.c doesn't correctly apply relocations. Each "PROGBITS" segment has a related "REL" segment.

The correlation between the two appears to be via naming convention, with the relocation segment having the same name as the code segment, but prefixed with ".rel".

As an example, the linux-source-5.4.0/samples/bpf/xdp1_kern.c sample once built has the following sections:

alan@alanjo-home:~/ubpf/vm$ readelf -S ~/src/linux-source-5.4.0/samples/bpf/xdp1_kern.o
There are 21 section headers, starting at offset 0x2e48:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .strtab           STRTAB           0000000000000000  00002d50
       00000000000000f7  0000000000000000           0     0     1
  [ 2] .text             PROGBITS         0000000000000000  00000040
       0000000000000000  0000000000000000  AX       0     0     4
  [ 3] xdp1              PROGBITS         0000000000000000  00000040
       00000000000001e8  0000000000000000  AX       0     0     8
  [ 4] .relxdp1          REL              0000000000000000  00002330
       0000000000000010  0000000000000010          20     3     8
  [ 5] .maps             PROGBITS         0000000000000000  00000228
       0000000000000020  0000000000000000  WA       0     0     8
  [ 6] license           PROGBITS         0000000000000000  00000248
       0000000000000004  0000000000000000  WA       0     0     1
  [ 7] .debug_str        PROGBITS         0000000000000000  0000024c
       00000000000002f4  0000000000000001  MS       0     0     1
  [ 8] .debug_loc        PROGBITS         0000000000000000  00000540
       0000000000000398  0000000000000000           0     0     1
  [ 9] .debug_abbrev     PROGBITS         0000000000000000  000008d8
       0000000000000199  0000000000000000           0     0     1
  [10] .debug_info       PROGBITS         0000000000000000  00000a71
       00000000000005a9  0000000000000000           0     0     1
  [11] .rel.debug_info   REL              0000000000000000  00002340
       00000000000007c0  0000000000000010          20    10     8
  [12] .BTF              PROGBITS         0000000000000000  0000101a
       0000000000000582  0000000000000000           0     0     1
  [13] .rel.BTF          REL              0000000000000000  00002b00
       0000000000000020  0000000000000010          20    12     8
  [14] .BTF.ext          PROGBITS         0000000000000000  0000159c
       0000000000000240  0000000000000000           0     0     1
  [15] .rel.BTF.ext      REL              0000000000000000  00002b20
       0000000000000210  0000000000000010          20    14     8
  [16] .eh_frame         PROGBITS         0000000000000000  000017e0
       0000000000000030  0000000000000000   A       0     0     8
  [17] .rel.eh_frame     REL              0000000000000000  00002d30
       0000000000000010  0000000000000010          20    16     8
  [18] .debug_line       PROGBITS         0000000000000000  00001810
       00000000000001d8  0000000000000000           0     0     1
  [19] .rel.debug_line   REL              0000000000000000  00002d40
       0000000000000010  0000000000000010          20    18     8
  [20] .symtab           SYMTAB           0000000000000000  000019e8
       0000000000000948  0000000000000018           1    96     8
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  p (processor specific)

"xdp1" is the code segment and ".relxdp1" is the corresponding relocation segment.

[Defense in depth] uBPF interpreter should defend against ROP attacks

The uBPF interpreter can be used by an ROP attack to launch the execution of arbitrary byte code.

Assume attacker can control return pointer and RDI (Linux) or RCX (Windows). They can then trigger the interpreter to run arbitrary pages of data as BPF instructions by setting return address = ubpf_exec and RDI/RCX as pointer to pointer to BPF byte code.

While not a bug in uBPF itself, it does make exploiting other bugs easier.

Proposed mitigation is:
Store a mask in a global variable.
Generate random mask on startup.
Store ubpf_vm::inst as the result of mask XOR pointer.
Load ubpf_vm::inst after mask XOR stored pointer.

As a result, a call to ubpf_exec with RDI/RCX pointing to byte code will fail as the decode of the pointer will result in an invalid memory address. The attacker would need to first read the mask, XOR the pointer, write the resulting value, which raises the bar significantly.

No install target in vm/Makefile

If other programs want to make use of libubpf.a it would be nice to have a make install target in the Makefile to put this .a and the associated ubpf.h in the appropriate places on the system.

uBPF should emit lfence to achieve speculative load hardening

Current generations of Intel CPU's are subject to side channel attacks due to conditional speculative loading being visible via cache line timing. This can lead to reading arbitrary privileged memory.

uBPF should either mask load address or emit an lfence when speculative load hardening is required.

See this link for details about the problem: https://googleprojectzero.blogspot.com/2018/01/reading-privileged-memory-with-side.html

See this link for details about the speculative load hardening: https://releases.llvm.org/8.0.1/docs/SpeculativeLoadHardening.html

CI/CD for aarch64

I noticed the current Travis CI only runs for x86_64. What would be necessary for it to run for 64-bit Arm as well?

Thanks

Failed to load code: bad relocation type 1

I compiled llvm + clang 3.9.1, and I compiled my sample like this...

~/local/bin/clang -target bpf -c src/benchmark/factor.c -o factor.o

and I get this error...

Failed to load code: bad relocation type 1

What does this mean, and how do I fix this?

JIT stack allocation

Hi everyone,

I've been doing few things with ubpf and ran into an issue yesterday with how the JIT load value to the stack. I'm not sure if it's something I'm doing that's wrong or a bug. I managed to modify the code to get it working locally but if it's a bug it would be nice to fix it :) and if it's something I'm doing wrong, it would be nice to know why!

First thing is the C code I'm using (compiled to bpf with clang). I modified the loader to do the map relocation from the elf and also added both map functions to the vm.

int bpf_prog1(void *pkt)
{
        long index = 1;
        long insertvalue = 0xdeadbeefdeadbeef;
        long value;
        int ret;

        bpf_map_update_elem(&my_map, &index, &insertvalue, 0);

        value = (long)bpf_map_lookup_elem(&my_map, &index);

        return (int)value;
}

The issue was that once calling the bpf_map_update_elem the two pointers index and insertvalue were clearly valid pointers but not containing the values. So therefore what I was interested in was:

lddw r1, 0x1
stxdw [r10-8], r1
lddw r1, 0xdeadbeefdeadbeef
stxdw [r10-16], r1

Going through the jit compiled code with a debugger I noticed that the two values were allocated outside the stack and therefore calling the bpf_map_update_elem (call rax) was overriding the stack and I was losing the two values.

My fix is to modify ubpf_jit_x86_64.c at line 95 and to copy the stack pointer to R10 before the stack space is allocated and the pointer incremented by the STACK_SIZE.

    /* Copy stack pointer to R10 */
    emit_mov(state, RSP, map_register(10));

    /* Allocate stack space */
    emit_alu64_imm32(state, 0x81, 5, RSP, STACK_SIZE);

Best regards,
Simon

uBPF Windows build should create a nuget package that can be published

uBPF Windows build should create a nuget package that can be published.

It would be easier for Windows developers to consume uBPF if there was a nuget package for it.

Pass the mem_len argument into the uBPF VM so it can be used by programs.

Currently the ubpf_exec function takes both a pointer to a memory structure and the length of that structure as inputs. However only the pointer is passed into the VM itself (via the R1 register). The length is used for range checking in the verifier step but not for anything else.

It would be useful when running programs to have the length of the input memory structure available. I propose that we add that via the R2 register. The eBPF spec allows R1-R5 to be used for such purposes.

Longer term a more generic approach to populating these registers would be nice. But we can start with mem_len and go from there.

I'll try and work up something for this.

uBPF jitter is should have option to apply constant blinding

uBPF jitter is should have option to apply constant blinding

Code generated by the uBPF jitter is susceptible to JIT spray attacks. See: https://www.usenix.org/sites/default/files/conference/protected-files/woot18_slides_gawlik.pdf

For a good description of the attack.

Proposed fix:
https://samsung.github.io/kspp-study/bpf.html#hardening-hostile-code-in-ebpf

uBPF should emit format string for log functions as literal in x64 machine code

ubpf/vm/ubpf_jit_x86_64.c

Line 495 in 9ee9e33

const char *div_by_zero_fmt = "uBPF error: division by zero at PC %u\n";

Address emitted by this code points to uBPF code page, not the jitted code page. The result is code that fails when relocated to a different address space.

ubpf needs to support maps

Adding support for maps to ubpf.

As near as I can tell from examining dumps of ELF eBPF programs it looks like:

1. All maps are in a section labelled ".maps"
2. Maps layout is sequentially.
3. Symbol table entries contain segment index, offset and size for maps.

Proposed behavior is then:

1. Look for a section name ".maps" that should be marked as "PROGBITS" and have the "write" and "allocate" flags set.
2. When processing section name "XYZ", find the corresponding relocation section named ".relXYZ" and process it's relocations.
3. For each relocation, check if it refers to an entry in the .maps section and apply the relocation to point to that location.

Maps appear to have relocation type R_BPF_64_64 / 1

uBPF should provide an option to emit assembly in place of machine code

uBPF should provide an option to emit assembly in place of machine code

Use case:
Generate an ELF or COFF formatted object file from an eBPF program, with relocations for helper functions and maps.

Proposed process:

1. uBPF accept ELF eBPF program, emits assembly
2. as/ml64 accepts assembly and generates either ELF or COFF file that can be linked into another executable.

Release initial version

I'd like to start depending on ubpf for my BPFnative library. I can make up arbitrary versions for ubpf, but I would prefer not to. It would be great if a version 0.1 (or 0.0.1) could be released, and then we follow Semver or similar and cut regular releases.

Clang/LLVM as code generator for uBPF

Clang/LLVM as code generator for uBPF

Overview

Clang/LLVM has a much richer tool chain for generating optimal ISA-specific instruction sequences. Proposal is to use LLVM to convert eBPF byte code to optimal native instruction sequences.

Background

Clang/LLVM has a language neutral internal representation (IR). Various front-ends produce IR that is then handed off to back-ends to produce optimal and secure ISA-specific machine code. LLVM code generation is platform aware and can take advantage of advanced processor features like vectored / SIMD instructions (via auto-vectorization) as well as having support for speculative load hardening and other security features.

Proposal

1. Write a Clang/LLVM front-end that translates BPF byte code to LLVM IR.
2. Detect CPU features at runtime and configure LLVM.
3. Use LLVM back-end to generate optimal native instruction sequence.
4. Optionally provide LLVM IR versions of map helper functions (to permit
   inlining of helper functions).

Expected benefits

1. LLVM already supports x86, ARM and ARM64, so uBPF will be able
   to JIT correctly on all these platforms.
2. LLVM can optimize the generated machine code as it has the entire control
   flow graph as IR.
3. LLVM mitigations for branch prediction side-channel attacks will be applied
   to the generated code.
4. As the BPF ISA grows (adding support of atomic instructions etc), the Clang/LLVM gets updated and the uBPF can benefit from it.

Challenges

1. It looks like LLVM doesn't currently support constant blinding (to mitigate JIT spray attacks).
2. Clang/LLVM is fairly large dependency, making it less than ideal for uBPF's goal of being a small light weight jitter/interpreter.

uBPF should support the standard 512 byte stack size

ubpf/vm/ubpf_int.h

Line 24 in 6492b7a

#define STACK_SIZE 128

uBPF bounds_check doesn't account for access to packet data

The bounds_check function doesn't correctly account for programs that access packet data.

As an example, xdp_md_t is defined as:

// XDP hook.  We use "struct xdp_md" for cross-platform compatibility.
typedef struct xdp_md
{
    void* data;         /*     0     8 */
    void* data_end;     /*     8     8 */
    uint64_t data_meta; /*     16    8 */

    /* size: 12, cachelines: 1, members: 3 */
    /* last cacheline: 12 bytes */
} xdp_md_t;

Programs are legal permitted to access memory pointed to by context->data up to context->data_end.

But the bounds_check only checks for:

1. Access is within range of [context, context+sizeof(context)]
2. Access is within range [stack, stack+stack_size]

uBPF should store byte code in read-only memory

uBPF should store byte code in read-only memory.

uBPF should use mmap / mprotect to store byte code in read-only pages.

Error in compiling

Hi guys,
I am very happy to find your Repo here and tried to install the uBPF on Raspberry Pi 3 with Ubuntu Xenial today.
After installing the requirements and using make -C vm I get this Error:

ubpf_vm.c: In function ‘ubpf_exec’:
ubpf_vm.c:369:27: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
if (!bounds_check((void *)reg[inst.src] + inst.offset, size, "load", cur_pc, mem, mem_len, stack)) {
^
ubpf_vm.c:381:13: note: in expansion of macro ‘BOUNDS_CHECK_LOAD’
BOUNDS_CHECK_LOAD(4);
^
ubpf_vm.c:369:27: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
if (!bounds_check((void *)reg[inst.src] + inst.offset, size, "load", cur_pc, mem, mem_len, stack)) {
^
ubpf_vm.c:385:13: note: in expansion of macro ‘BOUNDS_CHECK_LOAD’
BOUNDS_CHECK_LOAD(2);
^
ubpf_vm.c:369:27: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
if (!bounds_check((void *)reg[inst.src] + inst.offset, size, "load", cur_pc, mem, mem_len, stack)) {
^
ubpf_vm.c:389:13: note: in expansion of macro ‘BOUNDS_CHECK_LOAD’
BOUNDS_CHECK_LOAD(1);
^
ubpf_vm.c:369:27: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
if (!bounds_check((void *)reg[inst.src] + inst.offset, size, "load", cur_pc, mem, mem_len, stack)) {
^
ubpf_vm.c:393:13: note: in expansion of macro ‘BOUNDS_CHECK_LOAD’
BOUNDS_CHECK_LOAD(8);
^
ubpf_vm.c:375:27: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
if (!bounds_check((void *)reg[inst.dst] + inst.offset, size, "store", cur_pc, mem, mem_len, stack)) {
^
ubpf_vm.c:398:13: note: in expansion of macro ‘BOUNDS_CHECK_STORE’
BOUNDS_CHECK_STORE(4);
^
ubpf_vm.c:375:27: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
if (!bounds_check((void *)reg[inst.dst] + inst.offset, size, "store", cur_pc, mem, mem_len, stack)) {
^
ubpf_vm.c:402:13: note: in expansion of macro ‘BOUNDS_CHECK_STORE’
BOUNDS_CHECK_STORE(2);
^
ubpf_vm.c:375:27: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
if (!bounds_check((void *)reg[inst.dst] + inst.offset, size, "store", cur_pc, mem, mem_len, stack)) {
^
ubpf_vm.c:406:13: note: in expansion of macro ‘BOUNDS_CHECK_STORE’
BOUNDS_CHECK_STORE(1);
^
ubpf_vm.c:375:27: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
if (!bounds_check((void *)reg[inst.dst] + inst.offset, size, "store", cur_pc, mem, mem_len, stack)) {
^
ubpf_vm.c:410:13: note: in expansion of macro ‘BOUNDS_CHECK_STORE’
BOUNDS_CHECK_STORE(8);
^
ubpf_vm.c:375:27: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
if (!bounds_check((void *)reg[inst.dst] + inst.offset, size, "store", cur_pc, mem, mem_len, stack)) {
^
ubpf_vm.c:415:13: note: in expansion of macro ‘BOUNDS_CHECK_STORE’
BOUNDS_CHECK_STORE(4);
^
ubpf_vm.c:375:27: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
if (!bounds_check((void *)reg[inst.dst] + inst.offset, size, "store", cur_pc, mem, mem_len, stack)) {
^
ubpf_vm.c:419:13: note: in expansion of macro ‘BOUNDS_CHECK_STORE’
BOUNDS_CHECK_STORE(2);
^
ubpf_vm.c:375:27: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
if (!bounds_check((void *)reg[inst.dst] + inst.offset, size, "store", cur_pc, mem, mem_len, stack)) {
^
ubpf_vm.c:423:13: note: in expansion of macro ‘BOUNDS_CHECK_STORE’
BOUNDS_CHECK_STORE(1);
^
ubpf_vm.c:375:27: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
if (!bounds_check((void *)reg[inst.dst] + inst.offset, size, "store", cur_pc, mem, mem_len, stack)) {
^
ubpf_vm.c:427:13: note: in expansion of macro ‘BOUNDS_CHECK_STORE’
BOUNDS_CHECK_STORE(8);
^

After a closer look into the Code I tried to put an & before reg like this in line 369 and 375:

if (!bounds_check((void *)&reg[inst.src] + inst.offset, size, "load", cur_pc, mem, mem_len, stack)) { \
if (!bounds_check((void *)&reg[inst.dst] + inst.offset, size, "store", cur_pc, mem, mem_len, stack)) { \

I think it gets me a step further but I have the next Error:

cc -Wall -Werror -Iinc -O2 -g -c -o ubpf_vm.o ubpf_vm.c
cc -Wall -Werror -Iinc -O2 -g -c -o ubpf_jit_x86_64.o ubpf_jit_x86_64.c
cc -Wall -Werror -Iinc -O2 -g -c -o ubpf_loader.o ubpf_loader.c
ar rc libubpf.a ubpf_vm.o ubpf_jit_x86_64.o ubpf_loader.o
cc -Wall -Werror -Iinc -O2 -g -c -o test.o test.c
/tmp/ccdsP4py.s: Assembler messages:
/tmp/ccdsP4py.s:82: Error: ARM register expected -- mov $0xf0,%rax' /tmp/ccdsP4py.s:82: Error: ARM register expected -- mov $0xf1,%rcx'
/tmp/ccdsP4py.s:82: Error: ARM register expected -- mov $0xf2,%rdx' /tmp/ccdsP4py.s:82: Error: ARM register expected -- mov $0xf3,%rsi'
/tmp/ccdsP4py.s:82: Error: ARM register expected -- mov $0xf4,%rdi' /tmp/ccdsP4py.s:82: Error: ARM register expected -- mov $0xf5,%r8'
/tmp/ccdsP4py.s:82: Error: ARM register expected -- mov $0xf6,%r9' /tmp/ccdsP4py.s:82: Error: ARM register expected -- mov $0xf7,%r10'
/tmp/ccdsP4py.s:82: Error: ARM register expected -- `mov $0xf8,%r11'
: recipe for target 'test.o' failed
make: *** [test.o] Error 1

I am very new to Linux and tried to change from GCC to Clang but get the same result.
Do you have a way to fix it?
Thanks
Pascal

uBPF should support > 64 helper_id

ubpf/vm/ubpf_vm.c

Line 27 in 6492b7a

#define MAX_EXT_FUNCS 64

The ebpf-for-windows uses a disjointed helper-id space, allowing for both global helper functions and program type specific helper functions, with the following mapping:
0x1-0xFFFF - Global helper functions usable by any program type
0x1000 - 0x1FFF - Program type specific helper functions

ubpf fails to build against Alpine / MUSL C runtime

Looks like the library uses non-POSIX API's like htole*.

Going to create a pull request that adds local versions if the C runtime doesn't have it.

bash-5.0# make
cc -Wall -Werror -Iinc -O2 -g -Wunused-parameter   -c -o ubpf_vm.o ubpf_vm.c
ubpf_vm.c: In function 'ubpf_exec':
ubpf_vm.c:268:33: error: implicit declaration of function 'htole16' [-Werror=implicit-function-declaration]
  268 |                 reg[inst.dst] = htole16(reg[inst.dst]);
      |                                 ^~~~~~~
ubpf_vm.c:270:33: error: implicit declaration of function 'htole32' [-Werror=implicit-function-declaration]
  270 |                 reg[inst.dst] = htole32(reg[inst.dst]);
      |                                 ^~~~~~~
ubpf_vm.c:272:33: error: implicit declaration of function 'htole64' [-Werror=implicit-function-declaration]
  272 |                 reg[inst.dst] = htole64(reg[inst.dst]);
      |                                 ^~~~~~~
ubpf_vm.c:277:33: error: implicit declaration of function 'htobe16' [-Werror=implicit-function-declaration]
  277 |                 reg[inst.dst] = htobe16(reg[inst.dst]);
      |                                 ^~~~~~~
ubpf_vm.c:279:33: error: implicit declaration of function 'htobe32' [-Werror=implicit-function-declaration]
  279 |                 reg[inst.dst] = htobe32(reg[inst.dst]);
      |                                 ^~~~~~~
ubpf_vm.c:281:33: error: implicit declaration of function 'htobe64' [-Werror=implicit-function-declaration]
  281 |                 reg[inst.dst] = htobe64(reg[inst.dst]);
      |                                 ^~~~~~~
cc1: all warnings being treated as errors
make: *** [<builtin>: ubpf_vm.o] Error 1

Header files should contain SPDX-License-Identifier

As part of making it easier to track licensing, add SPDX-License-Identifier: to all files and add rules to verify it's present in the repo.

uBPF CI/CD should run tests and capture code coverage

uBPF CI/CD should run tests and capture code coverage

Maximum ebpf instruction count

Linux apparently has a limit of INT_MAX instructions since 5.2.
Earlier versions had a limit of 4096 instructions.

The ubpf jitter is currently capped at 64k of machine code.
The ubpf interpreter is currently capped at UINT32_MAX of bytes (i.e., UINT32_MAX / 8 instructions).
These are higher than the old Linux limit of 4096, but lower than the newer limit of INT_MAX instructions.

uBPF should provide capabilities to implement a debugger for eBPF programs

uBPF should provide capabilities to implement a debugger for eBPF programs.

eBPF programs are hard to debug. uBPF should add the ability to step through an eBPF program, instruction by instruction.

To facilitate the proposal is to add a synchronous callout before each instruction is executed that permits capturing the state of the VM (registers, PC, and stack).

Support for 32-bit jump instructions

eBPF in Linux kernel just got extended to use the 0x06 class of instructions for 32-bit jumps. See the merge commit and its parents. This should land in kernel 5.1.

To keep uBPF compatibility with Linux eBPF as close as possible, it would be nice to have support for those instructions for the interpreter as well as for the JIT in the future.

Add instruction counter capability to test.c

When benchmarking eBPF programs it is useful to have the test program output the number of eBPF instructions executed as well as the total number of instructions needed to run the VM (or the JIT when appropriate).

I'll take a stab at this and do a PR for it if I get something working that I like.

uBPF uses 128 byte stack by default - Should this match the common consensus of 512 bytes

uBPF uses 128 byte stack by default - Should this match the common consensus of 512 bytes?

Other projects (Linux, generic-epbf and others) all agree on a 512 byte stack size. Anyone know why uBPF has a default of 512?

ubpf won't operate in a Trusted Execution Environment (TEE) like OpenEnclave.

Problem:
ubpf_compile relies on mmap and mprotect that are not supported in a TEE.

Proposal:
Define new API: ubfp_translate that only does the translate/resolve and is called by ubpf_compile.
ubpf_translate would have no dependency on mmap or mprotect and would be usable in a trusted execution environment.
ubpf_translate would be passed a target buffer that it would translate into.

Justification:
Desire to move eBPF verifier / JIT code out if kernel into a TEE to improve stability of kernel and provide better isolation.

uBPF should support support the new atomic instructions

uBPF should support support the new atomic instructions

eBPF ABI was extended to add support for atomic operations:
https://lwn.net/Articles/842354/

This is important as it allow eBPF programs to update maps in a lock-free manner.