marcfrei / netsec-lightning-filter

LightningFilter (Open Source)

LightningFilter is a high-speed traffic filtering mechanism that performs authentication, rate limiting, and duplicate detection. LightningFilter uses the DPDK framework, enabling high-speed packet processing.

This repository contains the open-source version, which offers at least the core functionalities of a LightningFilter. The closed-source version provides additional functionalities.

License

The software is licensed under BSD-3.

The license might change to Apache v2 if possible.

3rd Party

DPDK (BSD 3) https://www.dpdk.org/

json-parser (BSD 2) https://github.com/json-parser/json-parser

murmurhash (public domain) https://github.com/aappleby/smhasher/blob/master/src/MurmurHash3.cpp

hashdict (MIT licensed) https://github.com/exebook/hashdict.c

Installation

See docs/Installation.md for detailed information.

There are mainly two possibilities to build LightingFilter: with the help of a docker container or natively on a machine. Both of the options rely on Ubuntu 22.04.

Native

To install dependencies, we use the script provided in usertools and then set the required environment variables:

./usertools/install_deps.sh
source dependencies/env_vars

To build LightningFilter we use CMake.

mkdir build
cd build
cmake ../
make

Docker Container

For the docker container build, install docker and add the user to the docker group. Then run the docker.sh script to create the docker image and container that builds LightningFilter.

./docker.sh build <CMAKE_ARGS>

Note: Because CMake flags are cached, once set, any following build call uses them.

Run

After compiling the application, the executable is in build/src/ and can be run as follows:

build/src/lf <EAL Parameters> -- <LF Parameters>

LightningFilter expects various parameters, which are divided into EAL and LF parameters. Script examples that run LightningFilter can be found in the test directory, e.g., in test/perfnet_ip.

DPDK defines the EAL parameters, which are described here. The application's help text describes the LF parameters.

E.g.:

build/src/lf --lcores (0-2)@(0-2),(3-7)@(3,7) --log-level lf:debug \
-- \
-p 0x1 --portmap "(0,0,o),(0,0,i)" --c lf_config.json

More info: docs/Parameters.md

Statistics

The DPDK telemetry API provides information and statistics of a running DPDK instance. Also, LightningFilter collects additional statistics and exposes them to the same interface.

Launch interactive client script:

sudo ./usertools/lf-telemetry.py

When using a file prefix, the file prefix is set with the -f flag:

sudo ./usertools/lf-telemetry.py -f "file_prefix"

More info: docs/Metrics.md

Runtime Interface

LightningFilter provides an interface through a Unix socket during runtime, just as for the statistics.

Launch interactive client script:

sudo ./usertools/lf-ipc.py

When running LightningFilter with a file prefix, set the file prefix as follows:

sudo ./usertools/lf-ipc.py -f "file_prefix"

The script also allows running single commands without starting the interactive mode:

sudo ./usertools/lf-ipc.py --cmd=<command> {--params=<parameters>}

Develop

To get quickly started with developing LightningFilter, we provide a Development Container setup (.devcontainer/devcontainer.json) with all required dependencies and some useful tools. When using VS Code, just install the Visual Studio Code Dev Containers extension and open the project in a container (>Dev Container: Reopen in Container).

Alternatively, the developer container can also be created with the docker.sh script:

./docker.sh dev_image && ./docker.sh dev_up

Tests

Unit Tests

(in the build directory)

make run_tests

The tests require additional packages:

sudo apt-get install bsdmainutils tmux

Integration Tests

SCION

(in test/testnet_scion)

sudo ./integration_test.sh ../../build/src/lf ~/scion

Requires an appropriate build (see README in directory).

IP

(in test/testnet_ip)

sudo ./integration_test.sh ../../build/src/lf

Requires an appropriate build (see README in directory).

Test Script

To run all of the unit and integration tests with different settings (compilation configurations), run the script tests.sh.

Performance Tests

For the performance tests, use the test configurations in the directories test/perfnet_scion and test/perfnet_ip. The README.md, in the corresponding directories, provides additional information on the setup and required adjustments.

Documentation

The docs directory contains a collection of documentation files. The following list provides an overview of them.

• Installation Description of the installation process and provider scripts.

  • DPDK Installation of DPDK (required by LightningFilter).
  • SCION Installation of a SCION setup (especially used for testing).

• Functionality Description of the core functionalities

  • Parameters Documentation on the application parameters.
  • Configuration Documentation of configuration file.
  • Metrics
    • Monitoring Monitoring setup with Grafana
  • IPC Inter process communication interface
  • Features Additional features (Jumbo Frames, Checksum Offloading)
  • Plugins Short description of the plugin system. -Control Traffic Overview of control traffic processing

• Implementation

  • Key Manager Overview of the keymanager module.
  • Ratelimiter Overview of the ratelimiter module.
  • Multi Core Multi core approach and distributor
  • Code Quality Code style and format, as well as linter
  • Optimizations Optimization approaches and effects

• Specification

  • SCION LF
  • IP LF

• Troubleshooting

  • Debug Debugging setup example (VSCode)
  • Profiling Profiling setup example (Perf, VTune)
  • Troubleshooting Common problems and solutions

Usertools

The directory usertools contains tools and scripts that allow the user to interact more easily with the LightningFilter.