nathansrf / juliet-test-suite-c

Juliet Test Suite for C/C++

Introduction

This is the Juliet Test Suite for C/C++ version 1.3 from https://samate.nist.gov/SARD/testsuite.php augmented with a build system for Unix-like OSes that supports automatically building test cases into individual executables and running those tests. The build system originally provided with the test suite supports building all test cases for a particular CWE into a monolithic executable. Building individual test cases supports the evaluation of projects like CHERI that facilitate memory safety for C/C++ programs at runtime.

Testcases are organized by CWE in the testcases subdirectory. juliet.py is the main script that supports building and running individual test cases - individual CWEs or the entire test suite can be targeted. To build executables, juliet.py copies CMakeLists.txt into the directories for targeted CWEs and runs cmake followed by make. Output appears by default in a bin subdirectory. Each targeted CWE has a bin/CWEXXX directory that is further divided into bin/CWEXXX/good and bin/CWEXXX/bad subdirectories. For each test case, a "good" binary that does not contain the error is built and placed into the good subdirectory and a "bad" binary that contains the error is built and placed into the bad subdirectory.

To run executables after they are built, juliet.py invokes the juliet-run.sh script, which is copied to the bin subdirectory during the build. It records exit codes in bin/CWEXXX/good.run and bin/CWEXXX/bad.run. Executables are run with a timeout so that test cases depending on user input timeout with exit code 124.

Note: Juliet C++ test cases that use namespace std and the bind() socket function didn't compile under c++11, which introduces std::bind(). This version of the test suite has replaced bind() calls in C++ source files with calls to ::bind().

Usage

Install the required dependencies for running on a Linux machine, here we assume Ubuntu:

sudo apt-get install build-essential cmake

Clone the repo:

git clone [email protected]:nathansrf/juliet-test-suite-c.git juliet && cd juliet

Run a specific CWE test case:

python3 juliet.py -cgmr 416
python3 parse-cwe-status.py bin/CWE416/bad.run

Run a specific CWE test case compiled with Clang and run the test in QEMU:

 ./juliet.py --c-compiler=clang-15 --cxx-compiler=clang++-15 -cmgrq 121