smr-llc / bitray

gigantic, file-backed, thread-safe, bit-resolution arrays

Overview

This is a prototype, work-in-progress library for gigantic, file-backed, thread-safe, bit-resolution arrays. The goal of this library is to provide useful abstractions and reasonable performance for tools that want to process arbitrarily large blobs of data, like hobbits.

Building

Requirements:

• CMake 3.16+
• A C++ compiler that supports C++17 (e.g. GCC 9+)
• (optional) Ninja

Clone and build the project:

git clone --recurse-submodules https://github.com/smr-llc/bitray.git
cd bitray
cmake . -Bbuild -GNinja -DCMAKE_BUILD_TYPE=Release -DBITRAY_BUILD_TESTS=ON
cmake --build build

Run a benchmark:

./build/tests/benchmarks/bm_byteray

Building the Documentation

Building the documentation requires:

• Python
• Doxygen
• Sphinx, Breathe, and sphinx_rtd_theme

Some of the requirements can be installed with pip:

pip install sphinx sphinx_rtd_theme breathe

Build the docs:

cmake . -Bbuild -GNinja -DBITRAY_BUILD_DOCS=ON
cmake --build build

bitray's Issues

Add read and write benchmarks

Simple benchmarks for reading and writing single bytes and various sizes of contiguous bytes should be added.

Add BitRay class

Make a bit-resolution wrapper around ByteRay.

I think that there probably needs to be a (read-only?) BitFrame class that can be used for speedily working with char* chunks.

Node.js bindings

Create Node.js bindings and deploy a package to the npm index.

This might need the API to be a little bit more mature.

Implement memory-only ByteRays

ByteRays that are small enough to easily fit into memory should not bother creating temporary files. This should make smaller ByteRays much closer to std::vectors in general performance.

Basic read/write mutli-threading tests

Tests need to be added to validate thread safety for reading and writing to a single ByteRay across multiple threads. The first set of tests can be fairly simple as long as they establish a good way to extend with additional tests.

Add Sphinx API docs and a nice way to build and publish the docs

Add a benchmark result display

The benchmarks will be producing a lot of data, so it will be good to display them in a useful way with some graphs that show performance trends over time as new optimizations and features are added.

The minimal starting point might just be a script that runs the benchmarks and produces images or data that can easily be dropped into a graphical tool for evaluation. Eventually it might be nice to have a web application that can organize the results from different sources (CI jobs, different developer machines) and present them nicely.

The output of google-benchmark's compare.py can probably be used:
https://github.com/google/benchmark/blob/main/docs/tools.md

Python bindings

Create Python bindings and deploy a package to PyPI.

This might need the API to be a little bit more mature.

Add GitHub Actions Tests

Add a GitHub Actions workflow that runs the tests and benchmarks.