bkille / bitlib

Provides a bit-vector, an optimized replacement of the infamous std::vector<:b:ool>. In addition to the bit-vector, the library also provides implementations of STL algorithms tailored for bit-vectors.

BitLib

This repository acts as an efficient replacement of std::vector<bool>. It provides implementations of many of the functions in <algorithms> optimized for containers of bits, in addition to providing a bit_vector class which has roughly the same interface as std::vector<bool>.

This project is built on "bit iterators" developed by Vincent Reverdy and many of the implementations in include/bit-algorithms come from some of my previous work with Vincent here.

Example

The code below is from example/src/example1.cpp. While the type of word that the bitvector is built off of is templated and you can use any unsigned type, it is likely that you'll want to use uint64_t or another 64 bit unsigned type, as that will leverage the most bit-parallelism.

#include <iostream>
#include "bitlib/bitlib.hpp"

int main() {
    bit::bit_vector<unsigned char> bv1 ("011111010010");
    std::cout << "Original bitvec:  " << bv1.debug_string() << std::endl;
    // Original bitvec:  01111101 0010

    // Same behavior as std::reverse
    bit::reverse(bv1.begin(), bv1.end());
    std::cout << "Reversed bitvec:  " << bv1.debug_string() << std::endl;
    // Reversed bitvec:  01001011 1110

    // Same behavior as std::rotate
    bit::rotate(bv1.begin(), bv1.begin() + 3, bv1.end());
    std::cout << "Rotated bitvec:   " << bv1.debug_string() << std::endl;
    // Rotated bitvec:   01011111 0010

    // Same behavior as the corresponding std::vector::push_back and std::vector::insert
    bv1.push_back(bit::bit0);
    bv1.insert(bv1.end(), 10, bit::bit1);
    std::cout << "Extended bitvec:  " << bv1.debug_string() << std::endl;
    // Extended bitvec:  01011111 00100111 1111111

    return 0;
}

Installation

BitLib is a header-only libarary. Currently, the BitLib library requires at least -std=c++17.

CMake

You can automatically fetch the library using Cmake's FetchContent.

include(FetchContent)
FetchContent_Declare(
    bitlib
    GIT_REPOSITORY https://github.com/bkille/bitlib.git
    GIT_TAG origin/master
)
FetchContent_MakeAvailable(bitlib)

add_executable(example example.cpp)
target_link_libraries(example bitlib::bitlib)

Manual include

Alternatively, you can copy the include/bitlib directory to somewhere in your include path.

SIMD support, testing and benchmarking

SIMD support (enabled via Google's highway library) can be enabled by defining BITLIB_HWY. For example, with cmake, you can run cmake -DBITLIB_HWY=1. Other options can be found in the CMakeLists.txt file:

option(BITLIB_HWY "Build with google highway SIMD extensions" OFF)
option(BITLIB_BENCHMARK "Build bitlib benchmarks" OFF)
option(BITLIB_EXAMPLE "Build bitlib examples" OFF)
option(BITLIB_TEST "Build bitlib tests" OFF)
option(BITLIB_PROFILE "Buid simple example for profiling" OFF)
option(BITLIB_COVERAGE "Compute test coverage" OFF)

Usage

The goal of BitLib is to be as similar to the C++ STL as possible. The interface of most functions and classes are the same as they are in the STL. Instead of the values being bool, we have bit::bit_value, which can take on either bit::bit0 or bit::bit1.

Containers

Right now, the only container I have implemented is the bitvector. bit::bit_vector<WordType> is essentially a wrapper around std::vector<WordType>. The interfaces are nearly identical. In addition to the normal vector constructors, you can also provide a string to construct your bitvector:

using WordType = uint64_t;
bit::bit_vector<WordType> bvec1 ("011111010010");

While the type of word that the bitvector is built off of is templated and you can use any unsigned type, it is likely that you'll want to use uint64_t or another 64 bit unsigned type, as that will leverage the most bit-parallelism.

Algorithms

The algorithms again work in the same manner as the STL. The functions provided here have the same interface as those in the STL, however under the hood, they take advantage of bit-parallelism. It should be noted that if there is an STL algorithm that is not supported yet by BitLib, you can still use the STL implementation. For example:

using WordType = uint64_t;
bit::bit_vector<WordType> bvec1 ("011111010010");
bit::bit_vector<WordType> bvec2 = bvec1;
bit::equal(bvec1.begin(), bvec1.end(), bvec2.begin(), bvec1.end());
std::equal(bvec1.begin(), bvec1.end(), bvec2.begin(), bvec1.end()); // Also works, but much slower as it works bit-by-bit

For algorithms which take a function (i.e. bit::transform), the function should have WordType as the input types as well as the return type. For example, to compute the intersection of two bitvectors:

using WordType = uint64_t;
auto binary_op = std::bit_and<WordType>();

// Store the AND of bitvec1 and bitvec2 in bitvec3
auto bitret = bit::transform(
        bitvec1.begin(),
        bitvec1.end(),
        bitvec2.begin(),
        bitvec3.begin()
        binary_op); 

Iterators

The bit-iterators are the foundation of the library. In most cases, users will only need to work w/ the bit::bit_vector::begin() and bit::bit_vector::end() methods to obtain iterators. However, constructing a bit iterator from any address is also straightforward:

using WordType = uint64_t;
std::array<WordType, 4> wordArr = {1,2,3,4};
bit::bit_iterator<WordType*>(&(wordArr[0])); // Constructs a bit iterator starting from the first bit from the first word of the vector
bit::bit_iterator<WordType*>(&(wordArr[0]), 1); // Constructs a bit iterator from the second bit (position 1) of the first word of the vector

In order to grab the underlying word that a bit pointed to by a bit_iterator comes from, you can use the bit_iterator.base() function.

It is worth noting that the "position" of a bit always increases from LSB to MSB. For those looking to create their own algorithms from bit_iterators, this can be a common "gotcha". For example, shifting a word to the right by k will eliminate the first k bits of the container. This is only important to those implementing their own algorithms. bit::shift_* works as described in the documentation i.e. shift_right shifts the container towards end() and shift_left shifts the container towards begin().

       MSB|<-----|LSB
Position: 76543210
Value:    01010001 --> Sequence: 10001010


// bit::shift_right by 2
       MSB|<-----|LSB
Position: 76543210
Value:    01000100 --> Sequence: 00100010

Documentation

Given that the majority of the library is focused on having the same interface as the C++ STL iterators, containers, and algorithms, users should use the official STL documentation website. We do plan on adding our own documentation in the future, however.

Performance Benchmarks

I used Google's benchmark library for computing benchmarks. Each benchmark is formatted as {bit, BitArray, std}::function (size) [(alignment-tags)].

* `bit` is for this library, `BitArray` is for the popular C-based [BitArray library](https://github.com/noporpoise/BitArray), [dynamic_bitset](https://github.com/pinam45/dynamic_bitset) is a header-only library similar to Boost's dynamic_bitset, and`std` is the standard library operating on the infamous `vector<bool>`. 

• (size) denotes the size of the container in bits. small = 1 << 8, medium= 1 << 16, large = 1 << 24, huge = 1 << 31
• (alignment-tags) refers to the memory alignment of the bit-iterators. U means the iterator does not fall on a word boundary, R means the iterator is placed at random, and A means the iterator is aligned with a word boundary.

For example, bit::rotate (large) (ARA) refers to our library's implementation of the rotate algorithm operating on a container of 65536 bits, where first and last are aligned but n_first is selected at random.

---------------------------------------------------------------------------------------
Benchmark                                             Time             CPU   Iterations
---------------------------------------------------------------------------------------
bit::set (large)                                   1.90 ns         1.90 ns    367974893
dynamic_bitset::set (large)                        2.37 ns         2.37 ns    296837879
bitarray::set (large)                              2.19 ns         2.19 ns    319133940
std::set (large)                                   2.39 ns         2.39 ns    293135332
bit::shift_left (small)                            26.8 ns         26.8 ns     25929070
bit::shift_left (small) (UU)                       22.4 ns         22.4 ns     31233265
dynamic_bitset::shift_left (small)                 13.1 ns         13.1 ns     53627207
bitarray::shift_left (small)                       38.2 ns         38.2 ns     18339126
std::shift_left (small)                             345 ns          345 ns      2029283
bit::shift_left (large)                          371224 ns       371211 ns         1886
bit::shift_left (large) (UU)                     371536 ns       371530 ns         1880
dynamic_bitset::shift_left (large)               638896 ns       638880 ns         1097
bitarray::shift_left (large)                    3156273 ns      3156003 ns          222
std::shift_left (large)                       105227752 ns    105223527 ns            7
bit::shift_right (small)                           26.9 ns         26.9 ns     25976563
bit::shift_right (small) (UU)                      39.3 ns         39.3 ns     17962533
dynamic_bitset::shift_right (small)                12.2 ns         12.2 ns     57419526
bitarray::shift_right (small)                      38.1 ns         38.1 ns     18325350
std::shift_right (small)                            504 ns          504 ns      1386280
bit::shift_right (large)                         413297 ns       413269 ns         1693
bit::shift_right (large) (UU)                    413692 ns       413655 ns         1682
dynamic_bitset::shift_right (large)              557287 ns       557305 ns         1257
bitarray::shift_right (large)                   3156463 ns      3156516 ns          222
std::shift_right (large)                      210100788 ns    210083631 ns            3
bit::reverse (small) (UU)                          43.4 ns         43.4 ns     16112098
bitarray::reverse (small) (UU)                     95.1 ns         95.1 ns      7387177
std::reverse (small)                                419 ns          419 ns      1677069
bit::reverse (large)                            1245260 ns      1245160 ns          563
bit::reverse (large) (UU)                       1800771 ns      1800680 ns          389
bitarray::reverse (large)                      16899481 ns     16898587 ns           41
bitarray::reverse (large) (UU)                 22719408 ns     22720393 ns           31
std::reverse (large)                          293563397 ns    293542850 ns            2
bit::transform(UnaryOp) (small)                    8.75 ns         8.75 ns     80079214
bit::transform(UnaryOp) (small) (UU)               16.6 ns         16.6 ns     42254961
dynamic_bitset::transform(UnaryOp) (small)         4.00 ns         4.00 ns    169219246
bitarray::transform(UnaryOp) (small)               8.39 ns         8.39 ns     83877004
std::transform(UnaryOp) (small)                     763 ns          763 ns       917975
bit::transform(UnaryOp) (large)                  373982 ns       373950 ns         1853
bit::transform(UnaryOp) (large) (UU)            2059234 ns      2059268 ns          339
dynamic_bitset::transform(UnaryOp) (large)       379368 ns       379368 ns         1805
bitarray::transform(UnaryOp) (large)             739552 ns       739544 ns          881
std::transform(UnaryOp) (large)               197977698 ns    197969224 ns            4
bit::transform(BinaryOp) (small)                   4.38 ns         4.38 ns    160002060
bit::transform(BinaryOp) (small) (UU)              42.1 ns         42.1 ns     16549758
dynamic_bitset::transform(BinaryOp) (small)        4.36 ns         4.36 ns    160692979
bitarray::transform(BinaryOp) (small)              10.7 ns         10.7 ns     66178974
std::transform(BinaryOp) (small)                    855 ns          855 ns       832115
bit::transform(BinaryOp) (large)                 763642 ns       763574 ns          912
bit::transform(BinaryOp) (large) (UU)          10966202 ns     10966406 ns           64
dynamic_bitset::transform(BinaryOp) (large)      758617 ns       758574 ns          906
bitarray::transform(BinaryOp) (large)            518286 ns       518267 ns         1177
std::transform(BinaryOp) (large)              802270688 ns    802303941 ns            1
bit::rotate (small)                                 131 ns          131 ns     16525922
std::rotate (small)                                1782 ns         1782 ns       417293
bit::rotate (large)                             7333284 ns      7333170 ns           96
std::rotate (large)                           514697313 ns    514718779 ns            1
bit::count (small)                                 8.14 ns         8.14 ns     86522765
dynamic_bitset::count (small)                      6.29 ns         6.29 ns    108878018
bitarray::count (small)                            5.47 ns         5.47 ns    133692569
std::count (small)                                  234 ns          234 ns      2997782
bit::count (large)                               365194 ns       365159 ns         1919
dynamic_bitset::count (large)                    365279 ns       365269 ns         1919
bitarray::count (large)                          917302 ns       917185 ns          764
std::count (large)                             58934071 ns     58931785 ns           12
bit::swap_ranges (small)                           9.58 ns         9.57 ns     73128377
bit::swap_ranges (small) (UU)                      19.7 ns         19.7 ns     35498474
std::swap_ranges (small)                            756 ns          756 ns       912041
bit::swap_ranges (large)                         852205 ns       852241 ns          821
bit::swap_ranges (large) (UU)                   5691899 ns      5692145 ns          123
std::swap_ranges (large)                      522198664 ns    522161939 ns            1
bit::copy (small) (UU)                             25.0 ns         25.0 ns     28200772
std::copy (small)                                   707 ns          707 ns       990757
bit::copy (large) (UU)                          5952278 ns      5951729 ns          116
std::copy (large)                             189551338 ns    189554366 ns            4
bit::equal (small) (UU)                            13.1 ns         13.1 ns     53616228
std::equal (small)                                  886 ns          886 ns       790035
bit::equal (large) (UU)                         1960399 ns      1960375 ns          357
std::equal (large)                            234389098 ns    234398907 ns            3
bit::move (small) (UU)                             23.5 ns         23.5 ns     29764745
std::move (small)                                   706 ns          706 ns       992054
bit::move (large) (UU)                          5135837 ns      5135619 ns          136
std::move (large)                             188961979 ns    188953500 ns            4
bit::copy_backward (small) (UU)                    39.0 ns         39.0 ns     17977387
std::copy_backward (small)                          527 ns          527 ns      1313265
bit::copy_backward (large) (UU)                 9163333 ns      9163038 ns           76
std::copy_backward (large)                    444362971 ns    444350668 ns            2
bit::fill (small) (UU)                             6.48 ns         6.48 ns    108934237
dynamic_bitset::fill (small)                       4.79 ns         4.79 ns    146205764
bitarray::fill (small)                             14.5 ns         14.5 ns     48030428
std::fill (small)                                  9.15 ns         9.15 ns     76612702
bit::fill (large) (UU)                           440400 ns       440396 ns         1590
dynamic_bitset::fill (large)                     429375 ns       429359 ns         1631
bitarray::fill (large)                           369732 ns       369736 ns         1964
std::fill (large)                                356517 ns       356488 ns         1894
bit::find (small) (UU)                             3.10 ns         3.10 ns    228714994
dynamic_bitset::find (small)                       3.05 ns         3.05 ns    229830138
bitarray::find (small)                             7.38 ns         7.38 ns     99039746
std::find (small)                                   110 ns          110 ns      6311725
bit::find (large) (UU)                           182002 ns       182006 ns         3850
dynamic_bitset::find (large)                     259896 ns       259908 ns         2696
bitarray::find (large)                           252434 ns       252445 ns         2774
std::find (large)                              28570723 ns     28567762 ns           25

bitlib's Issues

Add bit::shuffle

Add bit version of std::shuffle

• shuffle implementation
• shuffle tests
• shuffle benchmark

Add bit::fill

Add bit version of std::fill

• fill implementation
• fill tests
• fill benchmark

Consistent iterator template names

While the bitlib only supports random-access iterators at the moment, there are still remnants of the old library which supported other iterator types i.e. InputIt/OutputIt. While we may want to support those in the future, we should rename them for now to reflect their actual functionality.

A few issues with `bit_vector`

Pardon me for dumping several issues into a single ticket.

• bit_vector lacks a move constructor.
• The copy assignment should take the parameter by const reference, otherwise this fails:

  const bit_vector<unsigned int> foo();
  bit_vector<unsigned int> v;
  v = foo();

• digits should be static constexpr. There's no reason to store it in each vector instance.
• Move constructor and assignment need to be noexcept, otherwise when placed in a standard container such as std::vector, they'll be copied on resize instead of being moved; see How to enforce move semantics when a vector grows?
• The move assignment leaves the source vector in an inconsistent state, since its vector is now empty but length_ is non-zero. I would zero it. Same applies to the future move constructor.
  On a side note, the copy constructor an assignment can (and should) be = default;ed.
• Using std::ceil with floating-point division looks odd. It'll give inaccurate results at large sizes, and I expect it to be slower than the integral equivalent: (a + b - 1) / b.
• bit_vector::bit_vector(std::string_view s) does something weird. It interprets the string as bytes, and takes the LSB of each character. It probably should reject everything except '0' and '1', and there should be a separate constructor with a pair of bool iterators.

The rest is purely nitpicking:

• constexpr is missing on a few functions: non-const operator[] and the assignment operators.
• Compiling with -Wextra-semi would give a bunch of warnings on ;s after closing braces.

Add bit::search

Add bit version of std::search

• search implementation
• search tests
• search benchmark

Note: There are multiple different implementations of bit-searching algorithms, but any will do for the first pass

wrong number for 1 << 16

1 << 16 = 65536, need to fix in readme

Add bit::swap_ranges

Add bit version of std::swap_ranges

• swap_ranges implementation
• swap_ranges tests
• swap_ranges benchmark

Add bit::search_n

Add bit version of std::search_n

• search_n implementation
• search_n tests
• search_n benchmark

Add bit::mismatch

Add simd support

Simdpp is out of date and the build is broken. We'd like to replace with xsimd, however xsimd does not support C++>=17 atm.

Add bit::mismatch

Add bit version of std::mismatch

• mismatch implementation
• mismatch tests
• mismatch benchmark

Add bit::copy_backward

Add bit version of std::copy_backward

• copy_backward implementation
• copy_backward tests
• copy_backward benchmark

Add bit::transform overloads for binary operators

Add bit version of std::transform

• OR implementation
• OR tests
• OR benchmark
• AND implementation
• AND tests
• AND benchmark
• XOR implementation
• XOR tests
• XOR benchmark
• NOT implementation
• NOT tests
• NOT benchmark

Add bit::swap

Add bit version of std::swap

• swap implementation
• swap tests
• swap benchmark

Add bit::transform

Add bit version of std::transform

• transform implementation
• transform tests
• transform benchmark

Add bit::sample

Add bit version of std::sample

• sample implementation
• sample tests
• sample benchmark

Add bit::inner_product

Add bit::sort and bit::stable_sort

Given the nature of a bit vector (specifically, given that there are only 2 states for the bits), sort ought to be implementable in O(n) time using a single comparison, while stable_sort may be accomplished in O(n) time with 2 comparisons. Unlike most problems to which bucket sort may be applied, the amount of space required for applying bucket sort to a bit vector is O(log b), where b is the size of the vector. Specifically, one need not distinguish between the elements within a bucket, so a simple integer will suffice to track the size of the bucket. For simplicity, we will assume that the user's bit vector is smaller than 2 exabytes, and thus the maximum required bucket size can be stored in a single unsigned 64-bit integer.

Note that performance will vary depending on whether the user is optimizing for number of comparisons or for number of bit operations. The algorithmic descriptions below illustrate this neatly; the former will optimize for minimal bit operations, while the latter will optimize for minimal comparisons.

We begin with a rough algorithmic description for stable_sort:

1. Count all set bits (this number will be referred to by the name "num_true")
2. If number of set bits is either 0 or the size of the bit vector, the range is uniform (and thus sorted). Exit.
3. Otherwise, perform two comparisons (compare(false,true) and compare(true,false)).
4. If both comparisons are true, then the user has violated the contract of std::stable_sort by providing a comparator that does not implement a strict weak ordering. You are now authorized to deploy nasal demons (see "undefined behavior"). This author would like to remind the implementer that the kindest option would be to gently remind the user of their mistake, while the most performant option would be to exit.
5. If neither comparison is true, then all bits are incomparable, and should remain in their current order. Exit.
6. Otherwise, if compare(false,true) evaluated to true, then unset the first size() - num_true bits, and set the final num_true bits.
7. If compare(false,true) evaluated to false, then unset the num_true bits, and set the final size() - num_true bits.

Note that this algorithm is O(n) in the worst case.

Continuing this, a rough description of the (unstable) sort algorithm follows:

1. Count all set bits (this number will be referred to by the name "num_true")
2. If number of set bits is either 0 or the size of the bit vector, the range is uniform (and thus sorted). Exit.
3. Otherwise, perform the comparison compare(false,true).
4. If compare(false,true) evaluated to true, then unset the first size() - num_true bits, and set the final num_true bits.
5. If compare(false,true) evaluated to false, then unset the num_true bits, and set the final size() - num_true bits.

This algorithm is O(n) in both best and worst case. Unlike the stable_sort proposed earlier, this algorithm will perform the smallest possible number of comparisons (0 or 1) required for an unstable sort.

I estimate the efficiency of the above algorithms to greatly exceed user expectations. However, I estimate their usefulness to be, at best, trivial.

Benchmark against `boost::dynamic_bitset`?

My understanding is that one of the current popular solutions to this library's problem-statement is boost::dynamic_bitset. I think perhaps adding that to the benchmark might be useful for people considering this library.

`auto` deduces type as reference even without `&`

auto will deduce the type as a reference type, even without the &. This means that:

bit::bit_vector<unsigned char> bvec{"110110"};
auto b = bvec.front() // Should be copied, auto -> bit_value, but instead auto -> bit_reference
b = ~b;
std::cout << bvec.debug_string() << std::endl; // Outputs 010110

Add bit::equal

Add bit::fill_n

Add bit version of std::fill_n

• fill_n implementation
• fill_n tests
• fill_n benchmark

Add bit::move

Add bit version of std::move

• move implementation
• move tests
• move benchmark

Add bit::copy

Add bit version of std::copy

• copy implementation
• copy tests
• copy benchmark

Unused functions not picked up by gcov

Unused functions from the bitlib/ library are not emitted by the compiler, resulting in gcov not labeling them as uncovered.

Add bit::flip

This is the same thing as std::transform with a function [](auto b) { return ~b; }

• flip general implementation
• flip general tests
• flip general benchmark

Add bit::move_backward

Add bit version of std::move_backward

• move_backward implementation
• move_backward tests
• move_backward benchmark