thekvs / cpp-serializers

Benchmark comparing various data serialization libraries (thrift, protobuf etc.) for C++

About

Compare various data serialization libraries for C++.

• Thrift
• Protobuf
• Boost.Serialization
• Msgpack
• Cereal
• Avro
• Capnproto
• Flatbuffers
• YAS

Build

This project does not have any external serialization libraries dependencies. All (boost, thrift etc.) needed libraries are downloaded and built automatically, but you need enough free disk space (approx. 2.3G) to build all components. To build this project you need a compiler that supports C++14 features. Project was tested with Clang and GCC compilers.

1. git clone https://github.com/thekvs/cpp-serializers.git
2. cd cpp-serializers
3. mkdir build
4. cd build
5. cmake -DCMAKE_BUILD_TYPE=Release ..
6. cmake --build .

Usage

$ ./benchmark -h
Benchmark various C++ serializers
Usage:
  benchmark [OPTION...]

  -h, --help             show this help and exit
  -l, --list             show list of supported serializers
  -c, --csv              output in CSV format
  -i, --iterations arg   number of serialize/deserialize iterations
  -s, --serializers arg  comma separated list of serializers to benchmark

• Benchmark all serializers, run each serializer 100000 times:

$ ./benchmark -i 100000

• Benchmark only protobuf serializer, run it 100000 times:

$ ./benchmark -i 100000 -s protobuf

• Benchmark protobuf and cereal serializers only, run each of them 100000 times:

$ ./benchmark -i 100000 -s protobuf,cereal

Results

Following results were obtained running 1000000 serialize-deserialize operations 50 times and then averaging results on a typical desktop computer with Intel Core i7 processor running Ubuntu 16.04. Exact versions of libraries used are:

• thrift 0.12.0
• protobuf 3.7.0
• boost 1.69.0
• msgpack 3.1.1
• cereal 1.2.2
• avro 1.8.2
• capnproto 0.7.0
• flatbuffers 1.10.0
• YAS 7.0.2

serializer	object's size	avg. total time
thrift-binary	17017	1190.22
thrift-compact	13378	3474.32
protobuf	16116	2312.78
boost	17470	1195.04
msgpack	13402	2560.6
cereal	17416	1052.46
avro	16384	4488.18
yas	17416	302.7
yas-compact	13321	2063.34

Size

Time

For capnproto and flatbuffers since they already store data in a "serialized" form and serialization basically means getting pointer to the internal storage, we measure full build/serialize/deserialize cycle. In the case of other libraries we measure serialize/deserialize cycle of the already built data structure.

serializer	object's size	avg. total time
capnproto	17768	400.98
flatbuffers	17632	491.5

Size measured in bytes, time measured in milliseconds.

cpp-serializers's Issues

error: ‘numeric_limits’ is not a member of ‘std’

$ cmake -DCMAKE_BUILD_TYPE=Release ..
-- The CXX compiler identification is GNU 12.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Performing Test CXX14
-- Performing Test CXX14 - Success
-- Looking for include file inttypes.h
-- Looking for include file inttypes.h - found
-- Looking for include file netinet/in.h
-- Looking for include file netinet/in.h - found
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.13") 
-- Found autoreconf: /usr/bin/autoreconf
-- Found libtool: /usr/bin/libtoolize
CMake Warning (dev) at /usr/share/cmake-3.25/Modules/ExternalProject.cmake:3075 (message):
  The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
  not set.  The policy's OLD behavior will be used.  When using a URL
  download, the timestamps of extracted files should preferably be that of
  the time of extraction, otherwise code that depends on the extracted
  contents might not be rebuilt if the URL changes.  The OLD behavior
  preserves the timestamps from the archive instead, but this is usually not
  what you want.  Update your project to the NEW behavior or specify the
  DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
  robustness issue.
Call Stack (most recent call first):
  /usr/share/cmake-3.25/Modules/ExternalProject.cmake:4185 (_ep_add_download_command)
  CMakeLists.txt:55 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/share/cmake-3.25/Modules/ExternalProject.cmake:3075 (message):
  The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
  not set.  The policy's OLD behavior will be used.  When using a URL
  download, the timestamps of extracted files should preferably be that of
  the time of extraction, otherwise code that depends on the extracted
  contents might not be rebuilt if the URL changes.  The OLD behavior
  preserves the timestamps from the archive instead, but this is usually not
  what you want.  Update your project to the NEW behavior or specify the
  DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
  robustness issue.
Call Stack (most recent call first):
  /usr/share/cmake-3.25/Modules/ExternalProject.cmake:4185 (_ep_add_download_command)
  CMakeLists.txt:67 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Using Boost from /home/marek/git/cpp-serializers/build/external/boost/include
CMake Warning (dev) at /usr/share/cmake-3.25/Modules/ExternalProject.cmake:3075 (message):
  The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
  not set.  The policy's OLD behavior will be used.  When using a URL
  download, the timestamps of extracted files should preferably be that of
  the time of extraction, otherwise code that depends on the extracted
  contents might not be rebuilt if the URL changes.  The OLD behavior
  preserves the timestamps from the archive instead, but this is usually not
  what you want.  Update your project to the NEW behavior or specify the
  DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
  robustness issue.
Call Stack (most recent call first):
  /usr/share/cmake-3.25/Modules/ExternalProject.cmake:4185 (_ep_add_download_command)
  CMakeLists.txt:86 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/share/cmake-3.25/Modules/ExternalProject.cmake:3075 (message):
  The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
  not set.  The policy's OLD behavior will be used.  When using a URL
  download, the timestamps of extracted files should preferably be that of
  the time of extraction, otherwise code that depends on the extracted
  contents might not be rebuilt if the URL changes.  The OLD behavior
  preserves the timestamps from the archive instead, but this is usually not
  what you want.  Update your project to the NEW behavior or specify the
  DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
  robustness issue.
Call Stack (most recent call first):
  /usr/share/cmake-3.25/Modules/ExternalProject.cmake:4185 (_ep_add_download_command)
  CMakeLists.txt:104 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/share/cmake-3.25/Modules/ExternalProject.cmake:3075 (message):
  The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
  not set.  The policy's OLD behavior will be used.  When using a URL
  download, the timestamps of extracted files should preferably be that of
  the time of extraction, otherwise code that depends on the extracted
  contents might not be rebuilt if the URL changes.  The OLD behavior
  preserves the timestamps from the archive instead, but this is usually not
  what you want.  Update your project to the NEW behavior or specify the
  DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
  robustness issue.
Call Stack (most recent call first):
  /usr/share/cmake-3.25/Modules/ExternalProject.cmake:4185 (_ep_add_download_command)
  CMakeLists.txt:118 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/share/cmake-3.25/Modules/ExternalProject.cmake:3075 (message):
  The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
  not set.  The policy's OLD behavior will be used.  When using a URL
  download, the timestamps of extracted files should preferably be that of
  the time of extraction, otherwise code that depends on the extracted
  contents might not be rebuilt if the URL changes.  The OLD behavior
  preserves the timestamps from the archive instead, but this is usually not
  what you want.  Update your project to the NEW behavior or specify the
  DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
  robustness issue.
Call Stack (most recent call first):
  /usr/share/cmake-3.25/Modules/ExternalProject.cmake:4185 (_ep_add_download_command)
  CMakeLists.txt:135 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/share/cmake-3.25/Modules/ExternalProject.cmake:3075 (message):
  The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
  not set.  The policy's OLD behavior will be used.  When using a URL
  download, the timestamps of extracted files should preferably be that of
  the time of extraction, otherwise code that depends on the extracted
  contents might not be rebuilt if the URL changes.  The OLD behavior
  preserves the timestamps from the archive instead, but this is usually not
  what you want.  Update your project to the NEW behavior or specify the
  DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
  robustness issue.
Call Stack (most recent call first):
  /usr/share/cmake-3.25/Modules/ExternalProject.cmake:4185 (_ep_add_download_command)
  CMakeLists.txt:153 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/share/cmake-3.25/Modules/ExternalProject.cmake:3075 (message):
  The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
  not set.  The policy's OLD behavior will be used.  When using a URL
  download, the timestamps of extracted files should preferably be that of
  the time of extraction, otherwise code that depends on the extracted
  contents might not be rebuilt if the URL changes.  The OLD behavior
  preserves the timestamps from the archive instead, but this is usually not
  what you want.  Update your project to the NEW behavior or specify the
  DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
  robustness issue.
Call Stack (most recent call first):
  /usr/share/cmake-3.25/Modules/ExternalProject.cmake:4185 (_ep_add_download_command)
  CMakeLists.txt:164 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/share/cmake-3.25/Modules/ExternalProject.cmake:3075 (message):
  The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
  not set.  The policy's OLD behavior will be used.  When using a URL
  download, the timestamps of extracted files should preferably be that of
  the time of extraction, otherwise code that depends on the extracted
  contents might not be rebuilt if the URL changes.  The OLD behavior
  preserves the timestamps from the archive instead, but this is usually not
  what you want.  Update your project to the NEW behavior or specify the
  DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
  robustness issue.
Call Stack (most recent call first):
  /usr/share/cmake-3.25/Modules/ExternalProject.cmake:4185 (_ep_add_download_command)
  CMakeLists.txt:183 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/share/cmake-3.25/Modules/ExternalProject.cmake:3075 (message):
  The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
  not set.  The policy's OLD behavior will be used.  When using a URL
  download, the timestamps of extracted files should preferably be that of
  the time of extraction, otherwise code that depends on the extracted
  contents might not be rebuilt if the URL changes.  The OLD behavior
  preserves the timestamps from the archive instead, but this is usually not
  what you want.  Update your project to the NEW behavior or specify the
  DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
  robustness issue.
Call Stack (most recent call first):
  /usr/share/cmake-3.25/Modules/ExternalProject.cmake:4185 (_ep_add_download_command)
  CMakeLists.txt:197 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done
-- Generating done
-- Build files have been written to: /home/marek/git/cpp-serializers/build
$ cmake --build .
[  1%] Creating directories for 'cxxopts'
[  2%] Performing download step (download, verify and extract) for 'cxxopts'
-- Downloading...
   dst='/home/marek/git/cpp-serializers/build/external/cxxopts/src/v2.1.2.tar.gz'
   timeout='none'
   inactivity timeout='none'
-- Using src='https://github.com/jarro2783/cxxopts/archive/v2.1.2.tar.gz'
-- verifying file...
       file='/home/marek/git/cpp-serializers/build/external/cxxopts/src/v2.1.2.tar.gz'
-- Downloading... done
-- extracting...
     src='/home/marek/git/cpp-serializers/build/external/cxxopts/src/v2.1.2.tar.gz'
     dst='/home/marek/git/cpp-serializers/build/external/cxxopts/src/cxxopts'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[  3%] No update step for 'cxxopts'
[  4%] No patch step for 'cxxopts'
[  5%] Performing configure step for 'cxxopts'
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/marek/git/cpp-serializers/build/external/cxxopts/src/cxxopts-build
[  6%] Performing build step for 'cxxopts'
CMake Error at /home/marek/git/cpp-serializers/build/external/cxxopts/src/cxxopts-stamp/cxxopts-build-Release.cmake:49 (message):
  Command failed: 2

   '/usr/bin/gmake'

  See also

    /home/marek/git/cpp-serializers/build/external/cxxopts/src/cxxopts-stamp/cxxopts-build-*.log


gmake[2]: *** [CMakeFiles/cxxopts.dir/build.make:86: external/cxxopts/src/cxxopts-stamp/cxxopts-build] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:103: CMakeFiles/cxxopts.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2

cxxopts-build-err.log
cxxopts-build-out.log

What about cista++ ?

Could you include https://github.com/felixguendling/cista ?

I would be very interested in how it performs against the others...

Fails to compile flatbuffers tests: memest clearing an object of non-trivial type ‘struct MyGame::Example::Vec3'

When compiling this project as shown in the README, it gives

[ 17%] Completed 'capnproto'
[ 17%] Built target capnproto
Scanning dependencies of target flatbuffers
[ 18%] Creating directories for 'flatbuffers'
[ 19%] Performing download step (download, verify and extract) for 'flatbuffers'
-- Downloading...
   dst='/home/max/md_compile_test/cpp-serializers/build/external/flatbuffers/src/v1.10.0.tar.gz'
   timeout='none'
-- Using src='https://github.com/google/flatbuffers/archive/v1.10.0.tar.gz'
-- [download 100% complete]
-- verifying file...
       file='/home/max/md_compile_test/cpp-serializers/build/external/flatbuffers/src/v1.10.0.tar.gz'
-- Downloading... done
-- extracting...
     src='/home/max/md_compile_test/cpp-serializers/build/external/flatbuffers/src/v1.10.0.tar.gz'
     dst='/home/max/md_compile_test/cpp-serializers/build/external/flatbuffers/src/flatbuffers'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[ 20%] No patch step for 'flatbuffers'
[ 21%] No update step for 'flatbuffers'
[ 22%] Performing configure step for 'flatbuffers'
-- flatbuffers configure command succeeded.  See also /home/max/md_compile_test/cpp-serializers/build/external/flatbuffers/src/flatbuffers-stamp/flatbuffers-configure-*.log
[ 23%] Performing build step for 'flatbuffers'
CMake Error at /home/max/md_compile_test/cpp-serializers/build/external/flatbuffers/src/flatbuffers-stamp/flatbuffers-build-Release.cmake:49 (message):
  Command failed: 2

   '/usr/bin/make'

  See also

    /home/max/md_compile_test/cpp-serializers/build/external/flatbuffers/src/flatbuffers-stamp/flatbuffers-build-*.log


make[2]: *** [CMakeFiles/flatbuffers.dir/build.make:115: external/flatbuffers/src/flatbuffers-stamp/flatbuffers-build] Error 1
make[1]: *** [CMakeFiles/Makefile2:240: CMakeFiles/flatbuffers.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

with the further inner log being

$ cat /home/max/md_compile_test/cpp-serializers/build/external/flatbuffers/src/flatbuffers-stamp/flatbuffers-build-*.log
In file included from /home/max/md_compile_test/cpp-serializers/build/external/flatbuffers/src/flatbuffers/tests/test.cpp:32:
/home/max/md_compile_test/cpp-serializers/build/external/flatbuffers/src/flatbuffers/tests/monster_test_generated.h: In constructor ‘MyGame::Example::Vec3::Vec3()’:
/home/max/md_compile_test/cpp-serializers/build/external/flatbuffers/src/flatbuffers/tests/monster_test_generated.h:315:33: error: ‘void* memset(void*, int, size_t)’ clearing an object of non-trivial type ‘struct MyGame::Example::Vec3’; use assignment or value-initialization instead [-Werror=class-memaccess]
  315 |     memset(this, 0, sizeof(Vec3));
      |                                 ^
/home/max/md_compile_test/cpp-serializers/build/external/flatbuffers/src/flatbuffers/tests/monster_test_generated.h:301:41: note: ‘struct MyGame::Example::Vec3’ declared here
  301 | FLATBUFFERS_MANUALLY_ALIGNED_STRUCT(16) Vec3 FLATBUFFERS_FINAL_CLASS {
      |                                         ^~~~
cc1plus: all warnings being treated as errors
make[5]: *** [CMakeFiles/flattests.dir/build.make:145: CMakeFiles/flattests.dir/tests/test.cpp.o] Error 1
make[4]: *** [CMakeFiles/Makefile2:113: CMakeFiles/flattests.dir/all] Error 2
make[3]: *** [Makefile:141: all] Error 2

Machine is running Ubuntu Budgie 20.04, x64.

Misleading y-axis

Quick comment that your images make it appear that e.g. protobuf'd objects are 10x smaller than cap'n proto ones, which is untrue. This is because the y-axis doesn't start at 'zero' (which is considered bad practice for this reason).

How to compile & benchmark a subset of serializers?

I searched for a simple cmake command such as DISABLE_THRIFT or something similar to compile and benchmark a subset of serializers, but couldn't find any. Is there any simple way to avoid building all of these serializers?

Benchmark original object size not present

You provide nice benchmarks on your readme, however i'm curious what the original size of the object is before serialization.

Adding other formats into the test

See:

• https://github.com/ludocode/schemaless-benchmarks
• https://github.com/fraillt/cpp_serializers_benchmark
• https://github.com/felixguendling/cpp-serialization-benchmark
• https://github.com/alecthomas/go_serialization_benchmarks
• https://github.com/smallnest/gosercomp

Also:

• 18F/data-act-pilot#161
• https://leopard.in.ua/2013/10/13/binary-serialization-formats
• https://serde.rs/#data-formats
• https://luminousmen.com/post/big-data-file-formats
• https://towardsdatascience.com/the-best-format-to-save-pandas-data

Here is a list

• Binn
• BSON
• UBJSON
• bitsery
• zpp
• Cista++
• ORC
• Parquet
• BERT
• Blink protocol
• Bencode
• Gobs
• Abstract Syntax Notation One (ASN.1)
• Bincode
• CBOR
• Pickle
• RON
• JSON5
• postcard
• Envy
• HDF5
• Feather

Do size comparison after gzip

For anyone thinking about these formats for client-server communication, they will want to know what the size of them is after gzipping or applying other forms of compression. It can be very surprising - JSON can beat binary formats.

Create separate programs for each library

Currently, benchmark.cpp has everything lumped together in one file. It would be helpful if there were a separate program for each library being tested. That way the sizes of the programs could be compared.

zlib dependency not included

zlib is not included.

"This project does not have any external library dependencies. All (boost, thrift etc.) needed libraries are downloaded and built automatically, but you need enough free disk space to build all components"

Fails to build on Windows using CMake 3.7.2, MSVC 2015 x64

Avro benchmark numbers are not realistic at all

function ** avro_serialization_test** is not at all efficient. As can be seen, for every encode / decode new resources are initialised, not an optimal technique.

Also the the default avro::InputStream and avro::OutputStream implementations are very rudimentary. Best to use a custom implementation.

    for (size_t i = 0; i < iterations; i++) {
        auto out = avro::memoryOutputStream();
        auto encoder = avro::binaryEncoder();
        encoder->init(*out);
        avro::encode(*encoder, r1);

        auto in = avro::memoryInputStream(*out);
        auto decoder = avro::binaryDecoder();
        decoder->init(*in);
        avro::decode(*decoder, r2);
    }

I found avro to be as performant as protobuf after making necessary changes

Running on Mac

i am having problem compiling this project on Mac. is this tested working on Mac yet?

checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... configure: error: in /Users/xxx/Library/Caches/CLion2016.2/cmake/generated/cpp-serializers-master-d78257/d78257/Debug/external/protobuf/src/protobuf': configure: error: cannot run C compiled programs. If you meant to cross compile, use--host'.
See `config.log' for more details
make[2]: *** [external/protobuf/src/protobuf-stamp/protobuf-configure] Error 1
make[1]: *** [CMakeFiles/protobuf.dir/all] Error 2
make: *** [all] Error 2

ZeroBuf lib integration

Pretty nice benchmark
Did you see this library https://github.com/HBPVIS/ZeroBuf ?

Add ORC format

Would it be possible to add the ORC file format to the benchmarks here?

Protobuf usage not appropriate

For Google's protobuf, when a significant portion of negative numbers is expected, use sint64 instead of int64 in the proto definition will make it run much faster and give a much smaller size.

misleading graphs

From glancing at your graphs, for say capnproto vs flatbuffers, one might think that capnproto takes half the time.

Ditto for the other graphs. You should ensure that the bottom of the y axis is zero.