ascoderu / compression-benchmarks

Benchmark to compare serialization and compression algorithms

compression-benchmarks

What's this?

This repository contains serialization, compression and encryption benchmarks for the Lokole email data exchange protocol with the aim to find the method of encoding our dataset in the smallest possible size to save bandwidth for users of the system.

💡 Do you know a compression or serialization that could make our data smaller? Then open a pull request!

Setup

Install the requirements with pip install -r requirements.txt -r requirements-dev.txt.

Run the tests with python -m benchmarks.tests and run the linter with flake8 benchmarks.

Run the benchmarks via python -u -m benchmarks <emails-zip-url>.

Results

Benchmark results are kept up to date by Github Actions at ascoderu/compression-benchmarks.

compression-benchmarks's Issues

Fix BSON serializer

The BSON serializer currently crashes for many inputs. I believe this is because the serialized format for a specific email may contain the \n character, e.g. in attachments... but we also use this character as a record separator to be able to read/write multiple serialized emails to a single file. This means that when we try to deserialize an email, reading to the first \n character may give us back a corrupt BSON record if the original email contained a raw \n character.

A suggested fix is either to find a way to stream multiple records in a single file pointer (similar to the CBOR implementation), or find another record delimiter which is guaranteed not to clash with any data in the file. But of course, other approaches that fix the problem are welcome too!

Add CBOR serialization

https://cbor.io/

Parallelize execution

We generate one benchmark run per combination of serializer, compressor and encryptor (see code). The runs have no dependency on each other, so it should be possible to run them in parallel using multiprocessing.