t1mm3 / dgen

High-performance CSV data generator

dgen

dgen is a flexible data generator for testing and benchmarking purpose. It aims at producing in the desired format quickly and, hence, can be used for generating large amounts of data.

Performance

This benchmark compares optimized single-threaded naive implementations with dgen. The naive implementations generate integers in the given domain and directly write a row to stdout. Note that the naive implementations are hard-coded for this benchmark and do not allow the generation of arbitrary CSVs. lut_printf replaces the integer to string conversion by a lookup table assuming the domain is small enough (64k).

Observe that single-threaded dgen is a around 3.5x faster than the naive implementations whereas the parallel version of dgen using 4 threads is another 2x faster while still providing a deterministic output.

Further information

• Data specification

dgen's Issues

[Bug] In presence of multiple threads sequential generation becomes non-sequential

The problem is that we parallelize in chunks of n vectors. Each thread can then pick up such a chunk i.e. we lose order between chunks.

Idea:
During initialization, each chunk can get a dense ID assigned.
After processing has been done, we "commit" a chunk to a queue. This queue will be ordered (priority queue). So when finding the position of our chunk in the queue, we will visit all the chunks before and check whether the chunk ids are dense. In case the ids are dense and the first chunk exists, we can flush them. (At some point we will add our chunk to the queue)

Output is (sometimes) non-deterministic

If you run diff_test multiple times the results differ.
It is probably related to multi-threading

Output cardinalities are wrong

It seems that with the recent changes we lose tuples

[Performance] Reduce string copying

In the multi-threaded case it can happen that we spend a lot of time integrating the partial result into the chunk's final string i.e. memmove(). The problem is that this memmove() is redudant because, instead, we can just scatter the strings into the final string

Void full materialization in OutputQueue (worst-case)

When many threads are producing data concurrently it may happen that they allocate strings that are flushed very late (due to deterministic ordering constraints). To have bounds on memory, we could have a second queue (with buffers) that are allocated by the threads and pushed into the OutputQueue.
Using this second queue, we can control the amount of buffers in the system and the memory usage

[Performance] Compact data types

For faster operations on integers in limited domains we could exploit the smallest data type that fits.

For example int8_t (for [-75, 64]) instead of int64_t.

[Performance] Integer to string conversion

It appears that the division by powers of 10 are quite costly.
Maybe there is a way to replace them through multiplications by counting bits like in rounddown_log10()

[Performance] Improve ScatterOut

We can combine multiple scatters of subsequent values into a compound primitive i.e. they will access the same cacheline

Generate floating point numbers

Extend generator to handle floats