velvia / filo Goto Github PK

Open hash map using Hopscotch Hashing:
https://en.wikipedia.org/wiki/Hopscotch_hashing

Would be pretty impressive esp if it could be off heap.

Key questions:

1. Base type to use

• Proposal: http://docs.oracle.com/javase/7/docs/api/java/math/BigDecimal.html
• java.math.BigDecimal has the best compatibility across the data ecosystem( C*, Spark - which uses a BigDecimal-based variant, others)
• There is also Scala's BigDecimal, which has nice operators and nice functions for determining if it can fit into long/int/double etc., but has a slight overhead.

BigDecimal has two components: a variable precision unscaled value and a 32-bit scale.
2. Filo binary format

• Anything that fits in Long/Int/smaller (scale == 0 for all numbers, unscaled value < 64 bits) should use SimplePrimitiveVector / DiffPrimitiveVector for efficient storage.
• Otherwise, store unscaled values as binary blobs. Most of the time the size of these blobs are probably very similar, so consider something like FixedBinaryVector to save on space (8 bytes per element savings). Store the scale as a SimplePrimitiveVector since most scale numbers are very small and can be stored efficiently.
• Consider converting what we can to floats/doubles for efficiency -- if no loss when converting back and forth

Implement extremely fast summation and other math operations for integral and float/double vectors.
Some papers to look at:

https://developers.opengamma.com/articles/DGEMV.pdf

Scala's Traversable trait is really slow due to boxing and other issues. Instead, use one of these projects to get a much faster API:

https://github.com/denisrosset/metal

https://github.com/non/debox

Something to abstract the UNSAFE methods so that upgrade to JDK9 can be smoother.

https://github.com/real-logic/Agrona/tree/master/agrona/src/main/java/org/agrona

https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent0.java#L309

https://github.com/OpenHFT/Chronicle-Bytes/tree/master/src/main/java/net/openhft/chronicle/bytes

Agrona is on maven central:
https://mvnrepository.com/artifact/org.agrona/Agrona/0.9.1