Feature request: Jaccard statistic about roaringbitmap

[daveuu]

I've added a Block.orlen(), a bitsetunioncount(), a RoaringBitmap.union_len() and a RoaringBitmap.jaccard_dist() which are working for bitmaps containing "dense", "positive" or both kinds of blocks. I have not been able to generate a roaring bitmap that uses "inverted" blocks so I haven't been able to test that. For the other kinds of blocks:

from roaringbitmap import RoaringBitmap
import random
# dense blocks
A = RoaringBitmap(random.sample(range(100000),20000))
# positive blocks
B = RoaringBitmap(random.sample(range(10000),200))
# many dense blocks, positive at end
D = RoaringBitmap(random.sample(range(4000000),400000))

@andreasvc, how can I generate a bitmap that uses inverted blocks? (The code is a lot like positive blocks so probably works already). I've tried to maintain coding style etc. so this should be suitable for a pull request eventually.

[andreasvc]

That's great.
In the meantime I have been completely reworking the implementation to use structs and manual memory management instead of python objects. This hopefully will make pickling more efficient and will allow releasing the GIL. However, I'm running into hard-to-debug errors, so it still makes sense to merge your code.

An inverted block is created whenever it contains 61141 or more elements.
So in theory a=RoaringBitmap(range(61141)) creates one.
However... this runs into a memory error. What is needed is a constructor to efficiently add a range of elements, and the unittests should use this to test inverted blocks.

[daveuu]

Sounds good! I'll update the benchmarks and tests (only manual tests so far . . .) then submit a pull request (my first 😄). Should be sometime today (tonight).