mbrg / py-hyperminhash

HyperLogLog with intersection

HyperMinHash

This repository is a Python>=3.6 port of golang hyperminhash:

Besides being a compact and pretty speedy HyperLogLog implementation for cardinality counting, this modified HyperLogLog allows intersection and similarity estimation of different HyperLogLogs.

Install

pip install hyperminhash

Example Usage

from hyperminhash import HyperMinHash

sk1 = HyperMinHash()
sk2 = HyperMinHash()

for i in range(10000):
    sk1.add(i)

print(len(sk1))
# 10001 (should be 10000)

for i in range(3333, 23333):
    sk2.add(i)

print(len(sk2))         
# 19977 (should be 20000)

print(sk1.similarity(sk2))
# 0.284589082 (should be 0.2857326533)

print(sk1.intersection(sk2))
# 6623 (should be 6667)

sk1.merge(sk2)
print(sk1.cardinality())
# 23271 (should be 23333)

py-hyperminhash's Issues

Best practices

• Cleanup HyperMinHash class from utilities

• Implement Python set

• Check slow tests

Add docs about err rate and warning on too many items