gyson / blex Goto Github PK
View Code? Open in Web Editor NEWFast Bloom filter with concurrent accessibility, powered by :atomics module.
License: MIT License
blex's People
Contributors
Stargazers
Watchers
blex's Issues
false_positive_probability not holding for large bloomfilter
I was expecting the False Positive rate to stay below the level the bloomfilter was initialized with until the bloomfilter reaches capacity, but instead it vastly exceeds it.
Create a bloomfilter with 400 million capacity, FPP of 0.001, put 100 million things into the bloomfilter, then test 1 million things that weren't put into the bloomfilter.
I expected to find 1,000 or less false positives, but instead I find 23,242.
iex(1)> b = Blex.new(400000000, 0.001)
%Blex{
a: #Reference<0.3778163223.2071855107.97792>,
b: 30,
hash_fn: #Function<4.114810444/2 in Blex.get_hash_fn/1>,
hash_id: 202,
k: 10,
m: 1073741824
}
iex(2)> 1..100000000 |> Enum.map(fn x -> Blex.put(b, x) end)
[:ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok,
:ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok,
:ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok,
:ok, :ok, ...]
iex(3)> 101000001..102000000 |> Enum.map(fn x -> Blex.member?(b, x) end) |> Enum.filter(fn x -> x end) |> Enum.count
23242
I get the exact same false positive count when I initialize the bloomfilter with a capacity of 1.6 billion.
While experimenting I also tried creating a bloomfilter with capacity of 3.2 billion, but that resulted in a very different error that I'll open a separate issue for.
ArgumentError when putting anything into a very large bloomfilter (hash_id: 203)
iex(1)> b = Blex.new(3200000000, 0.01)
%Blex{
a: #Reference<0.830440789.1533673474.145158>,
b: 33,
hash_fn: #Function<3.114810444/2 in Blex.get_hash_fn/1>,
hash_id: 203,
k: 7,
m: 8589934592
}
iex(2)> Blex.put(b, 1)
** (ArgumentError) argument error
(blex 0.2.0) lib/blex.ex:209: anonymous fn/2 in Blex.get_hash_fn/1
(blex 0.2.0) lib/blex.ex:345: Blex.set_all/7
Line reference:
https://github.com/gyson/blex/blob/master/lib/blex.ex#L209
Unfortunately I'm pretty new to Elixir so the code's a bit beyond me to debug.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.