curiosity-ai / hnsw-sharp Goto Github PK
View Code? Open in Web Editor NEWC# library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs
License: MIT License
C# library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs
License: MIT License
When using this library, it seems like you often need to specify both M and LevelLambda, but the two values are typically related.
It would be nice if you could just set one or the other and the default would follow. For example:
class Parameters
{
private int? _m;
private double? _levelLambda;
public int M
{
get => this._m is { } m ? m
: this._levelLambda is { } levelLambda => (int)Math.Round(Math.Pow(Math.E, 1 / levelLambda), MidpointRounding.AwayFromZero)
: 10; // default
set => this._m = value;
}
public double LevelLambda
{
get => this._levelLambda ?? 1 / Math.Log(this.M);
set => this._levelLambda = value;
}
}
Looks like all the tests in SmallWorldTest.cs fail. I tried increasing the K parameter from 20 to 30 and now some of them pass reliably, but some still fail.
Minor usability request. I feel like empty graphs should be searchable as normal, just returning an empty result set. This is useful for applications where you are building up the graph as you go, e.g.
foreach (var item in items)
{
if (has close neighbor in graph) { continue; }
else { add item to graph }
}
See https://github.com/curiosity-ai/hnsw-sharp/blob/master/Src/HNSW.Net/Graph.cs#L161
Like for example:
At least the search example works. A pity I had to waste a lot of time on the initialization.
Is it a unwritten rule in ML related fields to not document or comment, anything, anywhere, ever? (MS's SPTAG is even worse in that regard)
I'm not 100% sure this is a bug, but I don't understand why any node in the graph would want self-links or duplicate edges.
Here's a unit test which demonstrates this:
[TestMethod]
public void TestGraphDoesNotHaveSelfLinksOrDuplicateEdges()
{
var items = SmallVectors().Take(100)
.Select((v, i) => (Id: (ulong)i, Vector: v))
.ToArray();
var graph = new SmallWorld<(ulong Id, float[] Vector), float>(
distance: (a, b) => CosineDistance.NonOptimized(a.Vector, b.Vector),
generator: new DeterministicGenerator(),
parameters: new()
);
graph.AddItems(items);
var underlyingGraph = (Graph<(ulong Id, float[] Vector), float>)graph.GetType()
.GetField("Graph", BindingFlags.Instance | BindingFlags.NonPublic)
.GetValue(graph);
List<(int From, int To, int Layer, string Type)> badEdges = new();
for (var i = 0; i < underlyingGraph.GraphCore.Nodes.Count; ++i)
{
var node = underlyingGraph.GraphCore.Nodes[i];
for (var l = 0; l <= node.MaxLayer; ++l)
{
var connections = node[l];
foreach (var group in connections.GroupBy(c => c))
{
if (group.Key == i) { badEdges.Add((i, group.Key, l, "self-link")); }
if (group.Count() > 1) { badEdges.Add((i, group.Key, l, "duplicate")); }
}
}
}
private IEnumerable<float[]> SmallVectors()
{
Random random = new(Seed: 1234);
while (true)
{
var vector = new float[10];
for (var j = 0; j < vector.Length; ++j)
{
vector[j] = random.NextSingle();
}
yield return vector;
}
}
internal sealed class DeterministicGenerator : IProvideRandomValues
{
private readonly Random _random = new(Seed: 12345);
public bool IsThreadSafe => false;
public int Next(int minValue, int maxValue) => this._random.Next(minValue, maxValue);
public float NextFloat() => this._random.NextSingle();
public void NextFloats(Span<float> buffer)
{
for (var i = 0; i < buffer.Length; ++i)
{
buffer[i] = this.NextFloat();
}
}
}
Big thanks for your implementation of HNSW. We have integrated this layer into DBreeze database . Now will be a ride of it. Here is documentation .
If you have some notes, please write me directly.
Hi
Do you have estimations for the performance of this code?
For example I ran it on GIST data (dim = 960, ef = 200, M = 32) and it takes about 3 hours to create the graph.
Maybe I am doing something wrong although the parameters are only few and there is no much place to do a mistake here.
not sure this is the righ tplace to submit the query but i will more very happy to hear from you
thanks
Samer
Hi,thank you for deploying the HNSW.Net library.
I try to build the program using HNSW.Net,but it has occured that namespace HnswIndex<> is not found error.
How do I fix this error ? Please teach me.
I don't know whether I have mistaken when installing the HNSW.net library package from Nuget.
Best Regards.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.