dadhi / imtools Goto Github PK
View Code? Open in Web Editor NEWFast and memory-efficient immutable collections and helper data structures
License: MIT License
Fast and memory-efficient immutable collections and helper data structures
License: MIT License
KV is not .NET/BCL. Even for newest nugets for .NET do not do such things. How would enterprise developer add reference to this assembly?
To improve the extensibility I have introduced the base GetOrAddEntry
and ReplaceEntry
methods.
Another important thing is to inversion the control, instead of passing the delegates to operate on the particular type of Entry (either Single entry or Conflicting Hash entry) the GetOrAddEntry
is returning the entry, then it can be updated or added based on the type on the consumer side, and then passed to the ReplaceEntry
.
It is much harder to work with old dotnet for me as VS Code does not runs package.config manually. Plus config and csproj files do conflict in old projects. Plus new projects may build nuget out of csproj.
Did some view onto, hope you find some useful.
ImTools/src/ImTools/ImTools.cs
Line 534 in 56b7f53
Also it is public, it may deliver some caveats to possible users:
ImTools/src/ImTools/ImTools.cs
Line 524 in 56b7f53
ImTools/src/ImTools/ImTools.cs
Line 489 in 56b7f53
reference equals
instead of equals
(to be crystal clear).ImTools/src/ImTools/ImTools.cs
Line 497 in 56b7f53
ImTools/src/ImTools/ImTools.cs
Line 498 in 56b7f53
==
https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/operators/equality-comparison-operator. Same seems here ImTools/src/ImTools/ImTools.cs
Line 531 in 56b7f53
public static ImMap<T> GetOrAdd<T>(
this ImMap<T> map,
int key, out T value
) where T : class, new()
{}
The BuildFromDifferent
is creating the final form of the map without the temporary allocated data structures.
The array will represent a root of data structure to do a initial lookup by hash & (arrLength - 1)
where arrLength
is power of 2.
The array item would be an Im(Hash)Map tree.
Space saving here:
The idea is for the small map we don't need to iterate on nodes and waste the memory but rather sort the input items and map them to the concrete ImMap representation right away. It will be big win for the small maps.
Somehow related to #41
Hello,
for my specific project I should be able to use int
type as key, therefore I decided to compare ImHashMap and MemoryOwner and what I got surprised me:
BenchmarkProject.zip
I was expecting MemoryOwner to be faster and better in terms of allocation, but I was not prepared for such big difference. So, I'm wondering if I have used ImTools library in the wrong way.
I feel like just using what seems to be the obvious choice is not a guarantee that could be the right choice. So I would like to know what a very skilled and talented developer like you would choose based on the following:
I initially tried with ImHashMap<Type, Type>
and I was surprised on how ImTools performed compared to other solutions. No mapping needed, no source generators needed and no waste of memory. However when I tried to use int keys, I saw a big increase in performance which became gigantic with MemoryOwner and now I'm in doubt on what could be the best solution. I feel like they boths have their pro and con which I'm unable to estimate in order to take an informed decision.
Your help would be really appreciated.
Thanks
Related #18
Here is the existing implementation to copy and adapt Ariadne
Test and verify via RelaSharp.
Here is the Fibonacci Hashing reveal to fit hash into smaller space, instead of current modulo approach:
https://probablydance.com/2018/06/16/fibonacci-hashing-the-optimization-that-the-world-forgot-or-a-better-alternative-to-integer-modulo/
As I understand ArrayTools consider arrays as immutable(while .NET devs have build Spans for arrays to be really immutable), may try to apply
ImTools/src/ImTools/ImTools.cs
Line 48 in 56b7f53
Would you accept migration to Span in some future version?
Methods to work with immutable arrays
replace with Methods to work with arrays as immutable
ImTools/src/ImTools/ImTools.cs
Line 47 in 56b7f53
Consider something existing which on arrays:
[ReliabilityContract(Consistency.WillNotCorruptState, Cer.MayFail)]
public static int BinarySearch<T>(T[] array, int index, int length, T value, IComparer<T> comparer);
[ReliabilityContract(Consistency.WillNotCorruptState, Cer.MayFail)]
[SecuritySafeCritical]
public static int LastIndexOf(Array array, object value, int startIndex, int count);
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public ReadOnlyMemory<T> Slice(int start)
It will save us some checks and will improve the performance in case we know that the key is in the map. And it is not a rare situation because the map is immutable and no one can remove the key without us noticing.
Related: dadhi/DryIoc#322
struct ImMapArray<V, Config> where Config : struct, IImArrayMapSlotConfig
{
public bool TryFind(int key, out V value) =>
Slots[key & default(Config).SlotMask].TryFind(key & default(Config).KeyMask, out value);
}
struct DefaultConfig : IImArrayMapSlotConfig
{
public int SlotCount => 32;
public int SlotMask => 31;
public int KeyMask => ~31;
}
Example of ToMermaidString<K, V>(this ImHashMap<K, V> map)
output:
graph TD
L5_-2122864884(`-2122864884`,`2`,`b`/`-2106679681`,`12`,`12`/`-1525889598`,`18`,`18`/`-1333024111`,`20`,`20`/`-1103748092`,`14`,`14`)
L5P_-2025051795>`-2025051795`,`22`,`22`]-->L5_-2122864884
L2_-756060967(`-756060967`,`7`,`7`/`-697679721`,`17`,`17`)
B2_-897298592[`-897298592`,`4`,`d`]-->L5P_-2025051795
B2_-897298592-->L2_-756060967
L5_-355422310(`-355422310`,`15`,`15`/`-159924239`,`11`,`11`/`-16327267`,`16`,`16`/`561526489`,`1`,`a`/`843106263`,`13`,`13`)
L5P_709486099>`709486099`,`23`,`23`]-->L5_-355422310
L5_1127870384(`1127870384`,`5`,`e`/`1319350685`,`19`,`19`/`1759459841`,`10`,`10`/`2002406165`,`9`,`9`/`2030411590`,`3`,`c`)
L5P_1959121513>`1959121513`,`21`,`21`]-->L5_1127870384
B2_1063643491[`1063643491`,`6`,`6`]-->L5P_709486099
B2_1063643491-->L5P_1959121513
B2_-377816965[`-377816965`,`8`,`8`]-->B2_-897298592
B2_-377816965-->B2_1063643491
Split from #12
Split from #7
Likely will fork https://github.com/denis-prodan/Switch-analyzer/blob/master/SwitchAnalyzer.Test/InterfaceSwitchAnalyzerTests.cs
For instance we can match over array of union vs F# sum-type.
Maybe try SOA UnionList as well.
Because we are creating useful tools to handle immutable values.
interface ILens<T, P>
{
void Set(T whole, P part);
P Get(T whole);
}
From the gist https://gist.github.com/dadhi/815cf2ef8e88b555c59be7e4be93e712
Develop roslyn analyzer to notify about missing switch cases, like this: https://github.com/dadhi/SwitchEnumAnalyzer
What we often need is invoking some method or delegate for each map element. We don't need a separate data structure for this - we may just pass the delegate to the iteration method. The cast of delegate creation usually is much smaller than support for IEnumerable/IEnumerator
implementation.
Include into ArrayTools faster analogs of Where, Select, FirstOrDefault for starter.
The list of opportunities:
_data
, benchmark vs inlined, compacted hierarchyEquals
and GetHashCode
. Select the right way.GetHashCode
Currently it uses HammingWeight algorithm for calculating number of set bits in a int value:
https://bitbucket.org/dadhi/dryioc/src/e788ffce78b727b35fda5355a2297d40f0e7731b/Net45/Playground/HashArrayMappedTrieTests.cs#lines-190
Here is the thing Aye, that's there System.Runtime.Intrinsics.X86.PopCount(uint/ulong)
First step will to BDN benchmark against Swar or HammingWeight with PopCount.
The newly added maps have a better performance and memory consumption than the V2 maps, so they are replacing the V2 maps and dropping the 23
suffix.
To efficoently represent array or list of Union
in memory.
May be revive ImArray
Should emphase that the class of union and case is just a wrapper, holder of the nested interface.
Should consider named and unnamed union.
Should consider the ergonomics of the fact that types are declared once but used often: names of classes in declaration may be longer, but names of implementation nested things shorter.
Should consider how is the final stored value access looks like:
case of<A> a: a.val
Experimental.ImMap234
name2-3-4 tree is a self-balanced tree with the much simplier and straightforward balancing comparing to AVL rotations (IMHO).
The plain 2, 3 leaf nodes and 3 branch node compared to AVL binary only branches promises the menory savings.
In addition I may apply some of the technics I did in AVL Experimental.ImMap
for further savings and speedup:
Prevent unnecessary temp node creation and construct the final structure. Compared to AVL, t234 has only split-3-leaf/branch case to consider.
Flatten the lower level structure to save space, e.g. Branch2 with the Leaf branches can flattened to the special 4, 5, 6, 7 leafs nodes. We can start with 4 and 5 leaf nodes because the addition won't split them, so we keep current split approach intact.
Moreover we can hold a single shape for 4 leafs - right-leaning, and hold the centralized shape for 5 leafs. This way the nodes will be locally rebalanced minimizing the braching split!
Here is the current wip tree structure, then I will describe the adjustments for the point 2.
class ImMap
{
static ImMap Empty = new ImMap();
class Entry : ImMap {}
class Leaf2 : ImMap {}
class Leaf3 : Leaf2 {}
class Branch2 : ImMap {}
class Branch3 : Branch2 {}
}
Adjustments:
class ImMap
{
static ImMap Empty = new ImMap();
class Entry : ImMap {}
class Leaf2 : ImMap {}
// maybe not need for virtual Leaf lookup *
class Leaf3 : Leaf2 {}
// don't inherit Leaf3 because it is checked for split
class Branch2Leaf4 : ImMap {}
class Branch2Leaf5 : Branch2Leaf4 {}
class Branch2 : ImMap {}
class Branch3 : Branch2 {}
}
*
- Regarding Lookup we may combine iteratition for branches and virtual call (polymorphism) for Leafs and BranchLeafs. But it should be proved by benchmark. So the Leaf3 : Leaf2
won't be needed to speedup lookup, likely it would be faster if the inheritance for Leaf3 is cut to ImMap.
I want a fast, configurable for the small sizes and less-allocating variant of HashMap in comparison with Dictionary and DictionarySlim.
Alternative implementation to look for and compare:
Tasks:
Results:
SmallMap
hash tableCurrent simpton of trying to use ImTool with other projects that encapsulates this library is impossible.
It may sounds strange, but makes sense when DryIoc and ImTools itself is on different upgrade path in a project. The container will always be in lower version due to safety and compatibility, while in other parts of the project the new functionality of the ImTools could be benefical.
I would suggest using compile macro usage to differentiate the base namespace between embedded and normal usage of the library, thus making it easy for everyone.
Any response is appreciated. Thanks!
Example:
class X : union<X, A.value, B.value> {}
class A : @case<A> {}
class B : @case<B, string> {}
The Branch2 thing at the right with small right leaf but full left leaf:
graph TD
L5_21030449(`21030449`,`21637`,`21637`/`131778175`,`24995`,`24995`/`221540030`,`53169`,`53169`/`507310657`,`41691`,`41691`/`631038222`,`80976`,`80976`)
L5P_364635560>`364635560`,`94562`,`94562`]-->L5_21030449
L5_761574385(`761574385`,`33461`,`33461`/`1049478692`,`15842`,`15842`/`1278461947`,`61688`,`61688`/`1593872025`,`54708`,`54708`/`2037997398`,`34187`,`34187`)
L5P_848679699>`848679699`,`7046`,`7046`]-->L5_761574385
L5PP_752405386>`752405386`,`56521`,`56521`]-->L5P_848679699
B2_648516453[`648516453`,`21318`,`21318`]-->L5P_364635560
B2_648516453-->L5PP_752405386
Pros:
The challenge is accessing the field fast by index from 1 to 16.
Use switch?
Or use Sequential layout and span like access?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.