tessil / hat-trie Goto Github PK

Ho Tessil, I have a core problem when I use hat-trie. This is the core information:
get (this=) at /usr/local/include/c++/6.3.0/bits/unique_ptr.h:308

It happened in the array_hash.h file at line 880.

Looking forward to your reply.

Thanks

possible to rewrite in rustlang?

Hi,
I am author HArray
https://github.com/Bazist/HArray
KV storage that also kind of Trie structure.
High performance and variety functionality in 8k code lines.
I want to test under your dataset my structure, but here is 404 error
https://dumps.wikimedia.org/enwiki/20170320/

Could you help me, where a I can download the dataset ?
Thanks.

Hi! Does the hat-trie have to be string keys only? Would be really great to have something like arbitrary type composite keys along the lines of MultiIndex container http://www.boost.org/doc/libs/1_64_0/libs/multi_index/doc/tutorial/key_extraction.html#composite_keys

This is more a question, than a real issue.

I am wondering, what is the most efficient way to intersect the keys of two or more hat-tries.

My naive approach would be to iterate over the keys of the HAT-trie with the smallest size and probe the others for the key. e.g.:

std::set<std::string> intersect(std::vector<tsl::htrie_set<char>> operands) {
    // Argument shouldn't be copied or changed. Only for demonstration purposes.
    sort(operands.begin(), operands.end(), 
        [&](const auto & a, const auto & b) { 
            return a.size() < b.size(); 
    });
    auto iterated_operand = operands[0];
    operands.erase(0);
    std::set<std::string> result{};
    for(auto it = iterated_operand.begin(); it != iterated_operand.end(); ++it) {
        bool skip = false;
        for(const auto &probe_operand : operands)
            if (not probe_operand.count(it.key())){
                skip = true;
                break;
            }
        if (not skip) 
            result.insert(it.key());
    }
    return result;
}

#include
#include <windows.h>
#include
#include "tsl/htrie_map.h"
int main()
{
std::cout << "Hello World!\n";
}

tsl\array-hash\array_growth_policy.h(96,48): warning C4003: not enough arguments for function-like macro invocation 'max'
tsl\array-hash\array_growth_policy.h(96,48): error C2589: '(': illegal token on right side of '::'
tsl\array-hash\array_growth_policy.h(133): message : see reference to class template instantiation 'tsl::ah::power_of_two_growth_policy' being compiled
tsl\array-hash\array_growth_policy.h(185,61): warning C4003: not enough arguments for function-like macro invocation 'max'
tsl\array-hash\array_growth_policy.h(185,61): error C2589: '(': illegal token on right side of '::'
tsl\array-hash\array_growth_policy.h(192): message : see reference to class template instantiation 'tsl::ah::mod_growth_policy' being compiled
tsl\array-hash\array_growth_policy.h(185,61): error C2059: syntax error: '('

https://stackoverflow.com/questions/2789481/problem-calling-stdmax

maybe add macro NOMINMAX in CMakeLists.txt

cmake ..
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Boost: /usr/local/include (found suitable version "1.70.0", minimum required is "1.54.0") found components: unit_test_framework
-- Configuring done
-- Generating done
-- Build files have been written to:

Scanning dependenccmake ..
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Boost: /usr/local/include (found suitable version "1.70.0", minimum required is "1.54.0") found components: unit_test_framework
-- Configuring done
-- Generating done
ies of target tsl_hat_trie_tests
[ 25%] Building CXX object CMakeFiles/tsl_hat_trie_tests.dir/main.cpp.o
[ 50%] Building CXX object CMakeFiles/tsl_hat_trie_tests.dir/trie_map_tests.cpp.o
/trie_map_tests.cpp: In member function 'void test_htrie_map::test_empty_map::test_method()':
/trie_map_tests.cpp:739:22: error: '((void)& last +65)' may be used uninitialized in this function [-Werror=maybe-uninitialized]
739 | BOOST_AUTO_TEST_CASE(test_empty_map) {
| ^~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
make[2]: *** [CMakeFiles/tsl_hat_trie_tests.dir/build.make:76: CMakeFiles/tsl_hat_trie_tests.dir/trie_map_tests.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:96: CMakeFiles/tsl_hat_trie_tests.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

possible to provide golang binding for golang?

great work by the way, only lacking bindings for golang and rust and nodejs

when creating a tsl::htrie_map<char, bool> and trying to insert or emplace an element i'm getting the following error:

array_hash.h:870:14: error: non-const lvalue reference to type 'bool' cannot bind to a temporary of type
      'std::__1::vector<bool, std::__1::allocator<bool> >::reference' (aka '__bit_reference<std::__1::vector<bool, std::__1::allocator<bool> > >')
      return this->m_array_hash->m_values[value_position()];
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

probably this is due to the difference of implementation of a bitvector, see std::vector<bool>

possible to write this to store in disk format?

by the way, what's emplace? is there a replace function and how to use? thx

is this production ready?

excuse me,i am a beginner.
Can i use Chinese as key?

how do u iterate through hat trie?

When the burst-threshold is set to smaller than 65536, the hat-trie behaves normally, and memory usage is 32%. However, when set to 65536 or larger, it causes my process to coredump, with memory usage only at 24%. Why is that?

Hi Thibaut,

The recent implement of std::hash use CityHash for 64 bit platforms and will fallback to Murmur hash for 32 bit platforms. Could you update README.md to reflect the new change in std::hash?

IMHO users should use std::hash by default and try a better hash function if needed. From these benchmark results (https://github.com/hungptit/clhash/blob/convert.to.header.only.library/benchmark/results.svg) we can see that std::hash is pretty fast i.e about 0.5 CPU cycle per byte.

BTW, thank you very much for your work I have started to replace std::hash with hopscotch_map/set in all of my projects.

Regards,
Hung

Great project! I'd like to use it in cquery for faster global symbol search; is it possible to add substring search, ie, the search Foo matches the string void Foo()? equal_prefix_range comes close but will not match because of the preceding void.

(Even better would be a fuzzy search, ie, vF matches void Foo(), or a custom match function, but I suspect that's out of scope).

e.g.
https://github.com/kephir4eg/trie
and
https://github.com/rafaelkallis/adaptive-radix-tree

?

Hi!

It would be great to have a visualizer for VS.
Visualizers are very useful. I failed to write by my own, the structure is rather complicated.

Thanks for the excellent implementation. I'm thinking of using the hat-trie in my project where I need to serialize the built map and later deserialize it. I'm working with tens of millions of strings, so would prefer to deserialize from disk than rebuild the map each time from raw data. What's a good way to serialize/deserialize to/from disk?

Hello,

I just found your implementation of a HAT-tree map and already love it. I use strings as keys and structs as values, and it is faster than a std::map / std::unordered_map for my use case and more memory efficient. However, I noticed that the iterators are a lot bigger than that of a std::map: 72 bytes compared to just 8 bytes for an iterator of a std::map. My software needs to remember the locations of a ton of items, so the bigger the size of a single iterator, the more memory is consumed. With a std::map, I can instruct my software to remember the items using a vector of iterators. I can easily retrieve the keys using i->first and the values using i->second. When I use htrie_map, the vector of iterators grows substantially, because the individual iterators are so big. Is there a more memory efficient way to store a list of locations, which lets me retrieve both the key and the value? Are there more lightweight iterators? Or pointers to items of the htrie_map from which key and value can be reconstructed? Simple pointers to the values are not an option, because I also need the key.

Many thanks in advance,
Sebastian

possible to implement a lru cache example using this hat trie?

According to the comment in code:

A trie node should at least have one child, except if it has a value node then no child is a possibility.

I add following assert to iterator htrie_hash::erase(iterator pos):

if(pos.m_read_trie_node_value) {
    tsl_assert(pos.m_current_trie_node != nullptr && pos.m_current_trie_node->val_node() != nullptr);
    tsl_assert(pos.m_current_trie_node->nb_children() > 0); // here
    pos.m_current_trie_node->val_node().reset(nullptr);
    m_nb_elements--;        
    return next_pos;
}

And write a test case:

BOOST_AUTO_TEST_CASE(stest_empty_trie_node) {
	tsl::htrie_set<char> set = { "k1", "k2", "k3", "k4" };
	set.burst_threshold(4);
	set.insert("k");
	set.erase("k1");
	set.erase("k2");
	set.erase("k3");
	set.erase("k4");
	set.erase("k");
}

It cause the assert fail. Is it a bug?

By the way, code coverage testing will make unit tests more complete.
Thanks for your implement.

Does this implementation has wildcard search support using the advantages of a trie (So that performance may increase, not needing to iterate over whole string) ?

By wildcard I mean * and ? included pattern search.

Hi,

Another request :)

The case is as follows, again for filesystem trees.
Right now a ::equal_prefix_range call gives all items that match the given prefix. For for a prefx of /home/a/Downloads it would give back the following (highlighted):
/home/a/Downloads/
/home/a/File
/home/a/Downloads/File2
/home/a/Downloads/SomeDir/
/home/a/Downloads/SomeDir/File

In other terms, basically a recursive list of files and folders for a given path.
That is fine and mostly what you need. However, in my case i only want to have the direct child items of a given prefix (same prefix as before), so the following:
/home/a/Downloads/
/home/a/File
/home/a/Downloads/File2
/home/a/Downloads/SomeDir/
/home/a/Downloads/SomeDir/File

And for a prefix of /home/a i would only want the following as a result:
/home/a/Downloads/
/home/a/File
/home/a/Downloads/File2
/home/a/Downloads/SomeDir/
/home/a/Downloads/SomeDir/File

There should not be a problem for this in terms of the tree structure behind all of this. At the moment it's simply returning too much where the end iterator would have to be set back some position for the match i want. But how to do this in a clear API way seems like a challenge.

It is in fact rather easy to do this with the current code, but a bit wasteful as well. I could for instance iterate over the results and look for the first occurrence that starts with "/", has some text before the "/" and use that point as my end iterator.

But putting something like that in the API is... nasty.

Oh well, just putting it on here, perhaps you like the idea and want to implement it :)

Best regards,
Mark

It would be quite useful if it was possible not just get an iterator to the longest prefix, but also a path leading to that prefix from the shortest prefix. Such a method would return a vector of iterators pointing to values that matched the first characters of the longest prefix argument. In other words, if a map contained these values:

/A
/A/B/C
/A/B/D
/A/B/E

, and the longest prefix would be requested as /A/B/D/E/F, then in a loop similar to the ones in equal_prefix_range_impl and filter_prefix it would test if keys of nodes that have values would match the beginning of the prefix string, so in the example above, it would return a vector of iterators pointing to /A and /A/B/D.

It would allow many useful operations for parent paths, which now can only be done via multiple calls with prefixes of different lengths.

A special "parent iterator" hiding the parent path vector and overloading -- would probably fit nicely in the interface.

Sorry if there is a way to achieve what I described and I missed it.

Hi there, just a question, do you have a trie storing integer keys?
Thanks, - greg

any difference between c++ tessil vs this rustlang crate?
https://lib.rs/crates/hat_trie

any benchmarks?

get_hash_node_for_char(...) function
m_max_load_factor = 0, so nb_buckets = inf .

I tried inserting few entires in a htrie_map,
tsl::htrie_map<char , std::vector> map;

But then when i tried to serialize it using this,
serializer serial("file_name");
map.serialize(serial);
i couldnt serialize. It throws some error.
It would be great if you could help me with this.

There's a use cases that all values are in specific order. And I wan to extract values by prefix , while keep values in correct order.

Hi,

Imagine the case of inserting a whole filesystem structure in the hat-trie. You have folders with a bunch of files, more folders and more files. At the moment in the current code each entry is looking up the appropriate point to insert the entry. For example a folder structure like:

/home/a/Downloads
/home/a/Documents

the insertion logic has to iterate the tree to find the correct spot. That is fine for the first entry, but could be optimized for the second entry and specially for all entries after the second.

I'd like to propose a insert function with a "fixed prefix" that can then take a list (vector or map, depending on the type of container you used for the hat-trie) to insert a bunch of elements at a given prefix. For instance something like this:

tsl::htrie_set::insert("/home/a/", {"Downloads", "Documents"})

This in only really beneficial for filesystem like structures, not for wordbook purposes. But it would still be a neat addition imho.

Best regards,
Mark

Hi,

I admit, i haven't tried it. But looking at the code it seems to be doing case-sensitive searching.
Imagine i put a bunch of files paths in the trie with structures like:

/home/a/Desktop
/home/a/Downloads
/home/a/Pictures

I'm guessing a prefix search of:

tsl::htrie_set<char> set = {"/home/a/Desktop", "/home/a/Downloads", "/home/a/Pictures"};
set.equal_prefix_range("/home/a/d");

note the lowercase d in the equal_prefix_range call, it will probably not match "/home/a/Desktop" and "/home/a/Downloads" while that would be desirable. Just lowercase everything would fix this, but then the data does not represent the actual paths anymore (both pats could exists in a linux/unix world as it is case sensitive). Using a map and have a lowercase -> uppercase mapping could potentially be a solution as well, but that also doubles the data usage (at the very least) thus kinda defeating the point of using a nice HAT trie in the first place (in terms of memory efficiency).

I do realize that there is a bit of trouble in making the equal_prefix_range case-insensitive (ideally optional, case-sensitive and case-insensitive). You don't know the type i'm putting in as string. It might be a std::string, might be a std::string_view, a byte array, a QString.. You just don't know therefore can't expect a call on a character (like toLowercase() for example) to exist.

So i have a bit of a request. Could you add a function that allows me to set how to compare a prefix?

I'm thinking of a:

tsl::htrie_set::set_character_compare(...);

Then in the compare functions you use whatever is set by the user.

Again, i could be completely wrong and it's already possible, but it doesn't look like that from reading the code ;)

I'm curious about your opinion!

Best regards,
Mark

Hi, currently all insert() / emplace() functions take const CharT* key, size_type key_size parameters.

I have a use-case to allow inserting via iterators instead (I need to insert a std::string / std::string_view backwards) and have started to add these functions. However, it's quickly snowballing so before I continue I'm wondering if this is something you have tried yourself? If it's ultimately not possible I'd like to know! If it's worth pursuing, and contributing back to you, it will most likely end up with the engine of the library being written to use iterators and supplying std::string / std::string_view, {const Char*T, size_type} wrappers over these new functions. What do you think about this? Would you accept a PR for such a large refactor?

Hello —

It's all a bit daft, but the Microsoft Visual Studio compilers report __cplusplus as 199711L by default, even when using modern C++ standards. This behavior can be remedied using the /Zc:__cplusplus switch, or by detecting the _MSVC_LANG macro (whose value will be greater than or equal to __cplusplus and more representative of the standard used).

https://docs.microsoft.com/en-us/cpp/build/reference/zc-cplusplus?view=msvc-160

This would allow for use of string_view in a "blank" MSVC project.

https://github.com/Tessil/hat-trie/blob/master/include/tsl/htrie_hash.h

1. is load factor 16 or greater possible?
2. What's the maximum load factor possible? what does load factor mean?
3. Possible to have mem usage lower than the actual size of the data?

Hi,

I am trying to use the hat_trie in my project.
But I am facing an issue on serializing the following:

tsl::htrie_map<char,set<pair<ll,string>>> data[N];

Here, it's a multidimensional map trying to store char as key and set of pair of integers and strings as a value to it.
Can you help me out?

Hi,

I'm using the tsl::htrie_map for inserting DNA fixed-size strings as keys (4-bases ACGT).
Is there any option to iterate over sorted keys? I didn't find anything related to that in the documentations.

Thanks.

I am missing a common feature of Trie APIs: searching for the longest string in the trie which is a prefix of input.

Example in pseudo code:

hattriemap = {"a", "as", "asdf"}
hattriemap.longest_prefix("asd") -> "as"

Is this possible with hat-tries? If yes, it would be a great addition to your already great library.

Hi,
in my application, a crash caused probably by the trie appeared. CLion's debugger shows the following trace beggining in equal_prefix_range method:

After veeeeery long recursion (btw, is the trie implemented using recursion?!), EXC_BAD_ACCESS raised:

For my data, the error appears always. Is it possible that it is a bug inside the trie?
Thank you

Marek

Is it possible to implement weighting of nodes for accuracy in text prediction?

Not an issue but a question. Oh and thanks for releasing a really excellent project. Great code!

Ideally I'd like to iterate through the entire trie, say a few hundred keys at a time. However iterators are invalidated after a mutation, so I was thinking of just recording the last few keys one visited then pick up where you left off. What do you think? You might get unlucky and get your key deleted so I was thinking of keeping a random set from the last iteration. It doesn't matter for me if you visit a key twice.

Any other ideas?

Hi Tessil, when I use hat-trie for prefix matching, there is a core happens:
#0 get (this=) at /usr/local/include/c++/6.3.0/bits/unique_ptr.h:308
308 { return std::get<0>(_M_t); }
(gdb) bt
#0 get (this=) at /usr/local/include/c++/6.3.0/bits/unique_ptr.h:308
#1 operator bool (this=) at /usr/local/include/c++/6.3.0/bits/unique_ptr.h:322
#2 operator!=<tsl::detail_htrie_hash::htrie_hash<char, long unsigned int, str_hash, short unsigned int>::anode, std::default_delete<tsl::detail_htrie_hash::htrie_hash<char, long unsigned int, str_hash, short unsigned int>::anode> > (__x=...) at /usr/local/include/c++/6.3.0/bits/unique_ptr.h:674
#3 first_child (this=0x0) at ./comsearch-search/websearch/svr/suggest_server/com-sugdirect/suggest/hat-trie/htrie_hash.h:338
#4 tsl::detail_htrie_hash::htrie_hash<char, unsigned long, str_hash, unsigned short>::htrie_hash_iterator<true, true>::operator++ (this=0x7f5a757f8480,
this@entry=<error reading variable: Cannot access memory at address 0x7f5a757f8338>)
at ./comsearch-search/websearch/svr/suggest_server/com-sugdirect/suggest/hat-trie/htrie_hash.h:664

I don't know how to fix it. looking forward to your reply

Here's a minimally reproducible example that causes a crash:

#include <iostream>
#include <tsl/htrie_map.h>

int main() {
    tsl::htrie_map<char, std::string> data_map;
    std::cout << "Inserting items into map..." << std::endl;
    data_map.emplace("data", "foo");

    for(size_t i = 0; i < 30000; i++) {
        std::string key = "data." + std::to_string(i);
        data_map.insert(key, "foo");
    }

    size_t count = 0;

    std::cout << "Erase prefix..." << std::endl;
    data_map.erase_prefix("data.");

    std::cout << "Start iteration..." << std::endl;

    for(auto it = data_map.begin(); it != data_map.end(); it++) {
        count++;
    }

    std::cout << "End iteration, count: " << count << std::endl;
    return 0;
}