merging maps about parallel-hashmap

[gf777]

terrific! It seems to be working well
Thank you!

[greg7mdp]

Hi @gf777 ,
Thanks for the kind workds, and also for using phmap.

You have the right idea, and indeed for your purpose this would work very well. The two tables to merge would have to have the same N template parameter (which is 4 by default), so that the two maps to merge have the same number of submaps, and the values are inserted into the submaps with the same index.

So say you are using N=4 and therefore you have 16 submaps, you could create 16 threads and have each thread merge the two matching submaps (with the same index). To access the submaps, you can use the get_inner function:

auto& inner = map1.get_inner(0);   // to retrieve the submap at index 0
auto& submap1 = inner.set_;        // can be a set or a map, depending on the type of map1

[gf777]

Thanks for the prompt feedback @greg7mdp
Dumb question then, how do I achieve this:

    auto& inner = map1->get_inner(subMapIndex);   // to retrieve the submap at given index
    auto& submap1 = inner.set_;        // can be a set or a map, depending on the type of map1
    inner = map2->get_inner(subMapIndex);
    auto& submap2 = inner.set_;
    
    for (auto pair : submap1) { // for each element in map1, find it in map2 and increase its value
        
        DBGkmer &dbgkmerMap = submap2[pair.first]; // insert or find this kmer in the hash table

Error:
src//kreeq.cpp:944:38: error: type 'EmbeddedSet' (aka 'raw_hash_set<phmap::priv::FlatHashMapPolicy<unsigned long long, DBGkmer>, phmap::Hash<uint64_t>, phmap::EqualTo, std::allocator<std::pair<const unsigned long long, DBGkmer>>>') does not provide a subscript operator

I tried a number of things with .insert and .find but to no avail..
Thank you for the help!

[gf777]

    auto& inner = map1->get_inner(subMapIndex);   // to retrieve the submap at given index
    auto& submap1 = inner.set_;        // can be a set or a map, depending on the type of map1
    auto& inner2 = map2->get_inner(subMapIndex);
    auto& submap2 = inner2.set_;
    
    for (auto pair : submap1) { // for each element in map1, find it in map2 and increase its value
        
        auto got = submap2.find(pair.first); // insert or find this kmer in the hash table
        if (got == submap2.end()){
            submap2.insert(pair);
        }else{

            DBGkmer& dbgkmerMap = got->second;

Apparently this works (verbose though)

[gf777]

One more question: if I now want to increase the number of maps in the template, say when I declare it here:
std::vector<phmap::parallel_flat_hash_map<uint64_t, VALUE>*> maps; // all hash maps where VALUES are stored
Is there an easy way of declaring it ?

[greg7mdp]

Hum, did you find the answer to your question? I was looking in Kreeq and I see that you figured out how to declare parallelMap with 256 submaps (using N=8), and also your code for merging the submaps looks good.

Let me know if you have any other question, I'm happy to help.

[gf777]

This seems to work like a charm, thanks! I would go as far as saying that it should be a core function XD
While I have your attention I could pick your brain on how to best estimate map size in constant time, currently I have this from another project:

uint64_t mapSize(parallelMap& m) {
    
    return (m.size() * (sizeof(DBGkmer) + sizeof(void*)) + // data list
     m.bucket_count() * (sizeof(void*) + sizeof(uint64_t))) // bucket index
    * 1.3; // estimated allocation overheads
    
}

I noticed you have some calculations in the readme as well. Thoughts?

[greg7mdp]

Correct size is:

uint64_t mapSize(parallelMap& m) {
   return m.capacity() * (sizeof(parallelMap::value_type) + 1) + sizeof(parallelMap);
}

Map is in essence an array of values + an array of bytes (hence the + 1). I think the above is pretty accurate.