Larger k and generic alphabets in SSHash about sshash

[jermp]

Hi @adamant-pwn,
and thank you for reaching out! The generalization to arbitrary alphabets would be very interesting.

Would it be of interest for SSHash to merge these changes upstream in the future?

Yeah, why not? :) My focus would still be on indexing DNA, hence I would prefer the code to be optimized for this specific usecase rather than being general. Right now, there aren't specific optimizations when dealing with 2-bit alphabets.

Do you have any advice on how this should be approached? Does the outlined approach make sense to you?

It does make sense to me. In my experiments, I haven't seen much difference in query time when using k=31 and k=63 because I'm relying on built-in data types (uint64_t and uint128_t respectively). I suspect a user-defined struct with 64-bit words to be fairly slower than this approach, but it is needed to use k > 63.

I expect someone already implemented a generic kmer class in C++ that does so, but I never searched on GitHub for one.
Perhaps, you could first see if there is a good one out there.

It seemed to me that most of the stuff that heavily relies on the alphabet being ACGT is currently encapsulated in util.hpp, but there are also some methods in e.g. dictionary.cpp. Are there any other places where you rely on the alphabet being ACGT directly, rather than implicitly via calls to util:: or dictionary methods?

It is mostly in util.hpp but also in other place where, essentially, we process 1 symbol at a time and thus read 2 bits from a bitvector. See for example include/buckets.hpp and include/bit_vector_iterator.hpp. So making SSHash general would need to carefully review the entire codebase.