Hash table possibility instead a list about dialetus-service

[mateusduboli]

Hi there :)

When you're dealing with text search, it is more interesting to have not a hash table, but a prefix tree or a ngram tree, so you can execute partial searches on terms.

A trie is fairly simple to implement with only vanilla libs, but specializing it to ngrams may need some external libs or way more work.

Some inspirational examples:

https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-ngram-tokenizer.html

https://whoosh.readthedocs.io/en/latest/ngrams.html

One other thing is in regards to Data/Search index model. In most search engines, you have a clear separation between the data you store and the search indexes you use. So a good thing on this idea is to turn the Data model into a K/V store, but keep an index for it in another data structure, so that the data is easily addressable via Key, but searchable via a more elaborate search index :)

[mateusduboli]

I think the JSON file idea is not bad at all (see the last paragraph), as it enables a KV like semantics in data retrieval which is ideal. The tree index could be a different PR to implement text searches on the document. I can hack a bit on that as it was sort of a side project of mine to implement a simplistic database.

…

On 3 Jul 2019, at 14:17, Matheus Marsiglio ***@***.***> wrote: @mateusduboli You're absolutely right about using trees for search purposes, I was being simplistic in my solution and didn't consider a better searchable solution for looking for words. 🤦‍♂ Instead of having a user readable JSON file, we would design some logic to retrieve data from searched characters. It makes sense to me. Is this aligned with the project long run expectations @mvfsillva? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[mvfsillva]

Wow, that is amazing! I liked the proposal, thinking about scalability, long-term and even about other projects that may arise in the future. Please send a PR for the project!

[mtmr0x]

I'll work on that, but I'm kinda slow these times working in something quite complex. I will start writing the implementation and tech specs for it. 🎉

[mtmr0x]

@mateusduboli You're absolutely right about using trees for search purposes, I was being simplistic in my solution and didn't consider a better searchable solution for looking for words. 🤦‍♂

Instead of having a user readable JSON file, we would design some logic to retrieve data from searched characters. It makes sense to me. Is this aligned with the project long run expectations @mvfsillva?

[mateusduboli]

I have spent some time studying what lucene does internally to get some inspiration. This weekend I'll open a PR :)

…

On 30 Sep 2019, at 21:58, Marcus Silva ***@***.***> wrote: Hey, guys, @mtmr0x @mateusduboli let's do it \0/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[mvfsillva]

I found it super interesting, I did not know the ngram tree, I think it completely aligns with the expectation of the project.

If I understand this correctly it will improve the performance to look for words and also in the semantics of the data storage

[mvfsillva]

Hey, guys, @mtmr0x @mateusduboli let's do it \0/