Custom tokenizer about tantiny

[baygeldin]

Hey @morygonzalez, currently Tantiny does not support custom tokenizers. I had some ideas how to implement it, but it's a complex issue to tackle due to the fact that it requires extending behaviour in runtime which is not easy to do with Rust (let alone it's interaction with Ruby).

However, it seems that lidera is quite a useful project and it might make sense to just add a new tokenizer type to Tantiny that uses it. This is much easier than dealing with custom tokenizers. What do you think?

[morygonzalez]

@baygeldin Thank you! That's cool. I'm happy with your suggestion!!

[baygeldin]

Okay, I'll see what I can do, but probably after I deal with aggregations (or you can make a PR yourself if you want).

[morygonzalez]

I see. I'll try to make a Pull Request though I'm quite new to Rust then it'll take some time.