Text highlighting: should we preserve density? about eli5

[kmike]

I don't see a big difference in these examples. For tokens it made it worse. For chars it probably made it more correct. I think it is a good option to have, +1 to add it and enable for char ngrams.

[kmike]

For char_wb and for token ngrams it is not clear what is better.

[lopuhin]

I agree that it's worse for tokens, but even for char ngrams, the main difference seems to be that bias (the largest feature) is less intense compared to shorter char n-grams, so they look brighter (about the same brightness as bias).
I'd rather have it off by default in all cases, but I'm not entirely sure.
One more questions: I just realised that currently weights that are displayed on mouse over in title are different depending on the preserve_density option value, that is clearly wrong, we should have original weights in title, right?

[kmike]

Let's take 'software' token in the following text as an example:

hi there, i am here looking for some help. my friend is a interior decor designer. he is from thailand. he is trying to find some graphics software on pc. any suggestion on which software to buy,where to buy and how much it costs ? he likes the most sophisticated software(the more features it has,the better)

In char-based example it is light-green. The total weight it gives to the result is quite large though: about 0.1 (a bit less for some of them) for each of char 3- and 4-grams (' sof', 'sof', 'soft', 'oft', 'oftw', 'ftw', 'ftwa', 'twa', 'twar', 'war', 'ware', 'are'); it sums up to a value larger than BIAS. But bias is much brighter here:

It has L value which is 2x brighter than the brightest part of 'software'. In token-based example 'software' is bright-green because it gets a single large weight.

[lopuhin]

Indeed, you are right, I see the point now!

[lopuhin]

One more questions: I just realised that currently weights that are displayed on mouse over in title are different depending on the preserve_density option value, that is clearly wrong, we should have original weights in title, right?

After thinking about it more, I think we should display the same weight in title as we use for highlighting, especially in the light of your last example.

[lopuhin]

Fixed in #32