[Feature Request] Should line endings be removed by decoders? about kernel-memory

[dluc]

Removing line endings is a tricky step because it affects the semantics. From a high level perspective I usually leave all the line endings in place, until the moment of calculating embeddings and saving the text chunk. I would avoid removing line endings while chunking sentences because it might join paragraphs together. Then there's the scenario where one might want to use "the chunk before" and "the chunk after", merged with the current chunk. In this case it would be better having the original line endings.

I'm a bit surprised that cosine similarity is affected so much by this detail to be honest, and I would not jump the gun without a thorough investigation with a considerable amount of test cases and a report comparing results. I'm worried we could make things worse overall.

There's also another way to look at it if you want: for each pieace of text we could have multiple versions (with line endings, without, etc), and calculate embeddings for all. It would increase cost, but could make things better for everyone.

[marcominerva]

I understand your point, so I think that the correct approach could be the feature I have discussed in #379, so everyone can makes the tests for a specific use case.