TextChunker doesn't handle Markdown Tables about kernel-memory

[dluc]

Tables in markdown need to be chunked in a single embedding, it doesn't make sense to split the content strictly based on token limit.

hi @Licantrop0, it's not that simple. What if a table is too big for the embedding model? I think a more advanced chunker would split by row, keeping the row header in each chunk. Even in this case a row might be too big, so there's need for more complex logic.

[Licantrop0]

I described how to approach the problem if the entire table doesn't fit the embedding model, down to the single cell.
The current chunking model makes table data completely unusable, as it breaks the structure and doesn't maintain context (the table headers).

It's also difficult with the current Memory APIs to get the nearby partitions like it was done in this old example: https://github.com/Azure-Samples/semantic-kernel-rag-chat/blob/cc51e164ac1e559e80437918c671ab6257e7c873/src/chapter2/Chapter2Function.cs#L45

For reference, the current approach is this:

kernel-memory/examples/207-dotnet-expanding-chunks-on-retrieval/Program.cs

Line 97 in 5c7f4ca

foreach (Citation.Partition partition in result.Partitions)

[dluc]

Thanks for the details. The existing chunker is a sample with its limitations, and we welcome improvements. The behavior with markdown file is a bare minimum implementation, and there are improvements that could be made with regards to tables, lists, headers etc. If someone wants to work on adding the feature described above or other improvements we'd be open to help with PR reviews.