birdca / richer Goto Github PK

5_Levels_Of_Text_Splitting

• Level 1: Character Splitting - Simple static character chunks of data
• Level 2: Recursive Character Text Splitting - Recursive chunking based on a list of separators
• Level 3: Document Specific Splitting - Various chunking methods for different document types (PDF, Python, Markdown)
• Level 4: Semantic Splitting - Embedding walk based chunking
• Level 5: Agentic Splitting - Experimental method of splitting text with an agent-like system. Good for if you believe that token cost will trend to $0.00
• Bonus Level: Alternative Representation Chunking + Indexing - Derivative representations of your raw text that will aid in retrieval and indexing

POC features:
query 看結果 (api return 該集所有 metadata) 是不是我們想要的
新增 meta data(transcript stamp: hh:mm:ss ), 存回
query 看結果是不是我們想要的

Return:
result1: channel -> cover, channel name, author name
result2: episode -> episode name, description ...