atisharma / llama_farm Goto Github PK

Hi,

first of all thank for your great project! Whenever I invoke "/ingest" and regardless of the given input (urls, filnames, dir) it throws an AttributeError, e.g. /ingest "text.txt" will throw

AttributeError: 'list' object has no attribute 'decode'

The trace is barely readable so could you please check if you are having the same issue?
Am I doing something wrong here?

FAISS works well. The Chroma db support should be tested or should be removed. If kept, an options should be presented in the config.

Some ideas are already in tasks.hy, but it's a quite a hard problem. Other projects are attempting it. Perhaps it should be out of scope?

Langchain is a quite large and sprawling dependency. It's currently used for:

• document loading (wrapping unstructured)
• document splitting (the document splitters are pretty good)
• wrapping the Chroma and FAISS dbs.

It's simple enough to replace the db wrappers and the loaders could use Unstructured directly, but I'm unsure about the document splitters.

tools.hy sort-of-works but is unreliable and could offer something like a plugin system. If it could be made more reliable and performant then it should be made available in the regular chat.