flagro / projecttextagent

Updater of text contents of project files for ProjectTextSuite

🚀 ProjectTextAgent

Part of a ProjectText Suite. ProjectTextAgent is an observer for your project directory for updating the Text and Table databases for future question-answering pipelines.

🌟 Features

• Implements concurrent Goroutines watchers to send updates when files are modified/deleted/created
• On events extracts text/tables from files with TextTableSpoon
• Updates the data in Vector database (VecMetaQ) and simple relational database (PostgreSQL) with new texts

🚀 Getting Started

Make sure to have docker installed on your system and then simply copy and initialize the .env file and do a docker compose up:

mv .env-example .env
docker compose up

📘 Usage

Running the image makes VecMetaQ and PostgreSQL databases available at the hosts you specified in the .env file and updates these databases with text contents of files in directory you specified to enable Question-Answering pipelines over these texts with LLMs (check out https://github.com/Flagro/ProjectTextQnA).

🤝 Collaboration & Issues

Open for collaboration; check the issues page for discussions.

🌟 Shoutout to the dependencies

This project uses:

• https://github.com/fsnotify/fsnotify for file update events handling
• https://gorm.io - orm to connect ProjectTextAgent with postgres for for the flat tables manipulations
• https://github.com/Flagro/VecMetaQ - wrapper over vector database for easy sever-like interface for text embeddings
• https://github.com/Flagro/TextTableScoop - CLI file-to-text parser

projecttextagent's Issues

New env variables in VecMetaQ fix is needed

Postgres connector implementation needed

Parser file with JSON encapsulation is needed

This will ensure that we are not limit for JSON output of the parser and will encapsulate this logic in parser

Vecmetaq connector implementation needed

Error handlers for unstat files and better logging

• Solve the issue with temporary files processing (could be excel tmp files, could be vim buffer files, etc)
• Replace most of the fatal logging with simple prints

Golang modules implementation is needed

Docker configuration is needed

The project should run be able to run in a docker container for several reasons:

• Ease of use and installation
• Grant the users the possibility to guarantee the limitation on the internet access of the suite
• To utilize the docker's volume as temporary folder.

Project README TODO implementation is needed

Observer design is needed

VecMetaQ delete request

JSON body approach is more secure and stable compared to query strings, especially when we are dealing with file names