flepied / second-brain-agent Goto Github PK
View Code? Open in Web Editor NEW馃 Second Brain AI agent
License: GNU General Public License v3.0
second-brain-agent's People
Stargazers
Watchers
second-brain-agent's Issues
update to streamlit 1.25.0 or later
check if content has been modified before re-indexing
This will support export workflows from some editors like https://www.amplenote.com/ and it will also prevent re-indexing stuff that didn't change.
store dates about documents
store the creation date and updated date in the metadata.
Add links to the context documents when answering
Should display a list of sources (url from the metadata of the documents used as context).
add some tests
create tests that inject each document type and do some simple queries to validate they have been injected.
Display the question from the user
This has been lost with in 0.2.0.
Enhance the way documents are used in the dialogs
Make the agent able to exchange about documents like find documents in the knowledge base and act on them. Actions could be questions, summarizing or comparing.
use the metadata from the markdown file in the extracted documents.
Obsidian supports Markdown metadata, this way:
---
date: 2023/06/18 10:06
---
Use these fields in the metadata of the vector store. date will be transformed into created_at.
Implement semantic splitting
Split the notes using semantic knowledge. First semantic split could be for project or journal oriented notes. They are using an history format based on dates. Split these parts from the doc in their own documents with the date metadata corresponding to the extracted section.
create CI jobs
add a language metadata field
Support multiple languages in your data by detecting which language is used and storing it as the language metadata.
support deleting documents
Document Amplenote Workflow
If we can identify a workflow that works well with Amplenote, that will unlock this tool for Amplenote users.
requirements.txt
Hello, I'd like to open an issue and was wondering if it would be possible to provide the 'requirements.txt' file to help with setting up the environment. This would greatly assist in configuring the environment correctly. Thank you!
Use dates in the similarity search
Improve the scoring of the document retriever by using the updated dates of the document as a parameter for similarity.
Needs #8
use ChromaDB in client/server mode
create a Dockerfile and docker-compose.yml to start the ChromaDB in server mode using the same version used in the agent and inderxer
Update the requirements
Update pyproject.toml to the lastest available versions and see if the workarounds (constraints) are still needed.
no newline in content is causing a backtrace
Backtrace:
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: Storing files under /home/fred/.second-brain
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: Traceback (most recent call last):
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: File "/home/fred/perso/second-brain-agent/./transform_txt.py", line 106, in <module>
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: main(sys.argv[1], sys.argv[2])
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: File "/home/fred/perso/second-brain-agent/./transform_txt.py", line 101, in main
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: process_file(os.path.join(in_dir, entry.name), out_dir, indexer, splitter)
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: File "/home/fred/perso/second-brain-agent/./transform_txt.py", line 70, in process_file
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: for chunk in splitter.split_text(content):
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: File "/home/fred/perso/second-brain-agent/.venv/lib/python3.11/site-packages/langchain/text_splitter.py", line 531, in split_text
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: return split_text_on_tokens(text=text, tokenizer=tokenizer)
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: File "/home/fred/perso/second-brain-agent/.venv/lib/python3.11/site-packages/langchain/text_splitter.py", line 474, in split_text_on_tokens
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: input_ids = tokenizer.encode(text)
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: ^^^^^^^^^^^^^^^^^^^^^^
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: File "/home/fred/perso/second-brain-agent/.venv/lib/python3.11/site-packages/langchain/text_splitter.py", line 518, in _encode
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: return self._tokenizer.encode(
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: ^^^^^^^^^^^^^^^^^^^^^^^
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: File "/home/fred/perso/second-brain-agent/.venv/lib64/python3.11/site-packages/tiktoken/core.py", line 117, in encode
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: raise_disallowed_special_token(match.group())
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: File "/home/fred/perso/second-brain-agent/.venv/lib64/python3.11/site-packages/tiktoken/core.py", line 351, in raise_disallowed_special_token
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: raise ValueError(
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: If you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|endoftext|>', ...}`.
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: If you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|endoftext|>'})`.
ao没t 22 00:15:57 laptop-fred.local sba-txt-service.sh[2698822]: To disable this check for all special tokens, pass `disallowed_special=()`.
a
add support for podcast
lookup the transcript or create one from a service.
add support for video
when there is no transcript for a Youtube video or a video without any, use a service to build one.
add support for multiple sessions
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
馃枛 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 馃搳馃搱馃帀
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google 鉂わ笍 Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.