As title, create a docker image for MeMaS so that it can easily be deployed.

Currently even when we raise exceptions in MeMaS, they end up as 500 Internal Server errors.

We need to properly implement and refactor exceptions, so they produce proper error codes as well as descriptive error infos.

This will also include API adjustments.

TRACKS: #20, #21

The core memas server does not contain any authentication, it trusts whoever the requester claims they are. (The namespace_pathname aka user_name)

This is mostly intended for now; MeMaS is not a user facing application, and we leave it up to developers to set up authentication for now.

However, we do need to implement/enable some basic authentications:

1. Make MeMaS support API key authentication.
2. Add support for authenticating to our dependency services (ES/cassandra/milvus). Right now we have very sus configs like xpack.security.enabled=false lol

TRACKS: #23, #24

As the default behavior, we'd like to enforce that CP APIs can only modify nodes below the user, such as user1 can have control over user1.A.B:C, but not over user2:X

Currently there are concerns of MeMaS's ability to handle large corpuses, such as the milvus file index size https://milvus.io/docs/v1.1.0/storage_concept.md

Investigate this issue through tests and benchmarking, possibly stress test each component (ES/Milvus) separately.

Create a github workflow so that integration tests are ran for each pull request. This will be similar to how we run unit tests right now with https://github.com/memas-ai/MeMaS/blob/main/.github/workflows/python-app.yml, but need to do research on how to run docker images

Currently we only implemented 2 CP APIs:

• create_user
• create_corpus

We need to implement the following sets of APIs:

• batch_memorize
• Various delete APIs
  • delete_corpus
  • delete_user
  • Maybe also delete_memory, TBD
• a namespace navigation API, similar to ls

And there are many more candidates, these are all up for discussion/debate, such as:

• update_corpus_configs
• copy_corpus
• export_corpus/import_corpus

TRACKS: #16, #17, #6

Implement delete_user CP API.

Simple as it sounds, this is not a straightforward implementation. Deleting a user will involve deleting all corpuses underneath them. It should also fail when they still have users underneath them.

We need to implement some sort task queue mechanism, where we delete the corpuses in the background. Also be wary of situations where a deletion is halted midway, we need to be able to restart the deletion and clean things up

Currently due to prioritizing progress, we didn't implement integration tests for the top level APIs. (I swear it's not as bad as it sounds, since we implemented integration tests for everything below the top layer...)

Implement integration tests for the following APIs:

• memorize
• recall
• create_user
• create_corpus
  And whichever other untested API exists at the time of tackling this issue. We will mandate integration tests moving forward :(

We've identified the remember operation to be really slow with large documents. Speed this up through batching the many operations

Integrate MeMaS both as memory and a retriever

Need to work with LangChain to get approval before hand, as well as confirm this form of integration

Explore and Design how memas users can host prepopulated corpuses. Would prefer platforms like hugging face, where we can use existing platforms and not worry about data storage/hosting issues.

We may need to look at creating our own index, akin to package indexes like pypi. Avoid if possible for now...

DEPENDS ON: #18

We need to populate memas.ai to host documentation as well as a simple intro.

This will include:

• API documentation (generate using swagger from api package)
• General documentation and FAQ
• Creating a webpage to host the above information
• And more demos

Implement delete_corpus CP API.

Simple as it sounds, this is not a straightforward implementation. Deleting a corpus will be slow and prone to interruptions.

We need to devise a plan on how to make sure data isn't orphaned, and how to resume deletion when interruptions happen.

Integrate with AutoGPT as a memory backend.

This maybe on hold due to Significant-Gravitas/AutoGPT#4280.

I would personally try advocating for a generic extendable memory interface, where AutoGPT expects simple interfaces like memorize and recall, and then it is up to people to create third party drivers. This should alleviate the stress of maintenance from AutoGPT's side, since the simpler the interface, the less likely things break.

Currently even when we raise exceptions in MeMaS, they end up as 500 Internal Server errors.

Design/Implement/Refactor proper Exception Handling, so that Exceptions CAN end up as Errors with proper error codes and descriptive error messages. This Story only includes the infrastructure/framework changes, actual error type refactors will be handled in another Story.

Standard ways that other flask projects do this is preferred.

Integrate MeMaS as a data store for LlamaIndex. MeMaS would be a very unique LlamaIndex data store, since it doesn't fall under any of the existing catagories: Document Store, Vector Store, and Index Store. Instead we would need to work with LlamaIndex to create a new category for us, something like a Memory Store.

Implement the batch_memorize api for memas. This will be used to mostly for mass import scenarios, like populating knowledge corpora.

The json payload for the api call will most likely look something like this:

{
    "corpus_pathname": "aaa.bbb:ccc",
    "cited_documents": [
        {
            "document": "...",
            "source_uri": "...",
            ...
        },
        {
            "document": "...",
            "source_uri": "...",
            ...
        }
    ]
}

For now, a basic loop and insert on the server side should suffice. Feel free to go above and beyond implementing a batch insert on the corpus level as well.

Similar to how people share training datasets for machine learning, we wish memas users can share commonly used knowledge corpuses.

The most primitive way is to provide a raw dataset, and have people run scripts to memorize all of it. Whilst feasible, this sounds highly inefficient.

Research, design, and implement a mechanism to prepopulate corpuses with a large set of knowledge.

Knowledge Corpuses right now need a number of improvements to be usable:

1. Speed is too slow
2. Implement shared corpus needed infrastructure
3. Size issues

Here are some of my initial thoughts on these directions.

Speed

We need to find ways to increase insertion speed so the full wikipedia can be inserted in hours not days. This will likely be a combination of:

• Batch API
• Parallel and more resource intensive solutions, like map reduce scripts and scaling up dependencies. These are not ideal, but still a reasonable solution
• Preprocessed corpuses or other ways we can import full corpuses

It'd be great if we can figure out a way to package and import an entire (preprocessed) corpus. Two major concerns with this are:

1. How much improvement can we get? I imagine it mostly cuts down on embedding time + memas processing logic time (which isn't much?).
2. How can we make it not exploitable? This may become a huge security risk

Shared Corpus Infrastructure

We want people to be able to freely and easily share knowledge corpuses, so that people can utilize common datasets similar to training datasets. Sharing corpuses is also essential to keep memas deployment size in control; we'd like to avoid extremes like needing hundreds of GBs per every single user.

Community shared corpuses will need a number of improvements in place, such as:

• Speed and preprocessed corpuses
• multicorpus search. In order to fully utilize shared corpuses, we need to be able to recall from multiple corpuses at once.
• A community hub. How much can we use existing platforms like hugging face?
• Community safety and trust mechanisms, maybe like upvoting useful knowledge sets, reporting malicious/problematic ones, etc.
• Potentially an export_corpus CP API.

Size issues

We need to investigate and alleviate most size issues, such as the milvus file index size https://milvus.io/docs/v1.1.0/storage_concept.md

We may also need to look into further sharding/splitting corpus storage. Currently across services like ES/Milvus/Cassandra, we fit all corpuses into a single index/collection/table. This is because most of these services do not scale well with these top level storage units. E.g. cassandra has a hard limit of about 1000 tables. This however creates noisy neighbor problems as we have more corpuses/bigger corpuses. So we may need to look at pushing these table number limits through sharding corpuses across tables etc.

TRACKS: #14, #15, #18, #22, #8

When using recall, we want to search multiple corpuses together. This is important for sharing/reusing corpuses; being able to search multiple corpuses together encourages users to create modular knowledge corpuses.

Currently we use the most suspicious settings when connecting to our dependency services like ES/scyalla/milvus, like xpack.security.enabled=false lol

Add support for properly authenticating to our dependency services, where users that deploy MeMaS can configure authentication to each of its dependencies (and MeMaS will properly pass on these dependencies).

We want to integrate with opensource projects such as:

• LangChain
• AutoGPT
• LlamaIndex

This may only be integrating the DP APIs, or also CP APIs. Up for discussion.

TRACKS: #26, #27, #28

The core memas server does not contain any authentication right now, it trusts whoever the requester claims they are. (The namespace_pathname aka user name)

However, even if we don't authenticate caller namespace_pathname, we should at least authenticate users so we know they are allowed to access MeMaS, not some random person on the internet.

Hence, we'd like to enable API Token Authentication for MeMaS, to grant blanket access to those that are allowed to call MeMaS APIs.

(It'd be great if we have separated chatbot API token and admin API token, but this is a plus.)

As the default behavior, we'd like to enforce the following:

• CP APIs can only modify nodes below the user, such as user1 can have control over user1.A.B:C, but not over user2:X
• DP API memorize can only write to corpuses directly below the user (? To be debated if this is necessary)
• DP API recall can read only from corpuses that belong to the user, or belong to direct ancestors. Such as companyA.team1.user1 can read from the corpus companyA:C, but not companyA.team2:X. (This is for sharing corpuses at a higher level)

These default behaviors are based on the structure of the namespace, and should be intuitive/natural in most cases.

Note that this is also a temporary implementation. In the future we will be doing policy/role based authorization, in which case even though we likely want to maintain these default behaviors, we would implement them as policy/roles.

TRACKS: #13

After the exception handling mechanism is in place:

• Refactor existing exception types to have appropriate REST error codes
• Refactor API specification and clients as well

DEPENDS ON: #20

Currently MeMaS does not check if the claimed user is authorized to do X. Even though we trust their claim of who the user is, we still need to verify if they have the rights/permissions to perform the action they request.

This at its early stages involve:

1. CP APIs can only modify nodes below the user, such as user1 can have control over user1.A.B:C, but not over user2.X
2. DP API memorize can only write to corpuses directly below the user (? To be debated if this is necessary)
3. DP API recall can read only from corpuses that belong to the user, or belong to direct ancestors. Such as companyA.team1.user1 can read from the corpus companyA:C, but not companyA.team2:X.
4. Enforce read/write permissions

TRACKS: #19, #25

Currently while we store some basic read/write permissions on a corpus level, this is not enforced, and the API doesn't allow specifying custom read write permissions.

For this story do the following:

• Add proper permission checks
• Add configurable permissions when calling create_corpus

NOTE: these permissions only apply to DP APIs