Kernel Memory (KM) is a multi-modal AI Service specialized in the efficient indexing of datasets through custom continuous data hybrid pipelines, with support for Retrieval Augmented Generation (RAG), synthetic memory, prompt engineering, and custom semantic memory processing.
KM includes a GPT Plugin, web clients, a .NET library for embedded applications, and as a Docker container.
Utilizing advanced embeddings and LLMs, the system enables Natural Language querying for obtaining answers from the indexed data, complete with citations and links to the original sources.
Designed for seamless integration as a Plugin with Semantic Kernel, Microsoft Copilot and ChatGPT, Kernel Memory enhances data-driven features in applications built for most popular AI platforms.
This repository presents best practices and a reference architecture for memory in specific AI and LLMs application scenarios. Please note that the provided code serves as a demonstration and is not an officially supported Microsoft offering.
Semantic Memory (SM) is a library for C#, Python, and Java that wraps direct calls to databases and supports vector search. It was developed as part of the Semantic Kernel (SK) project and serves as the first public iteration of long-term memory. The core library is maintained in three languages, while the list of supported storage engines (known as "connectors") varies across languages.
Kernel Memory (KM) is a service built on the feedback received and lessons learned from developing Semantic Kernel (SK) and Semantic Memory (SM). It provides several features that would otherwise have to be developed manually, such as storing files, extracting text from files, providing a framework to secure users' data, etc. The KM codebase is entirely in .NET, which eliminates the need to write and maintain features in multiple languages. As a service, KM can be used from any language, tool, or platform, e.g. browser extensions and ChatGPT assistants.
Here's a few notable differences:
| Feature | Semantic Memory | Kernel Memory |
|---|---|---|
| Data formats | Text only | Web pages, PDF, Images, Word, PowerPoint, Excel, Markdown, Text, JSON, more being added |
| Search | Cosine similarity | Cosine similarity, Hybrid search with filters, AND/OR conditions |
| Language support | C#, Python, Java | Any language, command line tools, browser extensions, low-code/no-code apps, chatbots, assistants, etc. |
| Storage engines | Azure AI Search, Chroma, DuckDB, Kusto, Milvus, MongoDB, Pinecone, Postgres, Qdrant, Redis, SQLite, Weaviate | Azure AI Search, Elasticsearch, MongoDB Atlas, Postgres, Qdrant, Redis, SQL Server, In memory KNN, On disk KNN. In progress: Azure Cosmos DB for MongoDB vCore, Chroma |
and features available only in Kernel Memory:
- RAG (Retrieval Augmented Generation)
- RAG sources lookup
- Summarization
- Security Filters (filter memory by users and groups)
- Long running ingestion, large documents, with retry logic and durable queues
- Custom tokenization
- Document storage
- OCR via Azure Document Intelligence
- LLMs (Large Language Models) with dedicated tokenization
- Cloud deployment
- OpenAPI
- Custom storage schema (partially implemented/work in progress)
- Short Term Memory (partially implemented/work in progress)
- Concurrent write to multiple vector DBs
- 📝 MS Office: Word, Excel, PowerPoint
- 📃 PDF documents
- 🌐 Fetch web pages and HTML files
- 🖼️ JPG/PNG/TIFF Images with text via OCR
- 📄 MarkDown and Raw plain text
- 💻 JSON files
- 💡 AI: Azure OpenAI, OpenAI, LLama - thanks to llama.cpp and LLamaSharp, LM Studio, Azure Document Intelligence
- 🧠 Vector storage: Azure AI Search, Postgres+pgvector, Qdrant, MSSQL Server (third party), Elasticsearch (third party), Redis, Chroma (work in progress), KNN vectors in memory (volatile), KNN vectors on disk (persistent).
- 🗂 Content storage: Azure Blobs, Local file system, In memory content (volatile).
- ⏳ Orchestration: Azure Queues, RabbitMQ, Local file based queues, In memory queues (volatile).
Kernel Memory works and scales at best when running as a service, allowing to ingest thousands of documents and information without blocking your app.
However, you can use Kernel Memory also serverless, embedding the MemoryServerless
class in your app.
var memory = new KernelMemoryBuilder() .WithOpenAIDefaults(Env.Var("OPENAI_API_KEY")) .Build<MemoryServerless>(); // Import a file await memory.ImportDocumentAsync("meeting-transcript.docx", tags: new() { { "user", "Blake" } }); // Import multiple files and apply multiple tags await memory.ImportDocumentAsync(new Document("file001") .AddFile("business-plan.docx") .AddFile("project-timeline.pdf") .AddTag("user", "Blake") .AddTag("collection", "business") .AddTag("collection", "plans") .AddTag("fiscalYear", "2023"));
var answer1 = await memory.AskAsync("How many people attended the meeting?"); var answer2 = await memory.AskAsync("what's the project timeline?", filter: new MemoryFilter().ByTag("user", "Blake"));
The code leverages the default documents ingestion pipeline:
- Extract text: recognize the file format and extract the information
- Partition the text in small chunks, to optimize search
- Extract embedding using an LLM embedding generator
- Save embedding into a vector index such as Azure AI Search, Qdrant or other DBs.
Documents are organized by users, safeguarding their private information. Furthermore, memories can be categorized and structured using tags, enabling efficient search and retrieval through faceted navigation.
All memories and answers are fully correlated to the data provided. When producing an answer, Kernel Memory includes all the information needed to verify its accuracy:
await memory.ImportFileAsync("NASA-news.pdf");
var answer = await memory.AskAsync("Any news from NASA about Orion?");
Console.WriteLine(answer.Result + "/n");
foreach (var x in answer.RelevantSources)
{
Console.WriteLine($" * {x.SourceName} -- {x.Partitions.First().LastUpdate:D}");
}Yes, there is news from NASA about the Orion spacecraft. NASA has invited the media to see a new test version of the Orion spacecraft and the hardware that will be used to recover the capsule and astronauts upon their return from space during the Artemis II mission. The event is scheduled to take place at Naval Base San Diego on Wednesday, August 2, at 11 a.m. PDT. Personnel from NASA, the U.S. Navy, and the U.S. Air Force will be available to speak with the media. Teams are currently conducting tests in the Pacific Ocean to demonstrate and evaluate the processes, procedures, and hardware for recovery operations for crewed Artemis missions. These tests will help prepare the team for Artemis II, which will be NASA's first crewed mission under the Artemis program. The Artemis II crew, consisting of NASA astronauts Reid Wiseman, Victor Glover, and Christina Koch, and Canadian Space Agency astronaut Jeremy Hansen, will participate in recovery testing at sea next year. For more information about the Artemis program, you can visit the NASA website.
- NASA-news.pdf -- Tuesday, August 1, 2023
Depending on your scenarios, you might want to run all the code locally inside your process, or remotely through an asynchronous service.
If you're importing small files, and need only C# and can block the process during the import, local-in-process execution can be fine, using the MemoryServerless seen above.
However, if you are in one of these scenarios:
- I'd just like a web service to import data and send queries to answer
- My app is written in TypeScript, Java, Rust, or some other language
- I want to define custom pipelines mixing multiple languages like Python, TypeScript, etc
- I'm importing big documents that can require minutes to process, and I don't want to block the user interface
- I need memory import to run independently, supporting failures and retry logic
then you can deploy Kernel Memory as a service, plugging in the default handlers, or your custom Python/TypeScript/Java/etc. handlers, and leveraging the asynchronous non-blocking memory encoding process, sending documents and asking questions using the MemoryWebClient.
Here you can find a complete set of instruction about how to run the Kernel Memory service.
If you want to give the service a quick test, use the following command to start the Kernel Memory Service using OpenAI:
docker run -e OPENAI_API_KEY="..." -it --rm -p 9001:9001 kernelmemory/serviceIf you prefer using custom settings and services such as Azure OpenAI, Azure
Document Intelligence, etc., you should create an appsettings.Development.json
file overriding the default values set in appsettings.json, or using the
configuration wizard included:
cd service/Service
dotnet run setup
Then run this command to start the Docker image with the configuration just created:
on Windows:
docker run --volume .\appsettings.Development.json:/app/appsettings.Production.json -it --rm -p 9001:9001 kernelmemory/service
on macOS/Linux:
docker run --volume ./appsettings.Development.json:/app/appsettings.Production.json -it --rm -p 9001:9001 kernelmemory/service
#reference clients/WebClient/WebClient.csproj var memory = new MemoryWebClient("http://127.0.0.1:9001"); // <== URL where the web service is running // Import a file (default user) await memory.ImportDocumentAsync("meeting-transcript.docx"); // Import a file specifying a Document ID, User and Tags await memory.ImportDocumentAsync("business-plan.docx", new DocumentDetails("[email protected]", "file001") .AddTag("collection", "business") .AddTag("collection", "plans") .AddTag("fiscalYear", "2023"));
curl http://127.0.0.1:9001/ask -d'{"query":"Any news from NASA about Orion?"}' -H 'Content-Type: application/json'{ "Query": "Any news from NASA about Orion?", "Text": "Yes, there is news from NASA about the Orion spacecraft. NASA has invited the media to see a new test version of the Orion spacecraft and the hardware that will be used to recover the capsule and astronauts upon their return from space during the Artemis II mission. The event is scheduled to take place at Naval Base San Diego on August 2nd at 11 a.m. PDT. Personnel from NASA, the U.S. Navy, and the U.S. Air Force will be available to speak with the media. Teams are currently conducting tests in the Pacific Ocean to demonstrate and evaluate the processes, procedures, and hardware for recovery operations for crewed Artemis missions. These tests will help prepare the team for Artemis II, which will be NASA's first crewed mission under the Artemis program. The Artemis II crew, consisting of NASA astronauts Reid Wiseman, Victor Glover, and Christina Koch, and Canadian Space Agency astronaut Jeremy Hansen, will participate in recovery testing at sea next year. For more information about the Artemis program, you can visit the NASA website.", "RelevantSources": [ { "Link": "...", "SourceContentType": "application/pdf", "SourceName": "file5-NASA-news.pdf", "Partitions": [ { "Text": "Skip to main content\nJul 28, 2023\nMEDIA ADVISORY M23-095\nNASA Invites Media to See Recovery Craft for\nArtemis Moon Mission\n(/sites/default/files/thumbnails/image/ksc-20230725-ph-fmx01_0003orig.jpg)\nAboard the USS John P. Murtha, NASA and Department of Defense personnel practice recovery operations for Artemis II in July. A\ncrew module test article is used to help verify the recovery team will be ready to recovery the Artemis II crew and the Orion spacecraft.\nCredits: NASA/Frank Michaux\nMedia are invited to see the new test version of NASA’s Orion spacecraft and the hardware teams will use\nto recover the capsule and astronauts upon their return from space during the Artemis II\n(http://www.nasa.gov/artemis-ii) mission. The event will take place at 11 a.m. PDT on Wednesday, Aug. 2,\nat Naval Base San Diego.\nPersonnel involved in recovery operations from NASA, the U.S. Navy, and the U.S. Air Force will be\navailable to speak with media.\nU.S. media interested in attending must RSVP by 4 p.m., Monday, July 31, to the Naval Base San Diego\nPublic Affairs (mailto:[email protected]) or 619-556-7359.\nOrion Spacecraft (/exploration/systems/orion/index.html)\nNASA Invites Media to See Recovery Craft for Artemis Moon Miss... https://www.nasa.gov/press-release/nasa-invites-media-to-see-recov...\n1 of 3 7/28/23, 4:51 PMTeams are currently conducting the first in a series of tests in the Pacific Ocean to demonstrate and\nevaluate the processes, procedures, and hardware for recovery operations (https://www.nasa.gov\n/exploration/systems/ground/index.html) for crewed Artemis missions. The tests will help prepare the\nteam for Artemis II, NASA’s first crewed mission under Artemis that will send four astronauts in Orion\naround the Moon to checkout systems ahead of future lunar missions.\nThe Artemis II crew – NASA astronauts Reid Wiseman, Victor Glover, and Christina Koch, and CSA\n(Canadian Space Agency) astronaut Jeremy Hansen – will participate in recovery testing at sea next year.\nFor more information about Artemis, visit:\nhttps://www.nasa.gov/artemis (https://www.nasa.gov/artemis)\n-end-\nRachel Kraft\nHeadquarters, Washington\n202-358-1100\[email protected] (mailto:[email protected])\nMadison Tuttle\nKennedy Space Center, Florida\n321-298-5868\[email protected] (mailto:[email protected])\nLast Updated: Jul 28, 2023\nEditor: Claire O’Shea\nTags: Artemis (/artemisprogram),Ground Systems (http://www.nasa.gov/exploration/systems/ground\n/index.html),Kennedy Space Center (/centers/kennedy/home/index.html),Moon to Mars (/topics/moon-to-\nmars/),Orion Spacecraft (/exploration/systems/orion/index.html)\nNASA Invites Media to See Recovery Craft for Artemis Moon Miss... https://www.nasa.gov/press-release/nasa-invites-media-to-see-recov...\n2 of 3 7/28/23, 4:51 PM", "Relevance": 0.8430657, "SizeInTokens": 863, "LastUpdate": "2023-08-01T08:15:02-07:00" } ] } ] }
You can find a full example here.
On the other hand, if you need a custom data pipeline, you can also customize the steps, which will be handled by your custom business logic:
// Memory setup, e.g. how to calculate and where to store embeddings
var memoryBuilder = new KernelMemoryBuilder()
.WithoutDefaultHandlers()
.WithOpenAIDefaults(Env.Var("OPENAI_API_KEY"));
var memory = memoryBuilder.Build();
// Plug in custom .NET handlers
memory.Orchestrator.AddHandler<MyHandler1>("step1");
memory.Orchestrator.AddHandler<MyHandler2>("step2");
memory.Orchestrator.AddHandler<MyHandler3>("step3");
// Use the custom handlers with the memory object
await memory.ImportDocumentAsync(
new Document("mytest001")
.AddFile("file1.docx")
.AddFile("file2.pdf"),
steps: new[] { "step1", "step2", "step3" });The API schema is available at http://127.0.0.1:9001/swagger/index.html when running the service locally with OpenAPI enabled.
- Collection of Jupyter notebooks with various scenarios
- Using Kernel Memory web service to upload documents and answer questions
- Using KM Plugin for Semantic Kernel
- Importing files and asking question without running the service (serverless mode)
- Processing files with custom logic (custom handlers) in serverless mode
- Processing files with custom logic (custom handlers) in asynchronous mode
- Upload files and ask questions from command line using curl
- Customizing RAG and summarization prompts
- Custom partitioning/text chunking options
- Using a custom embedding/vector generator
- Using custom LLMs
- Using LLama
- Summarizing documents, using synthetic memories
- Writing and using a custom ingestion handler
- Running a single asynchronous pipeline handler as a standalone service
- Test project using KM package from nuget.org
- Integrating Memory with ASP.NET applications and controllers
- Sample code showing how to extract text from files
- .NET configuration and logging
- Expanding chunks retrieving adjacent partitions
- Using local models via LM Studio
- .NET appsettings.json generator
- Curl script to upload files
- Curl script to ask questions
- Curl script to search documents
- Script to start Qdrant for development tasks
- Script to start Elasticsearch for development tasks
- Script to start MS SQL Server for development tasks
- Script to start Redis for development tasks
- Script to start RabbitMQ for development tasks
- Script to start MongoDB Atlas for development tasks
-
Microsoft.KernelMemory.WebClient: The web client library, can be used to call a running instance of the Memory web service. .NET Standard 2.0 compatible.
-
Microsoft.KernelMemory.SemanticKernelPlugin: a Memory plugin for Semantic Kernel, replacing the original Semantic Memory available in SK. .NET Standard 2.0 compatible.
-
Microsoft.KernelMemory.Abstractions: The internal interfaces and models shared by all packages, used to extend KM to support third party services. .NET Standard 2.0 compatible.
-
Microsoft.KernelMemory.MemoryDb.AzureAISearch: Memory storage using Azure AI Search.
-
Microsoft.KernelMemory.MemoryDb.Postgres: Memory storage using PostgreSQL.
-
Microsoft.KernelMemory.MemoryDb.Qdrant: Memory storage using Qdrant.
-
Microsoft.KernelMemory.AI.AzureOpenAI: Integration with Azure OpenAI LLMs.
-
Microsoft.KernelMemory.AI.LlamaSharp: Integration with LLama LLMs.
-
Microsoft.KernelMemory.AI.OpenAI: Integration with OpenAI LLMs.
-
Microsoft.KernelMemory.DataFormats.AzureAIDocIntel: Integration with Azure AI Document Intelligence.
-
Microsoft.KernelMemory.Orchestration.AzureQueues: Ingestion and synthetic memory pipelines via Azure Queue Storage.
-
Microsoft.KernelMemory.Orchestration.RabbitMQ: Ingestion and synthetic memory pipelines via RabbitMQ.
-
Microsoft.KernelMemory.ContentStorage.AzureBlobs: Used to store content on Azure Storage Blobs.
-
Microsoft.KernelMemory.Core: The core library, can be used to build custom pipelines and handlers, and contains a serverless client to use memory in a synchronous way, without the web service. .NET 6+.
Kernel Memory service offers a Web API out of the box, including the OpenAPI swagger documentation that you can leverage to test the API and create custom web clients. For instance, after starting the service locally, see http://127.0.0.1:9001/swagger/index.html.
A .NET Web Client and a Semantic Kernel plugin are available, see the nugets packages above.
A python package with a Web Client and Semantic Kernel plugin will soon be available. We also welcome PR contributions to support more languages.
 |
 |
 |
 |
 |
 |
|---|---|---|---|---|---|
| afederici75 | alexibraimov | alkampfergit | amomra | anthonypuppo | cherchyk |
 |
 |
 |
 |
 |
 |
|---|---|---|---|---|---|
| crickman | dluc | DM-98 | GraemeJones104 | kbeaugrand | lecramr |
 |
 |
 |
 |
 |
 |
|---|---|---|---|---|---|
| luismanez | marcominerva | pascalberger | pawarsum12 | qihangnet | slapointe |
 |
 |
 |
 |
 |
|---|---|---|---|---|
| slorello89 | TaoChenOSU | teresaqhoang | vicperdana | xbotter |
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent