Comments (2)
hi @martinoss the service returns tags and other details, which you should see also in the MemoryAnswer
object if you're using C# WebClient.
Here's an example of a response, after using the ImportWebPageAsync
API to upload a web page:
{
"question": "which storage engines can I use with Kernel Memory?",
"noResult": false,
"text": "The storage engines you can use with Kernel Memory include Azure AI Search, Elasticsearch, Postgres, Qdrant, Redis, In memory KNN, and On disk KNN.",
"relevantSources": [
{
"link": "default/doc001/c8e2d982611649fba242c06c8ee1b37f",
"index": "default",
"documentId": "doc001",
"fileId": "c8e2d982611649fba242c06c8ee1b37f",
"sourceContentType": "text/x-uri",
"sourceName": "content.url",
"sourceUrl": "https://microsoft.github.io/kernel-memory/",
"partitions": [
{
"text": "Storage engines Azure AI Search, Chroma, DuckDB, Kusto, Milvus, MongoDB, Pinecone, Postgres, Qdrant, Redis, SQLite, Weaviate Azure AI Search, Elasticsearch, Postgres, Qdrant, Redis, In memory KNN, On disk KNN and features available only in Kernel Memory: RAG (Retrieval Augmented Generation) RAG sources lookup Summarization Security filters (filter memory by users and groups) Long running ingestion, large documents, with retry logic and durable queues Custom tokenization Document storage OCR via Azure Document Intelligence LLMs (Large Language Models) with dedicated tokenization Cloud deployment OpenAPI Custom storage schema (partially implemented/work in progress) Short Term Memory (partially implemented/work in progress)(*) Partially implemented and/or work in progress. Topics Quickstart: test KM in few minutes Memory service, web clients and plugins Memory API, memory ingestion and information retrieval KM Extensions: vector DBs, AI models, Data formats, Orchestration, Content storage Embedding serverless memory in .NET apps Security, service and users How-to guides, customizing KM and examples Concepts, KM glossary KM packages Edit this page",
"relevance": 0.8469289,
"lastUpdate": "2024-01-08T23:45:15-08:00",
"tags": {
"__document_id": [
"doc001"
],
"__file_type": [
"text/x-uri"
],
"__file_id": [
"c8e2d982611649fba242c06c8ee1b37f"
],
"__file_part": [
"36a60baa3976456f92850dc07adf43d0"
],
"type": [
"manual"
]
}
},
{
"text": "Overview | Kernel Memory Skip to main content Link Menu Expand (external link) Document Search Copy Copied Kernel Memory Overview QuickstartConfigurationStart the servicePython exampleC# exampleJava exampleJavaScript exampleBash and Curl examples ServiceArchitectureConfigurationOpenAPI - Web APIWeb ClientSK Plugin Memory APIStore memoryAnswer questions (RAG)Search memory Extensions Memory DBs Azure AI Search Qdrant PostgreSQL Elastic Search Redis Simple memory AI Azure OpenAI OpenAI LLama Data formats Azure AI Document Intelligence Orchestration Azure Queues RabbitMQ Simple queues Content storage Azure Blobs Simple storage Serverless (.NET)ComponentsVolatile memoryKernel BuilderSK Plugin SecuritySecurity FiltersService API Keys How-to guidesPartitioning & chunkingCustom promptsCustom pipelinesHugging Face models ConceptsIndexDocumentMemoryTagLLMEmbeddingCosine SimilarityVector SearchTokensF.A.Q.Packagesupload-file.\nsh KM on GitHub KM on Discord GitHub Discord Kernel Memory Kernel Memory (KM) is a multi-modal AI Service specialized in the efficient indexing of documents and information through custom continuous data pipelines, with support for Retrieval Augmented Generation (RAG), synthetic memory, prompt engineering, and custom semantic memory processing. KM supports PDF and Word documents, PowerPoint presentations, Images, Spreadsheets and more, extracting information and generating memories by leveraging Large Language Models (LLMs), Embeddings and Vector storage.\nUtilizing advanced embeddings, LLMs and prompt engineering, the system enables Natural Language querying for obtaining answers from the information stored, complete with citations and links to the original sources. Kernel Memory is designed for seamless integration with any programming language, providing a web service that can also be consumed as an OpenAPI endpoint for ChatGPT, web clients ready to use, and a Plugin for Microsoft Copilot and Semantic Kernel. Kernel Memory (KM) and Semantic Memory (SM) Semantic Memory (SM) is a library for C#, Python, and Java that wraps direct calls to databases and supports vector search. It was developed as part of the Semantic Kernel (SK) project and serves as the first public iteration of long-term memory.\nThe core library is maintained in three languages, while the list of supported storage engines (known as “connectors”) varies across languages. Kernel Memory (KM) is a service built on the feedback received and lessons learned from developing Semantic Kernel (SK) and Semantic Memory (SM). It provides several features that would otherwise have to be developed manually, such as storing files, extracting text from files, providing a framework to secure users’ data, etc. The KM codebase is entirely in .NET, which eliminates the need to write and maintain features in multiple languages. As a service, KM can be used from any language, tool, or platform, e.g. browser extensions and ChatGPT assistants. Here’s a few notable differences: Feature Semantic Memory Kernel Memory Data formats Text only Web pages, PDF, Images, Word, PowerPoint, Excel, Markdown, Text, JSON, more being added Search Cosine similarity Cosine similarity, Hybrid search with filters, AND/OR conditions Language support C#, Python, Java Any language, command line tools, browser extensions, low-code/no-code apps, chatbots, assistants, etc. Storage engines Azure AI Search, Chroma, DuckDB, Kusto, Milvus, MongoDB, Pinecone, Postgres, Qdrant, Redis, SQLite, Weaviate Azure AI Search, Elasticsearch, Postgres, Qdrant, Redis, In memory KNN, On disk KNN and features available only in Kernel Memory:",
"relevance": 0.81711054,
"lastUpdate": "2024-01-08T23:45:15-08:00",
"tags": {
"__document_id": [
"doc001"
],
"__file_type": [
"text/x-uri"
],
"__file_id": [
"c8e2d982611649fba242c06c8ee1b37f"
],
"__file_part": [
"fc75380e8f354b3389df8ce7dc5a1225"
],
"type": [
"manual"
]
}
}
]
}
]
}
from kernel-memory.
Hi @dluc thank you!
My fault, I haven't drilled into partitions, sorry for that.
from kernel-memory.
Related Issues (20)
- [Bug] Microsoft.AspNetCore.Http.Abstractions is deprecated HOT 2
- [Feature Request ]I want to know,do we push python version for kernel-memory in the future?
- [Bug] Exception when using Qdrant v1.8.0 HOT 5
- [Bug] Issue with default index name management HOT 1
- [Feature Request] vector support milvus
- [Question] Possible to get an example using AzureQueue ?
- [Bug] Documents duplicated with Qdrant HOT 5
- [Question] qdrant HOT 8
- [Question] Is it possible to import from a web page that requires auth? HOT 2
- Running server with Simple Disk storage seems always to use volatile memory HOT 2
- [Question] Should document upload status endpoint signal when the files were ignored by the text extractor?
- [Bug] NotSupportedException when referencing Microsoft.SemanticKernel library v1.6.1 HOT 7
- Non of the serverless examples run without Azure and/or OpenAI endpoints. HOT 6
- [Bug] An item with the same key has already been added. Key: XXXXXX.docx.partition.0.txt.AI.OpenAI.OpenAITextEmbeddingGenerator.TODO.text_embedding HOT 1
- [Question] TextExtractionHandler HOT 1
- [Question] I want to use the RAG function offline, but I only found the KEY for OPENAI in the documents and search. HOT 2
- [Bug] System.ArgumentException: Input span arguments must all have the same length. HOT 8
- TextChunker CPU usage HOT 2
- TextChunker doesn't handle Markdown Tables HOT 3
- Different similarity results when using text-embedding-3-small or text-embedding-3-large models HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kernel-memory.