Misc improvements to handlers and web api

.gitignore

@@ -2,6 +2,7 @@ KernelMemoryDev.*
 
 dotnet/.config
 tmp/
+_tmp/
 tmp-*/
 out/
 _files/

Directory.Packages.props

@@ -14,6 +14,7 @@
     <PackageVersion Include="LLamaSharp" Version="0.10.0" />
     <PackageVersion Include="LLamaSharp.Backend.Cpu" Version="0.10.0" />
     <PackageVersion Include="LLamaSharp.Backend.Cuda12" Version="0.10.0" />
+    <PackageVersion Include="Microsoft.AspNetCore.Http.Abstractions" Version="2.2.0" />
     <PackageVersion Include="Microsoft.Bcl.AsyncInterfaces" Version="8.0.0" />
     <PackageVersion Include="Microsoft.Extensions.Configuration" Version="8.0.0" />
     <PackageVersion Include="Microsoft.Extensions.Configuration.Json" Version="8.0.0" />
@@ -29,7 +30,6 @@
     <PackageVersion Include="Polly.Core" Version="8.3.0" />
     <PackageVersion Include="RabbitMQ.Client" Version="6.8.1" />
     <PackageVersion Include="NRedisStack" Version="0.11.0" />
-    <PackageVersion Include="Swashbuckle.AspNetCore" Version="6.5.0" />
     <PackageVersion Include="System.Linq.Async" Version="6.0.1" />
     <PackageVersion Include="System.Memory.Data" Version="8.0.0" />
     <PackageVersion Include="System.Numerics.Tensors" Version="8.0.0" />

KernelMemory.sln.DotSettings

@@ -244,6 +244,7 @@ public void It$SOMENAME$()
 	<s:Boolean x:Key="/Default/UserDictionary/Words/=keyvault/@EntryIndexedValue">True</s:Boolean>
 	<s:Boolean x:Key="/Default/UserDictionary/Words/=Kitto/@EntryIndexedValue">True</s:Boolean>
 	<s:Boolean x:Key="/Default/UserDictionary/Words/=lessthan/@EntryIndexedValue">True</s:Boolean>
+	<s:Boolean x:Key="/Default/UserDictionary/Words/=liveness/@EntryIndexedValue">True</s:Boolean>
 	<s:Boolean x:Key="/Default/UserDictionary/Words/=Logit/@EntryIndexedValue">True</s:Boolean>
 	<s:Boolean x:Key="/Default/UserDictionary/Words/=Mddhhmmssfffffff/@EntryIndexedValue">True</s:Boolean>
 	<s:Boolean x:Key="/Default/UserDictionary/Words/=mergeresults/@EntryIndexedValue">True</s:Boolean>

README.md

@@ -76,39 +76,34 @@ and **features available only in Kernel Memory**:
 
 # Supported Data formats and Backends
 
-* MS Office: Word, Excel, PowerPoint
-* PDF documents
-* Web pages
-* JPG/PNG/TIFF Images with text via OCR
-* MarkDown and Raw plain text
-* JSON files
-* 🧠 AI
-    * [Azure OpenAI](https://learn.microsoft.com/azure/ai-services/openai/concepts/models)
-    * [OpenAI](https://platform.openai.com/docs/models)
-    * LLama - thanks to [llama.cpp](https://github.com/ggerganov/llama.cpp) and [LLamaSharp](https://github.com/SciSharp/LLamaSharp)
-    * [Azure Document Intelligence](https://azure.microsoft.com/products/ai-services/ai-document-intelligence)
-
-* ↗️ Vector storage
-    * [Azure AI Search](https://azure.microsoft.com/products/ai-services/ai-search)
-    * [Postgres+pgvector](https://github.com/microsoft/kernel-memory/extensions/postgres)
-    * [Qdrant](https://qdrant.tech)
-    * [MSSQL Server (third party)](https://www.nuget.org/packages/KernelMemory.MemoryStorage.SqlServer)
-    * [Elasticsearch (third party)](https://www.nuget.org/packages/FreeMindLabs.KernelMemory.Elasticsearch)
-    * [Redis](https://redis.io)
-    * [Chroma (work in progress)](https://www.trychroma.com)
-    * In memory KNN vectors (volatile)
-    * On disk KNN vectors
-
-* 📀 Content storage
-    * [Azure Blobs](https://learn.microsoft.com/azure/storage/blobs/storage-blobs-introduction)
-    * Local file system
-    * In memory, volatile content
-
-* ⏳ Orchestration
-    * [Azure Queues](https://learn.microsoft.com/azure/storage/queues/storage-queues-introduction)
-    * [RabbitMQ](https://www.rabbitmq.com)
-    * Local file based queue
-    * In memory queues (volatile)
+* 📝 MS Office: Word, Excel, PowerPoint
+* 📃 PDF documents
+* 🌐 Web pages
+* 🖼️ JPG/PNG/TIFF Images with text via OCR
+* 📄 MarkDown and Raw plain text
+* 💻 JSON files
+* 💡 AI:
+  [Azure OpenAI](https://learn.microsoft.com/azure/ai-services/openai/concepts/models),
+  [OpenAI](https://platform.openai.com/docs/models),
+  LLama - thanks to [llama.cpp](https://github.com/ggerganov/llama.cpp) and [LLamaSharp](https://github.com/SciSharp/LLamaSharp),
+  [Azure Document Intelligence](https://azure.microsoft.com/products/ai-services/ai-document-intelligence)
+* 🧠 Vector storage:
+  [Azure AI Search](https://azure.microsoft.com/products/ai-services/ai-search),
+  [Postgres+pgvector](https://github.com/microsoft/kernel-memory/extensions/postgres),
+  [Qdrant](https://qdrant.tech),
+  [MSSQL Server (third party)](https://www.nuget.org/packages/KernelMemory.MemoryStorage.SqlServer),
+  [Elasticsearch (third party)](https://www.nuget.org/packages/FreeMindLabs.KernelMemory.Elasticsearch),
+  [Redis](https://redis.io),
+  [Chroma (work in progress)](https://www.trychroma.com),
+  KNN vectors in memory (volatile),
+  KNN vectors on disk (persistent).
+* 🗂 Content storage: [Azure Blobs](https://learn.microsoft.com/azure/storage/blobs/storage-blobs-introduction),
+  Local file system,
+  In memory content (volatile).
+* ⏳ Orchestration: [Azure Queues](https://learn.microsoft.com/azure/storage/queues/storage-queues-introduction),
+  [RabbitMQ](https://www.rabbitmq.com),
+  Local file based queues,
+  In memory queues (volatile).
 
 # Kernel Memory in serverless mode
 
@@ -217,7 +212,7 @@ However, if you are in one of these scenarios:
   logic**
 
 then you can deploy Kernel Memory as a service, plugging in the
-default handlers or your custom Python/TypeScript/Java/etc. handlers,
+default handlers, or your custom Python/TypeScript/Java/etc. handlers,
 and leveraging the asynchronous non-blocking memory encoding process,
 sending documents and asking questions using the **MemoryWebClient**.
 
@@ -300,30 +295,23 @@ customize the steps, which will be handled by your custom business logic:
 
 ```csharp
 // Memory setup, e.g. how to calculate and where to store embeddings
-var memoryBuilder = new KernelMemoryBuilder().WithOpenAIDefaults(Env.Var("OPENAI_API_KEY"));
-memoryBuilder.Build();
-var orchestrator = memoryBuilder.GetOrchestrator();
-
-// Define custom .NET handlers
-var step1 = new MyHandler1("step1", orchestrator);
-var step2 = new MyHandler2("step2", orchestrator);
-var step3 = new MyHandler3("step3", orchestrator);
-await orchestrator.AddHandlerAsync(step1);
-await orchestrator.AddHandlerAsync(step2);
-await orchestrator.AddHandlerAsync(step3);
-
-// Instantiate a custom pipeline
-var pipeline = orchestrator
-    .PrepareNewFileUploadPipeline("user-id-1", "mytest", new[] { "memory-collection" })
-    .AddUploadFile("file1", "file1.docx", "file1.docx")
-    .AddUploadFile("file2", "file2.pdf", "file2.pdf")
-    .Then("step1")
-    .Then("step2")
-    .Then("step3")
-    .Build();
-
-// Execute in process, process all files with all the handlers
-await orchestrator.RunPipelineAsync(pipeline);
+var memoryBuilder = new KernelMemoryBuilder()
+    .WithoutDefaultHandlers()
+    .WithOpenAIDefaults(Env.Var("OPENAI_API_KEY"));
+
+var memory = memoryBuilder.Build();
+
+// Plug in custom .NET handlers
+memory.Orchestrator.AddHandler<MyHandler1>("step1");
+memory.Orchestrator.AddHandler<MyHandler2>("step2");
+memory.Orchestrator.AddHandler<MyHandler3>("step3");
+
+// Use the custom handlers with the memory object
+await memory.ImportDocumentAsync(
+    new Document("mytest001")
+        .AddFile("file1.docx")
+        .AddFile("file2.pdf"),
+    steps: new[] { "step1", "step2", "step3" });
 ```
 
 # Web API specs
@@ -337,31 +325,35 @@ running the service locally with OpenAPI enabled.
 
 1. [Collection of Jupyter notebooks with various scenarios](examples/000-notebooks)
 2. [Using Kernel Memory web service to upload documents and answer questions](examples/001-dotnet-WebClient)
-3. [Using KM Plugin for Semantic Kernel](examples/002-dotnet-SemanticKernelPlugin)
+3. [Using KM Plugin for Semantic Kernel](examples/002-dotnet-SemanticKernel-plugin)
 4. [Importing files and asking question without running the service (serverless mode)](examples/003-dotnet-Serverless)
-5. [Processing files with custom steps](examples/004-dotnet-ServerlessCustomPipeline)
-6. [Upload files and ask questions from command line using curl](examples/005-curl-calling-webservice)
-7. [Customizing RAG and summarization prompts](examples/101-dotnet-custom-Prompts)
-8. [Custom partitioning/text chunking options](examples/102-dotnet-custom-partitioning-options)
-9. [Using a custom embedding/vector generator](examples/103-dotnet-custom-EmbeddingGenerator)
-10. [Using custom LLMs](examples/104-dotnet-custom-LLM)
-11. [Using LLama](examples/105-dotnet-serverless-llamasharp)
-12. [Summarizing documents](examples/106-dotnet-retrieve-synthetics)
-11. [Natural language to SQL examples](examples/200-dotnet-nl2sql)
-12. [Writing and using a custom ingestion handler](examples/201-dotnet-InProcessMemoryWithCustomHandler)
-13. [Running a single asynchronous pipeline handler as a standalone service](examples/202-dotnet-CustomHandlerAsAService)
-14. [Test project linked to KM package from nuget.org](examples/203-dotnet-using-core-nuget)
-15. [Integrating Memory with ASP.NET applications and controllers](examples/204-dotnet-ASP.NET-MVC-integration)
-16. [Sample code showing how to extract text from files](examples/205-dotnet-extract-text-from-docs)
+5. [Processing files with custom logic (custom handlers) in serverless mode](examples/004-dotnet-serverless-custom-pipeline)
+6. [Processing files with custom logic (custom handlers) in asynchronous mode](examples/005-dotnet-AsyncMemoryCustomPipeline)
+7. [Upload files and ask questions from command line using curl](examples/006-curl-calling-webservice)
+8. [Customizing RAG and summarization prompts](examples/101-dotnet-custom-Prompts)
+9. [Custom partitioning/text chunking options](examples/102-dotnet-custom-partitioning-options)
+10. [Using a custom embedding/vector generator](examples/103-dotnet-custom-EmbeddingGenerator)
+11. [Using custom LLMs](examples/104-dotnet-custom-LLM)
+12. [Using LLama](examples/105-dotnet-serverless-llamasharp)
+13. [Summarizing documents, using synthetic memories](examples/106-dotnet-retrieve-synthetics)
+14. [Writing and using a custom ingestion handler](examples/201-dotnet-serverless-custom-handler)
+15. [Running a single asynchronous pipeline handler as a standalone service](examples/202-dotnet-custom-handler-as-a-service)
+16. [Test project using KM package from nuget.org](examples/203-dotnet-using-core-nuget)
+17. [Integrating Memory with ASP.NET applications and controllers](examples/204-dotnet-ASP.NET-MVC-integration)
+18. [Sample code showing how to extract text from files](examples/205-dotnet-extract-text-from-docs)
+19. [Expanding chunks retrieving adjacent partitions](examples/206-dotnet-configuration-and-logging)
 
 ## Tools
 
-1. [Curl script to upload files](tools/upload-file.sh)
-2. [Curl script to ask questions](tools/ask.sh)
-3. [Curl script to search documents](tools/search.sh)
-4. [Script to start Qdrant for development tasks](tools/run-qdrant.sh)
-5. [Script to start RabbitMQ for development tasks](tools/run-rabbitmq.sh)
-6. [.NET appsettings.json generator](tools/InteractiveSetup)
+1. [.NET appsettings.json generator](tools/InteractiveSetup)
+2. [Curl script to upload files](tools/upload-file.sh)
+3. [Curl script to ask questions](tools/ask.sh)
+4. [Curl script to search documents](tools/search.sh)
+5. [Script to start Qdrant for development tasks](tools/run-qdrant.sh)
+6. [Script to start Elasticsearch for development tasks](tools/run-elasticsearch.sh)
+7. [Script to start MS SQL Server for development tasks](tools/run-mssql.sh)
+8. [Script to start Redis for development tasks](tools/run-redis.sh)
+9. [Script to start RabbitMQ for development tasks](tools/run-rabbitmq.sh)
 
 ### .NET packages
 

examples/000-notebooks/.env.example

@@ -0,0 +1,2 @@
+OPENAI_API_KEY=sk-...
+MEMORY_API_KEY=...

examples/000-notebooks/001-upload-and-ask.ipynb

@@ -26,15 +26,15 @@
     {
      "data": {
       "text/html": [
-       "<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>Microsoft.KernelMemory.Core, 0.23.231218.1</span></li></ul></div></div>"
+       "<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>Microsoft.KernelMemory.Core, 0.29.240219.2</span></li></ul></div></div>"
       ]
      },
      "metadata": {},
      "output_type": "display_data"
     }
    ],
    "source": [
-    "#r \"nuget: Microsoft.KernelMemory.Core, 0.23.231218.1\"\n",
+    "#r \"nuget: Microsoft.KernelMemory.Core, 0.29.240219.2\"\n",
     "\n",
     "using Microsoft.KernelMemory;"
    ]
@@ -141,7 +141,7 @@
    "source": [
     "At this point we have the memory client ready, we can import data and ask questions!\n",
     "\n",
-    "Let's import a PDF file with some details about Semantic Kernel:"
+    "Let's import a PDF file with some details about Kernel Memory:"
    ]
   },
   {
@@ -157,14 +157,14 @@
    },
    "outputs": [],
    "source": [
-    "await memory.ImportDocumentAsync(\"sample-SK-Readme.pdf\", documentId: \"doc001\");"
+    "await memory.ImportDocumentAsync(\"sample-KM-Readme.pdf\", documentId: \"doc001\");"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now, let's ask a question about Semantic Kernel:"
+    "Now, let's ask a question about Kernel Memory:"
    ]
   },
   {
@@ -183,14 +183,14 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Question: What's Semantic Kernel?\n",
+      "Question: What's Kernel Memory?\n",
       "\n",
-      "Answer: Semantic Kernel is a lightweight SDK (Software Development Kit) developed by Microsoft. It enables the integration of AI Large Language Models (LLMs) with conventional programming languages. The SDK combines natural language semantic functions, traditional code native functions, and embeddings-based memory to add value and unlock new potential in applications with AI. Semantic Kernel supports prompt templating, function chaining, vectorized memory, and intelligent planning capabilities. It encapsulates several design patterns from the latest AI research, allowing developers to infuse their applications with features like prompt chaining, recursive reasoning, summarization, zero/few-shot learning, contextual memory, long-term memory, embeddings, semantic indexing, planning, retrieval-augmented generation, and accessing external knowledge stores. Semantic Kernel is available for use with C# and Python programming languages. It is an open-source project, and developers are invited to contribute to its development through GitHub discussions, opening issues, sending pull requests, and joining the Discord community.\n"
+      "Answer: Kernel Memory is a multi-modal AI service that specializes in efficient indexing of datasets through custom continuous data hybrid pipelines. It supports features such as Retrieval Augmented Generation (RAG), synthetic memory, prompt engineering, and custom semantic memory processing. It includes a GPT Plugin, web clients, a .NET library for embedded applications, and can be deployed as a Docker container. Kernel Memory enables natural language querying for obtaining answers from indexed data, complete with citations and links to the original sources. It is designed for seamless integration with popular AI platforms and offers support for various data formats and backends. It also provides features like security filters, long-running ingestion, custom tokenization, document storage, OCR via Azure Document Intelligence, and more. Kernel Memory can be used as a service from any language, tool, or platform, including browser extensions, chatbots, and assistants.\n"
      ]
     }
    ],
    "source": [
-    "var question = \"What's Semantic Kernel?\";\n",
+    "var question = \"What's Kernel Memory?\";\n",
     "\n",
     "var answer = await memory.AskAsync(question);\n",
     "\n",
@@ -222,7 +222,7 @@
      "text": [
       "Sources:\n",
       "\n",
-      "  - sample-SK-Readme.pdf  - default/doc001/a6c00dad170140178e805c9a0c267a7f [Tuesday, December 19, 2023]\n"
+      "  - sample-KM-Readme.pdf  - default/doc001/3e0c889e762b413594085c715593bbc7 [Tuesday, February 27, 2024]\n"
      ]
     }
    ],