Merge pull request #73 from llm-tools/mongodb-vector

Mongodb vector database

README.md

@@ -105,6 +105,7 @@ The author(s) are looking to add core maintainers for this opensource project. R
     -   [HNSWLib](#hnswlib)
     -   [Weaviate](#weaviate)
     -   [Qdrant](#qdrant)
+    -   [MongoDB](#mongodb)
     -   [Bring your own database](#bring-your-own-database)
     -   [More databases coming soon](#more-databases-coming-soon)
 -   [Caches](#caches)
@@ -902,6 +903,34 @@ import { QdrantDb } from '@llm-tools/embedjs/vectorDb/qdrant';
 .setVectorDb(new QdrantDb({ apiKey: '...'; url: '...'; clusterName: '...' }))
 ```
 
+## MongoDB
+
+[MongoDB](https://www.mongodb.com/products/platform/atlas-vector-search) is an open source document database. They offer a managed cloud offering **MongoDB Atlas**. As of right now, only the Atlas version supports vector search while the open source version does not.
+
+To use MongoDB as your vector database, follow these steps -
+
+-   Sign up for a MongoDB Atlas account if you haven't already. Once you have signed up, you will need to spin up a new cluster (or use an existing one)
+
+**Note:** you will need to provision a M10 (or higher) instance type to use Atlas vector search. Cheaper instance types or the free version (M0) give an error when vector indexes are created programatically.
+
+-   The cluster creation takes a few minutes. Once the cluster is ready, click on the connect button on the dashboard to get the connection string.
+
+**Note:** You will need to add a users separately and allow IP access from your relevant development and production environments.
+
+-   Set MongoDB database as your choice of `vectorDb`
+
+```TS
+import { MongoDb } from '@llm-tools/embedjs/vectorDb/mongodb';
+
+.setVectorDb(
+    new MongoDb({
+        connectionString: 'mongodb+srv://<username>:<password>@<url>',
+    }),
+)
+```
+
+**Note:** you can also optionally configure the database and collection name the library will use with the constructor parameters `dbName` and `collectionName`. Default values are used if these are not provided.
+
 ## Bring your own database
 
 You can pass along your vector database to the `setVectorDb` method by implementing the interface `BaseDb`. Here's how that would look like -
@@ -1046,11 +1075,9 @@ export LANGCHAIN_API_KEY="<api key>"
 
 Here's a list of projects / examples built with RagKit
 
-| **Project**                                                                      | **Description**                                                         |
-| -------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
-| [nestjs-embedjs-template](https://github.com/llm-tools/nestjs-embedjs-template)  | A NestJS server side template                                           |
-| [nextjs-chatbot-template](https://github.com/llm-tools/chat-bot-nextjs-template) | A NextJS chat template - including a chat UI                            |
-| [slack-bot](https://github.com/llm-tools/slack-bot-template)                     | A slack bot that can answer questions based on learnt confluence spaces |
+| **Project**                                                  | **Description**                                                        |
+| ------------------------------------------------------------ | ---------------------------------------------------------------------- |
+| [slack-bot](https://github.com/llm-tools/slack-bot-template) | A NestJs based slack bot that can answer questions based on confluence |
 
 # Contributing
 

examples/chroma/src/index.ts

@@ -10,7 +10,7 @@ const llmApplication = await new RAGApplicationBuilder()
 
 await llmApplication.addLoader(new YoutubeLoader({ videoIdOrUrl: 'pQiT2U5E9tI' }));
 await llmApplication.addLoader(new SitemapLoader({ url: 'https://tesla-info.com/sitemap.xml' }));
-await llmApplication.addLoader(new WebLoader({ url: 'https://en.wikipedia.org/wiki/Tesla,_Inc.' }));
+await llmApplication.addLoader(new WebLoader({ urlOrContent: 'https://en.wikipedia.org/wiki/Tesla,_Inc.' }));
 
 console.log((await llmApplication.query('Who founded Tesla?')).result);
 // The founder of Tesla is Elon Musk. He co-founded the company with JB Straubel, Martin Eberhard, Marc Tarpenning, and Ian Wright in 2003. Elon Musk is also the CEO of SpaceX and Neuralink.

examples/cohere/src/index.ts

@@ -12,7 +12,7 @@ const llmApplication = await new RAGApplicationBuilder()
 
 await llmApplication.addLoader(new YoutubeLoader({ videoIdOrUrl: 'pQiT2U5E9tI' }));
 await llmApplication.addLoader(new SitemapLoader({ url: 'https://tesla-info.com/sitemap.xml' }));
-await llmApplication.addLoader(new WebLoader({ url: 'https://en.wikipedia.org/wiki/Tesla,_Inc.' }));
+await llmApplication.addLoader(new WebLoader({ urlOrContent: 'https://en.wikipedia.org/wiki/Tesla,_Inc.' }));
 
 console.log((await llmApplication.query('Who founded Tesla?')).result);
 // The founder of Tesla is Elon Musk. He co-founded the company with JB Straubel, Martin Eberhard, Marc Tarpenning, and Ian Wright in 2003. Elon Musk is also the CEO of SpaceX and Neuralink.

examples/ollama/src/index.ts

@@ -24,7 +24,7 @@ const llmApplication = await new RAGApplicationBuilder()
 
 await llmApplication.addLoader(new YoutubeLoader({ videoIdOrUrl: 'pQiT2U5E9tI' }));
 await llmApplication.addLoader(new SitemapLoader({ url: 'https://tesla-info.com/sitemap.xml' }));
-await llmApplication.addLoader(new WebLoader({ url: 'https://en.wikipedia.org/wiki/Tesla,_Inc.' }));
+await llmApplication.addLoader(new WebLoader({ urlOrContent: 'https://en.wikipedia.org/wiki/Tesla,_Inc.' }));
 
 console.log((await llmApplication.query('Who founded Tesla?')).result);
 // The founder of Tesla is Elon Musk. He co-founded the company with JB Straubel, Martin Eberhard, Marc Tarpenning, and Ian Wright in 2003. Elon Musk is also the CEO of SpaceX and Neuralink.

examples/pinecone/src/index.ts

@@ -25,7 +25,7 @@ const llmApplication = await new RAGApplicationBuilder()
 
 await llmApplication.addLoader(new YoutubeLoader({ videoIdOrUrl: 'pQiT2U5E9tI' }));
 await llmApplication.addLoader(new SitemapLoader({ url: 'https://tesla-info.com/sitemap.xml' }));
-await llmApplication.addLoader(new WebLoader({ url: 'https://en.wikipedia.org/wiki/Tesla,_Inc.' }));
+await llmApplication.addLoader(new WebLoader({ urlOrContent: 'https://en.wikipedia.org/wiki/Tesla,_Inc.' }));
 
 console.log((await llmApplication.query('Who founded Tesla?')).result);
 // The founder of Tesla is Elon Musk. He co-founded the company with JB Straubel, Martin Eberhard, Marc Tarpenning, and Ian Wright in 2003. Elon Musk is also the CEO of SpaceX and Neuralink.

examples/simple/src/index.ts

@@ -7,7 +7,7 @@ const llmApplication = await new RAGApplicationBuilder().setSearchResultCount(30
 
 await llmApplication.addLoader(new YoutubeLoader({ videoIdOrUrl: 'pQiT2U5E9tI' }));
 await llmApplication.addLoader(new SitemapLoader({ url: 'https://tesla-info.com/sitemap.xml' }));
-await llmApplication.addLoader(new WebLoader({ url: 'https://en.wikipedia.org/wiki/Tesla,_Inc.' }));
+await llmApplication.addLoader(new WebLoader({ urlOrContent: 'https://en.wikipedia.org/wiki/Tesla,_Inc.' }));
 
 console.log((await llmApplication.query('Who founded Tesla?')).result);
 // The founder of Tesla is Elon Musk. He co-founded the company with JB Straubel, Martin Eberhard, Marc Tarpenning, and Ian Wright in 2003. Elon Musk is also the CEO of SpaceX and Neuralink.

examples/vertexai/src/index.ts

@@ -13,7 +13,7 @@ const llmApplication = await new RAGApplicationBuilder()
 
 await llmApplication.addLoader(new YoutubeLoader({ videoIdOrUrl: 'pQiT2U5E9tI' }));
 await llmApplication.addLoader(new SitemapLoader({ url: 'https://tesla-info.com/sitemap.xml' }));
-await llmApplication.addLoader(new WebLoader({ url: 'https://en.wikipedia.org/wiki/Tesla,_Inc.' }));
+await llmApplication.addLoader(new WebLoader({ urlOrContent: 'https://en.wikipedia.org/wiki/Tesla,_Inc.' }));
 
 let question = 'Who founded Tesla?';
 console.log('[QUESTION]', question);