Loading multiple documents of the same type

[ashryanbeats]

Hello!

Context

I'm up and running with the following basic setup:

const ragApplication = await new RAGApplicationBuilder()
  .setModel(SIMPLE_MODELS["OPENAI_GPT3.5_TURBO"])
  .setEmbeddingModel(new OpenAi3SmallEmbeddings())
  .setVectorDb(new LanceDb({ path: path.resolve("./db") }))
  .setCache(new LmdbCache({ path: path.resolve("./cache") }))
  .addLoader(new WebLoader({ urlOrContent: "https://www.somesite.whatevs" }))
  .build();

This is working great.

Question

Say for example, I have multiple webpages on the same domain, or multiple text files that I want to pass in.

What is the intended code pattern for loading multiple documents of the same type?

Should I call addLoader() with a new XYZLoader for each document before I build?

[ashryanbeats]

I was just writing about EmbedJS here and may have answered my own question.

From the EmbedJS readme:

You can add new loaders at any point dynamically (even after calling the build function on RAGApplicationBuilder). To do this, simply call the addLoader method -

await ragApplication.addLoader(new YoutubeLoader({ videoIdOrUrl: 'pQiT2U5E9tI' }));

So, instead of adding all loaders in the builder, you can probably just wrap the addLoader() call in a function that iterates over an array of loaders later in your application.

[adhityan]

Hello @ashryanbeats, thank you for helping others learn to build quick RAG applications using embedJs. You are indeed correct, one can add loaders at any time using addLoader on the RAGApplication instance.

This can be inside a for await loop or even via Promise.all. If you want to add a whole

[ashryanbeats]

Awesome! I tried it out and wrote a blog post explaining it here:

https://www.ashryan.io/batch-loading-resources-in-embedjs-for-rag/

[adhityan]

Sorry I missed my thank you note @ashryanbeats. The library has made progress since then and is easier to use now.