Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

getting error "conversation not found" #147

Closed
jooles opened this issue Oct 30, 2024 · 8 comments
Closed

getting error "conversation not found" #147

jooles opened this issue Oct 30, 2024 · 8 comments
Assignees
Labels
bug Something isn't working

Comments

@jooles
Copy link

jooles commented Oct 30, 2024

🐛 Describe the bug

I'll use a simple example:

import { RAGApplicationBuilder } from '@llm-tools/embedjs';
import { OpenAi, OpenAiEmbeddings } from '@llm-tools/embedjs-openai';
import { WebLoader } from '@llm-tools/embedjs-loader-web';
import { HNSWDb } from '@llm-tools/embedjs-hnswlib';
import { RedisCache } from '@llm-tools/embedjs-redis';
import path from 'path';

//Replace this with your OpenAI key
process.env.OPENAI_API_KEY = "..."

const ragApplication = await new RAGApplicationBuilder()
.setModel(new OpenAi({ modelName: 'gpt-4' }))
.setEmbeddingModel(new OpenAiEmbeddings())
.setVectorDb(new HNSWDb())
.setCache(new RedisCache({ redisUrl: 'redis://localhost:6379', keyPrefix: 'rag-demo' }))
.build();

let loader1 = await ragApplication.addLoader(new WebLoader({ urlOrContent: 'https://www.forbes.com/profile/elon-musk' }));

let result = await ragApplication.query('What is the net worth of Elon Musk today?')
console.log(result)
//Answer: The net worth of Elon Musk today is $258.7 billion.

Unfortunately, I get the following error when executing:

            throw new Error('Conversation not found');
                  ^

Error: Conversation not found
    at RedisCache.getConversation ...

what am I doing wrong?

@adhityan
Copy link
Collaborator

Looks like this is a bug. I think I have addressed this and published a new version - 0.1.13. Could you update to this version and rerun your app and let me know if this is resolved?

@adhityan adhityan added the bug Something isn't working label Oct 30, 2024
@jooles
Copy link
Author

jooles commented Oct 30, 2024

@adhityan Thank you! it works now!
I can see the conversation with a specified id being stored.

However, shouldn't any subsequent run (with the same query and same conversation id) be much faster due to the cache?

I see different outputs from the llm and it's "slow", so it looks like no cache is being used.
Am I missing somehing?

@adhityan
Copy link
Collaborator

I see what you mean and perhaps the documentation should make this clearer / we should really rename this feature. The cache is not used to cache the query response against a query seen already.

It's used to store the metadata around loaders, conversations, etc. It makes sure that a document or URL (or any already seen loader) is not re-processed. It also stores info around if a particular chunk is already seen and avoid creating new embedding for it. Apart from this, it stores conversation data so that users can get more contextual response. A better name from cache is database and maybe we should really rename it down the line.

The app sends all conversations in a topic (upto a limit after which it prunes older entries) to the LLM as history so the LLM knows what the user is talking about. This is useful if you ask follow-up questions. By default, if no optional conversationId is provided - it uses a default conversationId called default. You should use one conversationId per user in your app ideally (unless its a single user app in which case it doesn't matter). In this way, every user gets a personalized response.

Tl;dr - the more you chat with the app, the more the content sent to the LLM and thus the slower it gets. The next version of embedJs will allow an option to turn off the conversations feature altogether - allowing you to have no history.

Hope this was helpful.

@jooles
Copy link
Author

jooles commented Oct 30, 2024

@adhityan Thank you for the clarification, that was indeed very helpful.

But then something else, maybe I'm understanding it completely wrong.
When I keep running the application, it seems to create the embeddings again and again. This obviously costs tokens and is slow.
Shouldn't the cache prevent this from happening, if I understood you correctly?

Currently, each query (regardless of whether cache is active or not, regardless of whether it's a new conversationId or not) with the code above takes about 2 seconds, or even longer.

I came across this library through some searching and it looked very promising. Because I actually want to use it to build my own "knowledge bases" for a twilio voice bot (e.g. with the realtimeAPI).

Thanks again for a response and information regarding this as well.

@adhityan
Copy link
Collaborator

It should not be re-creating any embedding again when you have the cache set. I will check if there is a bug here and if so address it with a quick bugfix release tom. Will keep this ticket open in the meantime.

The response from the LLM can take some time but its usually fast. There is a work in progress to allow streaming responses from the LLM which should bring the time to first words lower.

@jooles
Copy link
Author

jooles commented Oct 30, 2024

Thank you for looking into it!

Is there a way to debug

  • to see if the embeddings are only created once
  • the request and response times from the LLM
    ?

You're support is really kind. Really appreciate it! 🙏

@adhityan
Copy link
Collaborator

adhityan commented Nov 1, 2024

You can get a lot of debug logs by setting the following environment variable DEBUG=embedjs:*.

I pushed across an old refactor topic which renames caches to stores along with a number of internal changes. This issue made me realize its important to address this before new loaders / databases are added in. If you update to version 0.1.14 you will need to make a few changes. Read the updated documentation and sample app for more info,

@adhityan adhityan closed this as completed Nov 2, 2024
@jooles
Copy link
Author

jooles commented Nov 2, 2024

Thank you very much @adhityan!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants