Add experimental question rephrase and chunks rerank

[fkesheh]

Question rephraser, changes the user question before the embeddings, improving what is retrieved. It also allows multi-turn conversation as it rephrases the question using the context and the prompt. Very useful for assitants.

Untitled.video.-.Made.with.Clipchamp.3.mp4

[mckaywrigley]

Love the idea. Going to dig into this a bit more on Thursday.

[fkesheh]

Love the idea. Going to dig into this a bit more on Thursday.

Right, then I will try also push in the reranker by today.

[fkesheh]

Added the re-ranker in this PR as well. The re-ranker will take all the chunks and evaluate them under the question light and will return the Top chunks to the LLM. It's based on this post: https://medium.com/@foadmk/enhancing-data-retrieval-with-vector-databases-and-gpt-3-5-reranking-c58ec6061bde

[fkesheh]

Hello, indeed, this functionality is activated only when there are files present in the chat.

The purpose of the rephraser is to modify your query by considering the surrounding conversation. The parameters you mentioned adjusts how much of the conversation history it's allowed to view. This is particularly handy for follow-up questions where the context might be minimal, such as when asking for further details on a specific point. The rephraser also proves useful in cases where the initial question lacks sufficient context. You can find more details about its application here (https://twitter.com/FKesheh84/status/1767184356009710029?t=68zSSXNMdQV-0ty5c5Z66w&s=19). Essentially, the rephraser aims to generate text that improves the retrieval of relevant information from databases.

On the other hand, the reranker operates towards the end of the process. It assesses the information chunks pulled from the database and selects the most appropriate ones in response to the query. This function is somewhat similar to the reranking process by Cohere. Due to its need to process extensive information, the reranker utilizes nearly the entire context capacity (16k tokens), which incurs costs. While this may be costly in GPT-4, it is manageable in GPT-3.5. For a detailed explanation of the reranker, refer to this article: https://medium.com/@foadmk/enhancing-data-retrieval-with-vector-databases-and-gpt-3-5-reranking-c58ec6061bde.

After the reranker, then the selected chunks are passed back to the user selected model (i.e. GPT-4) to generate the final answer.

These methodologies significantly enhance the Retrieval-Augmented Generation (RAG), making it effective and reliable for answering straightforward questions. However, this approach is not flawless. A more robust alternative for production environments might be the ReAct agent.

[fkesheh]

This is the sequence diagram:

I will double check if there is any issue and left a comment here

[spammenotinoz]

Thank-you, this is working really well!!

[ivanfioravanti]

Amazing job @fkesheh

[ndroo]

This is certainly a scope creep suggestion but is there any chance there's the ability for something like this to be re-factored to customize the model responding? Ie if the user wants to ask about an image, but the model isnt 4o, can we make it send the request to 4o instead so the user gets a response that isn't just "I cant do that..." ?