When rag's chatbot is called through the api, the llm context is lost

[jiandanfeng]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.7

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

1. Create a RAG chatbot
   2.Call via api
   3.Tracing logs show that the llm context is lost
   
   
   4.During the debug and preview execution of the orchestration, view the trace log, the context is normal, as shown in the following figure
   

✔️ Expected Behavior

The context of the llm is not lost

❌ Actual Behavior

The context of the llm is lost

[dosubot]

The issue with the LLM context getting lost when calling your RAG chatbot via the API might be due to a mismatch in the context_size parameter or how the context is managed in the API request. The context_size for the meta/llama2-70b model is set to 4096 tokens, so ensure your API call's context does not exceed this limit. Additionally, review the handling of the context in your API requests, especially since the context appears normal during debug and preview executions but is lost during actual API calls.

For a self-hosted Docker instance of Dify, consider that the handling of context might differ from standard deployments. Check the docker-compose.middleware.yaml for Docker-specific configurations that could affect context management. Also, ensure you're following Dify's best practices for managing the LLM context during API calls, including parameters like max_tokens, temperature, top_p, frequency_penalty, and presence_penalty.

If the issue persists, providing more details about your API call, including the exact request and response, along with any relevant code or configuration details, will be crucial for further troubleshooting.

Details

Sources

• Context gets lost between statements in a chat, makes chat unreliable.
• api/core/model_runtime/model_providers/leptonai/llm/mixtral-8x7b.yaml
• api/core/model_runtime/model_providers/nvidia/llm/llama2-70b.yaml
• en/user-guide/launching-dify-apps/developing-with-apis/README.md
• en/features/workflow/nodes/llm.md
• en/user-guide/creating-dify-apps/llms-use-faq.md
• en/getting-started/install-self-hosted/environments.md
• docker/docker-compose.middleware.yaml
• en/getting-started/install-self-hosted/start-the-frontend-docker-container.md

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[jiandanfeng]

@dosu API call sample:
curl --location --request POST 'http://xxxxxx/v1/chat-messages'
--header 'Content-Type: application/json'
--header 'Authorization: Bearer app-xxxxx'
--data-raw '{
"inputs":{},
"query": "哪些情况下不能退订服务？",
"response_mode": "blocking",
"conversation_id":"",
"user": "abc-1234"
}'

response:

{
"event": "message",
"task_id": "b670aebb-02dc-4eaf-8fe0-66531c92cec7",
"id": "09fed329-1033-4254-8cdd-fa22c6a2a355",
"message_id": "09fed329-1033-4254-8cdd-fa22c6a2a355",
"conversation_id": "4259b53b-4674-41a5-8e7a-1d8071f4a55c",
"mode": "advanced-chat",
"answer": "通常情况下，不能退订服务的情况可能包括以下几种：\n\n1. 服务合同中有不可退订条款：如果在签订服务合同时，合同中明确规定了不可退订的条款，那么在合同有效期内，您可能无法退订该服务。\n\n2. 服务已经开始或正在进行中：某些服务一旦开始提供，可能不允许退订，特别是那些具有时效性或定制化的服务。例如，已经开始的课程、咨询会话或定制的产品等。\n\n3. 退订窗口期已过：很多服务会提供一个退订窗口期，比如试用期内可以退订。一旦这个窗口期过去，可能就不再接受退订请求。\n\n4. 技术或系统限制：有时候，由于技术原因或系统限制，可能暂时无法进行退订操作。这通常发生在系统维护、更新或故障时。\n\n5. 法律和监管要求：在某些情况下，由于法律或监管要求，服务提供商可能不得不拒绝退订请求。例如，某些金融服务或公共事业服务可能受到特定法规的约束。\n\n请注意，具体情况还需根据您所使用服务的具体条款和政策来确定。如果您不确定自己是否能够退订某项服务，建议您查阅相关的服务协议或联系服务提供商的客服部门进行咨询。",
"metadata": {
"usage": {
"prompt_tokens": 110,
"prompt_unit_price": "0.0",
"prompt_price_unit": "0.0",
"prompt_price": "0.0",
"completion_tokens": 250,
"completion_unit_price": "0.0",
"completion_price_unit": "0.0",
"completion_price": "0.0",
"total_tokens": 360,
"total_price": "0.0",
"currency": "USD",
"latency": 22.998089984059334
}
},
"created_at": 1715842244
}

[dosubot]

To address the issue of the LLM context getting lost during API calls in your self-hosted Docker setup with Dify, consider the following steps:

1. Check Model Configuration: Ensure your llama2-70b.yaml configuration correctly sets mode: chat and context_size: 4096. This is essential for maintaining context in chat scenarios.

2. Review Parameter Rules: Make sure the parameter_rules in the YAML configuration are correctly applied in your API calls, particularly max_tokens, temperature, top_p, frequency_penalty, and presence_penalty.

3. Docker Configuration: Verify your docker-compose.middleware.yaml for any settings that might affect context management. Ensure the Docker container has sufficient memory and CPU resources. If integrating with Ollama, set the Base URL correctly to avoid connection issues, using http://host.docker.internal:11434 for Docker deployments.

4. Session Management: Implement or refine session management to ensure continuity across API calls. This may involve storing and retrieving context for each interaction.

5. Logging: Use logging to monitor API requests and responses for insights into where the context might be getting lost.

6. Ollama Service Accessibility: If using Ollama, ensure it's accessible from the Docker container, using host.docker.internal instead of localhost or 127.0.0.1.

By following these steps and ensuring your setup is correctly configured, you should be able to resolve the context loss issue.

Details

Sources

• api/core/model_runtime/model_providers/nvidia/llm/llama2-70b.yaml
• en/tutorials/model-configuration/ollama.md
• en/user-guide/launching-dify-apps/developing-with-apis/README.md
• api/core/workflow/nodes/llm/llm_node.py
• docker/docker-compose.middleware.yaml

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[jiandanfeng]

context is lost when the knowledge base is output to the llm @dosu

[jiandanfeng]

When recreate a new rag chatbot by import dsl, api call is ok.