Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

api推理,正常打印info,无报错但无响应 #6576

Closed
1 task done
BreezeHavana opened this issue Jan 9, 2025 · 3 comments
Closed
1 task done

api推理,正常打印info,无报错但无响应 #6576

BreezeHavana opened this issue Jan 9, 2025 · 3 comments
Labels
invalid This doesn't seem right

Comments

@BreezeHavana
Copy link

Reminder

  • I have read the README and searched the existing issues.

System Info

  • llamafactory version: 0.9.2.dev0
  • Platform: Linux-5.4.0-150-generic-x86_64-with-glibc2.31
  • Python version: 3.11.5
  • PyTorch version: 2.5.1+cu124 (GPU)
  • Transformers version: 4.46.1
  • Datasets version: 3.1.0
  • Accelerate version: 1.0.1
  • PEFT version: 0.12.0
  • TRL version: 0.9.6
  • GPU type: NVIDIA A100 80GB PCIe
  • vLLM version: 0.6.6.post1

Reproduction

Put your message here.

Others

No response

@github-actions github-actions bot added the pending This problem is yet to be addressed label Jan 9, 2025
@BreezeHavana BreezeHavana changed the title api推理,正常打印info api推理,正常打印info,无报错但无响应 Jan 9, 2025
@BreezeHavana BreezeHavana reopened this Jan 9, 2025
@BreezeHavana
Copy link
Author

服务器使用API_PORT=8000 CUDA_VISIBLE_DEVICES=0 llamafactory-cli api examples/inference/llama3.yaml,正常启动,无报错;

@BreezeHavana
Copy link
Author

终端调用代码:
from openai import OpenAI
client = OpenAI(api_key="0",base_url="http://服务器IP:8000/v1")
messages = [{"role": "user", "content": "Who are you?"}]
result = client.chat.completions.create(messages=messages, model="服务器上模型地址")
print(result.choices[0].message)

@BreezeHavana
Copy link
Author

问题描述:服务器正常打印info,显示message,但终端无响应。
尝试排错:1、在服务器上直接调用API,修改ip为0.0.0.0,仍无响应无报错;
2、直接使用transformers.pipeline调用预训练模型,模型正常推理,预训练文件正常

@hiyouga hiyouga added invalid This doesn't seem right and removed pending This problem is yet to be addressed labels Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
invalid This doesn't seem right
Projects
None yet
Development

No branches or pull requests

2 participants