Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: nemoguardrails server api streaming is not working #893

Open
4 tasks done
whisper-bye opened this issue Dec 4, 2024 · 1 comment
Open
4 tasks done

bug: nemoguardrails server api streaming is not working #893

whisper-bye opened this issue Dec 4, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@whisper-bye
Copy link

Did you check docs and existing issues?

  • I have read all the NeMo-Guardrails docs
  • I have updated the package to the latest version before submitting this issue
  • (optional) I have used the develop branch
  • I have searched the existing issues of NeMo-Guardrails

Python version (python --version)

3.11

Operating system/version

15

NeMo-Guardrails version (if you must use a specific version and not the latest

0.11.0

Describe the bug

I started a server with the following command, but streaming mode is not working

python -m nemoguardrails server --config=./config --default-config-id=config

config.yml

models:
  - type: main
    engine: openai
    model: gpt-4o-mini

Steps To Reproduce

my test code

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()

client = OpenAI(
    base_url="http://localhost:8000/v1"
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Hello"},
    ],
    stream=True,
    extra_body={
        "config_id": "config"
    }
)

for chunk in response:
    print(chunk.choices[0].delta.content)

Expected Behavior

print stream chunks

Actual Behavior

print nothing

@whisper-bye whisper-bye added bug Something isn't working status: needs triage New issues that have not yet been reviewed or categorized. labels Dec 4, 2024
@Pouyanpi
Copy link
Collaborator

Pouyanpi commented Dec 5, 2024

Hi @whisper-bye, thanks for opening this issue.

The problem is that the server is not OpenAI compatibe (It does not implement OpenAI's interface). We will integrate the OpenAI conpatible version at some point in future.

@Pouyanpi Pouyanpi removed the status: needs triage New issues that have not yet been reviewed or categorized. label Dec 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants