-
Notifications
You must be signed in to change notification settings - Fork 316
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Logits Processors Guide
integration will be buggy when len(tokens) > 1
in a Write
instruction
#855
Comments
Guide
integration will fail be buggy when len(tokens) > 1
in a Write
instructionGuide
integration will be buggy when len(tokens) > 1
in a Write
instruction
Hi @br3no , I have a couple of questions on this Issue. Can you pls share more detail on these?
|
ff-tokens are fast-forward tokens. When you are generating guided output, e.g. a json object, there are moments when you don't really need an LLM to generate the next tokens, because the next tokens are specified by the guide. This reduces the load on the GPU and is generally much faster, as you only need to traverse the state-machine.
|
Thank you @br3no ! Much appreciated! |
Describe the issue as clearly as possible:
See:
outlines/outlines/integrations/vllm.py
Line 110 in d6a2b79
Here the
tokens
field of the next instruction is treated equally regardless of whether it is of typeGenerate
orWrite
.If a
Write
instruction has atokens
field with length > 1, this means we will accept any of the next ff-tokens as the token in the next step. This is incorrect.Steps/code to reproduce the bug:
Expected result:
Error message:
No response
Outlines/Python version information:
Version information
Context for the issue:
Bug was discussed in a call with @rlouf.
The text was updated successfully, but these errors were encountered: