Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pydantic example from README crashes #841

Closed
crclark opened this issue Apr 27, 2024 · 2 comments
Closed

Pydantic example from README crashes #841

crclark opened this issue Apr 27, 2024 · 2 comments

Comments

@crclark
Copy link

crclark commented Apr 27, 2024

Describe the issue as clearly as possible:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[1], line 40
     37 rng = torch.Generator(device="cuda")
     38 rng.manual_seed(789001)
---> 40 character = generator("Give me a character description", rng=rng)
     42 print(repr(character))
     43 # Character(name='Anderson', age=28, armor=<Armor.chainmail: 'chainmail'>, weapon=<Weapon.sword: 'sword'>, strength=8)

File ~/<redacted>/lib/python3.10/site-packages/outlines/generate/api.py:207, in SequenceGenerator.__call__(self, prompts, max_tokens, stop_at, rng)
    205 while True:
    206     try:
--> 207         last_state = next(states)
    208         if max_tokens or stop_sequences:
    209             token_ids = last_state.token_ids

File ~/<redacted>/lib/python3.10/site-packages/outlines/generate/generator.py:81, in sequence_generator(model, sampler, fsms, token_ids, sequence_weights, attention_masks, fsm_states, rng)
     76     raise ContextLengthExceededError(
     77         "The input length exceeds the context length of the model."
     78     )
     80 allowed_tokens = get_allowed_tokens(fsms, fsm_states)
---> 81 biased_logits = bias_logits(logits, allowed_tokens)
     82 next_token_ids, ancestors, sequence_weights = sampler(
     83     biased_logits, sequence_weights, rng
     84 )
...
    304 for i, ids in enumerate(allowed_token_ids):
--> 305     biased_logits[i, ids] = logits[i, ids]
    306 return biased_logits

IndexError: index 54732 is out of bounds for dimension 0 with size 32000

Steps/code to reproduce the bug:

Copy this example from the readme into a notebook.

Expected result:

no crash

Error message:

see above

Outlines/Python version information:

Python 3.10.14

Library versions (let me know if you need more, there are many others that are non-ML that just add noise)

outlines                  0.0.40
torch                     2.3.0+cu118
transformers              4.40.1

Context for the issue:

No response

@lapp0
Copy link
Contributor

lapp0 commented May 16, 2024

Unfortunately I cannot reproduce. The only difference is I'm running torch==2.3.0 with torch.version.cuda == 12.1

My output is

Character(name='Alex Chen', age=28, armor=<Armor.leather: 'leather'>, weapon=<Weapon.bow: 'bow'>, strength=65)
Character(name='Boo', age=27, armor=<Armor.chainmail: 'chainmail'>, weapon=<Weapon.bow: 'bow'>, strength=70)

Could you please try the following:

    1. delete the model folder (it's somewhere in ~/.cache/huggingface/hub) so it's re-downloaded
    1. delete your outlines cache via import outlines.caching; outlines.caching.clear_cache()
    1. pip install --upgrade outlines

If that doesn't work, I'd like to know your full pip freeze, nvcc --version, nvidia-smi, and the model of your device please.

@rlouf
Copy link
Member

rlouf commented May 19, 2024

Closing for now, as it is not reproducible. Feel free to re-open if the error persists.

@rlouf rlouf closed this as not planned Won't fix, can't repro, duplicate, stale May 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants