You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What behavior of the library made you think about the improvement?
The current structured generation code is creating a -inf copy of the logits array and setting the allowed token ID indices to the corresponding values in the original logits array. See here.
How would you like it to behave?
When possible, the original logits array should be updated in-place and completely avoid creating a new array. This change would likely require the set of disallowed token IDs instead of the allowed ones.
The text was updated successfully, but these errors were encountered:
What behavior of the library made you think about the improvement?
The current structured generation code is creating a
-inf
copy of the logits array and setting the allowed token ID indices to the corresponding values in the original logits array. See here.How would you like it to behave?
When possible, the original logits array should be updated in-place and completely avoid creating a new array. This change would likely require the set of disallowed token IDs instead of the allowed ones.
The text was updated successfully, but these errors were encountered: