Provide more restrictive defaults for white space patterns in JSON #839
Labels
Comments
rlouf
pushed a commit
that referenced
this issue
May 24, 2024
Fixes #839 #908 #690 #450 ## Problem A major problem, especially with smaller language models, is the repetition problem. For example, let's say a model is generating json and must provide 12 space tokens for indentation in json output. Often a language model will assign a high probability to a 13th space token, and do the same for a 14th space, and then enter an infinite space generation loop. This is a problem with NLG that has been known for half a decade, but only has mitigations (mirostat, repetition penalty, using hundreds of billions of weights, etc), no absolute solutions (except for **structured generation**) ## Solution For structured json generation, we set a sane default whitespace pattern of `r"[ ]?"`. This removes all newlines and indentation. It disallows any syntactic whitespace beyond a single space separator. Users can still set the argument `whitespace_pattern=` if they want different behavior
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Repeated new lines is a common issue when generating JSON so many users end up setting the value of the
whitespace_patternargument. We should probably set a more restrictive limit on the number of consecutive white spaces and new lines.The text was updated successfully, but these errors were encountered: