END token enforcing

[wiiiktor]

1// Is this possible to enforce the LLM to finish the generation? At some point of time, LLM decides to finish it by itself, but can WE enforce it by putting a growing pressure to finish the ouptut? I expect that there is something like END token and we can push pressure on the LLM to use it. Just using stop=... is not that useful, because the output is brutally cut.

2// Are there any methods of enforcing approximate length of the LLM output, at all?..

[wiiiktor]

3// Is there a way to discourage LLM from giving us any other specific output? Let's say, I use a word "polite" in the instructions (just an example) and observe that LLM is quoting this instruction in the output (which I do not want). Can I somehow discourage LLM from generating the "polite" token?

[wiiiktor]

I see it on this discussion from 2020:
https://community.openai.com/t/get-all-requested-max-tokens-with-gpt-3-5-turbo-instruct/381922/7
I am mainly interested in LlamaCpp solution, not OpenAI.

[gmonair]

I was going to create a new thread, but since this seems close to what I wanted to add, might as well write it here. All of these would be nice to have, and would allow for even better control of local models when they go haywire in generation.

1. Adding the ability to stop generation based on a callback (gen gets a callback fn with some token_len / time_based - and calls that fn after tokens generated or time elapsed. The callback can decide if gen continues or if it stops).
2. Any way to set the equivalent of "repetition penalty" from other projects. I find that some codegen models get really weirded out by # comments and just repeat stuff (more than general models).