Stop using is_final_state and final_states

[lapp0]

What behavior of the library made you think about the improvement?

The verbiage is a bit confusing. RegexGuide.is_final_state currently returns both EOS-required AND EOS-accepting states.

This has resulted in a few bugs

• Endless generation bug popped up during migration to Guide in vLLM integration #856
• Remove broken final state loop #874 (comment)

The ambiguity has resulted in bugs at nearly every step of development, from the previous FSM class, to the original set of CFG LogitsProcessors, just from my recollection.

How would you like it to behave?

1. Investigate whether is_termination_accepting_state is necessary anywhere. It probably is necessary to determine whether EOS is legal in some places.
2. Implement is_halted_state. This is the set of states which must produce EOS
3. Replace dependencies on is_final_state s.t. they use the appropriate new function, e.g. https://github.com/outlines-dev/outlines/blob/4f8433d8d6633b0780c3a6c27981f9adffbe49f5/outlines/generate/generator.py#L94