You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently distilabel handles the formatting as part of the class Prompt that expects a system_prompt and a formatted_prompt, since it was the most suitable solution for the initial approach i.e. UltraFeedback, as we have a one-turn interaction where we have a system_prompt and a default formatting suited for a given task, in this case UltraFeedback. The Prompt class also contains a method named format_as that contains some standard formats, to go from system_prompt and formatted_prompt to the formatting expected by a given model, where the formatted_prompt is the first user message within a chat-like application, and the assistant follow-up is the completion to it i.e. the expected output.
But this approach is a bit weak under some scenarios, as multi-turn is not supported, and the formatting may get complex if i.e. there's not a system_prompt, the formatted_output is a sequence of messages, etc.
So on, IMHO we should refactor that to support only the following scenarios:
Pre-defined tasks such as UltraFeedbackTask produce a Prompt following OpenAI formatting, meaning a one-turn only; and the format conversion is handled within the Prompt (similarly to what's being done with the Chat class in Add ChatTask for multi-turn and follow-up generation #203)
Some helper functions can be provided to easily go from one formatting to another i.e. from Zephyr to OpenAI
This way we end up with a more custom experience, as the task is the one that decides how the formatting happens and the standard output is formatted as OpenAI specifies. Meaning that the input for a given generate_prompt function within a Task could handle both multi-turn and one-turn prompts if needed.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Description
Currently
distilabel
handles the formatting as part of the classPrompt
that expects asystem_prompt
and aformatted_prompt
, since it was the most suitable solution for the initial approach i.e. UltraFeedback, as we have a one-turn interaction where we have asystem_prompt
and a default formatting suited for a given task, in this case UltraFeedback. ThePrompt
class also contains a method namedformat_as
that contains some standard formats, to go fromsystem_prompt
andformatted_prompt
to the formatting expected by a given model, where theformatted_prompt
is the first user message within a chat-like application, and the assistant follow-up is the completion to it i.e. the expected output.But this approach is a bit weak under some scenarios, as multi-turn is not supported, and the formatting may get complex if i.e. there's not a
system_prompt
, theformatted_output
is a sequence of messages, etc.So on, IMHO we should refactor that to support only the following scenarios:
UltraFeedbackTask
produce aPrompt
following OpenAI formatting, meaning a one-turn only; and the format conversion is handled within thePrompt
(similarly to what's being done with theChat
class in AddChatTask
for multi-turn and follow-up generation #203)This way we end up with a more custom experience, as the task is the one that decides how the formatting happens and the standard output is formatted as OpenAI specifies. Meaning that the input for a given
generate_prompt
function within aTask
could handle both multi-turn and one-turn prompts if needed.Beta Was this translation helpful? Give feedback.
All reactions