fix(colang): use input rails output for intent generation

[niels-garve]

Related to #866

Description

Hi @drazvan,

I am creating an AI solution based on the interaction loop of Colang v2. To improve the intent detection for user messages like confirmations ("yes", "no", etc.) I'd like to add Query Rewriting; basically an LLM call prompting to rephrase the user message with context from the conversation.

Example:

User: Do you have any bikes?
Bot: Yes, we have a wide range of bikes. Would you like to see them?
User: sure

Now, query rewriting would transform the "sure" into "Show me some bikes!".

The Problem

The action generate_user_intent is called with user_action which is the final transcript of the UtteranceUserActionFinished. However, the input rails set the $user_message and $last_user_message through the context.

Here's my debugger:

The action has a typo: "Shoe me bikes", whereas the context has been rewritten to "show me bikes".

The solution

Let me first show my debugger again:

Same user input as above with typo: "Shoe me bikes".

Now, the context is consistent, and the example lookup works on the rewritten query.

Example and current workaround.

Currently, I am working around this issue by overriding flow _user_said_something_unexpected and performing three new steps:

1. match UtteranceUserActionFinished
2. Rewrite the transcript.
3. send UtteranceUserActionFinished with same action_uid and is_success as before but with updated transcript

Here's the code:

@override
flow _user_said_something_unexpected -> $event
    """Given the following conversation and a follow up input, rephrase the follow up input to be a standalone input,
    in its original language. Generate the output in the following format:

    $rephrased_user_message = '''<<the response>>'''

    Conversation:
    {{ history | co_v2 }}

    Question: {{ question }}

    Your Response:"""
    global $user_message
    global $last_user_message
    match UnhandledEvent(event="UtteranceUserActionFinished", loop_ids={$self.loop_id}) as $e
    $question = $e.final_transcript
    ...
    send UtteranceUserActionFinished(
        final_transcript=$rephrased_user_message, action_uid=$e.action_uid, is_success=$e.is_success
    ) as $event
    $user_message = $rephrased_user_message
    $last_user_message = $rephrased_user_message

Note: Please also refer to the custom prompt in config.yml, where I tried to do query rewriting and intent matching in one step, but examples would still be wrong without this fix.

Note: I'll gladly enhance the README of the query rewriting example once we find a solution.

Happy to hear your feedback!

Cheers,
Niels

Checklist

• I've read the CONTRIBUTING guidelines.
• I've updated the documentation if applicable.
• I've added tests if applicable.
• @mentions of the person or team responsible for reviewing proposed changes.

[schuellc-nvidia]

Hi @niels-garve, let me know if this answer resolves your issue. In general, you should not need to rewrite the user input to get context sensitive user intent detection.