Erroring with Pydantic defeats the purpose of Marvin

[landmann]

First check

• I added a descriptive title to this issue.
• I used the GitHub search to look for a similar issue and didn't find it.
• I searched the Marvin documentation for this feature.

Describe the current behavior

Seems like having Marvin error out when the response is outside the bounds of the responses defeats the purpose of marvin. If I'm giving marvin some bounds, it should use the numeric bounds to give me the response. At least, there should be a setting that lets me strictly adhere to said bounds. Otherwise there's no way for me to make sure the responses are valid, erroring my application.

Describe the proposed behavior

Said above.

Example Use

No response

Additional context

Traceback (most recent call last):
    process_text_data(url)
    ages = marvin.cast(
           ^^^^^^^^^^^^
    return run_sync(
           ^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/marvin/utilities/asyncio.py", line 106, in run_sync
    return context.run(asyncio.run, coroutine)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/marvin/ai/text.py", line 321, in cast_async
    return await _generate_typed_llm_response_with_tool(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/marvin/ai/text.py", line 187, in _generate_typed_llm_response_with_tool
    response = await generate_llm_response(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/marvin/ai/text.py", line 126, in generate_llm_response
    tool_outputs = _get_tool_outputs(request, response)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/marvin/ai/text.py", line 134, in _get_tool_outputs
    tool_output = marvin.utilities.tools.call_function_tool(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/marvin/utilities/tools.py", line 176, in call_function_tool
    output = tool.function._python_fn(**arguments)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/marvin/utilities/tools.py", line 85, in tool_fn
    return TypeAdapter(model).validate_python(data).value
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/pydantic/type_adapter.py", line 260, in validate_python
    return self.validator.validate_python(object, strict=strict, from_attributes=from_attributes, context=context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for FormatResponse
value.age
  Input should be less than or equal to 64 [type=less_than_equal, input_value=65, input_type=int]
    For further information visit https://errors.pydantic.dev/2.7/v/less_than_equal

[zzstoatzz]

hi @landmann - thanks for the issue

can you provide the example you used to produce that stack trace?

[pietz]

Yeah, I got the same error. I think it happens when you have a field defined like so:

rating: int = Field(..., ge=1, le=5, description="A rating from 1 to 5")

And the LLM returns a value outside this range. Are the <, >, <=, >= validators stored int he schema the model gets?