Detailed information on missing response fields

[viktorvsk-dualentry]

Consider the following example

class UserOut(Schema):
    id: int
    name: str
    
@router.get("/", response=list[UserOut])
def list_users(request):
    return User.objects.all().values("id", "name")

Now imagine by some reason (a bug for example) some users names are None. We will get entire request failed and the output would be something like:

["missing response.0.name: Field required"]

The question is, what are the best practices to debug here? Because 0 in response.0 here means a position of the failing record in the resulting list. But we are actually interested in figuring out a User id in order to investigate why was its name set to None and being able to update it.

The best of course would be to alter error message to:

["missing response.<User id=1>.name: Field required"]

But its probably too complicated to achieve, so maybe its possible to intercept this error and log entire response object before validation?

I've looked through the website, issues and discussions and haven't found anything similar. If its not possible to achieve right now, could you please point me roughly to the code responsible for handling errored responses so I would try to come up with a PR ?

[vitalik]

Now imagine by some reason (a bug for example) some users names are None

I guess you should make you name to accept None values:

class UserOut(Schema):
    id: int
    name: Optional[str]

[viktorvsk-dualentry]

Yeah, optional name works, but I'm thinking of exceptional cases where something went wrong because we missed it.
In such a case we should name every field optional which won't help a lot.

This example is simplified, on practice we have multiple fields on multiple endpoints, some of them are computable etc.

Right now we intercept ValidationError exceptions but they are probably missing information of the raw records if I understand correctly

[vitalik]

The question is, what are the best practices to debug here?

well it's a matter on what you want to achieve

if you want your data to be clean - then having error is better - the sooner detected the better

if you want to silence it - you can add pydantic validador on that field - replace it with some "" value but log it into some system to check later

[viktorvsk-dualentry]

I want data to be clean and having error sooner is great. However, after I see those errors, there is no context to understand what exact records have those errors. And for that I have to go to the database and basically search through all of the records to find out which exact record (or a set of records) caused this error. As in the example above, if I see that some User doesn't have a name, I have to go to the database, filter users without a name and start debugging. Instead, I would like to have and ID of a user without a name closer to the error message.

Right now I'm using this monkey patch for that:

# Here we augment Exception with the actual result that caused for better logging and debugging.
def patch_ninja():
    import contextlib
    from typing import Any

    from django.http import HttpRequest
    from django.http.response import HttpResponseBase
    from ninja.operation import Operation

    def patched_run(self, request: HttpRequest, **kw: Any) -> HttpResponseBase:  # noqa: ANN401
        error = self._run_checks(request)
        if error:
            return error
        try:
            temporal_response = self.api.create_temporal_response(request)
            values = self._get_values(request, kw, temporal_response)
            result = self.view_func(request, **values)
            return self._result_to_response(request, result, temporal_response)
        except Exception as e:
            if isinstance(e, TypeError) and "required positional argument" in str(e):
                msg = "Did you fail to use functools.wraps() in a decorator?"
                msg = f"{e.args[0]}: {msg}" if e.args else msg
                e.args = (msg,) + e.args[1:]

            # THIS WAS ADDED
            # result variable may be undefined in case an exception happend in self.view_func
            with contextlib.suppress(NameError):
                e.add_note(str(result))
            # THIS WAS ADDED

            return self.api.on_exception(request, e)

    Operation.run = patched_run

What do you think about it? Is there a cleaner way to achieve this with the current code? Can it become useful for somebody else and make it to the upstream to make it a PR ?