Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs should point out that only the EXTRACTIVE answer style will produce a "content" field if the answerable probability is low #83

Open
roschler opened this issue Apr 3, 2024 · 1 comment
Assignees
Labels
component:documentation Improvements or additions to documentation status:triaged Issue/PR triaged to the corresponding sub-team type:bug Something isn't working

Comments

@roschler
Copy link

roschler commented Apr 3, 2024

On this page:

https://ai.google.dev/api/rest/v1beta/models/generateAnswer

In the Response Body section, there is this text for the answer field:

"Candidate answer from the model.

Note: The model always attempts to provide a grounded answer, even when the answer is unlikely to be answerable from the given passages. In that case, a low-quality or ungrounded answer may be provided, along with a low answerableProbability."

While it is true that you will always get a Candidate answer object back, in the ABSTRACTIVE and VERBOSE answer styles, where the model is forced to synthesize text, you you will not get a content child object field containing answer text. When the answerableProbabillity is too low, you will only get a content child object with answer text in the EXTRACTIVE answer style, since the model just has to produce the grounding answer that is the closest match and does not have to synthesize any text to serve as a reply. So if you use the word "answer" in the context of a Candidate object, as the docs use it, then the docs are correct. But in the traditional sense of an answer containing text that replies to a question, the doc text is less helpful then it could be.

I suggest mentioning this in the docs, because I spent a fair amount of time trying to figure out why I didn't get any usable answer text due to this model behavior, until I wrote a rigorous test harness that tries the exact same user query and inline grounding passages set with each of the 3 AnswerStyle values. A short doc note about this could save other developers some confusion.

@roschler roschler changed the title Docs should point out that only the EXTRACTIVE answer style will produce content if the answerable probability is low Docs should point out that only the EXTRACTIVE answer style will produce a "content" field if the answerable probability is low Apr 4, 2024
@singhniraj08 singhniraj08 added type:bug Something isn't working component:documentation Improvements or additions to documentation status:triaged Issue/PR triaged to the corresponding sub-team labels Apr 4, 2024
@hsubox76
Copy link
Collaborator

This is for the REST API, as this SDK doesn't currently have a generateAnswer method. Let me see if I can find out where to route this bug to.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:documentation Improvements or additions to documentation status:triaged Issue/PR triaged to the corresponding sub-team type:bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants