Support "Grounding with Google Search" in Google provider

[wong2]

Feature Description

The Gemini API has a new feature where you can ground results with Google search which reduces hallucinations and provides references to content:
https://ai.google.dev/gemini-api/docs/grounding

[adolphnov]

I have tried vertex is valid, it will report an error in Google AI Studio, which does not support this parameter. And note that when used in the vertex, no tools can be included, otherwise an error will also be reported.
I hope this is helpful to you.

[lgrammel]

@shaper do we support this with the latest provider versions?

[shaper]

We do support grounding with the Google Vertex 2.0+ google-vertex provider if you pass useSearchGrounding: true in the provider settings. It is still early support, we would value your input on how you need or would like to use it.

I also observed that grounding and tool use are mutually exclusive -- if you specify grounding you cannot specify additional tools with a Google-API-side restriction error. I believe the Vertex provider will ignore other tools if you specify grounding.

Notes on the current implementation:

• we don't currently parse and include the grounding metadata in the response yet with the references to content (we should do this of course)
• the configuration allowing you to specify a threshold looks new and we don't support this yet:

"dynamicRetrievalConfig": {
  "mode": "MODE_DYNAMIC",
  "dynamicThreshold": DYNAMIC_THRESHOLD
}

• per the above the GenAI API (google provider) may now support grounding, we'd need to validate/add support for that, at the time of implementation I believe it was Vertex-only and so it is not present in google
• there is add'l support possible in Vertex to allow grounding vs your own enterprise data which would require further work

[shaper]

The above changes were just published to the package versions in #4081

This enables search grounding support in the Google Generative AI google package, similar to what we've had in Google Vertex.

We plan to add support for customizing the dynamic retrieval config soon. At that time I would consider this issue complete. If there is interest in the enterprise data support mentioned above please file that as a separate issue.

[wottpal]

Hey there, dope addition! Just played with it a bit and got it to work pretty well.

Some observations though:

1. It doesn't seem to work with generateObject, only generateText. At least I was not able to force the model to actually DO search when generating object output. Is that related to mutual exclusiveness with tool calls, you're mentioning above, @shaper? If yes, it should probably be mentioned in the docs.
2. Being able to set the dynamic retrieval threshold is crucial for a lot of use cases IMO. Because in my observations the model sometimes tends to NOT google even though I nudge it towards it in my prompt a lot. Would be amazing to see this soon.

Lmk whether I should open separate issues for those if you want to close this one.

[shaper]

Hey there, dope addition! Just played with it a bit and got it to work pretty well.

Some observations though:

1. It doesn't seem to work with generateObject, only generateText. At least I was not able to force the model to actually DO search when generating object output. Is that related to mutual exclusiveness with tool calls, you're mentioning above, @shaper? If yes, it should probably be mentioned in the docs.

Yes, probably -- I will check this and update docs if needed.

2. Being able to set the dynamic retrieval threshold is crucial for a lot of use cases IMO. Because in my observations the model sometimes tends to NOT google even though I nudge it towards it in my prompt a lot. Would be amazing to see this soon.

It is on my radar, will try to get to it soon.

Lmk whether I should open separate issues for those if you want to close this one.

No need, let's keep this one for now.

[sroze]

I just created another issue (and a proposed implementation plan) for the "enterprise" feature of grounding in your own data:
#4230