FEATURE REQUEST: Support for structured response

[BJG87]

I was just wondering if you had any plans to integrate the option to request a structured response in the form of JSON from Gemini by providing schema?

I have used schemas in the past using the node js library and they can be really effective when you need to force a specific response as sometimes Gemini can waffle on even when you ask it to be specific. Link to Gemini Structured Response

I could possibly look into adding it myself but I didn't want to waste my time if it's already been tried and abandoned for some reason I have not yet realised.

[mhawksey]

Hi Brett - a useful request. I've pushed an update to a branch for you to test it hasn't broken any of your existing uses of the GeminiApp library. Here is the updated library code and a test you can try

[BJG87]

Thanks very much. I've tested the new library code and it doesn't seem to have affected any old usage. I am getting an error though when trying to run the test code your provided.

Request to Gemini failed with response code 400 - {
  "error": {
    "code": 400,
    "message": "Invalid JSON payload received. Unknown name \"responseMimeType\" at 'generation_config': Cannot find field.\nInvalid JSON payload received. Unknown name \"responseSchema\" at 'generation_config': Cannot find field.",
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.BadRequest",
        "fieldViolations": [
          {
            "field": "generation_config",
            "description": "Invalid JSON payload received. Unknown name \"responseMimeType\" at 'generation_config': Cannot find field."
          },
          {
            "field": "generation_config",
            "description": "Invalid JSON payload received. Unknown name \"responseSchema\" at 'generation_config': Cannot find field."
          }
        ]
      }
    ]
  }
}

I'm not sure if it has something to do with the generationConfig object being in camel case compared to the generation_config object mentioned in the error message being in snake case.

The documentation seems a little unclear about this and when I tried other generationConfig options like "maxOutputTokens" it worked fine. It just doesn't like "responseMimeType" and "responseSchema".

I also tried different Gemini models but I got the same error message.

[mhawksey]

ah - if you are using a api key from Google AI Studio then the issue is likely to be you need to declare apiVersion: 'v1beta' like this (catches everyone out as the documentation isn't clear)

  const model = genAI.getGenerativeModel({
    model: "gemini-1.5-pro", 
    apiVersion: 'v1beta',
    generationConfig: {
      responseMimeType: "application/json",
      responseSchema: schema
    }
  });

[BJG87]

That almost worked but it did put me on the right track. Where you put the apiVersion:'v1beta was in modelParams rather than requestOptions.

This worked for me:

const model = genAI.getGenerativeModel({
    model: "gemini-1.5-pro", 
    generationConfig: {
      responseMimeType: "application/json",
      responseSchema: schema
    }
  }, {
    apiVersion: 'v1beta',
  });

[mhawksey]

Losing the plot - glad you figured it out :)