[Bug]: Vertex AI - Code Gecko stream not working

[ishaan-jaff]

What happened?

trying to use code-gecko from Vertex - The stream parameter doesn't work:

Relevant log output

curl https://llm/v1/completions --header "Authorization: Bearer xxx" --header 'Content-Type: application/json' -d '{"model": "code-gecko@002", "stream": true, "prompt": "def is_num_prime(val):", "temperature": 0.7, "max_tokens": 100}'
data: {"error": {"message": "litellm.APIConnectionError: 500 Internal error encountered.\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/google/api_core/grpc_helpers_async.py\", line 77, in wait_for_connection\n    await self._call.wait_for_connection()\n  File \"/usr/local/lib/python3.11/site-packages/grpc/aio/_call.py\", line 659, in wait_for_connection\n    await self._raise_for_status()\n  File \"/usr/local/lib/python3.11/site-packages/grpc/aio/_call.py\", line 272, in _raise_for_status\n    raise _create_rpc_error(\ngrpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:\n\tstatus = StatusCode.INTERNAL\n\tdetails = \"Internal error encountered.\"\n\tdebug_error_string = \"UNKNOWN:Error received from peer ipv4:108.177.15.95:443 {grpc_message:\"Internal error encountered.\", grpc_status:13, created_time:\"2024-12-17T00:38:04.577879316+00:00\"}\"\n>\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.55.9

Twitter / LinkedIn details

No response

[ishaan-jaff]

• we should move code-gecko to using vertex_httpx

[emerzon]

Issue also happens to other vertex models, such as codestral - It seems that they do not support the stream feature at all, but the Vertex streaming API just fails with Server Error if you try to use it.