gemini/multimodal-live-api/intro_multimodal_live_api.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oXnEutuDQa9c"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Getting Started with the Multimodal Live API in Vertex AI\n",
        "\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/intro_multimodal_live_api.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fmultimodal-live-api%2Fintro_multimodal_live_api.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/multimodal-live-api/intro_multimodal_live_api.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/intro_multimodal_live_api.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://upload.wikimedia.org/wikipedia/commons/9/91/Octicons-mark-github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/intro_multimodal_live_api.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/intro_multimodal_live_api.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/intro_multimodal_live_api.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/53/X_logo_2023_original.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/intro_multimodal_live_api.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/intro_multimodal_live_api.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "84f0f73a0f76"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "| Author(s) |  [Eric Dong](https://github.com/gericdong), [Holt Skinner](https://github.com/holtskinner) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "The Multimodal Live API enables low-latency bidirectional voice and video interactions with Gemini. The API can process text, audio, and video input, and it can provide text and audio output. This tutorial demonstrates the following simple examples to help you get started with the Multimodal Live API in Vertex AI.\n",
        "\n",
        "- Text-to-text generation\n",
        "- Text-to-audio generation\n",
        "- Text-to-audio conversation\n",
        "\n",
        "See the [Multimodal Live API](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/multimodal-live) page for more details."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gPiTOAHURvTM"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CHRZUpfWSEpp"
      },
      "source": [
        "### Install libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sG3_LKsWSD3A"
      },
      "outputs": [],
      "source": [
        "%pip install --quiet websockets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HlMVjiAWSMNX"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you are running this notebook on Google Colab, run the cell below to authenticate your environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "12fnq4V0SNV3"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "41oBMp0YraPr"
      },
      "source": [
        "### Set Google Cloud project information\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mRv0DzDjrigq"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type: \"string\"}\n",
        "\n",
        "HOST = \"us-central1-aiplatform.googleapis.com\"\n",
        "SERVICE_URL = f\"wss://{HOST}/ws/google.cloud.aiplatform.v1beta1.LlmBidiService/BidiGenerateContent\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N1mXI5aDB8vl"
      },
      "source": [
        "## Generate an access token\n",
        "\n",
        "`gcloud auth application-default print-access-token` generates and prints an access token for the current Application Default Credential. The default access token lifetime is 3600 seconds. This access token will be used to connect to the WebSocket server."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Nb_bHsEhe-37"
      },
      "outputs": [],
      "source": [
        "bearer_token = !gcloud auth application-default print-access-token"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5M7EKckIYVFy"
      },
      "source": [
        "### Use the Gemini 2.0 Flash model\n",
        "\n",
        "Multimodal Live API is a new capability introduced with the [Gemini 2.0 Flash model](https://cloud.google.com/vertex-ai/generative-ai/docs/gemini-v2)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-coEslfWPrxo"
      },
      "outputs": [],
      "source": [
        "MODEL_ID = \"gemini-2.0-flash-exp\"  # @param {type: \"string\"}\n",
        "\n",
        "MODEL = (\n",
        "    f\"projects/{PROJECT_ID}/locations/{LOCATION}/publishers/google/models/{MODEL_ID}\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Ef0zVX-X9Bg"
      },
      "source": [
        "### Import libraries\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QNxC25Pg4Hfr"
      },
      "outputs": [],
      "source": [
        "import base64\n",
        "import json\n",
        "\n",
        "from IPython.display import Audio, Markdown, display\n",
        "import numpy as np\n",
        "from websockets.asyncio.client import connect"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k9jAArxzClXz"
      },
      "source": [
        "## Use the Multimodal Live API\n",
        "\n",
        "Multimodal Live API is a stateful API that uses [WebSockets](https://en.wikipedia.org/wiki/WebSocket). This section shows some basic examples of how to use Multimodal Live API for text-to-text and text-to-audio generation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q1DE3s_LIUuE"
      },
      "source": [
        "### **Example 1**: Text-to-text generation\n",
        "\n",
        "You send a text prompt and receive a text message.\n",
        "\n",
        "**Notes**\n",
        "- A session `ws` represents a single WebSocket connection between the client and the server.\n",
        "- After a new connection is initiated, the session can exchange messages with the server.\n",
        "- Messages are JSON-formatted strings exchanged over the WebSocket connection.\n",
        "- The first message to be sent should be a [`setup`](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/multimodal-live#bidigeneratecontentsetup) message that contains the model, generation parameters, system instructions, and tools.\n",
        "  - `response_modalities` accepts `TEXT` or `AUDIO`.\n",
        "- To receive messages from the server, listen for the WebSocket `message` event, and then parse the result according to the definition of supported server messages.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sAFvSv2IecqA"
      },
      "outputs": [],
      "source": [
        "# Set model generation_config\n",
        "CONFIG = {\"response_modalities\": [\"TEXT\"]}\n",
        "\n",
        "headers = {\n",
        "    \"Content-Type\": \"application/json\",\n",
        "    \"Authorization\": f\"Bearer {bearer_token[0]}\",\n",
        "}\n",
        "\n",
        "# Connect to the server\n",
        "async with connect(SERVICE_URL, additional_headers=headers) as ws:\n",
        "    # Setup the session\n",
        "    await ws.send(\n",
        "        json.dumps(\n",
        "            {\n",
        "                \"setup\": {\n",
        "                    \"model\": MODEL,\n",
        "                    \"generation_config\": CONFIG,\n",
        "                }\n",
        "            }\n",
        "        )\n",
        "    )\n",
        "\n",
        "    # Receive setup response\n",
        "    raw_response = await ws.recv(decode=False)\n",
        "    setup_response = json.loads(raw_response.decode(\"ascii\"))\n",
        "\n",
        "    # Send text message\n",
        "    text_input = \"Hello? Gemini are you there?\"\n",
        "    display(Markdown(f\"**Input:** {text_input}\"))\n",
        "\n",
        "    msg = {\n",
        "        \"client_content\": {\n",
        "            \"turns\": [{\"role\": \"user\", \"parts\": [{\"text\": text_input}]}],\n",
        "            \"turn_complete\": True,\n",
        "        }\n",
        "    }\n",
        "\n",
        "    await ws.send(json.dumps(msg))\n",
        "\n",
        "    responses = []\n",
        "\n",
        "    # Receive chucks of server response\n",
        "    async for raw_response in ws:\n",
        "        response = json.loads(raw_response.decode())\n",
        "        server_content = response.pop(\"serverContent\", None)\n",
        "        if server_content is None:\n",
        "            break\n",
        "\n",
        "        model_turn = server_content.pop(\"modelTurn\", None)\n",
        "        if model_turn is not None:\n",
        "            parts = model_turn.pop(\"parts\", None)\n",
        "            if parts is not None:\n",
        "                responses.append(parts[0][\"text\"])\n",
        "\n",
        "        # End of turn\n",
        "        turn_complete = server_content.pop(\"turnComplete\", None)\n",
        "        if turn_complete:\n",
        "            break\n",
        "\n",
        "    # Print the server response\n",
        "    display(Markdown(f\"**Response >** {''.join(responses)}\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cG3346aA9sRR"
      },
      "source": [
        "### **Example 2**: Text-to-audio generation\n",
        "\n",
        "You send a text prompt and receive a model response in audio.\n",
        "\n",
        "**Notes**\n",
        "- Multimodal Live API supports the following voices:\n",
        "  - Puck\n",
        "  - Charon\n",
        "  - Kore\n",
        "  - Fenrir\n",
        "  - Aoede\n",
        "- To specify a voice, set the `voice_name` within the `speech_config` object, as part of your session configuration.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Iz3OkQ-a51QM"
      },
      "outputs": [],
      "source": [
        "# Set model generation_config\n",
        "CONFIG = {\n",
        "    \"response_modalities\": [\"AUDIO\"],\n",
        "    \"speech_config\": {\n",
        "        \"voice_config\": {\"prebuilt_voice_config\": {\"voice_name\": \"Aoede\"}}\n",
        "    },\n",
        "}\n",
        "\n",
        "headers = {\n",
        "    \"Content-Type\": \"application/json\",\n",
        "    \"Authorization\": f\"Bearer {bearer_token[0]}\",\n",
        "}\n",
        "\n",
        "# Connect to the server\n",
        "async with connect(SERVICE_URL, additional_headers=headers) as ws:\n",
        "    # Setup the session\n",
        "    await ws.send(\n",
        "        json.dumps(\n",
        "            {\n",
        "                \"setup\": {\n",
        "                    \"model\": MODEL,\n",
        "                    \"generation_config\": CONFIG,\n",
        "                }\n",
        "            }\n",
        "        )\n",
        "    )\n",
        "\n",
        "    # Receive setup response\n",
        "    raw_response = await ws.recv(decode=False)\n",
        "    setup_response = json.loads(raw_response.decode(\"ascii\"))\n",
        "\n",
        "    # Send text message\n",
        "    text_input = \"Hello? Gemini are you there?\"\n",
        "    display(Markdown(f\"**Input:** {text_input}\"))\n",
        "\n",
        "    msg = {\n",
        "        \"client_content\": {\n",
        "            \"turns\": [{\"role\": \"user\", \"parts\": [{\"text\": text_input}]}],\n",
        "            \"turn_complete\": True,\n",
        "        }\n",
        "    }\n",
        "\n",
        "    await ws.send(json.dumps(msg))\n",
        "\n",
        "    responses = []\n",
        "\n",
        "    # Receive chucks of server response\n",
        "    async for raw_response in ws:\n",
        "        response = json.loads(raw_response.decode())\n",
        "        server_content = response.pop(\"serverContent\", None)\n",
        "        if server_content is None:\n",
        "            break\n",
        "\n",
        "        model_turn = server_content.pop(\"modelTurn\", None)\n",
        "        if model_turn is not None:\n",
        "            parts = model_turn.pop(\"parts\", None)\n",
        "            if parts is not None:\n",
        "                for part in parts:\n",
        "                    pcm_data = base64.b64decode(part[\"inlineData\"][\"data\"])\n",
        "                    responses.append(np.frombuffer(pcm_data, dtype=np.int16))\n",
        "\n",
        "        # End of turn\n",
        "        turn_complete = server_content.pop(\"turnComplete\", None)\n",
        "        if turn_complete:\n",
        "            break\n",
        "\n",
        "    # Play the returned audio message\n",
        "    display(Audio(np.concatenate(responses), rate=24000, autoplay=True))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JOBlWf566HOx"
      },
      "source": [
        "### **Example 3**: Text-to-audio conversation\n",
        "\n",
        "**Step 1**: You set up a conversation with the API that allows you to send text prompts and receive audio responses.\n",
        "\n",
        "**Notes**\n",
        "\n",
        "- While the model keeps track of in-session interactions, explicit session history accessible through the API isn't available yet. When a session is terminated the corresponding context is erased.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-eu4QL0L3HZi"
      },
      "outputs": [],
      "source": [
        "# Set model generation_config\n",
        "CONFIG = {\"response_modalities\": [\"AUDIO\"]}\n",
        "\n",
        "headers = {\n",
        "    \"Content-Type\": \"application/json\",\n",
        "    \"Authorization\": f\"Bearer {bearer_token[0]}\",\n",
        "}\n",
        "\n",
        "\n",
        "async def main() -> None:\n",
        "    # Connect to the server\n",
        "    async with connect(SERVICE_URL, additional_headers=headers) as ws:\n",
        "\n",
        "        # Setup the session\n",
        "        async def setup() -> None:\n",
        "            await ws.send(\n",
        "                json.dumps(\n",
        "                    {\n",
        "                        \"setup\": {\n",
        "                            \"model\": MODEL,\n",
        "                            \"generation_config\": CONFIG,\n",
        "                        }\n",
        "                    }\n",
        "                )\n",
        "            )\n",
        "\n",
        "            # Receive setup response\n",
        "            raw_response = await ws.recv(decode=False)\n",
        "            setup_response = json.loads(raw_response.decode(\"ascii\"))\n",
        "            print(f\"Connected: {setup_response}\")\n",
        "            return\n",
        "\n",
        "        # Send text message\n",
        "        async def send() -> bool:\n",
        "            text_input = input(\"Input > \")\n",
        "            if text_input.lower() in (\"q\", \"quit\", \"exit\"):\n",
        "                return False\n",
        "\n",
        "            msg = {\n",
        "                \"client_content\": {\n",
        "                    \"turns\": [{\"role\": \"user\", \"parts\": [{\"text\": text_input}]}],\n",
        "                    \"turn_complete\": True,\n",
        "                }\n",
        "            }\n",
        "\n",
        "            await ws.send(json.dumps(msg))\n",
        "            return True\n",
        "\n",
        "        # Receive server response\n",
        "        async def receive() -> None:\n",
        "            responses = []\n",
        "\n",
        "            # Receive chucks of server response\n",
        "            async for raw_response in ws:\n",
        "                response = json.loads(raw_response.decode())\n",
        "                server_content = response.pop(\"serverContent\", None)\n",
        "                if server_content is None:\n",
        "                    break\n",
        "\n",
        "                model_turn = server_content.pop(\"modelTurn\", None)\n",
        "                if model_turn is not None:\n",
        "                    parts = model_turn.pop(\"parts\", None)\n",
        "                    if parts is not None:\n",
        "                        for part in parts:\n",
        "                            pcm_data = base64.b64decode(part[\"inlineData\"][\"data\"])\n",
        "                            responses.append(np.frombuffer(pcm_data, dtype=np.int16))\n",
        "\n",
        "                # End of turn\n",
        "                turn_complete = server_content.pop(\"turnComplete\", None)\n",
        "                if turn_complete:\n",
        "                    break\n",
        "\n",
        "            # Play the returned audio message\n",
        "            display(Markdown(\"**Response >**\"))\n",
        "            display(Audio(np.concatenate(responses), rate=24000, autoplay=True))\n",
        "            return\n",
        "\n",
        "        await setup()\n",
        "\n",
        "        while True:\n",
        "            if not await send():\n",
        "                break\n",
        "            await receive()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "94IeUUb3e90M"
      },
      "source": [
        "**Step 2** Start the conversation, input your prompts, or type `q`, `quit` or `exit` to exit.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2UvgUDIYJqfw"
      },
      "outputs": [],
      "source": [
        "await main()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "usjiqTDXfk_6"
      },
      "source": [
        "## What's next\n",
        "\n",
        "\n",
        "- Try [getting started with the Multimodal Live API with the Gen AI SDK](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_genai_sdk.ipynb)\n",
        "- Learn how to [build a web application that enables you to use your voice and camera to talk to Gemini 2.0 through the Multimodal Live API.](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/gemini/multimodal-live-api/websocket-demo-app)\n",
        "- See the [Multimodal Live API reference docs](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/multimodal-live).\n",
        "- Explore other notebooks in the [Google Cloud Generative AI GitHub repository](https://github.com/GoogleCloudPlatform/generative-ai)."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "intro_multimodal_live_api.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}