notebooks/official/pipelines/multicontender_vs_champion_deployment_method.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2023 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Multicontender vs Champion methodology for model deployment into production\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/multicontender_vs_champion_deployment_method.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fpipelines%2Fmulticontender_vs_champion_deployment_method.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/pipelines/multicontender_vs_champion_deployment_method.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/multicontender_vs_champion_deployment_method.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This tutorial shows how to use Vertex AI Pipelines for deploying the next version of a model into production using the multicontender vs champion method.\n",
        "\n",
        "**_NOTE_**: This notebook uses KFP 1.x and GCPC 1.x. It's recommended to use 2.x.\n",
        "\n",
        "**_NOTE_**: This notebook has been tested in the following environment:\n",
        "\n",
        "* Python version = 3.9"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d975e698c9a4"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to construct a Vertex AI pipeline, which evaluates new production data from a deployed (production) model against other versions (contenders) of the model, to determine if a contender model becomes the champion model for replacement in production.\n",
        "\n",
        "This tutorial uses the following Vertex AI services and resources:\n",
        "\n",
        "- Vertex AI Pipeline\n",
        "- Vertex AI Model Evaluation\n",
        "- Vertex AI Model Registry\n",
        "- Vertex AI Endpoints\n",
        "\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Import a pretrained (champion) model to the Vertex AI Model Registry.\n",
        "- Import synthetic model training evaluation metrics to the corresponding (champion) model.\n",
        "- Create a Vertex AI endpoint resource.\n",
        "- Deploy the champion model to the endpoint resource.\n",
        "- Import additional (contender) versions of the deployed model.\n",
        "- Import synthetic model training evaluation metrics to the corresponding (contender) models.\n",
        "- Create a Vertex AI Pipeline that runs the following steps:\n",
        "    - Get the champion model.\n",
        "    - (Fake) Fine-tune champion model with production data.\n",
        "    - Import synthetic train+production evaluation metrics for the champion model.\n",
        "    - Get the contender models.\n",
        "    - (Fake) Fine-tune contender model with production data.\n",
        "    - Import synthetic train+production evaluation metrics for the contenders models.\n",
        "    - Compare the evaluations of the contenders to the champion and set the new champion as the default.\n",
        "    - Deploy the new champion model.\n",
        "\n",
        "Learn more about [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "08d289fa873f"
      },
      "source": [
        "### Model\n",
        "\n",
        "This tutorial uses a pre-trained image classification model from TensorFlow Hub, which is trained on ImageNet dataset.\n",
        "\n",
        "Learn more about [ResNet V2 pretained model](https://tfhub.dev/google/imagenet/resnet_v2_101/classification/5). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aed92deeb4a0"
      },
      "source": [
        "### Costs \n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
        "and [Cloud Storage pricing](https://cloud.google.com/storage/pricing), \n",
        "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d1ea81ac77f0"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2b4ef9b72d43"
      },
      "outputs": [],
      "source": [
        "# Install the packages\n",
        "! pip3 install --quiet --upgrade google-cloud-aiplatform \\\n",
        "                                 'google-cloud-pipeline-components<2'\n",
        "\n",
        "! pip3 install --quiet           tensorflow==2.5 \\\n",
        "                                 tensorflow_hub\n",
        "\n",
        "! pip3 install --quiet --upgrade 'kfp<2'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58707a750154"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f200f10a1da3"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c87a2a5d7e35"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5dccb1c8feb6"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cc7251520a07"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BF1j6f9HApxa"
      },
      "source": [
        "### Set Google Cloud project information\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-EcIXiGsCePi"
      },
      "source": [
        "**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account"
      },
      "source": [
        "### Service Account\n",
        "\n",
        "You use a service account to create Vertex AI Pipeline jobs.\n",
        "\n",
        "If you don't want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account"
      },
      "outputs": [],
      "source": [
        "SERVICE_ACCOUNT = \"[your-service-account]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_service_account"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "IS_COLAB = \"google.colab\" in sys.modules\n",
        "if (\n",
        "    SERVICE_ACCOUNT == \"\"\n",
        "    or SERVICE_ACCOUNT is None\n",
        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
        "):\n",
        "    # Get your service account from gcloud\n",
        "    if not IS_COLAB:\n",
        "        shell_output = !gcloud auth list 2>/dev/null\n",
        "        SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
        "\n",
        "    else:  # IS_COLAB:\n",
        "        shell_output = ! gcloud projects describe  $PROJECT_ID\n",
        "        project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n",
        "        SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n",
        "\n",
        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "source": [
        "#### Set service account access for Vertex AI Pipelines\n",
        "\n",
        "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step. You only need to run these once per service account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "outputs": [],
      "source": [
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
        "\n",
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "960505627ddf"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PyQmSRbKA8r-"
      },
      "outputs": [],
      "source": [
        "import kfp\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "from google.cloud import aiplatform\n",
        "from google.cloud.aiplatform import gapic\n",
        "from kfp.v2 import compiler\n",
        "from kfp.v2.dsl import component"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "To get started using Vertex AI, you must [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com) in your Google Cloud project.\n",
        "\n",
        "Then, initialize the Vertex AI SDK for Python for your project."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "outputs": [],
      "source": [
        "aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "source": [
        "#### Set hardware accelerators\n",
        "\n",
        "You can set hardware accelerators for training and prediction.\n",
        "\n",
        "Set the variables `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa T4 GPUs allocated to each VM, you would specify:\n",
        "\n",
        "    (aip.gapic.AcceleratorType.NVIDIA_TESLA_T4, 4)\n",
        "\n",
        "\n",
        "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n",
        "\n",
        "Learn more about [hardware accelerator support for your region](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "outputs": [],
      "source": [
        "DEPLOY_GPU, DEPLOY_NGPU = (None, None)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "container:training,prediction"
      },
      "source": [
        "#### Set pre-built containers\n",
        "\n",
        "Set the pre-built Docker container image for prediction.\n",
        "\n",
        "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "container:training,prediction"
      },
      "outputs": [],
      "source": [
        "TF = \"2.5\".replace(\".\", \"-\")\n",
        "\n",
        "if DEPLOY_GPU:\n",
        "    DEPLOY_VERSION = \"tf2-gpu.{}\".format(TF)\n",
        "else:\n",
        "    DEPLOY_VERSION = \"tf2-cpu.{}\".format(TF)\n",
        "\n",
        "DEPLOY_IMAGE = \"{}-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(\n",
        "    LOCATION.split(\"-\")[0], DEPLOY_VERSION\n",
        ")\n",
        "\n",
        "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "machine:training"
      },
      "source": [
        "#### Set machine type\n",
        "\n",
        "Next, set the machine type to use for deployment.\n",
        "\n",
        "- Set the variable `DEPLOY_COMPUTE` to configure  the compute resources for the VMs you use for deployment. For this, you specify:\n",
        "     - `machine_type`\n",
        "         - `n1-standard`: 3.75GB of memory per vCPU.\n",
        "         - `n1-highmem`: 6.5GB of memory per vCPU\n",
        "         - `n1-highcpu`: 0.9 GB of memory per vCPU\n",
        "     - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n",
        "\n",
        "**Note**: The following aren't supported.\n",
        "\n",
        " - `standard`: 2 vCPUs\n",
        " - `highcpu`: 2, 4 and 8 vCPUs\n",
        "\n",
        "**Note**: You may also use n2 and e2 machine types for training and deployment, but they don't support GPUs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "machine:training"
      },
      "outputs": [],
      "source": [
        "machine_type = \"n1-standard\"\n",
        "vCPUs = \"4\"\n",
        "DEPLOY_COMPUTE = f\"{machine_type}-{vCPUs}\"\n",
        "print(\"Deploy machine type\", DEPLOY_COMPUTE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d8128b8ff025"
      },
      "source": [
        "## Get pretrained model from TensorFlow Hub\n",
        "\n",
        "For demonstration purposes, this tutorial uses a pretrained model from TensorFlow Hub (TFHub), which is then uploaded to Vertex AI Model Registry. Once you have a Vertex AI model resource, the model can be deployed to a Vertex AI endpoint resource.\n",
        "\n",
        "### Download the pretrained model\n",
        "\n",
        "First, you download the pretrained model from TensorFlow Hub. The model gets downloaded as a TF.Keras layer. To finalize the model, in this example, you create a `Sequential()` model with the downloaded TFHub model as a layer, and specify the input shape to the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c55fa4c826f7"
      },
      "outputs": [],
      "source": [
        "tfhub_model = tf.keras.Sequential(\n",
        "    [hub.KerasLayer(\"https://tfhub.dev/google/imagenet/resnet_v2_101/classification/5\")]\n",
        ")\n",
        "\n",
        "tfhub_model.build([None, 32, 32, 3])\n",
        "\n",
        "tfhub_model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dcee58290f19"
      },
      "source": [
        "### Save the model artifacts\n",
        "\n",
        "At this point, the model is in memory. Next, save the model artifacts to a Cloud Storage location."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "64618c713db9"
      },
      "outputs": [],
      "source": [
        "# Save the model to Cloud Storage path\n",
        "MODEL_DIR = BUCKET_URI + \"/model\"\n",
        "tfhub_model.save(MODEL_DIR)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e8ce91147c93"
      },
      "source": [
        "## Upload the TensorFlow Hub model to Vertex AI Model Registry\n",
        "\n",
        "Finally, you upload the model artifacts from the TFHub model into Vertex AI Model Registry and get a model resource object using the method `upload()`, with the following parameters:\n",
        "\n",
        "- `display_name`: A human readable name for the model resource.\n",
        "- `artifact_uri`: The Cloud Storage location of the model package.\n",
        "- `serving_container_image_uri`: The serving container image.\n",
        "\n",
        "Uploading a model into a Vertex AI Model Registry returns a long running operation, since it may take a few moments.\n",
        "\n",
        "**Note:** When you upload the model artifacts to Vertex AI Model Registry, specify the corresponding deployment container image."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ad61e1429512"
      },
      "outputs": [],
      "source": [
        "champion_model = aiplatform.Model.upload(\n",
        "    display_name=\"resnet-v1\",\n",
        "    artifact_uri=MODEL_DIR,\n",
        "    serving_container_image_uri=DEPLOY_IMAGE,\n",
        "    is_default_version=True,\n",
        "    version_aliases=[\"v1\"],\n",
        ")\n",
        "\n",
        "print(champion_model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c11e98ef5391"
      },
      "source": [
        "## Create a model evaluation\n",
        "\n",
        "First, you create a model evaluation in a format that corresponds to one of the predefined schemas for model evaluations. In this example, you use the schema for a classification metric, and specify the following subset of evaluation metrics as a dictionary:\n",
        "\n",
        "- `logLoss`: The log loss.\n",
        "- `auPrc`: The accuracy.\n",
        "\n",
        "You then construct the `ModelEvaluation` object with the following parameters:\n",
        "\n",
        "- `display_name`: The human readable name for the evaluation metric.\n",
        "- `metrics_schema_uri`: The schema for the specific type of evaluation metrics.\n",
        "- `metrics`: The dictionary with the evaluation metrics.\n",
        "\n",
        "Learn more about [Schemas for evaluation metrics](https://cloud.google.com/vertex-ai/docs/evaluation/introduction#features)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e9af222db292"
      },
      "outputs": [],
      "source": [
        "metrics = {\"logLoss\": 1.4, \"auPrc\": 0.85}\n",
        "print(metrics)\n",
        "\n",
        "champion_eval = gapic.ModelEvaluation(\n",
        "    display_name=\"train-v1\",\n",
        "    metrics_schema_uri=\"gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml\",\n",
        "    metrics=metrics,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "68870bd8194d"
      },
      "source": [
        "### Upload the evaluation metrics to the Model Registry\n",
        "\n",
        "Next, upload the model's evaluation from the custom training job to the corresponding entry in the Vertex AI Model Registry.\n",
        "\n",
        "Currently, the Python SDK does not support for this method. Instead, use the lower level GAPIC API interface."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3044788848af"
      },
      "outputs": [],
      "source": [
        "API_ENDPOINT = f\"{LOCATION}-aiplatform.googleapis.com\"\n",
        "client = gapic.ModelServiceClient(client_options={\"api_endpoint\": API_ENDPOINT})\n",
        "\n",
        "client.import_model_evaluation(\n",
        "    parent=champion_model.resource_name, model_evaluation=champion_eval\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "628de0914ba1"
      },
      "source": [
        "## Creating an endpoint resource\n",
        "\n",
        "You create an endpoint resource using the `Endpoint.create()` method. At a minimum, you specify the display name for the endpoint. Optionally, you can specify the project and location (region); otherwise the settings are inherited by the values you set when you initialized the Vertex AI SDK with the `init()` method.\n",
        "\n",
        "In this example, the following parameters are specified:\n",
        "\n",
        "- `display_name`: A human readable name for the endpoint resource.\n",
        "- `project`: Your project ID.\n",
        "- `location`: Your region.\n",
        "\n",
        "This method returns an `Endpoint` object.\n",
        "\n",
        "Learn more about [Vertex AI Endpoints](https://cloud.google.com/vertex-ai/docs/predictions/deploy-model-api)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0ea443f9593b"
      },
      "outputs": [],
      "source": [
        "endpoint = aiplatform.Endpoint.create(\n",
        "    display_name=\"production\", project=PROJECT_ID, location=LOCATION\n",
        ")\n",
        "\n",
        "print(endpoint)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ca3fa3f6a894"
      },
      "source": [
        "## Deploy the model resource to the endpoint resource\n",
        "\n",
        "Next, you deploy the blessed Vertex AI model resource to a Vertex AI endpoint resource. The container image defined for the Vertex AI model resource is used for deployment. To deploy, you specify the following additional configuration settings:\n",
        "\n",
        "- The machine type.\n",
        "- The (if any) type and number of GPUs.\n",
        "- Static, manual or auto-scaling of VM instances.\n",
        "\n",
        "In this example, you deploy the model with the minimal amount of specified parameters, as follows:\n",
        "\n",
        "- `model`: The model resource.\n",
        "- `deployed_model_displayed_name`: The human readable name for the deployed model instance.\n",
        "- `machine_type`: The machine type for each VM instance.\n",
        "\n",
        "This may take a few minutes due to the provisioning of the configured resources."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4e93b034a72f"
      },
      "outputs": [],
      "source": [
        "response = endpoint.deploy(\n",
        "    model=champion_model,\n",
        "    deployed_model_display_name=\"champion\",\n",
        "    machine_type=DEPLOY_COMPUTE,\n",
        ")\n",
        "\n",
        "print(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "75cd089dd792"
      },
      "source": [
        "## Create multiple contender versions of the deployed model\n",
        "\n",
        "Next, you create plural synthetic versions of the deployed champion model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8e654013bde0"
      },
      "outputs": [],
      "source": [
        "contender_model_1 = aiplatform.Model.upload(\n",
        "    display_name=\"resnet-v2\",\n",
        "    artifact_uri=MODEL_DIR,\n",
        "    serving_container_image_uri=DEPLOY_IMAGE,\n",
        "    parent_model=champion_model.resource_name,\n",
        "    is_default_version=False,\n",
        "    version_aliases=[\"v2\"],\n",
        ")\n",
        "\n",
        "contender_model_2 = aiplatform.Model.upload(\n",
        "    display_name=\"resnet-v3\",\n",
        "    artifact_uri=MODEL_DIR,\n",
        "    serving_container_image_uri=DEPLOY_IMAGE,\n",
        "    parent_model=champion_model.resource_name,\n",
        "    is_default_version=False,\n",
        "    version_aliases=[\"v3\"],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c11e98ef5391"
      },
      "source": [
        "### Create a model evaluation for the contender models\n",
        "\n",
        "Next, you create a model evaluation for each of the contender models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e9af222db292"
      },
      "outputs": [],
      "source": [
        "metrics = {\"logLoss\": 1.5, \"auPrc\": 0.83}\n",
        "\n",
        "contender_1_eval = gapic.ModelEvaluation(\n",
        "    display_name=\"train-v2\",\n",
        "    metrics_schema_uri=\"gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml\",\n",
        "    metrics=metrics,\n",
        ")\n",
        "\n",
        "# workaround for bug\n",
        "contender_model_1.versioning_registry.add_version_aliases(\n",
        "    new_aliases=[\"default\"], version=contender_model_1.version_id\n",
        ")\n",
        "\n",
        "client.import_model_evaluation(\n",
        "    parent=contender_model_1.resource_name, model_evaluation=contender_1_eval\n",
        ")\n",
        "\n",
        "metrics = {\"logLoss\": 1.6, \"auPrc\": 0.82}\n",
        "\n",
        "contender_2_eval = gapic.ModelEvaluation(\n",
        "    display_name=\"train-v3\",\n",
        "    metrics_schema_uri=\"gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml\",\n",
        "    metrics=metrics,\n",
        ")\n",
        "\n",
        "# workaround for bug\n",
        "contender_model_2.versioning_registry.add_version_aliases(\n",
        "    new_aliases=[\"default\"], version=contender_model_2.version_id\n",
        ")\n",
        "\n",
        "client.import_model_evaluation(\n",
        "    parent=contender_model_2.resource_name, model_evaluation=contender_2_eval\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c57974c5e5f9"
      },
      "source": [
        "## Create custom components for pipeline\n",
        "\n",
        "Next, you create several custom components you use in your pipeline.\n",
        "\n",
        "### Create component to import classification metrics\n",
        "\n",
        "First, you define a component to import the evaluation metrics for the challenger model to the Model Registry. The component takes the following arguments:\n",
        "\n",
        "- `display_name`: Human readable name for the evaluation metrics\n",
        "- `metrics`: The evaluation metrics formatted for classification.\n",
        "- `parent_model_resource`: The full resource name for the challenger model version.\n",
        "- `region`: The region where you want to create or use the resources."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0f416e2eaf3c"
      },
      "outputs": [],
      "source": [
        "@component(packages_to_install=[\"google-cloud-aiplatform\"])\n",
        "def import_classification_metrics(\n",
        "    display_name: str,\n",
        "    metrics: dict,\n",
        "    parent_model_resource: str,\n",
        "    project: str,\n",
        "    region: str,\n",
        "):\n",
        "    from google.cloud import aiplatform\n",
        "    from google.cloud.aiplatform import gapic\n",
        "\n",
        "    print(\"DISPLAY\", display_name)\n",
        "\n",
        "    evaluation = gapic.ModelEvaluation(\n",
        "        display_name=display_name,\n",
        "        metrics_schema_uri=\"gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml\",\n",
        "        metrics=metrics,\n",
        "    )\n",
        "\n",
        "    # workaround for bug\n",
        "    aiplatform.init(project=project, location=region)\n",
        "    parent_model = aiplatform.Model(parent_model_resource)\n",
        "    parent_model.versioning_registry.add_version_aliases(\n",
        "        new_aliases=[\"default\"], version=parent_model.version_id\n",
        "    )\n",
        "\n",
        "    API_ENDPOINT = f\"{region}-aiplatform.googleapis.com\"\n",
        "    client = gapic.ModelServiceClient(client_options={\"api_endpoint\": API_ENDPOINT})\n",
        "    client.import_model_evaluation(\n",
        "        parent=parent_model_resource, model_evaluation=evaluation\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b19203fb3003"
      },
      "source": [
        "###  Create component to compare metrics\n",
        "\n",
        "Next, you define a component to compare the **auPrc** metric between the champion and contender versions of the model.  Whichever has the best **auPrc** value is set as the default model. When you subsequently deploy, the default model is deployed. The component takes the following arguments:\n",
        "\n",
        "- `champion_resource_name`: The full resource name of the champion model.\n",
        "- `contender_model_resource_names`: A list of the full resource namea of the contender models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8dc5304a0fd1"
      },
      "outputs": [],
      "source": [
        "@component(packages_to_install=[\"google-cloud-aiplatform\"])\n",
        "def compare_metrics(\n",
        "    champion_model_resource_name: str, contender_model_resource_names: list\n",
        "):\n",
        "    from google.cloud import aiplatform\n",
        "\n",
        "    # Get the metrics for the blessed model\n",
        "    champion_model = aiplatform.Model(champion_model_resource_name)\n",
        "    champion_eval = champion_model.list_model_evaluations()[\n",
        "        1\n",
        "    ]  # index 1 is the production eval data\n",
        "    champion_auPrc = champion_eval.metrics[\"auPrc\"]\n",
        "\n",
        "    champion_model.versioning_registry.add_version_aliases(\n",
        "        new_aliases=[\"default\"], version=champion_model.version_id\n",
        "    )\n",
        "\n",
        "    # Get the metrics for the challenger model\n",
        "    for contender in contender_model_resource_names:\n",
        "        contender_model = aiplatform.Model(contender)\n",
        "        contender_eval = contender_model.list_model_evaluations()[1]\n",
        "        contender_auPrc = contender_eval.metrics[\"auPrc\"]\n",
        "\n",
        "        # Which model has the best accuracy becomes the default model\n",
        "        if contender_auPrc > champion_auPrc:\n",
        "            contender_model.versioning_registry.add_version_aliases(\n",
        "                new_aliases=[\"default\"], version=contender_model.version_id\n",
        "            )\n",
        "            champion_auPrc = contender_auPrc"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1a61dc08ba46"
      },
      "source": [
        "## Champion vs multi-contender pipeline\n",
        "\n",
        "In this section, you construct a pipeline to fetch the champion, and the challenger models and compare them to deploy the best one to the production endpoint.\n",
        "\n",
        "### Define the pipeline\n",
        "\n",
        "Define your pipeline to run the following tasks:\n",
        "\n",
        "- Get the champion version of a model.\n",
        "- Get the endpoint for the deployed champion model.\n",
        "- Re-train and evaluate (faked) the champion model with production data.\n",
        "- Import the champion model's evaluation metrics for the production data.\n",
        "- For each contender model:\n",
        "  - Get the contender version of the model.\n",
        "  - Re-train and evaluate (faked) the contender model with production data.\n",
        "- Compare the champion and contenders evaluation metrics, and set the default accordingly.\n",
        "- Import the existing production endpoint.\n",
        "- Deploy the new default model to the production endpoint."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "beeb768408bf"
      },
      "outputs": [],
      "source": [
        "@kfp.dsl.pipeline(name=\"multicontender-vs-champion\")\n",
        "def pipeline(\n",
        "    champion_model_resource: str,\n",
        "    contender_model_resources: list,\n",
        "    serving_container: str,\n",
        "    machine_type: str,\n",
        "    endpoint_resource_name: str,\n",
        "    endpoint_resource_uri: str,\n",
        "    project: str = PROJECT_ID,\n",
        "    region: str = LOCATION,\n",
        "):\n",
        "    from google_cloud_pipeline_components.experimental.evaluation import \\\n",
        "        GetVertexModelOp\n",
        "    from google_cloud_pipeline_components.types import artifact_types\n",
        "    from google_cloud_pipeline_components.v1.endpoint import ModelDeployOp\n",
        "    from kfp.v2.components import importer_node\n",
        "\n",
        "    # Get the Vertex AI model resource of the blessed model\n",
        "    champion = GetVertexModelOp(model_resource_name=champion_model_resource)\n",
        "\n",
        "    # pretend to retrain and evaluate the champion with production data\n",
        "    champion_metrics = {\"logLoss\": 1.3, \"auPrc\": 0.86}\n",
        "\n",
        "    # upload the metrics for the champion version\n",
        "    import_champion_metrics = import_classification_metrics(\n",
        "        display_name=\"production\",\n",
        "        metrics=champion_metrics,\n",
        "        parent_model_resource=champion_model_resource,\n",
        "        project=project,\n",
        "        region=region,\n",
        "    ).after(champion)\n",
        "\n",
        "    ix = 0\n",
        "    with kfp.dsl.ParallelFor(contender_model_resources).after(\n",
        "        import_champion_metrics\n",
        "    ) as contender_model_resource:\n",
        "        contender = GetVertexModelOp(model_resource_name=contender_model_resource)\n",
        "\n",
        "        # pretend to retrain and evaluate the champion with production data\n",
        "        contender_metrics = {\"logLoss\": 1.1, \"auPrc\": 0.88}\n",
        "\n",
        "        ix += 1\n",
        "        import_contender_metrics = import_classification_metrics(\n",
        "            display_name=f\"production_{ix}\",\n",
        "            metrics=contender_metrics,\n",
        "            parent_model_resource=contender_model_resource,\n",
        "            project=project,\n",
        "            region=region,\n",
        "        ).after(contender)\n",
        "\n",
        "    # Select the best model\n",
        "    compare = compare_metrics(\n",
        "        champion_model_resource,\n",
        "        contender_model_resources,\n",
        "    ).after(import_contender_metrics)\n",
        "\n",
        "    # import the production Endpoint\n",
        "    endpoint = importer_node.importer(\n",
        "        artifact_uri=endpoint_resource_uri,\n",
        "        artifact_class=artifact_types.VertexEndpoint,\n",
        "        metadata={\"resourceName\": endpoint_resource_name},\n",
        "    )\n",
        "\n",
        "    # deploy model to endpoint\n",
        "    _ = ModelDeployOp(\n",
        "        model=champion.outputs[\"model\"],\n",
        "        endpoint=endpoint.output,\n",
        "        dedicated_resources_min_replica_count=1,\n",
        "        dedicated_resources_max_replica_count=1,\n",
        "        dedicated_resources_machine_type=machine_type,\n",
        "        traffic_split={\"0\": 100},\n",
        "    ).after(compare)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_custom_train_pipeline:icn"
      },
      "source": [
        "### Compile the pipeline\n",
        "\n",
        "Next, you compile the pipeline to a JSON file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cbc1dc2bd4e0"
      },
      "outputs": [],
      "source": [
        "# Compile the pipeline to a json file\n",
        "compiler.Compiler().compile(\n",
        "    pipeline_func=pipeline, package_path=\"multicontender_vs_champion.json\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bfa9c32ee12e"
      },
      "source": [
        "### Execute the pipeline\n",
        "\n",
        "Finally, you execute your pipeline, passing the following pipeline parameter values:\n",
        "\n",
        "- `blessed_model_resource`: The full resource name of the current blessed version of the model.\n",
        "- `serving_container`: The serving container for deploying the model.\n",
        "- `machine_type`: The machine type for deploying the model.\n",
        "- `endpoint_resource_name`: The full resource name of the production endpoint.\n",
        "- `endpoint_resource_uri`: The full URI for the production endpoint.\n",
        "- `project`:The project ID.\n",
        "- `region`: The region where you want to run your pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "run_pipeline:control"
      },
      "outputs": [],
      "source": [
        "PIPELINE_ROOT = \"{}/pipeline_root/control\".format(BUCKET_URI)\n",
        "\n",
        "job = aiplatform.PipelineJob(\n",
        "    display_name=\"multicontender_vs_champion\",\n",
        "    template_path=\"multicontender_vs_champion.json\",\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        "    parameter_values={\n",
        "        \"champion_model_resource\": champion_model.resource_name,\n",
        "        \"contender_model_resources\": [\n",
        "            contender_model_1.resource_name,\n",
        "            contender_model_2.resource_name,\n",
        "        ],\n",
        "        \"serving_container\": DEPLOY_IMAGE,\n",
        "        \"machine_type\": DEPLOY_COMPUTE,\n",
        "        \"endpoint_resource_name\": endpoint.resource_name,\n",
        "        \"endpoint_resource_uri\": \"https://us-central1-aiplatform.googleapis.com/v1/\"\n",
        "        + endpoint.resource_name,\n",
        "        \"project\": PROJECT_ID,\n",
        "        \"region\": LOCATION,\n",
        "    },\n",
        "    enable_caching=False,\n",
        ")\n",
        "\n",
        "job.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "326552fd358d"
      },
      "source": [
        "### Get the latest state of the production endpoint\n",
        "\n",
        "Once your pipeline execution is finished, notice that the contender (version 3) model has replaced the previous champion model on the production endpoint.\n",
        "\n",
        "Now, display the latest details of the deployed models from the production endpoint, and then display the traffic split. Notice that the resource ID for the 100% entry is the resource ID for the contender (verson 3) model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3383a2bb45a1"
      },
      "outputs": [],
      "source": [
        "gca_resource = endpoint.list(filter=\"display_name=production\")[0].gca_resource\n",
        "print(\"Deployed Models\", gca_resource.deployed_models)\n",
        "print(\"\\n\")\n",
        "print(\"Traffic Split\", gca_resource.traffic_split)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sx_vKniMq9ZX"
      },
      "outputs": [],
      "source": [
        "# Undeploy the model from endpoint\n",
        "endpoint.undeploy_all()\n",
        "\n",
        "# Delete the endpoint resource\n",
        "endpoint.delete()\n",
        "\n",
        "# Delete model resource\n",
        "champion_model.delete()\n",
        "\n",
        "# Delete the pipeline job resource\n",
        "job.delete()\n",
        "\n",
        "# Delete Cloud Storage objects that were created\n",
        "delete_bucket = True\n",
        "if delete_bucket:\n",
        "    ! gsutil -m rm -r $BUCKET_URI\n",
        "\n",
        "# Remove the local pipeline package file\n",
        "! rm multicontender_vs_champion.json"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "multicontender_vs_champion_deployment_method.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}