demo_with_real_data_notebook.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "A9YRrFYNY37s",
      "metadata": {
        "id": "A9YRrFYNY37s"
      },
      "source": [
        "# DISCLAIMER\n",
        "Copyright 2021 Google LLC. \n",
        "\n",
        "*This solution, including any related sample code or data, is made available on an “as is,” “as available,” and “with all faults” basis, solely for illustrative purposes, and without warranty or representation of any kind. This solution is experimental, unsupported and provided solely for your convenience. Your use of it is subject to your agreements with Google, as applicable, and may constitute a beta feature as defined under those agreements. To the extent that you make any data available to Google in connection with your use of the solution, you represent and warrant that you have all necessary and appropriate rights, consents and permissions to permit Google to use and process that data. By using any portion of this solution, you acknowledge, assume and accept all risks, known and unknown, associated with its usage, including with respect to your deployment of any portion of this solution in your systems, or usage in connection with your business, if at all.*"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "iTsKaxLPY37t",
      "metadata": {
        "id": "iTsKaxLPY37t"
      },
      "source": [
        "# Crystalvalue Demo: Predictive Customer LifeTime Value for a Retail Store\n",
        "\n",
        "Crystalvalue is a best practice comprehensive solution for running predictive LTV solutions leveraging Google Cloud Vertex AI. \n",
        "\n",
        "This demo runs the Crystalvalue python library in a notebook, from feature engineering to scheduling predictions. It uses the [Online Retail II data set from Kaggle](https://www.kaggle.com/mashlyn/online-retail-ii-uci) which contains transactions for a UK retail store between 2009 and 2011. Enable the [Vertex API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,storage-component.googleapis.com) for this demo to run.\n",
        "\n",
        "This notebook assumes that it is being run from within a [Google Cloud Platform AI Notebook](https://console.cloud.google.com/vertex-ai/notebooks/list/instances) with a Compute Engine default service account (the default setting when an AI Notebook is created) and TensorFlow backend. Ensure that the [Compute Engine default service account API](https://console.cloud.google.com/flows/enableapi?apiid=compute.googleapis.com) is enabled. When running it on your own data, we recommend [setting up your own service account](https://cloud.google.com/vertex-ai/docs/pipelines/configure-project).\n",
        "\n",
        "If you would like to share feedback about Crystalvalue, please email crystalvalue@google.com."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1_M6mv5_Al18",
      "metadata": {
        "id": "1_M6mv5_Al18"
      },
      "source": [
        "# Clone the Crystalvalue codebase"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "hnySWWD2DbJ8",
      "metadata": {
        "id": "hnySWWD2DbJ8"
      },
      "source": [
        "Start by cloning the Crystalvalue codebase and running a demo notebook from the root directory. To run Crystalvalue on your own data, simply change the parameters to it works on your data."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a6QSkrxSFsCB",
      "metadata": {
        "id": "a6QSkrxSFsCB"
      },
      "source": [
        "```git clone https://github.com/google/crystalvalue```"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "WJajLx_EtUKR",
      "metadata": {
        "id": "WJajLx_EtUKR"
      },
      "source": [
        "# Set up - Downloading the dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "iojPU3e4tW5z",
      "metadata": {
        "id": "iojPU3e4tW5z"
      },
      "source": [
        "In order to use the Kaggle’s public API, you must first authenticate using an API token. You can do this by visiting your Kaggle account and click 'Create New API Token' (See https://www.kaggle.com/docs/api). This will download an API token (called kaggle.json). Put this file in your working directory and run the following commands from your AI Notebook. Kaggle requires the json to be in a specific folder called 'kaggle'."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "duYMfWZelG0f",
      "metadata": {
        "id": "duYMfWZelG0f"
      },
      "outputs": [],
      "source": [
        "!pip install kaggle\n",
        "!mkdir ~/.kaggle\n",
        "!cp kaggle.json ~/.kaggle/kaggle.json\n",
        "!kaggle datasets download -d mashlyn/online-retail-ii-uci\n",
        "!sudo apt-get install unzip\n",
        "!unzip online-retail-ii-uci.zip -d data/"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "sDA4zRGytlAx",
      "metadata": {
        "id": "sDA4zRGytlAx"
      },
      "source": [
        "This creates a `online_retail_II.csv` in `/data` which we will import into BigQuery in the next steps."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1O4OhKV5Y37u",
      "metadata": {
        "id": "1O4OhKV5Y37u"
      },
      "source": [
        "# Installing dependencies and initializing Crystalvalue"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "XMYviqrPHbKD",
      "metadata": {
        "id": "XMYviqrPHbKD"
      },
      "source": [
        "First create a dataset in [Bigquery](https://console.cloud.google.com/bigquery) that will be used for this analysis if you don't already have one. The dataset location should be in a [location that Vertex AI services are available](https://cloud.google.com/vertex-ai/docs/general/locations#available_regions). "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "m6gkj-aXY37u",
      "metadata": {
        "id": "m6gkj-aXY37u"
      },
      "outputs": [],
      "source": [
        "%pip install -q -r 'requirements.txt'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "mKWzgxQ1Y37v",
      "metadata": {
        "id": "mKWzgxQ1Y37v"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "\n",
        "from src import crystalvalue"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "gCVphBgXY37w",
      "metadata": {
        "id": "gCVphBgXY37w"
      },
      "outputs": [],
      "source": [
        "# Initiate the CrystalValue class with the relevant parameters.\n",
        "pipeline = crystalvalue.CrystalValue(\n",
        "  project_id='your_project_name',  # The GCP project id.\n",
        "  dataset_id='your_dataset_name',  # The name of the pre-created dataset to work in. \n",
        "  customer_id_column='CustomerID',  # The customer ID column.\n",
        "  date_column='InvoiceDate',  # The transaction date column.\n",
        "  value_column='value',  #  Column to use for LTV calculation (i.e. profit or revenue).\n",
        "  days_lookback=90,  #  How many days in the past to use for feature engineering.\n",
        "  days_lookahead=365,  #  How many days in the future to use for value prediction.\n",
        "  ignore_columns=['Invoice'],  #  A list of columns in your input table to ignore.\n",
        "  location='europe-west4',  # This is the location of your dataset in Bigquery.\n",
        "  write_parameters=True,  #  Write parameters to local file so they can be retrieved for prediction.\n",
        "  credentials=None,  # The (optional) credentials to authenticate Bigquery and AIPlatform clients.\n",
        ")  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ySWcHL7DcKrd",
      "metadata": {
        "id": "ySWcHL7DcKrd"
      },
      "outputs": [],
      "source": [
        "# Read the data and rename the columns to be BiqQuery friendly (no spaces).\n",
        "data = pd.read_csv('./data/online_retail_II.csv')\n",
        "data.columns = data.columns.str.replace(' ', '')\n",
        "data['value'] = data['Price'] * data['Quantity']  # Calculate the value for transactions.\n",
        "data.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0qlNEnCjcL9f",
      "metadata": {
        "id": "0qlNEnCjcL9f"
      },
      "outputs": [],
      "source": [
        "# Load the data to Bigquery.\n",
        "TABLE_NAME = 'online_retail_data'\n",
        "\n",
        "pipeline.load_dataframe_to_bigquery(\n",
        "    data=data,\n",
        "    bigquery_table_name=TABLE_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "FHC77gHNcRDP",
      "metadata": {
        "id": "FHC77gHNcRDP"
      },
      "source": [
        "# Data Checks (Optional)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xIfjG7zNcVtx",
      "metadata": {
        "id": "xIfjG7zNcVtx"
      },
      "source": [
        "CrystalValue will run some checks on your data to check if the data is suitable for LTV modelling and raise errors if not. This will also output a new BigQuery table in your dataset called `crystalvalue_data_statistics` with key information such as the number of customers, customer return rate, transactions and analysis time period. This information can be used to check for outliers or anomalies (e.g. negative prices). "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "CTiZxBvrcSeV",
      "metadata": {
        "id": "CTiZxBvrcSeV"
      },
      "outputs": [],
      "source": [
        "summary_statistics = pipeline.run_data_checks(\n",
        "    transaction_table_name=TABLE_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "emCUVYhLFnla",
      "metadata": {
        "id": "emCUVYhLFnla"
      },
      "source": [
        "If a custom data cleaning routine has to be implemented use the `.run_query()` method. The example below removes transactions with negative prices. This method could also be used to run custom feature engineering scripts instead of the automated `.feature_engineering()` method in the next step. This data cleaning routine can be scheduled as part of the pipeline that we will define later (for model training and prediction)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "q_hFsfRGFjxG",
      "metadata": {
        "id": "q_hFsfRGFjxG"
      },
      "outputs": [],
      "source": [
        "query = f\"\"\"\n",
        "SELECT *\n",
        "FROM {pipeline.project_id}.{pipeline.dataset_id}.{TABLE_NAME}\n",
        "WHERE Price \u003e 0\n",
        "\"\"\"\n",
        "\n",
        "pipeline.run_query(\n",
        "    query_sql=query,\n",
        "    destination_table_name=TABLE_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5Bcu6lMlY37w",
      "metadata": {
        "id": "5Bcu6lMlY37w"
      },
      "source": [
        "# Feature Engineering\n",
        "\n",
        "Crystalvalue takes a transaction or browsing level dataset and creates a machine learning-ready dataset that can be ingested by AutoML. Data types are automatically inferred from the BigQuery schema unless the features are provided using the `feature_types` parameter in the `.feature_engineer()` method. Data transformations are applied automatically depending on the data type. The data crunching happens in BigQuery and the executed script can be optionally written to your directory. The features will be created in a BigQuery table called `crystalvalue_train_data` by default."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "yK5Lfb1SY37w",
      "metadata": {
        "id": "yK5Lfb1SY37w"
      },
      "outputs": [],
      "source": [
        "crystalvalue_train_data = pipeline.feature_engineer(\n",
        "  transaction_table_name=TABLE_NAME,\n",
        "  write_executed_query_file='src/executed_train_query.sql'  # (Optional) File path to write the executed SQL query.\n",
        ")  "
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xQ0yaFbBY37x",
      "metadata": {
        "id": "xQ0yaFbBY37x"
      },
      "source": [
        "# Model Training"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3JCdyprSY37x",
      "metadata": {
        "id": "3JCdyprSY37x"
      },
      "source": [
        "Crystalvalue leverages [Vertex AI (Tabular) AutoML](https://cloud.google.com/vertex-ai/docs/training/automl-api) which requires a\n",
        "[Vertex AI Dataset](https://cloud.google.com/vertex-ai/docs/datasets/create-dataset-api) as an input. CrystalValue automatically creates a Vertex AI Dataset from your input table as part of the training step of the pipeline. The training process typically takes about 2 or more hours to run. The Vertex AI Dataset will have a display name `crystalvalue_dataset`. The model will have a display name `crystalvalue_model` but it will also receive a model ID (so even if you train multiple models they will not be overwritten and can be identified using these IDs). By default CrystalValue chooses the following parameters:\n",
        "*  Predefined split with random 15% of users as test, 15% in validation and 70% in training.\n",
        "*  Optimization objective as Minimize root-mean-squared error (RMSE).\n",
        "*  1 node hour of training (1000 milli node hours), which we recommend starting with. [Modify this in line with the number of rows](https://cloud.google.com/automl-tables/docs/train#training_a_model) in the dataset when you are ready for productionising. See information here about [pricing](https://cloud.google.com/automl-tables/pricing).\n",
        "\n",
        "In this example we keep all the default settings so training the model is as simple as calling `pipeline.train_automl_model()`.\n",
        "\n",
        "In order to make fast predictions later, you can deploy the model using the `.deploy_model()` method."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "PvryAnPFiuWG",
      "metadata": {
        "id": "PvryAnPFiuWG"
      },
      "source": [
        "Once you start the training, you can view your model training progress here:  \n",
        "https://console.cloud.google.com/vertex-ai/training/training-pipelines  \n",
        "Once the training is finished, check out your Dataset (with statistics and distributions) and Model (with feature importance) in the UI:  \n",
        " https://console.cloud.google.com/vertex-ai/datasets   \n",
        " https://console.cloud.google.com/vertex-ai/models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "CSK1FtHVY37y",
      "metadata": {
        "id": "CSK1FtHVY37y"
      },
      "outputs": [],
      "source": [
        "model_object = pipeline.train_automl_model()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "dxA48EdTcs3c",
      "metadata": {
        "id": "dxA48EdTcs3c"
      },
      "outputs": [],
      "source": [
        "model_object = pipeline.deploy_model()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Dok2Wu6UY370",
      "metadata": {
        "id": "Dok2Wu6UY370"
      },
      "source": [
        "# Model Evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "wsJdx-MvY370",
      "metadata": {
        "id": "wsJdx-MvY370"
      },
      "source": [
        "To evaluate a model, we use the following criteria:\n",
        "\n",
        "* The spearman correlation, a measure of how well the model **ranked** the Liftetime value of customers in the test set. This is measured between -1 (worse) and 1 (better).\n",
        "* The normalised Gini coefficient, another measure of how well the model **ranked** the Lifetime value of customers in the test set compared to random ranking. This is measured between 0 (worse) and 1 (better). \n",
        "* The normalised Mean Average Error (MAE%). This is a measure of the **error** of the model's predictions for Lifetime value in the test set. \n",
        "* top_x_percent_predicted_customer_value_share: The proportion of value (i.e. total profit or revenue) in the test set that is accounted for by the top x% model-predicted customers. \n",
        "\n",
        "These outputs are sent to a BigQuery table (by default called `crystalvalue_evaluation`). Subsequent model evaluations append model performance evaluation metrics to this table to allow for comparison across models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "TmTV4LayjXo9",
      "metadata": {
        "id": "TmTV4LayjXo9"
      },
      "outputs": [],
      "source": [
        "metrics = pipeline.evaluate_model()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "oJ6ZFLwrY37y",
      "metadata": {
        "id": "oJ6ZFLwrY37y"
      },
      "source": [
        "# Generating predictions"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "jEzDVyiMY37z",
      "metadata": {
        "id": "jEzDVyiMY37z"
      },
      "source": [
        "Once model training is done, you can generate predictions. Features need to be engineered (the exact same as were used for model training) before prediction. This is done using the `.feature_engineer()` method by setting the parameter `query_type='predict_query'`. The features will be created in a BigQuery table called `crystalvalue_predict_data` by default. The model will make predictions for all customers in the provided input table that have any activity during the lookback window. The pLTV predictions will be for the period starting from the last date in the input table (not today's date).  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Z2sOiMID4X2-",
      "metadata": {
        "id": "Z2sOiMID4X2-"
      },
      "outputs": [],
      "source": [
        "crystalvalue_predict_data = pipeline.feature_engineer(\n",
        "    transaction_table_name=TABLE_NAME,  # An existing bigquery table in your dataset id containing the data to predict with.\n",
        "    query_type='predict_query')\n",
        "\n",
        "\n",
        "predictions = pipeline.predict(\n",
        "    input_table=crystalvalue_predict_data,\n",
        "    destination_table='crystalvalue_predictions'  # The bigquery table to append predictions to. It will be created if it does not exist yet.\n",
        "    )  "
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bDF0AE95vvuv",
      "metadata": {
        "id": "bDF0AE95vvuv"
      },
      "source": [
        "# Scheduling daily predictions"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "uWrKLbrEv0xo",
      "metadata": {
        "id": "uWrKLbrEv0xo"
      },
      "source": [
        "Crystalvalue uses [Vertex Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction) to schedule and monitor machine learning predictions. It can also be used for model retraining. The example below demonstrates how to set up the model to automatically create predictions using new input data from the source BigQuery table every day at 1am. The frequency and timing of the schedule can be altered using the chron schedule below. Once this pipeline is set up, you can view it [here](https://console.cloud.google.com/vertex-ai/pipelines). If you want a tutorial on how to set up Vertex Pipelines [this guide](https://cloud.google.com/vertex-ai/docs/pipelines/build-pipeline)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ZECtoCAyxbxX",
      "metadata": {
        "id": "ZECtoCAyxbxX"
      },
      "source": [
        "In order to use Vertex AI pipelines, we need a cloud storage bucket. Use the code below to create a cloud storage bucket. Note that you may have to grant Storage Object Admin to your service account to ensure the pipeline can run."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1zuWtrdwxjKX",
      "metadata": {
        "id": "1zuWtrdwxjKX"
      },
      "outputs": [],
      "source": [
        "BUCKET_NAME = 'crystalvalue_bucket'\n",
        "storage_bucket = pipeline.create_storage_bucket(bucket_name=BUCKET_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "pj3GIIJr5TqC",
      "metadata": {
        "id": "pj3GIIJr5TqC"
      },
      "source": [
        "In order to use Vertex AI pipelines with Crystalvalue we also need to create a docker container which will be stored in Google Cloud Container Registry. The following code builds a docker container and pushes it to your [GCP Container Registry](https://cloud.google.com/container-registry). \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "xCEA0QZo5ale",
      "metadata": {
        "id": "xCEA0QZo5ale"
      },
      "outputs": [],
      "source": [
        "!docker build -t crystalvalue .\n",
        "!docker tag crystalvalue gcr.io/$pipeline.project_id/crystalvalue\n",
        "!docker push gcr.io/$pipeline.project_id/crystalvalue"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "JlfrRkrF_jhR",
      "metadata": {
        "id": "JlfrRkrF_jhR"
      },
      "source": [
        "The Kubeflow components contains self-contained functions. Read about [Kubeflow components](https://www.kubeflow.org/docs/components/pipelines/sdk/component-development/).  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "jT1zxEqGwG17",
      "metadata": {
        "id": "jT1zxEqGwG17"
      },
      "outputs": [],
      "source": [
        "from kfp import dsl\n",
        "from kfp import compiler\n",
        "from kfp.dsl import component\n",
        "\n",
        "\n",
        "@component(base_image=f\"gcr.io/{pipeline.project_id}/crystalvalue:latest\")\n",
        "def pipeline_function():  \n",
        "  from src import crystalvalue\n",
        "  parameters = crystalvalue.load_parameters_from_file()\n",
        "  pipeline = crystalvalue.CrystalValue(**parameters)\n",
        "  TRANSACTION_TABLE = 'online_retail_data'  # Add your input table name.\n",
        "  pipeline.run_data_checks(transaction_table_name=TRANSACTION_TABLE)  \n",
        "  features = pipeline.feature_engineer(transaction_table_name=TRANSACTION_TABLE,\n",
        "                                       query_type='predict_query')\n",
        "  pipeline.predict(features)\n",
        "\n",
        "\n",
        "@dsl.pipeline(\n",
        "    name=\"crystalvaluepipeline\",\n",
        "    pipeline_root=f\"gs://{BUCKET_NAME}/pipeline_root\",\n",
        ")\n",
        "def crystalvalue_pipeline():\n",
        "    pipeline_function()\n",
        "    \n",
        "compiler.Compiler().compile(\n",
        "  pipeline_func=crystalvalue_pipeline,\n",
        "  package_path=\"crystalvaluepipeline.json\"\n",
        ")"
      ]
    },
    {
      "metadata": {
        "id": "zNSjgatukOKR"
      },
      "cell_type": "code",
      "source": [
        "from google.cloud import aiplatform\n",
        "\n",
        "# Choose a region compatible with Vertex Pipelines. \n",
        "# This doesn't have to be the same as your data location.\n",
        "\n",
        "PROJECT_ID = pipeline.project_id,\n",
        "REGION=pipeline.location,\n",
        "DISPLAY_NAME='crystalvalue_pipeline',\n",
        "PIPELINE_ROOT=f\"gs://{BUCKET_NAME}/pipeline_root\",\n",
        "PACKAGE_PATH=\"crystalvaluepipeline.json\",\n",
        "aiplatform.init(\n",
        "    project=PROJECT_ID,\n",
        "    location=REGION,\n",
        ")\n",
        "\n",
        "pipeline_job = aiplatform.PipelineJob(\n",
        "    display_name=DISPLAY_NAME,\n",
        "    template_path=PACKAGE_PATH,\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        ")\n",
        "\n",
        "## Check if your pipeline is running.\n",
        "job.submit()"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "id": "e_uER_PJ_fNW",
      "metadata": {
        "id": "e_uER_PJ_fNW"
      },
      "source": [
        "Create the scheduled pipeline. Adjust time zone and cron schedule as necessary."
      ]
    },
    {
      "metadata": {
        "id": "t7L0QRKOaLYM"
      },
      "cell_type": "code",
      "source": [
        "pipeline_job_schedule = pipeline_job.create_schedule(\n",
        "  display_name=\"crystalvalue_pipeline_schedule\",\n",
        "  cron=\"TZ=CRON\",\n",
        "  max_concurrent_run_count=MAX_CONCURRENT_RUN_COUNT,\n",
        "  max_run_count=MAX_RUN_COUNT,\n",
        ")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "id": "sxeSm7BxA5so",
      "metadata": {
        "id": "sxeSm7BxA5so"
      },
      "source": [
        "You can view your running and scheduled pipelines at:\n",
        "https://console.cloud.google.com/vertex-ai/pipelines or by adjusting the code below."
      ]
    },
    {
      "metadata": {
        "id": "ac4RlBqXaLYM"
      },
      "cell_type": "code",
      "source": [
        "aiplatform.PipelineJobSchedule.list(\n",
        "  filter='display_name=\"DISPLAY_NAME\"',\n",
        "  order_by='create_time desc'\n",
        ")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "id": "yNZpQxNU-cIi",
      "metadata": {
        "id": "yNZpQxNU-cIi"
      },
      "source": [
        "# (Optional) Get insights into the relationship between your features and customer LTV"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ZXZUlaZV_Ms9",
      "metadata": {
        "id": "ZXZUlaZV_Ms9"
      },
      "source": [
        "To get insights into how your model is making predictions based on your features using the [What-If Tool](https://pair-code.github.io/what-if-tool/). Check out an [online demo here](https://pair-code.github.io/what-if-tool/demos/age.html)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0HyVMxQj-jfl",
      "metadata": {
        "id": "0HyVMxQj-jfl"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "from witwidget.notebook.visualization import WitConfigBuilder\n",
        "from witwidget.notebook.visualization import WitWidget"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "CUK1kcr0-pS7",
      "metadata": {
        "id": "CUK1kcr0-pS7"
      },
      "outputs": [],
      "source": [
        "features_with_predictions = pd.concat([\n",
        "    crystalvalue_predict_data.iloc[:,7:],\n",
        "    predictions['predicted_value']], axis=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "s8hedfxF-qcn",
      "metadata": {
        "id": "s8hedfxF-qcn"
      },
      "outputs": [],
      "source": [
        "config_builder = WitConfigBuilder(\n",
        "    np.array(features_with_predictions[0:1000]).tolist(),\n",
        "    list(features_with_predictions)\n",
        ")\n",
        "WitWidget(config_builder, height=1000)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "R7RczTD6S1_O",
      "metadata": {
        "id": "R7RczTD6S1_O"
      },
      "source": [
        "# Clean Up"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6FyiJ93rTWhW",
      "metadata": {
        "id": "6FyiJ93rTWhW"
      },
      "source": [
        "To clean up tables created during this demo, delete the BigQuery tables that were created. All Vertex AI resources can be removed from the [Vertex AI console](https://console.cloud.google.com/vertex-ai). If you set up a Vertex Pipeline then also remove any relevant resources from [Cloud Storage](https://console.cloud.google.com/storage) and [Container Registry](https://console.cloud.google.com//gcr/images/). "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1wQI6E8US3Hc",
      "metadata": {
        "id": "1wQI6E8US3Hc"
      },
      "outputs": [],
      "source": [
        "pipeline.delete_table('crystalvalue_data_statistics')\n",
        "pipeline.delete_table('crystalvalue_evaluation')\n",
        "pipeline.delete_table('crystalvalue_train_data')\n",
        "pipeline.delete_table('crystalvalue_predict_data')\n",
        "pipeline.delete_table('crystalvalue_predictions')"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "last_runtime": {
        "build_target": "//corp/gtech/ads/infrastructure/colab_utils/ds_runtime:ds_colab",
        "kind": "private"
      },
      "name": "demo_with_real_data_notebook.ipynb",
      "private_outputs": true,
      "provenance": [
        {
          "file_id": "/piper/depot/google3/third_party/professional_services/solutions/crystalvalue/crystalvalue_demo_notebook.ipynb?workspaceId=sumedhamenon:CrystalValue::citc",
          "timestamp": 1627656627042
        },
        {
          "file_id": "/piper/depot/google3/third_party/professional_services/solutions/crystalvalue/crystalvalue_demo_notebook.ipynb?workspaceId=sumedhamenon:CrystalValue::citc",
          "timestamp": 1627656460877
        },
        {
          "file_id": "/piper/depot/google3/experimental/gtech_prem_data_science/projects/trac/treatwell/Crystal_Value_Demo_Notebook.ipynb?workspaceId=sumedhamenon:CrystalValue::citc",
          "timestamp": 1627648785641
        },
        {
          "file_id": "1JGQRDc1_luQsaMxx9ZZavddHBrgO7Xst",
          "timestamp": 1627648471137
        }
      ]
    },
    "environment": {
      "name": "common-cpu.mnightly-2021-01-05-debian-10-test",
      "type": "gcloud",
      "uri": "gcr.io/deeplearning-platform-release/base-cpu:mnightly-2021-01-05-debian-10-test"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}