C3_W4_Lab_4_GradCam.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "C3_W4_Lab_4_GradCam.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/vsnupoudel/AdvancedCvCourses/blob/main/C3_W4_Lab_4_GradCam.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cjJN-ms8aoZg"
      },
      "source": [
        "# Ungraded Lab: GradCAM\n",
        "\n",
        "This lab will walk you through generating gradient-weighted class activation maps (GradCAMs) for model predictions. \n",
        "- This is similar to the CAMs you generated before except:\n",
        "  - GradCAMs uses gradients instead of the global average pooling weights to weight the activations."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jz7WEt5dK3VL"
      },
      "source": [
        "## Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7B7AHdVuCA25"
      },
      "source": [
        "%tensorflow_version 2.x\n",
        "\n",
        "import warnings \n",
        "warnings.filterwarnings(\"ignore\")\n",
        "\n",
        "import os\n",
        "import glob\n",
        "import cv2\n",
        "from pathlib import Path\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "from skimage.io import imread, imsave\n",
        "from skimage.transform import resize \n",
        "from sklearn.model_selection import train_test_split\n",
        "from tensorflow.keras.models import Model\n",
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras.applications import vgg16\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "from tensorflow.keras.optimizers import SGD, Adam, RMSprop\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow.keras.backend as K\n",
        "import tensorflow_datasets as tfds\n",
        "import tensorflow_hub as hub\n",
        "\n",
        "import imgaug as ia\n",
        "from imgaug import augmenters as iaa"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nw6uOrWbvE-z"
      },
      "source": [
        "## Download and Prepare the Dataset\n",
        "\n",
        "You will use the Cats vs Dogs dataset again for this exercise. The following will prepare the train, test, and eval sets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w4hV5x9RvJxW"
      },
      "source": [
        "tfds.disable_progress_bar()\n",
        "\n",
        "splits = ['train[:80%]', 'train[80%:90%]', 'train[90%:]']\n",
        "\n",
        "# load the dataset given the splits defined above\n",
        "splits, info = tfds.load('cats_vs_dogs', with_info=True, as_supervised=True, split = splits)\n",
        "\n",
        "(train_examples, validation_examples, test_examples) = splits\n",
        "\n",
        "num_examples = info.splits['train'].num_examples\n",
        "num_classes = info.features['label'].num_classes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8GBF2PGyvmln"
      },
      "source": [
        "BATCH_SIZE = 32\n",
        "IMAGE_SIZE = (224, 224)\n",
        "\n",
        "# resizes the image and normalizes the pixel values\n",
        "def format_image(image, label):\n",
        "  image = tf.image.resize(image, IMAGE_SIZE) / 255.0\n",
        "  return  image, label\n",
        "\n",
        "# prepare batches\n",
        "train_batches = train_examples.shuffle(num_examples // 4).map(format_image).batch(BATCH_SIZE).prefetch(tf.data.experimental.AUTOTUNE)\n",
        "validation_batches = validation_examples.map(format_image).batch(BATCH_SIZE).prefetch(tf.data.experimental.AUTOTUNE)\n",
        "test_batches = test_examples.map(format_image).batch(1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W5v4-HIdV2lz"
      },
      "source": [
        "## Modelling\n",
        "\n",
        "You will use a pre-trained VGG16 network as your base model for the classifier. This will be followed by a global average pooling (GAP) and a 2-neuron Dense layer with softmax activation for the output. The earlier VGG blocks will be frozen and we will just fine-tune the final layers during training. These steps are shown in the utility function below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EanWlELFpxk2"
      },
      "source": [
        "def build_model():\n",
        "  # load the base VGG16 model\n",
        "  base_model = vgg16.VGG16(input_shape=IMAGE_SIZE + (3,), \n",
        "                      weights='imagenet', \n",
        "                      include_top=False)\n",
        "  \n",
        "  # add a GAP layer\n",
        "  output = layers.GlobalAveragePooling2D()(base_model.output)\n",
        "\n",
        "  # output has two neurons for the 2 classes (cats and dogs)\n",
        "  output = layers.Dense(2, activation='softmax')(output)\n",
        "\n",
        "  # set the inputs and outputs of the model\n",
        "  model = Model(base_model.input, output)\n",
        "\n",
        "  # freeze the earlier layers\n",
        "  for layer in base_model.layers[:-4]:\n",
        "      layer.trainable=False\n",
        "\n",
        "  # choose the optimizer\n",
        "  optimizer = tf.keras.optimizers.RMSprop(0.001)\n",
        "\n",
        "  # configure the model for training\n",
        "  model.compile(loss='sparse_categorical_crossentropy', \n",
        "                optimizer=optimizer, \n",
        "                metrics=['accuracy'])\n",
        "  \n",
        "  # display the summary\n",
        "  model.summary()\n",
        "  \n",
        "  return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8j3nULUDcs9i"
      },
      "source": [
        "model = build_model()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LI6XSP96PAO6"
      },
      "source": [
        "You can now train the model. This will take around 10 minutes to run."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kLR1ZgEXrMRX"
      },
      "source": [
        "EPOCHS = 3\n",
        "model.fit(train_batches,\n",
        "          epochs=EPOCHS,\n",
        "          validation_data=validation_batches)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eNnIA33ohDkf"
      },
      "source": [
        "## Model Interpretability\n",
        "\n",
        "Let's now go through the steps to generate the class activation maps. You will start by specifying the layers you want to visualize.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EgR_j1sGwT86"
      },
      "source": [
        "# select all the layers for which you want to visualize the outputs and store it in a list\n",
        "outputs = [layer.output for layer in model.layers[1:18]]\n",
        "\n",
        "# Define a new model that generates the above output\n",
        "vis_model = Model(model.input ,  outputs)\n",
        "\n",
        "# store the layer names we are interested in\n",
        "layer_names = []\n",
        "for layer in outputs:\n",
        "    # print(layer.name)\n",
        "    layer_names.append(layer.name.split(\"/\")[0])\n",
        "\n",
        "    \n",
        "print(\"Layers that will be used for visualization: \")\n",
        "print(layer_names)\n",
        "print( layer_names[16])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4cMozBwAt2Y5"
      },
      "source": [
        "### Class activation maps (GradCAM)\n",
        "\n",
        "We'll define a few more functions to output the maps. `get_CAM()` is the function highlighted in the lectures and takes care of generating the heatmap of gradient weighted features. `show_random_sample()` takes care of plotting the results."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wfb6lPuyjfc4"
      },
      "source": [
        "def get_CAM(processed_image, actual_label, layer_name='block5_conv3'):\n",
        "    model_grad = Model([model.inputs], \n",
        "                       [model.get_layer(layer_name).output, model.output])\n",
        "    \n",
        "    with tf.GradientTape() as tape:\n",
        "        conv_output_values, predictions = model_grad(processed_image)\n",
        "\n",
        "        # watch the conv_output_values\n",
        "        tape.watch(conv_output_values)\n",
        "\n",
        "        ## Use binary cross entropy loss\n",
        "        ## actual_label is 0 if cat, 1 if dog\n",
        "        # get prediction probability of dog\n",
        "        # If model does well, \n",
        "        # pred_prob should be close to 0 if cat, close to 1 if dog\n",
        "        pred_prob = predictions[:,1] \n",
        "        \n",
        "        # make sure actual_label is a float, like the rest of the loss calculation\n",
        "        actual_label = tf.cast(actual_label, dtype=tf.float32)\n",
        "        \n",
        "        # add a tiny value to avoid log of 0\n",
        "        smoothing = 0.00001 \n",
        "        \n",
        "        # Calculate loss as binary cross entropy\n",
        "        loss = -1 * (actual_label * tf.math.log(pred_prob + smoothing) + (1 - actual_label) * tf.math.log(1 - pred_prob + smoothing))\n",
        "        print(f\"binary loss: {loss}\")\n",
        "    \n",
        "    # get the gradient of the loss with respect to the outputs of the last conv layer\n",
        "    grads_values = tape.gradient(loss, conv_output_values)\n",
        "    grads_values = K.mean(grads_values, axis=(0,1,2))\n",
        "    \n",
        "    conv_output_values = np.squeeze(conv_output_values.numpy())\n",
        "    grads_values = grads_values.numpy()\n",
        "    \n",
        "    # weight the convolution outputs with the computed gradients\n",
        "    for i in range(512): \n",
        "        conv_output_values[:,:,i] *= grads_values[i]\n",
        "    heatmap = np.mean(conv_output_values, axis=-1)\n",
        "    \n",
        "    heatmap = np.maximum(heatmap, 0)\n",
        "    heatmap /= heatmap.max()\n",
        "    \n",
        "    del model_grad, conv_output_values, grads_values, loss\n",
        "   \n",
        "    return heatmap"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s2zkEe_9l6hH"
      },
      "source": [
        "def show_sample(idx=None):\n",
        "    \n",
        "    # if image index is specified, get that image\n",
        "    if idx:\n",
        "        for img, label in test_batches.take(idx):\n",
        "            sample_image = img[0]\n",
        "            sample_label = label[0]\n",
        "    # otherwise if idx is not specified, get a random image\n",
        "    else:\n",
        "        for img, label in test_batches.shuffle(1000).take(1):\n",
        "            sample_image = img[0]\n",
        "            sample_label = label[0]\n",
        "    \n",
        "    sample_image_processed = np.expand_dims(sample_image, axis=0)\n",
        "    \n",
        "    activations = vis_model.predict(sample_image_processed)\n",
        "    \n",
        "    pred_label = np.argmax(model.predict(sample_image_processed), axis=-1)[0]\n",
        "    \n",
        "    sample_activation = activations[0][0,:,:,16]\n",
        "    \n",
        "    sample_activation -= sample_activation.mean()\n",
        "    sample_activation /= sample_activation.std()\n",
        "    \n",
        "    sample_activation *=255\n",
        "    sample_activation = np.clip(sample_activation, 0, 255).astype(np.uint8)\n",
        "    \n",
        "    heatmap = get_CAM(sample_image_processed, sample_label)\n",
        "    heatmap = cv2.resize(heatmap, (sample_image.shape[0], sample_image.shape[1]))\n",
        "    heatmap = heatmap *255\n",
        "    heatmap = np.clip(heatmap, 0, 255).astype(np.uint8)\n",
        "    heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_HOT)\n",
        "    converted_img = sample_image.numpy()\n",
        "    super_imposed_image = cv2.addWeighted(converted_img, 0.8, heatmap.astype('float32'), 2e-3, 0.0)\n",
        "\n",
        "    f,ax = plt.subplots(2,2, figsize=(15,8))\n",
        "\n",
        "    ax[0,0].imshow(sample_image)\n",
        "    ax[0,0].set_title(f\"True label: {sample_label} \\n Predicted label: {pred_label}\")\n",
        "    ax[0,0].axis('off')\n",
        "    \n",
        "    ax[0,1].imshow(sample_activation)\n",
        "    ax[0,1].set_title(\"Random feature map\")\n",
        "    ax[0,1].axis('off')\n",
        "    \n",
        "    ax[1,0].imshow(heatmap)\n",
        "    ax[1,0].set_title(\"Class Activation Map\")\n",
        "    ax[1,0].axis('off')\n",
        "    \n",
        "    ax[1,1].imshow(super_imposed_image)\n",
        "    ax[1,1].set_title(\"Activation map superimposed\")\n",
        "    ax[1,1].axis('off')\n",
        "    plt.tight_layout()\n",
        "    plt.show()\n",
        "  \n",
        "    return activations"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zSaeGyNjnA7x"
      },
      "source": [
        "### Time to visualize the results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bHfizQifnMHn"
      },
      "source": [
        "# Choose an image index to show, or leave it as None to get a random image\n",
        "activations = show_sample(idx=None)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "niwu52jOnXMX"
      },
      "source": [
        "### Intermediate activations of layers\n",
        "\n",
        "You can use the utility function below to visualize the activations in the intermediate layers you defined earlier. This plots the feature side by side for each convolution layer starting from the earliest layer all the way to the final convolution layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n2khEVdunQ9N"
      },
      "source": [
        "def visualize_intermediate_activations(layer_names, activations):\n",
        "    assert len(layer_names)==len(activations), \"Make sure layers and activation values match\"\n",
        "    images_per_row=16\n",
        "    \n",
        "    for layer_name, layer_activation in zip(layer_names, activations):\n",
        "        nb_features = layer_activation.shape[-1]\n",
        "        size= layer_activation.shape[1]\n",
        "\n",
        "        nb_cols = nb_features // images_per_row\n",
        "        grid = np.zeros((size*nb_cols, size*images_per_row))\n",
        "\n",
        "        for col in range(nb_cols):\n",
        "            for row in range(images_per_row):\n",
        "                feature_map = layer_activation[0,:,:,col*images_per_row + row]\n",
        "                feature_map -= feature_map.mean()\n",
        "                feature_map /= feature_map.std()\n",
        "                feature_map *=255\n",
        "                feature_map = np.clip(feature_map, 0, 255).astype(np.uint8)\n",
        "\n",
        "                grid[col*size:(col+1)*size, row*size:(row+1)*size] = feature_map\n",
        "\n",
        "        scale = 1./size\n",
        "        plt.figure(figsize=(scale*grid.shape[1], scale*grid.shape[0]))\n",
        "        plt.title(layer_name)\n",
        "        plt.grid(False)\n",
        "        plt.axis('off')\n",
        "        plt.imshow(grid, aspect='auto', cmap='viridis')\n",
        "    plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0hE4PXkOnjRO"
      },
      "source": [
        "visualize_intermediate_activations(activations=activations, \n",
        "                                   layer_names=layer_names)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WQszzrg6TePY"
      },
      "source": [
        "If you scroll all the way down to see the outputs of the final conv layer, you'll see that there are very few active features and these are mostly located in the face of the cat. This is the region of the image that your model looks at when determining the class."
      ]
    }
  ]
}