C3_W4_Lab_1_FashionMNIST_CAM.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "C3_W4_Lab_1_FashionMNIST-CAM.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/vsnupoudel/AdvancedCvCourses/blob/main/C3_W4_Lab_1_FashionMNIST_CAM.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LhWiLK0IEsqX"
      },
      "source": [
        "# Ungraded Lab: Class Activation Maps with Fashion MNIST\n",
        "\n",
        "In this lab, you will see how to implement a simple class activation map (CAM) of a model trained on the [Fashion MNIST dataset](https://github.com/zalandoresearch/fashion-mnist). This will show what parts of the image the model was paying attention to when deciding the class of the image. Let's begin!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wOAlJAfRIY50"
      },
      "source": [
        "## Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zSyMHuCVys-O"
      },
      "source": [
        "import keras\n",
        "from keras.datasets import fashion_mnist\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from keras.models import Sequential,Model\n",
        "from keras.layers import Dense, Conv2D, MaxPooling2D, GlobalAveragePooling2D\n",
        "import scipy as sp"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uMIJYr6FIbZt"
      },
      "source": [
        "## Download and Prepare the Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "01974419yy5W"
      },
      "source": [
        "# load the Fashion MNIST dataset\n",
        "(X_train,Y_train),(X_test,Y_test)  = fashion_mnist.load_data()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VfVB6x6Oy1yF"
      },
      "source": [
        "# Put an additional axis for the channels of the image.\n",
        "# Fashion MNIST is grayscale so we place 1 at the end. Other datasets\n",
        "# will need 3 if it's in RGB.\n",
        "X_train = X_train.reshape(60000,28,28,1)\n",
        "X_test = X_test.reshape(10000,28,28,1)\n",
        "\n",
        "# Normalize the pixel values from 0 to 1\n",
        "X_train = X_train/255\n",
        "X_test  = X_test/255\n",
        "\n",
        "# Cast to float\n",
        "X_train = X_train.astype('float')\n",
        "X_test  = X_test.astype('float')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SDx99oRCzfTr"
      },
      "source": [
        "def show_img(img):\n",
        "    '''utility function for reshaping and displaying an image'''\n",
        "\n",
        "    # convert to float array if img is not yet preprocessed\n",
        "    img  = np.array(img,dtype='float')\n",
        "\n",
        "    # remove channel dimension\n",
        "    img = img.reshape((28,28))\n",
        "\n",
        "    # display image\n",
        "    plt.imshow(img)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rAL6lD1PLlj2"
      },
      "source": [
        "# test the function for the first train image. you can vary the index of X_train\n",
        "# below to see other images\n",
        "\n",
        "show_img(X_train[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1xPggGPyOhm5"
      },
      "source": [
        "## Build the Classifier\n",
        "\n",
        "Let's quickly recap how we can build a simple classifier with this dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ds9n85HmJySy"
      },
      "source": [
        "### Define the Model\n",
        "\n",
        "You can build the classifier with the model below. The image will go through 4 convolutions followed by pooling layers. The final Dense layer will output the probabilities for each class."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QyCoMd93zpc_"
      },
      "source": [
        "# use the Sequential API\n",
        "model = Sequential()\n",
        "\n",
        "# notice the padding parameter to recover the lost border pixels when doing the convolution\n",
        "model.add(Conv2D(16,input_shape=(28,28,1),kernel_size=(3,3),activation='relu',padding='same'))\n",
        "# pooling layer with a stride of 2 will reduce the image dimensions by half\n",
        "model.add(MaxPooling2D(pool_size=(2,2)))\n",
        "\n",
        "# pass through more convolutions with increasing filters\n",
        "model.add(Conv2D(32,kernel_size=(3,3),activation='relu',padding='same'))\n",
        "model.add(MaxPooling2D(pool_size=(2,2)))\n",
        "\n",
        "model.add(Conv2D(64,kernel_size=(3,3),activation='relu',padding='same'))\n",
        "model.add(MaxPooling2D(pool_size=(2,2)))\n",
        "\n",
        "model.add(Conv2D(128,kernel_size=(3,3),activation='relu',padding='same'))\n",
        "\n",
        "# use global average pooling to take into account lesser intensity pixels\n",
        "model.add(GlobalAveragePooling2D())\n",
        "\n",
        "# output class probabilities\n",
        "model.add(Dense(10,activation='softmax'))\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fxsip1WuOk0Q"
      },
      "source": [
        "### Train the Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t8LswgblLvc8"
      },
      "source": [
        "# configure the training\n",
        "model.compile(loss='sparse_categorical_crossentropy',metrics=['accuracy'],optimizer='adam')\n",
        "\n",
        "# train the model. just run a few epochs for this test run. you can adjust later.\n",
        "model.fit(X_train,Y_train,batch_size=32, epochs=5, validation_split=0.1, shuffle=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eriUpnzxQkus"
      },
      "source": [
        "## Generate the Class Activation Map\n",
        "\n",
        "To generate the class activation map, we want to get the features detected in the last convolution layer and see which ones are most active when generating the output probabilities. In our model above, we are interested in the layers shown below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UblFRpsK0Gd7"
      },
      "source": [
        "# final convolution layer\n",
        "print(model.layers[-3].name)\n",
        "\n",
        "# global average pooling layer\n",
        "print(model.layers[-2].name)\n",
        "\n",
        "# output of the classifier\n",
        "print(model.layers[-1].name)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cXAYTZZVrZFB"
      },
      "source": [
        "You can now create your CAM model as shown below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JSc42Fd4rZFB"
      },
      "source": [
        "# same as previous model but with an additional output\n",
        "cam_model  = Model(inputs=model.input,outputs=(model.layers[-3].output,model.layers[-1].output))\n",
        "cam_model.summary()\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MvWbe6sLrZFB"
      },
      "source": [
        "Use the CAM model to predict on the test set, so that it generates the features and the predicted probability for each class (`results`)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R8625UmNrxOU"
      },
      "source": [
        "# get the features and results of the test images using the newly created model\n",
        "features, results = cam_model.predict(X_test)\n",
        "\n",
        "# shape of the features\n",
        "print(\"features shape: \", features.shape)\n",
        "print(\"results shape\", results.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bfQMCOtmTaDX"
      },
      "source": [
        "You can generate the CAM by getting the dot product of the class activation features and the class activation weights.\n",
        "\n",
        "You will need the weights from the Global Average Pooling layer (GAP) to calculate the activations of each feature given a particular class.\n",
        "- Note that you'll get the weights from the dense layer that follows the global average pooling layer.\n",
        "  - The last conv2D layer has (h,w,depth) of (3 x 3 x 128), so there are 128 features.\n",
        "  - The global average pooling layer collapses the h,w,f (3 x 3 x 128) into a dense layer of 128 neurons (1 neuron per feature).\n",
        "  - The activations from the global average pooling layer get passed to the last dense layer.\n",
        "  - The last dense layer assigns weights to each of those 128 features (for each of the 10 classes),\n",
        "  - So the weights of the last dense layer (which immmediately follows the global average pooling layer) are referred to in this context as the \"weights of the global average pooling layer\".\n",
        "\n",
        "For each of the 10 classes, there are 128 features, so there are 128 feature weights, one weight per feature."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UwfG1vyprZFC"
      },
      "source": [
        "# these are the weights going into the softmax layer\n",
        "last_dense_layer = model.layers[-1]\n",
        "\n",
        "# get the weights list.  index 0 contains the weights, index 1 contains the biases\n",
        "gap_weights_l = last_dense_layer.get_weights()\n",
        "\n",
        "print(\"gap_weights_l index 0 contains weights \", gap_weights_l[0].shape)\n",
        "print(\"gap_weights_l index 1 contains biases \", gap_weights_l[1].shape)\n",
        "\n",
        "# shows the number of features per class, and the total number of classes\n",
        "# Store the weights\n",
        "gap_weights = gap_weights_l[0]\n",
        "\n",
        "print(f\"There are {gap_weights.shape[0]} feature weights and {gap_weights.shape[1]} classes.\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y7GtR-4NrZFC"
      },
      "source": [
        "Now, get the features for a specific image, indexed between 0 and 999."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x2aA6_ZTrZFD"
      },
      "source": [
        "# Get the features for the image at index 0\n",
        "idx = 0\n",
        "features_for_img = features[idx,:,:,:]\n",
        "\n",
        "print(f\"The features for image index {idx} has shape (height, width, num of feature channels) : \", features_for_img.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0OSiDiTnrZFD"
      },
      "source": [
        "The features have height and width of 3 by 3.  Scale them up to the original image height and width, which is 28 by 28."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1sNUNDLDrZFD"
      },
      "source": [
        "features_for_img_scaled = sp.ndimage.zoom(features_for_img, (28/3, 28/3,1), order=2)\n",
        "\n",
        "# Check the shape after scaling up to 28 by 28 (still 128 feature channels)\n",
        "print(\"features_for_img_scaled up to 28 by 28 height and width:\", features_for_img_scaled.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AYj5w9NhrZFD"
      },
      "source": [
        "For a particular class (0...9), get the 128 weights.\n",
        "\n",
        "Take the dot product with the scaled features for this selected image with the weights.\n",
        "\n",
        "The shapes are:\n",
        "scaled features: (h,w,depth) of (28 x 28 x 128).\n",
        "weights for one class: 128\n",
        "\n",
        "The dot product produces the class activation map, with the shape equal to the height and width of the image: 28 x 28."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sNj4D8FprZFD"
      },
      "source": [
        "# Select the weights that are used for a specific class (0...9)\n",
        "class_id = 0\n",
        "# take the dot product between the scaled image features and the weights for \n",
        "gap_weights_for_one_class = gap_weights[:,class_id]\n",
        "\n",
        "print(\"features_for_img_scaled has shape \", features_for_img_scaled.shape)\n",
        "print(\"gap_weights_for_one_class has shape \", gap_weights_for_one_class.shape)\n",
        "# take the dot product between the scaled features and the weights for one class\n",
        "cam = np.dot(features_for_img_scaled, gap_weights_for_one_class)\n",
        "\n",
        "print(\"class activation map shape \", cam.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vag5enPdrZFE"
      },
      "source": [
        "### Conceptual interpretation\n",
        "To think conceptually about what what you're doing and why:\n",
        "- In the 28 x 28 x 128 feature map, each of the 128 feature filters is tailored to look for a specific set of features (for example, a shoelace).  \n",
        "  - The actual features are learned, not selected by you directly.\n",
        "- Each of the 128 weights for a particular class decide how much weight to give to each of the 128 features, for that class.\n",
        "  - For instance, for the \"shoe\" class, it may have a higher weight for the feature filters that look for shoelaces.\n",
        "- At each of the 28 by 28 pixels, you can take the vector of 128 features and compare them with the vector of 128 weights.  \n",
        "  - You can do this comparison with a dot product.\n",
        "  - The dot product results in a scalar value at each pixel.\n",
        "  - Apply this dot product across all of the 28 x 28 pixels.\n",
        "  - The scalar result of the dot product will be larger when the image both has the particular feature (e.g. shoelace), and that feature is also weighted more heavily for the particular class (e.g shoe).\n",
        "  \n",
        "So you've created a matrix with the same number of pixels as the image, where the value at each pixel is higher when that pixel is relevant to the prediction of a particular class."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G-e9U5poVBis"
      },
      "source": [
        "Here is the function that implements the Class activation map calculations that you just saw."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YByJ8J1008Ms"
      },
      "source": [
        "def show_cam(image_index):\n",
        "  '''displays the class activation map of a particular image'''\n",
        "\n",
        "  # takes the features of the chosen image\n",
        "  features_for_img = features[image_index,:,:,:]\n",
        "\n",
        "  # get the class with the highest output probability\n",
        "  prediction = np.argmax(results[image_index])\n",
        "\n",
        "  # get the gap weights at the predicted class\n",
        "  class_activation_weights = gap_weights[:,prediction]\n",
        "\n",
        "  # upsample the features to the image's original size (28 x 28)\n",
        "  class_activation_features = sp.ndimage.zoom(features_for_img, (28/3, 28/3, 1), order=2)\n",
        "\n",
        "  # compute the intensity of each feature in the CAM\n",
        "  cam_output  = np.dot(class_activation_features,class_activation_weights)\n",
        "  \n",
        "  print('Predicted Class = ' +str(prediction)+ ', Probability = ' + str(results[image_index][prediction]))\n",
        "  \n",
        "  # show the upsampled image\n",
        "  plt.imshow(np.squeeze(X_test[image_index],-1), alpha=0.5)\n",
        "  \n",
        "  # strongly classified (95% probability) images will be in green, else red\n",
        "  if results[image_index][prediction]>0.95:\n",
        "    cmap_str = 'Greens'\n",
        "  else:\n",
        "    cmap_str = 'Reds'\n",
        "\n",
        "  # overlay the cam output\n",
        "  plt.imshow(cam_output, cmap=cmap_str, alpha=0.5)\n",
        "\n",
        "  # display the image\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9fjIdEmlXbez"
      },
      "source": [
        "You can now test generating class activation maps. Let's use the utility function below. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8iQjMfg9U03_"
      },
      "source": [
        "def show_maps(desired_class, num_maps):\n",
        "    '''\n",
        "    goes through the first 10,000 test images and generates CAMs \n",
        "    for the first `num_maps`(int) of the `desired_class`(int)\n",
        "    '''\n",
        "\n",
        "    counter = 0\n",
        "\n",
        "    if desired_class < 10:\n",
        "        print(\"please choose a class less than 10\")\n",
        "\n",
        "    # go through the first 10000 images\n",
        "    for i in range(0,10000):\n",
        "        # break if we already displayed the specified number of maps\n",
        "        if counter == num_maps:\n",
        "            break\n",
        "\n",
        "        # images that match the class will be shown\n",
        "        if np.argmax(results[i]) == desired_class:\n",
        "            counter += 1\n",
        "            show_cam(i)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ctjDEfzrjXa5"
      },
      "source": [
        "For class 8 (handbag), you'll notice that most of the images have dark spots in the middle and right side. \n",
        "- This means that these areas were given less importance when categorizing the image. \n",
        "- The other parts such as the outline or handle contribute more when deciding if an image is a handbag or not. \n",
        "\n",
        "Observe the other classes and see if there are also other common areas that the model uses more in determining the class of the image."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AkPMvVurezkb"
      },
      "source": [
        "show_maps(desired_class=8, num_maps=6)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d9Jm20L-9GM2"
      },
      "source": [
        "show_maps(desired_class=0, num_maps=3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NJ1qUVjJ9Otf"
      },
      "source": [
        "show_maps(desired_class=1, num_maps=3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o3Xub5iq9SXp"
      },
      "source": [
        "show_maps(desired_class=2, num_maps=10)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4E2hK8cf99ge"
      },
      "source": [
        "**Above, T shirts are activating more in the middle part, rather than the outlines.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XEBwbTyj9UXE"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}