evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "<a href=\"http://camma.u-strasbg.fr/\">\n",
    "<img src=\"lib/camma_logo.png\" width=\"18%\">\n",
    "</a>\n",
    "</div>\n",
    "\n",
    "\n",
    "\n",
    "Weakly-supervised ConvLSTM Surgical Tool Tracker\n",
    "================\n",
    "------\n",
    "**A re-implementation of the surgical tool tracker in** :<br>\n",
    "<i>Nwoye, C. I., Mutter, D., Marescaux, J., & Padoy, N. (2019). \n",
    "    Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos. \n",
    "    International journal of computer assisted radiology and surgery, 14(6), 1059-1067.<br></i>\n",
    "(c) Research Group CAMMA, University of Strasbourg, France<br>\n",
    "Website: http://camma.u-strasbg.fr<br>\n",
    "Code author: Chinedu Nwoye <br>\n",
    "    \n",
    "-----\n",
    "\n",
    "The model is built using the `tf.contrib` lib. Hence, TensorFlow version > 1.15 is discouraged."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Download code and libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/CAMMA-public/ConvLSTM-Surgical-Tool-Tracker.git\n",
    "%cd ConvLSTM-Surgical-Tool-Tracker\n",
    "\n",
    "print(\"Repo cloned and extracted ...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Download sample video data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --content-disposition https://s3.unistra.fr/camma_public/github/convlstm_tracker/data.zip\n",
    "!unzip data.zip\n",
    "\n",
    "print(\"Download completed ...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Download model weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --content-disposition https://s3.unistra.fr/camma_public/github/convlstm_tracker/ckpt.zip\n",
    "!unzip ckpt.zip\n",
    "\n",
    "print(\"Download completed ...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Some important installationns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'google.colab' in str(get_ipython()):  # colab installs tf.2.2 on default.\n",
    "    !pip uninstall -y tensorflow\n",
    "    !pip install tensorflow-gpu==1.14\n",
    "!pip install imageio\n",
    "!pip install imageio-ffmpeg\n",
    "\n",
    "print(\"Installations completed ...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import model\n",
    "import tensorflow as tf\n",
    "import os\n",
    "import numpy as np\n",
    "import cv2\n",
    "import imageio\n",
    "import sys\n",
    "from matplotlib import animation, rc, pyplot as plt\n",
    "plt.rcParams['animation.ffmpeg_path'] = '/usr/bin/ffmpeg'\n",
    "from IPython.display import HTML\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "%matplotlib inline\n",
    "\n",
    "print(\"imports success...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Variables & Device setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_height   = 480 #@param {type:\"integer\"}\n",
    "img_width    = 854 #@param {type:\"integer\"}\n",
    "img_channel  = 3   #@param {type:\"integer\"}\n",
    "num_classes  = 7   #@param {type:\"integer\"}\n",
    "offset_x     = 20  #@param {type:\"integer\"}\n",
    "offset_y     = 11  #@param {type:\"integer\"}\n",
    "data_path    = 'data/surgical_video.avi' #@param {type:\"string\"} you can modify this if you evaluate on a different video\n",
    "ckpt_path    = 'ckpt' #@param {type:\"string\"}\n",
    "\n",
    "print(\"Model and device variables set .. \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Model architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_ph  = tf.placeholder(dtype=tf.float32, shape=[None,None,3], name='inputs')\n",
    "x       = tf.expand_dims(img_ph, 0)   \n",
    "x       = tf.image.resize_bilinear(x, size=(480,854))             \n",
    "seek_ph = tf.placeholder(dtype=tf.int64, shape=[None], name='inputs')\n",
    "network = model.Model(images=x, seek=seek_ph, num_classes=num_classes)\n",
    "logits, lhmaps  = network.build_model() \n",
    "logits  = tf.cast(tf.round(tf.sigmoid(logits)), tf.int32)\n",
    "lhmaps  = lhmaps * tf.cast(logits, tf.float32)\n",
    "\n",
    "print(\"Model loaded successfully...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Saver and weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.name_scope(\"saver_and_writer\"):                  \n",
    "    saver = tf.train.Saver()  \n",
    "    state = tf.train.get_checkpoint_state(ckpt_path)\n",
    "    ckpt  = state.model_checkpoint_path\n",
    "\n",
    "print('Loading checkpoint from :',ckpt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Evaluate on video dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PREDICTIONS    = []\n",
    "CLASS_LHMAPS   = []\n",
    "reader         = imageio.get_reader(data_path)\n",
    "sess_config    = tf.ConfigProto(gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.9), allow_soft_placement = True, log_device_placement = False) \n",
    "with tf.Session(config=sess_config) as sess:   \n",
    "    sess.run([tf.local_variables_initializer(), tf.global_variables_initializer()])\n",
    "    saver.restore(sess, ckpt)\n",
    "    for seek, frame in enumerate(reader):\n",
    "        predict, lhmap = sess.run([logits, lhmaps], feed_dict={img_ph:frame, seek_ph:[seek]})\n",
    "        PREDICTIONS.append(predict)\n",
    "        CLASS_LHMAPS.append(lhmap)\n",
    "        \n",
    "print(\"Evaluation done...\")   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Some visualization helper functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get coordinates\n",
    "\n",
    "def get_center_coordinates(lhmap):\n",
    "    coord = np.where(lhmap == lhmap.max()) \n",
    "    cx    = (coord[1][0] * img_width // 107) + offset_x\n",
    "    cy    = (coord[0][0] * img_height // 60) + offset_y\n",
    "    return (cx, cy)\n",
    "\n",
    "def get_box_coordinates(lhmap):\n",
    "    coord = np.where(lhmap>0)\n",
    "    if len(coord[0])>0 and len(coord[1])>0 :\n",
    "        x0 = (coord[1].min() * img_width // 107) - offset_x\n",
    "        x1 = (coord[1].max() * img_width // 107) + offset_x\n",
    "        y0 = (coord[0].min() * img_height // 60) - offset_y\n",
    "        y1 = (coord[0].max() * img_height // 60) + offset_y\n",
    "    else:\n",
    "        x0,x1,y0,y1 = -1,-1,-1,-1\n",
    "    return (x0,y0,x1,y1)\n",
    "\n",
    "\n",
    "# Build animators\n",
    "def build_animators():\n",
    "    BUFFER_BOX_CENTER = []\n",
    "    colors    = [(255,0,0),(255,255,0),(0,0,255),(255,0,255),(255,128,0),(0,255,255),(0,255,0)] \n",
    "    radius    = 28\n",
    "    thickness = 4\n",
    "    reader    = imageio.get_reader(data_path)\n",
    "    fig       = plt.figure()\n",
    "    for k, (img, predict, lhmap) in enumerate(zip(reader, PREDICTIONS, CLASS_LHMAPS)):\n",
    "        img_overlay     = img.copy()\n",
    "        for i in range(num_classes):\n",
    "            cam         = lhmap[0,:,:,i]\n",
    "            x1,y1,x2,y2 = get_box_coordinates(cam)\n",
    "            cx,cy       = get_center_coordinates(cam)\n",
    "            color       = colors[i]\n",
    "            cv2.rectangle(img_overlay, (x1,y1), (x2,y2), color, thickness)\n",
    "            cv2.circle(img_overlay, (cx,cy), radius, color, -1)\n",
    "        cv2.circle(img_overlay, (offset_x,offset_y), radius, (0,0,0), -1)\n",
    "        BUFFER_BOX_CENTER.append([plt.imshow(img_overlay)])\n",
    "    return fig, BUFFER_BOX_CENTER\n",
    "        \n",
    "\n",
    "# Colorizer\n",
    "def cstr(s, color='black'):\n",
    "    return \"<text style=color:{}>{}</text>\".format(color, s)\n",
    "\n",
    "print(\"Model ready to track...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "#### Tracking the video\n",
    "Build animator to display the tool trajectory (_Colormap displays the legend for the tracker_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, OVERLAY = build_animators()\n",
    "\n",
    "HTML('='*20+\"> [  Tool Colormap:                                       \"\n",
    "           +cstr(\"Grasper\", \"red\") +\" | \"+cstr(\"Bipolar\", \"yellow\") +\"  |  \"+cstr(\"Hook\", \"blue\")+\"  |  \"\n",
    "           +cstr(\"Scissors\", \"violet\")+\"  |  \" +cstr(\"Clipper\", \"orange\") \n",
    "           +\"  |  \"+cstr(\"Irrigator\", \"mouve\") +\"  |  \"+cstr(\"Specimen bag  \", \"green\")+'  ] <'+'='*20 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Let's track the instruments in the video<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "anim = animation.ArtistAnimation(fig, OVERLAY, interval=160, blit=True, repeat_delay=1000)\n",
    "HTML(anim.to_html5_video())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "End"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}