tfvos/fstream_online_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `FStream` Online Training\n",
    "\n",
    "**[THIS IS WORK IN PROGRESS]**\n",
    "\n",
    "This notebook performs online training of the **flow stream parent model** on the **car-shadow** sequence, so make sure you've run the [`FStream` Offline Training](fstream_offline_training.ipynb) notebook before running this one.\n",
    "\n",
    "\n",
    "The online training of the `FStream` network is done by finetuning the parent model **on the first frame** of the video sequence. This is the only frame for which a mask is provided. It is augmented using scaling and vertical flipping. The network is trained for 500 iterations using the same training parameters as during offline training, except that deep supervision is disabled.\n",
    "\n",
    "![](img/osvos_child.png)\n",
    "\n",
    "To monitor training, run:\n",
    "```\n",
    "tensorboard --logdir E:\\repos\\tf-video-seg\\tfvos\\models\\fstream_car-shadow\n",
    "http://localhost:6006\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "fstream_online_training.ipynb\n",
    "\n",
    "FStream online trainer\n",
    "\n",
    "Written by Phil Ferriere\n",
    "\n",
    "Licensed under the MIT License (see LICENSE for details)\n",
    "\n",
    "Based on:\n",
    "  - https://github.com/scaelles/OSVOS-TensorFlow/blob/master/osvos_parent_demo.py\n",
    "    Written by Sergi Caelles (scaelles@vision.ee.ethz.ch)\n",
    "    This file is part of the OSVOS paper presented in:\n",
    "      Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixe, Daniel Cremers, Luc Van Gool\n",
    "      One-Shot Video Object Segmentation\n",
    "      CVPR 2017\n",
    "    Unknown code license\n",
    "\"\"\"\n",
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import os, sys\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "slim = tf.contrib.slim\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import model files\n",
    "import model\n",
    "import datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Model paths\n",
    "seq_name = \"car-shadow\"\n",
    "segnet_stream = 'fstream'\n",
    "parent_path = 'models/' + segnet_stream + '_parent/' + segnet_stream + '_parent.ckpt-50000'\n",
    "ckpt_name = segnet_stream + '_' + seq_name\n",
    "logs_path = 'models/' + ckpt_name\n",
    "\n",
    "# Online training parameters\n",
    "gpu_id = 0\n",
    "max_training_iters = 500\n",
    "learning_rate = 1e-8\n",
    "save_step = max_training_iters\n",
    "side_supervision = 3\n",
    "display_step = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initializing dataset...\n",
      "['JPEGImages/480p/car-shadow/00000.jpg Annotations/480p/car-shadow/00000.png']\n",
      "Loading training images and masks...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 26.30it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...done loading training images and masks.\n",
      "Performing scaling data augmentation on frames/masks...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 999.83it/s]\n",
      "100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 500.99it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... done performing scaling data augmentation on frames/masks.\n",
      "Performing flipping data augmentation on frames/masks...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████| 3/3 [00:00<00:00, 749.43it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... done performing flipping data augmentation on frames/masks.\n",
      "Converting images and masks to numpy arrays...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████| 6/6 [00:00<00:00, 667.16it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...done converting images and masks to numpy arrays.\n",
      "Computing optical flows and warping masks...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "video: 100%|█████████████████████████████████████| 6/6 [00:00<00:00, 752.68it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...done with computing optical flows and warping masks.\n",
      "Loading testing images...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████| 40/40 [00:00<00:00, 79.62it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...done loading testing images.\n",
      "...done initializing Dataset\n"
     ]
    }
   ],
   "source": [
    "# Load the DAVIS 2016 sequence\n",
    "options = datasets._DEFAULT_DAVIS16_OPTIONS\n",
    "options['use_cache'] = False\n",
    "options['data_aug'] = True\n",
    "# Set the following to wherever you have downloaded the DAVIS 2016 dataset\n",
    "dataset_root = 'E:/datasets/davis2016/' if sys.platform.startswith(\"win\") else '/media/EDrive/datasets/davis2016/'\n",
    "test_frames = sorted(os.listdir(dataset_root + 'JPEGImages/480p/' + seq_name))\n",
    "test_imgs = ['JPEGImages/480p/' + seq_name + '/' + frame for frame in test_frames]\n",
    "train_imgs = ['JPEGImages/480p/' + seq_name + '/' + '00000.jpg ' + 'Annotations/480p/' + seq_name + '/' + '00000.png']\n",
    "dataset = datasets.davis16(train_imgs, test_imgs, dataset_root, options)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Configuration:\n",
      "  in_memory            True\n",
      "  data_aug             True\n",
      "  use_cache            False\n",
      "  use_optical_flow     True\n",
      "  use_warped_masks     True\n",
      "  use_bboxes           True\n",
      "  optical_flow_mgr     pyflow\n"
     ]
    }
   ],
   "source": [
    "# Display dataset configuration\n",
    "dataset.print_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Online Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network Layers:\n",
      "   name = fstream/conv1/conv1_1/Relu:0, shape = (1, ?, ?, 64)\n",
      "   name = fstream/conv1/conv1_2/Relu:0, shape = (1, ?, ?, 64)\n",
      "   name = fstream/pool1/MaxPool:0, shape = (1, ?, ?, 64)\n",
      "   name = fstream/conv2/conv2_1/Relu:0, shape = (1, ?, ?, 128)\n",
      "   name = fstream/conv2/conv2_2/Relu:0, shape = (1, ?, ?, 128)\n",
      "   name = fstream/pool2/MaxPool:0, shape = (1, ?, ?, 128)\n",
      "   name = fstream/conv3/conv3_1/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/conv3/conv3_2/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/conv3/conv3_3/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/pool3/MaxPool:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/conv4/conv4_1/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv4/conv4_2/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv4/conv4_3/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/pool4/MaxPool:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv5/conv5_1/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv5/conv5_2/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv5/conv5_3/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv2_2_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/conv3_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/conv4_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/conv5_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-dsn_2/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_3/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_4/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_5/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_2-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_3-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape_1:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_4-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape_2:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_5-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape_3:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-multi2-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_4:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-multi3-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_5:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-multi4-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_6:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-multi5-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_7:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/upscore-fuse/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "Network Parameters:\n",
      "   name = fstream/conv1/conv1_1/weights:0, shape = (3, 3, 3, 64)\n",
      "   name = fstream/conv1/conv1_1/biases:0, shape = (64,)\n",
      "   name = fstream/conv1/conv1_2/weights:0, shape = (3, 3, 64, 64)\n",
      "   name = fstream/conv1/conv1_2/biases:0, shape = (64,)\n",
      "   name = fstream/conv2/conv2_1/weights:0, shape = (3, 3, 64, 128)\n",
      "   name = fstream/conv2/conv2_1/biases:0, shape = (128,)\n",
      "   name = fstream/conv2/conv2_2/weights:0, shape = (3, 3, 128, 128)\n",
      "   name = fstream/conv2/conv2_2/biases:0, shape = (128,)\n",
      "   name = fstream/conv3/conv3_1/weights:0, shape = (3, 3, 128, 256)\n",
      "   name = fstream/conv3/conv3_1/biases:0, shape = (256,)\n",
      "   name = fstream/conv3/conv3_2/weights:0, shape = (3, 3, 256, 256)\n",
      "   name = fstream/conv3/conv3_2/biases:0, shape = (256,)\n",
      "   name = fstream/conv3/conv3_3/weights:0, shape = (3, 3, 256, 256)\n",
      "   name = fstream/conv3/conv3_3/biases:0, shape = (256,)\n",
      "   name = fstream/conv4/conv4_1/weights:0, shape = (3, 3, 256, 512)\n",
      "   name = fstream/conv4/conv4_1/biases:0, shape = (512,)\n",
      "   name = fstream/conv4/conv4_2/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv4/conv4_2/biases:0, shape = (512,)\n",
      "   name = fstream/conv4/conv4_3/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv4/conv4_3/biases:0, shape = (512,)\n",
      "   name = fstream/conv5/conv5_1/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv5/conv5_1/biases:0, shape = (512,)\n",
      "   name = fstream/conv5/conv5_2/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv5/conv5_2/biases:0, shape = (512,)\n",
      "   name = fstream/conv5/conv5_3/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv5/conv5_3/biases:0, shape = (512,)\n",
      "   name = fstream/conv2_2_16/weights:0, shape = (3, 3, 128, 16)\n",
      "   name = fstream/conv2_2_16/biases:0, shape = (16,)\n",
      "   name = fstream/conv3_3_16/weights:0, shape = (3, 3, 256, 16)\n",
      "   name = fstream/conv3_3_16/biases:0, shape = (16,)\n",
      "   name = fstream/conv4_3_16/weights:0, shape = (3, 3, 512, 16)\n",
      "   name = fstream/conv4_3_16/biases:0, shape = (16,)\n",
      "   name = fstream/conv5_3_16/weights:0, shape = (3, 3, 512, 16)\n",
      "   name = fstream/conv5_3_16/biases:0, shape = (16,)\n",
      "   name = fstream/score-dsn_2/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_2/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_3/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_3/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_4/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_4/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_5/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_5/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_2-up/weights:0, shape = (4, 4, 1, 1)\n",
      "   name = fstream/score-dsn_3-up/weights:0, shape = (8, 8, 1, 1)\n",
      "   name = fstream/score-dsn_4-up/weights:0, shape = (16, 16, 1, 1)\n",
      "   name = fstream/score-dsn_5-up/weights:0, shape = (32, 32, 1, 1)\n",
      "   name = fstream/score-multi2-up/weights:0, shape = (4, 4, 16, 16)\n",
      "   name = fstream/score-multi3-up/weights:0, shape = (8, 8, 16, 16)\n",
      "   name = fstream/score-multi4-up/weights:0, shape = (16, 16, 16, 16)\n",
      "   name = fstream/score-multi5-up/weights:0, shape = (32, 32, 16, 16)\n",
      "   name = fstream/upscore-fuse/weights:0, shape = (1, 1, 64, 1)\n",
      "   name = fstream/upscore-fuse/biases:0, shape = (1,)\n",
      "Init variable\n",
      "Initializing from specified pre-trained model...\n",
      "INFO:tensorflow:Restoring parameters from models/fstream_parent/fstream_parent.ckpt-50000\n",
      "Weights initialized\n",
      "Start training\n",
      "2018-02-02 09:18:13.531993 Iter 10: Training Loss = 569.6891\n",
      "2018-02-02 09:18:15.746674 Iter 20: Training Loss = 339.9303\n",
      "2018-02-02 09:18:17.874067 Iter 30: Training Loss = 411.8366\n",
      "2018-02-02 09:18:20.096919 Iter 40: Training Loss = 173.0713\n",
      "2018-02-02 09:18:22.219696 Iter 50: Training Loss = 168.0783\n",
      "2018-02-02 09:18:24.546178 Iter 60: Training Loss = 352.7663\n",
      "2018-02-02 09:18:26.787858 Iter 70: Training Loss = 266.9531\n",
      "2018-02-02 09:18:29.014979 Iter 80: Training Loss = 327.4535\n",
      "2018-02-02 09:18:31.250700 Iter 90: Training Loss = 140.8070\n",
      "2018-02-02 09:18:33.476138 Iter 100: Training Loss = 249.2887\n",
      "2018-02-02 09:18:35.473987 Iter 110: Training Loss = 141.6752\n",
      "2018-02-02 09:18:37.909898 Iter 120: Training Loss = 132.8425\n",
      "2018-02-02 09:18:39.920605 Iter 130: Training Loss = 135.7357\n",
      "2018-02-02 09:18:42.557159 Iter 140: Training Loss = 273.6926\n",
      "2018-02-02 09:18:44.563315 Iter 150: Training Loss = 125.5738\n",
      "2018-02-02 09:18:46.676816 Iter 160: Training Loss = 221.7289\n",
      "2018-02-02 09:18:48.911636 Iter 170: Training Loss = 215.1107\n",
      "2018-02-02 09:18:51.250475 Iter 180: Training Loss = 214.7981\n",
      "2018-02-02 09:18:53.483244 Iter 190: Training Loss = 256.1548\n",
      "2018-02-02 09:18:55.611791 Iter 200: Training Loss = 116.1278\n",
      "2018-02-02 09:18:57.952465 Iter 210: Training Loss = 206.1201\n",
      "2018-02-02 09:19:00.074813 Iter 220: Training Loss = 113.1943\n",
      "2018-02-02 09:19:02.209976 Iter 230: Training Loss = 111.5827\n",
      "2018-02-02 09:19:04.656724 Iter 240: Training Loss = 228.0945\n",
      "2018-02-02 09:19:06.770126 Iter 250: Training Loss = 109.2165\n",
      "2018-02-02 09:19:09.091927 Iter 260: Training Loss = 112.8695\n",
      "2018-02-02 09:19:11.320634 Iter 270: Training Loss = 225.8829\n",
      "2018-02-02 09:19:13.642433 Iter 280: Training Loss = 183.6381\n",
      "2018-02-02 09:19:15.643466 Iter 290: Training Loss = 185.2786\n",
      "2018-02-02 09:19:17.978461 Iter 300: Training Loss = 179.0473\n",
      "2018-02-02 09:19:20.222647 Iter 310: Training Loss = 103.0942\n",
      "2018-02-02 09:19:22.650020 Iter 320: Training Loss = 210.6018\n",
      "2018-02-02 09:19:24.850377 Iter 330: Training Loss = 177.0227\n",
      "2018-02-02 09:19:27.161622 Iter 340: Training Loss = 104.5873\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-02-02 09:19:29.456114 Iter 350: Training Loss = 172.9282\n",
      "2018-02-02 09:19:31.696558 Iter 360: Training Loss = 98.5829\n",
      "2018-02-02 09:19:33.947753 Iter 370: Training Loss = 191.5003\n",
      "2018-02-02 09:19:36.088972 Iter 380: Training Loss = 97.0496\n",
      "2018-02-02 09:19:38.448853 Iter 390: Training Loss = 96.4809\n",
      "2018-02-02 09:19:40.683812 Iter 400: Training Loss = 99.6578\n",
      "2018-02-02 09:19:43.013047 Iter 410: Training Loss = 182.6668\n",
      "2018-02-02 09:19:45.130933 Iter 420: Training Loss = 180.7202\n",
      "2018-02-02 09:19:47.357807 Iter 430: Training Loss = 97.6054\n",
      "2018-02-02 09:19:49.689024 Iter 440: Training Loss = 158.0362\n",
      "2018-02-02 09:19:51.795934 Iter 450: Training Loss = 175.0791\n",
      "2018-02-02 09:19:54.008078 Iter 460: Training Loss = 173.5029\n",
      "2018-02-02 09:19:56.260132 Iter 470: Training Loss = 153.9403\n",
      "2018-02-02 09:19:58.494076 Iter 480: Training Loss = 149.3242\n",
      "2018-02-02 09:20:00.610344 Iter 490: Training Loss = 93.8205\n",
      "2018-02-02 09:20:02.708245 Iter 500: Training Loss = 89.5559\n",
      "INFO:tensorflow:models/fstream_car-shadow\\fstream_car-shadow.ckpt-500 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models/fstream_car-shadow\\fstream_car-shadow.ckpt-500\n",
      "Finished training.\n"
     ]
    }
   ],
   "source": [
    "# Finetune this branch of the binary segmentation network\n",
    "with tf.Graph().as_default():\n",
    "    with tf.device('/gpu:' + str(gpu_id)):\n",
    "        global_step = tf.Variable(0, name='global_step', trainable=False)\n",
    "        model.train_finetune(dataset, parent_path, side_supervision, learning_rate, logs_path, max_training_iters,\n",
    "                             save_step, display_step, global_step, segnet_stream, iter_mean_grad=1, ckpt_name=ckpt_name)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training losses & learning rate\n",
    "You should get training curves similar to the following:\n",
    "![](img/fstream_car-shadow_main_loss.png)\n",
    "![](img/fstream_car-shadow_total_loss.png)\n",
    "![](img/fstream_car-shadow_learning_rate.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Result path (if you want to check how well this branch is doing on its own)\n",
    "result_path = dataset_folder + 'Results/Segmentations/480p/' + ckpt_name\n",
    "\n",
    "# Test this branch of the network\n",
    "with tf.Graph().as_default():\n",
    "    with tf.device('/gpu:' + str(gpu_id)):\n",
    "        ckpt_path = logs_path + '/' + ckpt_name + '.ckpt-' + str(max_training_iters))\n",
    "        model.test(dataset, ckpt_path, result_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Log output should be similar to the following:\n",
    "```\n",
    "INFO:tensorflow:Restoring parameters from models\\car-shadow_new\\car-shadow_new.ckpt-500\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00000.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00001.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00002.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00003.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00004.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00005.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00006.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00007.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00008.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00009.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00010.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00011.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00012.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00013.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00014.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00015.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00016.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00017.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00018.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00019.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00020.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00021.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00022.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00023.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00024.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00025.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00026.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00027.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00028.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00029.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00030.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00031.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00032.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00033.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00034.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00035.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00036.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00037.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00038.png\n",
    "Saving E:/datasets/davis2016\\Results\\Segmentations\\480p\\OSVOS\\car-shadow_new\\00039.png\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visual evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the original images and their predicted masks to get an idea of how this branch of the network is doing. Note that the first mask is displayed in red overlay, as it is given to us. The predicted masks are displayed using a green overlay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load results\n",
    "frames = []\n",
    "predicted_masks=[]\n",
    "for test_frame in test_frames:\n",
    "    frame_num = test_frame.split('.')[0]\n",
    "    frame = np.array(Image.open(dataset_root + 'JPEGImages/480p/' + seq_name + '/' + test_frame))\n",
    "    predicted_mask = np.array(Image.open(result_path + frame_num +'.png'))\n",
    "    frames.append(frame)\n",
    "    predicted_masks.append(predicted_mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Overlay the masks on top of the frames\n",
    "frames_with_predictions = visualize.overlay_frames_with_predictions(frames, predicted_masks')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display individual frames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "visualize.display_images(frames_with_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display results as a video clip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set path to video clips\n",
    "video_clip_folder = dataset_root + 'clips/'\n",
    "video_clip = video_clip_folder + ckpt_name + '.mp4'\n",
    "\n",
    "# Combine images in a video clip\n",
    "visualize.make_clip(video_clip, frames_with_predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Display video\n",
    "visualize.show_clip(video_clip)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}