hellotensorflow_demo_end.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Intro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Names and execution in Python and TensorFlow\n",
    "* The simplest TensorFlow neuron\n",
    "* Making the neuron learn\n",
    "* Training diagnostics in TensorBoard\n",
    "* Flowing onward\n",
    "\n",
    "*courtesy of [Aaron Schumacher](https://www.oreilly.com/learning/hello-tensorflow)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Names and execution in Python and TensorFlow\n",
    "#### A name in Python is separated from the object. A name points at the object and not the other way around."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"img/an_object_has_no_name.jpg\" alt=\"An object has no name.\" style=\"width: 400px;\"/>\n",
    "\n",
    "\n",
    "\n",
    "*Image courtesy of [Hadley Wickham](https://twitter.com/hadleywickham/status/732288980549390336).*\n",
    "\n",
    "Let's see a code example of foo pointing to an object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "foo = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bar = foo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "foo == bar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "foo is bar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36658424"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id(foo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36658424"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id(bar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now see that id(foo) and id(bar) are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Nesting lists is one way to represent a graph structure like a TensorFlow computation graph.\n",
    "foo.append(bar) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[...]]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# and now foo is inside itself...\n",
    "foo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python is managing foo and bar and both names point at the same thing.\n",
    "\n",
    "<img src=\"img/foo_loop.png\" alt=\"foo points to itself\" style=\"width: 400px;\"/>\n",
    "\n",
    "*Image made with [draw.io](https://draw.io/).*\n",
    "\n",
    "\n",
    "Meaning that...\n",
    "### Tensorflow has also a system to keep track of things somewhere and giving access via names.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's start by the simplest TensorFlow neuron\n",
    "*Will just have an input and a weight and we will be able to multiply that*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Inport tensorflow, create the graph and define the values\n",
    "import tensorflow as tf\n",
    "sess = tf.Session()\n",
    "graph = tf.get_default_graph()\n",
    "input_value = tf.constant(1.0)\n",
    "weight = tf.Variable(0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We define our operation in the graph\n",
    "output_value = weight * input_value\n",
    "op = graph.get_operations()[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is the last operation in the graph? Multiplication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'mul'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "op.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.80000001"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generates an operation which will initialize all our variables a finally run it\n",
    "\n",
    "sess.run(tf.global_variables_initializer())\n",
    "sess.run(output_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Recall that's 0.8 x 1.0 with 32-bit floats, and 32-bit floats have a hard time with 0.8; 0.80000001 is as close as they can get.*\n",
    "\n",
    "This is the neuron's \"inference\" or \"forward pass\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neat! let's look at our graph in TensorBoard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reset previous stuff\n",
    "tf.reset_default_graph()\n",
    "sess = tf.Session()\n",
    "!rm -rf log_simple_graph\n",
    "!rm -rf log_simple_stat\n",
    "\n",
    "x = tf.constant(1.0, name='input')\n",
    "w = tf.Variable(0.8, name='weight')\n",
    "y = tf.multiply(w, x, name='output')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FileWriter needs an output directory.\n",
    "summary_writer = tf.summary.FileWriter('log_simple_graph', sess.graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorBoard 0.1.5 at http://ricardo-T440s:6006 (Press CTRL+C to quit) ^C\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!tensorboard --logdir=log_simple_graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To continue with the notebook, interrupt the kernel by using the square \"stop\" button or by typing `esc`, `i`, `i`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making the neuron learn with gradient descent\n",
    "Now the system takes the input one and returns 0.8, which is worg. We need a way to measure how wrong the system is. We'll call that measure the “loss” and give our system the goal of minimizing the loss. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_ = tf.constant(0.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We make the loss the **square of the difference** between the **current output** and the **desired output**, so it doesn't get to a negative values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss = (y - y_)**2\n",
    "\n",
    "# Optimizer with a learning rate of 0.025.\n",
    "optim = tf.train.GradientDescentOptimizer(learning_rate=0.025)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/plot.png\" alt=\"plot of loss and derivative.\" style=\"width: 400px;\"/>\n",
    "<center>Plot of Loss and Derivative</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.6"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grads_and_vars = optim.compute_gradients(loss)\n",
    "sess.run(tf.global_variables_initializer())\n",
    "sess.run(grads_and_vars[0][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why is the value of the gradient 1.6? Our loss is error squared, and the derivative of that is two times the error. Currently the system says 0.8 instead of 0, so the error is 0.8, and two times 0.8 is 1.6. It's working!\n",
    "\n",
    "We have `y = 0.8` and `y_ = 0`.\n",
    "\n",
    "So the loss is `(0.8 - 0)^2 = 0.64`.\n",
    "\n",
    "And the derivative of the loss is `2*(0.8 - 0)`.\n",
    "\n",
    "#### Let's apply the gradient, finishing the backpropagation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.75999999"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sess.run(optim.apply_gradients(grads_and_vars))\n",
    "sess.run(w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The weight decreased by 0.04 because the optimizer subtracted the gradient times the learning rate, 1.6 * 0.025, pushing the weight in the right direction.\n",
    "\n",
    "`0.8 - 0.025*1.6 = 0.8 - 0.04 = 0.76`\n",
    "\n",
    "Instead of hand-holding the optimizer like this, we can make one operation that calculates and applies the gradients: the `train_step`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0044996012"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_step = tf.train.GradientDescentOptimizer(0.025).minimize(loss)\n",
    "\n",
    "for i in range(100):\n",
    "    sess.run(train_step)\n",
    "\n",
    "sess.run(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the training step many times, the weight and the output value are now very close to zero. \n",
    "\n",
    "The neuron has learned!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Last Step: The all example on Tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# reset everything \n",
    "tf.reset_default_graph()\n",
    "!rm -rf log_simple_stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The script below is designed to stand alone, and will also work here with the default graph reset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "x = tf.constant(1.0, name='input')\n",
    "w = tf.Variable(0.8, name='weight')\n",
    "y = tf.multiply(w, x, name='output')\n",
    "y_ = tf.constant(0.0, name='correct_value')\n",
    "loss = tf.pow(y - y_, 2, name='loss')\n",
    "train_step = tf.train.GradientDescentOptimizer(0.025).minimize(loss)\n",
    "\n",
    "for value in [x, w, y, y_, loss]:\n",
    "    tf.summary.scalar(value.op.name, value)\n",
    "\n",
    "summaries = tf.summary.merge_all()\n",
    "\n",
    "session = tf.Session()\n",
    "summary_writer = tf.summary.FileWriter('log_simple_stats', session.graph)\n",
    "\n",
    "session.run(tf.global_variables_initializer())\n",
    "for i in range(100):\n",
    "    summary_writer.add_summary(session.run(summaries), i)\n",
    "    session.run(train_step)\n",
    "\n",
    "summary_writer.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorBoard 0.1.5 at http://ricardo-T440s:6006 (Press CTRL+C to quit) ^C\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!tensorboard --logdir=log_simple_stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After startup, go to http://localhost:6006/#events to see the interface.\n",
    "\n",
    "To continue with the notebook, interrupt the kernel by using the square \"stop\" button or by typing `esc`, `i`, `i`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Next Demo: http://localhost:8888/notebooks/final/ImageClassifier_GPU.ipynb \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}