PythonNNExampleFromSirajology.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Annotations for the Sirajology Python NN Example\n",
    "\n",
    "This code comes from a demo NN program from the YouTube video https://youtu.be/h3l4qz76JhQ. The program creates an neural network that simulates the [XOR (exclusive OR)](https://en.wikipedia.org/wiki/XOR_gate) function with two inputs and one output. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np  # Note: there is a typo on this line in the video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following is a function definition of the sigmoid function, which is the type of non-linearity chosen for this neural net. It is not the only type of non-linearity that can be chosen, but is has nice analytical features and is easy to teach with. In practice, large-scale deep learning systems use piecewise-linear functions because they are much less expensive to evaluate. \n",
    "\n",
    "The implementation of this function does double duty. If the deriv=True flag is passed in, the function instead calculates the derivative of the function, which is used in the error backpropogation step. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def nonlin(x, deriv=False):  # Note: there is a typo on this line in the video\n",
    "    if(deriv==True):\n",
    "        return (x*(1-x))\n",
    "    \n",
    "    return 1/(1+np.exp(-x))  # Note: there is a typo on this line in the video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code creates the input matrix. Although not mentioned in the video, the third column is for accommodating the bias term and is not part of the input. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#input data\n",
    "X = np.array([[0,0,1],  # Note: there is a typo on this line in the video\n",
    "            [0,1,1],\n",
    "            [1,0,1],\n",
    "            [1,1,1]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of the exclusive OR function follows. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#output data\n",
    "y = np.array([[0],\n",
    "             [1],\n",
    "             [1],\n",
    "             [0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The seed for the random generator is set so that it will return the same random numbers each time, which is sometimes useful for debugging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we intialize the weights to random values. syn0 are the weights between the input layer and the hidden layer.  It is a 3x4 matrix because there are two input weights plus a bias term (=3) and four nodes in the hidden layer (=4). syn1 are the weights between the hidden layer and the output layer. It is a 4x1 matrix because there are 4 nodes in the hidden layer and one output. Note that there is no bias term feeding the output layer in this example. The weights are initially generated randomly because optimization tends not to work well when all the weights start at the same value. Note that neither of the neural networks shown in the video describe the example. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#synapses\n",
    "syn0 = 2*np.random.random((3,4)) - 1  # 3x4 matrix of weights ((2 inputs + 1 bias) x 4 nodes in the hidden layer)\n",
    "syn1 = 2*np.random.random((4,1)) - 1  # 4x1 matrix of weights. (4 nodes x 1 output) - no bias term in the hidden layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the main training loop. The output shows the evolution of the error between the model and desired. The error steadily decreases. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error: 0.496410031903\n",
      "Error: 0.00858452565325\n",
      "Error: 0.00578945986251\n",
      "Error: 0.00462917677677\n",
      "Error: 0.00395876528027\n",
      "Error: 0.00351012256786\n",
      "Output after training\n",
      "[[ 0.00260572]\n",
      " [ 0.99672209]\n",
      " [ 0.99701711]\n",
      " [ 0.00386759]]\n"
     ]
    }
   ],
   "source": [
    "#training step\n",
    "# Python2 Note: In the follow command, you may improve \n",
    "#   performance by replacing 'range' with 'xrange'. \n",
    "for j in range(60000):  \n",
    "    \n",
    "    # Calculate forward through the network.\n",
    "    l0 = X\n",
    "    l1 = nonlin(np.dot(l0, syn0))\n",
    "    l2 = nonlin(np.dot(l1, syn1))\n",
    "    \n",
    "    # Back propagation of errors using the chain rule. \n",
    "    l2_error = y - l2\n",
    "    if(j % 10000) == 0:   # Only print the error every 10000 steps, to save time and limit the amount of output. \n",
    "        print(\"Error: \" + str(np.mean(np.abs(l2_error))))\n",
    "        \n",
    "    l2_delta = l2_error*nonlin(l2, deriv=True)\n",
    "    \n",
    "    l1_error = l2_delta.dot(syn1.T)\n",
    "    \n",
    "    l1_delta = l1_error * nonlin(l1,deriv=True)\n",
    "    \n",
    "    #update weights (no learning rate term)\n",
    "    syn1 += l1.T.dot(l2_delta)\n",
    "    syn0 += l0.T.dot(l1_delta)\n",
    "    \n",
    "print(\"Output after training\")\n",
    "print(l2)\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how the final output closely approximates the true output [0, 1, 1, 0]. If you increase the number of interations in the training loop (currently 60000), the final output will be even closer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "#The following line is for embedding the YouTube video \n",
       "#   in this Jupyter Notebook. You may remove it without peril. \n",
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/h3l4qz76JhQ\" frameborder=\"0\" allowfullscreen></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "#The following line is for embedding the YouTube video \n",
    "#   in this Jupyter Notebook. You may remove it without peril. \n",
    "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/h3l4qz76JhQ\" frameborder=\"0\" allowfullscreen></iframe>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}