TempConv.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Temporal Convolution\n",
    "\n",
    "We introduce a new neural network architecture for sentence classification using temporal convolution. This is a more powerful model than the simple bag of words model introduced before.\n",
    "\n",
    "We set up our sample data as before, but also include padding at the beginning of the sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "V = {[\"*padding*\"]=1, [\"I\"]= 2, [\"am\"]= 3, [\"a\"]= 4, [\"he\"]=5,\n",
    "     [\"it\"]=6, [\"dog\"]=7, [\"is\"]=8, [\"she\"]=9} \n",
    "nV = 10\n",
    "\n",
    "function make_data(sent, n, start_pad) \n",
    "    out = {}\n",
    "    for i = 1, start_pad do\n",
    "        v = V[\"*padding*\"]\n",
    "        table.insert(out, v)\n",
    "    end\n",
    "    for i = 1, n - start_pad do\n",
    "        if i <= #sent then \n",
    "            v = V[sent[i]]\n",
    "        else\n",
    "            v = V[\"*padding*\"]\n",
    "        end\n",
    "        table.insert(out, v)\n",
    "    end\n",
    "    return out\n",
    "end\n",
    "\n",
    "indata = {}\n",
    "outdata = {}\n",
    "table.insert(indata, make_data({\"I\", \"am\", \"a\", \"dog\"}, 10, 3))\n",
    "table.insert(outdata, 1)\n",
    "table.insert(indata, make_data({\"he\", \"is\", \"a\", \"dog\"}, 10, 3))\n",
    "table.insert(outdata, 2)\n",
    "table.insert(indata, make_data({\"she\", \"is\", \"a\", \"dog\"}, 10, 3))\n",
    "table.insert(outdata, 2)\n",
    "table.insert(indata, make_data({\"it\", \"is\", \"a\", \"dog\"}, 10, 3))\n",
    "table.insert(outdata, 2)\n",
    "\n",
    "\n",
    "X = torch.DoubleTensor(indata)\n",
    "y = torch.DoubleTensor(outdata)\n",
    "nY = 2 -- Two classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have $J$ data points, and\n",
    "\n",
    "* $\\mathbf{X} \\in \\mathcal{V}^{J\\times n}$ is our input data\n",
    "* $\\mathbf{y} \\in \\mathcal{Y}^{J}$ is out output data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Model\n",
    "\n",
    "The key step in our architecture is a convolution. Our input is a sequence of vectors $V_i \\in \\mathbb{R}^{d}$ for $i = 1, \\ldots, n$, i.e. the $d$-dimensional word embeddings of our sentence. A convolution then involves:\n",
    "\n",
    "* a _filter_ $\\mathbf{w} \\in \\mathbb{R}^{hd}$, where $h$ is the filter size,\n",
    "* $[V_{j}, V_{j+1}, \\ldots, V_{j+h-1}] \\in \\mathbb{R}^{hd}$, the concatenation of vectors $j$ through $j+h-1$\n",
    "\n",
    "We then take a dot product of $\\mathbf{w}$ with our concatenated word vectors $j$ through $j+h-1$ to get a single value. Additionally, we include a bias term $b \\in \\mathbb{R}$ and a nonlinear activation function $f$ such as $\\tanh(x)$ or the rectilinear unit $f(x) = \\max(0, x)$, so that our value is\n",
    "\n",
    "$$c_j = f(w \\cdot [V_j, V_{j+1}, \\ldots, V_{j+h-1}] + b)$$\n",
    "\n",
    "If we do this for every $j$, we get a feature map $\\mathbf{c} \\in \\mathcal{R}^{n-h+1}$ with entries $c_j$.\n",
    "\n",
    "With multiple filters $\\mathbf{w_1}, \\ldots, \\mathbf{w_{d'}}$, we can form a matrix of feature maps $[\\mathbf{c_1}; \\mathbf{c_2}; \\ldots; \\mathbf{c_{n-h+1}}] \\in \\mathbb{R}^{(n-h+1) \\times d'}$. In our approach, we do a max-over-time pooling for each feature map to get one feature per feature map:\n",
    "\n",
    "$$\\widehat{\\mathbf{c_i}} = \\max_j (\\mathbf{c_i})_j$$\n",
    "\n",
    "Then we have a vector of features for our sentence (one per filter): $[\\widehat{\\mathbf{c_1}}, \\widehat{\\mathbf{c_2}}, \\ldots, \\widehat{\\mathbf{c_{d'}}}] \\in \\mathbb{R}^{d'}$\n",
    "\n",
    "Now let's build this in torch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nn = require \"nn\"\n",
    "d = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start with the word embeddings layer, as usual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = nn.Sequential()\n",
    "matrixV = nn.LookupTable(nV, d)\n",
    "model:add(matrixV)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then add the convolution. Torch has convenient built-in methods for the convolution described above, such as `nn.TemporalConvolution`. This takes the dimensionality of the previous layer ($d$), the number of filters we use ($nd$, or $d'$ in the above), and the filter size ($h$). We can also do a max pooling with `nn.Max`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nd = 10\n",
    "h = 3\n",
    "conv = nn.Sequential()\n",
    "conv:add(nn.TemporalConvolution(d, nd, h))\n",
    "conv:add(nn.ReLU())\n",
    "conv:add(nn.Max(2))\n",
    "\n",
    "model:add(conv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The output of the above gives us our feature map $[\\widehat{\\mathbf{c_1}}, \\widehat{\\mathbf{c_2}}, \\ldots, \\widehat{\\mathbf{c_{d'}}}]$. Finally we add a logistic regression layer for predicting the sentiment from this vector of features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "logistic = nn.Sequential()\n",
    "\n",
    "logistic:add(nn.Linear(nd, nY))\n",
    "logistic:add(nn.LogSoftMax())\n",
    "\n",
    "model:add(logistic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test our model on the input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.6319 -0.7584\n",
       "-0.5740 -0.8285\n",
       "-0.6088 -0.7853\n",
       "-0.7404 -0.6481\n",
       "[torch.DoubleTensor of size 4x2]\n",
       "\n"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model:forward(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, we get (log) prediction probabilities for 2 classes for each input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Include a negative-log-likelihood criterion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "criterion = nn.ClassNLLCriterion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also implement these modules on GPUs. Specifically, we include the `cudnn` package, which has some GPU optimized versions of some of the above modules. One thing that requires modification is the convolution step - `cudnn` has no built in `TemporalConvolution` module, so we have to adapt the `SpatialConvolution` by reshaping our feature map matrix.\n",
    "\n",
    "Here's the full implementation on `cudnn` (using batch mode):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "require 'cutorch'\n",
    "require 'cudnn'\n",
    "\n",
    "cudnn_model = nn.Sequential()\n",
    "matrixV = nn.LookupTable(nV, d)\n",
    "model:add(matrixV)\n",
    "\n",
    "nd = 10\n",
    "h = 3\n",
    "S = 10\n",
    "conv = nn.Sequential()\n",
    "conv:add(nn.Reshape(1, S, d, false))\n",
    "conv:add(cudnn.SpatialConvolution(1, nd, d, h))\n",
    "conv:add(nn.Reshape(nd, S-h+1, false))\n",
    "conv:add(cudnn.ReLU())\n",
    "conv:add(nn.Max(3))\n",
    "\n",
    "cudnn_model:add(conv)\n",
    "\n",
    "logistic = nn.Sequential()\n",
    "\n",
    "logistic:add(nn.Linear(nd, nY))\n",
    "logistic:add(cudnn.LogSoftMax())\n",
    "\n",
    "cudnn_model:add(logistic)\n",
    "\n",
    "criterion = nn.ClassNLLCriterion()\n",
    "\n",
    "-- Move to GPU\n",
    "cudnn_model:cuda()\n",
    "criterion:cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We perform training with `adadelta`. In each epoch, we create a closure that returns the gradient updates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Epoch:\t1\t1.0637046591443\t\n",
       "Epoch:\t2\t0.92574699271742\t\n",
       "Epoch:\t3\t0.80757029787584\t\n"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Epoch:\t4\t0.70017095481893\t\n",
       "Epoch:\t5\t0.62009300393375\t\n",
       "Epoch:\t6\t0.53988474604789\t\n",
       "Epoch:\t7\t0.48039989542535\t\n"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Epoch:\t8\t0.42860197930536\t\n",
       "Epoch:\t9\t0.38423555419475\t\n",
       "Epoch:\t10\t0.34843210201123\t\n",
       "Epoch:\t11\t0.31601698906774\t\n",
       "Epoch:\t12\t0.29034451988978\t\n"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Epoch:\t13\t0.26565687644075\t\n",
       "Epoch:\t14\t0.24537762059148\t\n",
       "Epoch:\t15\t0.22573977925808\t\n",
       "Epoch:\t16\t0.20809219151101\t\n",
       "Epoch:\t17\t0.19274937869357\t\n"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Epoch:\t18\t0.17790250866323\t\n",
       "Epoch:\t19\t0.16478886826202\t\n",
       "Epoch:\t20\t0.15308249381824\t\n"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "require 'optim'\n",
    "model:reset()\n",
    "model:training()\n",
    "\n",
    "params, grads = model:getParameters()\n",
    "\n",
    "config = { rho = 0.95, eps = 1e-6 }\n",
    "state = {}\n",
    "for epoch = 1, 20 do\n",
    "    func = function(x)\n",
    "        if x ~= params then\n",
    "            params:copy(x)\n",
    "        end\n",
    "        grads:zero()\n",
    "        \n",
    "        out = model:forward(X)\n",
    "        err = criterion:forward(out, y)\n",
    "        \n",
    "        dout = criterion:backward(out, y)\n",
    "        model:backward(X, dout)\n",
    "        \n",
    "        return err, grads\n",
    "    end\n",
    "    \n",
    "    optim.adadelta(func, params, config, state)\n",
    "    print(\"Epoch:\", epoch, err)\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Note that training error goes down after every epoch, as expected."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "iTorch",
   "language": "lua",
   "name": "itorch"
  },
  "language_info": {
   "name": "lua",
   "version": "20100"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}