8-translation-transformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is adapted from [this one](https://github.com/fastai/fastai_docs/blob/master/dev_course/dl2/translation_transformer.ipynb) created by Sylvain Gugger.\n",
    "\n",
    "See also [The Annotated Transformer](http://nlp.seas.harvard.edu/2018/04/03/attention.html) from Harvard NLP."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Attention and the Transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nvidia AI researcher [Chip Huyen](https://huyenchip.com/) wrote a great post [Top 8 trends from ICLR 2019](https://huyenchip.com/2019/05/12/top-8-trends-from-iclr-2019.html) in which one of the trends is that *RNN is losing its luster with researchers*.\n",
    "\n",
    "There's good reason for this, RNNs can be a pain: parallelization can be tricky and they can be difficult to debug. Since language is recursive, it seemed like RNNs were a good conceptual fit with NLP, but recently methods using *attention* have been achieving state of the art results on NLP.\n",
    "\n",
    "This is still an area of very active research, for instance, a recent paper [Pay Less Attention with Lightweight and Dynamic Convolutions](https://arxiv.org/abs/1901.10430) showed that convolutions can beat attention on some tasks, including English to German translation.  More research is needed on the various strenghts of RNNs, CNNs, and transformers/attention, and perhaps on approaches to combine the best of each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/cc.en.300.bin'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/cc.fr.300.bin'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/data_save.pkl'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/giga-fren.release2.fixed.en'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/giga-fren.release2.fixed.fr'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/models'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/questions_easy.csv')]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = Config().data_path()/'giga-fren'\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We reuse the same functions as in the translation notebook to load our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def seq2seq_collate(samples, pad_idx=1, pad_first=True, backwards=False):\n",
    "    \"Function that collect samples and adds padding. Flips token order if needed\"\n",
    "    samples = to_data(samples)\n",
    "    max_len_x,max_len_y = max([len(s[0]) for s in samples]),max([len(s[1]) for s in samples])\n",
    "    res_x = torch.zeros(len(samples), max_len_x).long() + pad_idx\n",
    "    res_y = torch.zeros(len(samples), max_len_y).long() + pad_idx\n",
    "    if backwards: pad_first = not pad_first\n",
    "    for i,s in enumerate(samples):\n",
    "        if pad_first: \n",
    "            res_x[i,-len(s[0]):],res_y[i,-len(s[1]):] = LongTensor(s[0]),LongTensor(s[1])\n",
    "        else:         \n",
    "            res_x[i, :len(s[0])],res_y[i, :len(s[1])] = LongTensor(s[0]),LongTensor(s[1])\n",
    "    if backwards: res_x,res_y = res_x.flip(1),res_y.flip(1)\n",
    "    return res_x, res_y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqDataBunch(TextDataBunch):\n",
    "    \"Create a `TextDataBunch` suitable for training an RNN classifier.\"\n",
    "    @classmethod\n",
    "    def create(cls, train_ds, valid_ds, test_ds=None, path='.', bs=32, val_bs=None, pad_idx=1,\n",
    "               dl_tfms=None, pad_first=False, device=None, no_check=False, backwards=False, **dl_kwargs):\n",
    "        \"Function that transform the `datasets` in a `DataBunch` for classification. Passes `**dl_kwargs` on to `DataLoader()`\"\n",
    "        datasets = cls._init_ds(train_ds, valid_ds, test_ds)\n",
    "        val_bs = ifnone(val_bs, bs)\n",
    "        collate_fn = partial(seq2seq_collate, pad_idx=pad_idx, pad_first=pad_first, backwards=backwards)\n",
    "        train_sampler = SortishSampler(datasets[0].x, key=lambda t: len(datasets[0][t][0].data), bs=bs//2)\n",
    "        train_dl = DataLoader(datasets[0], batch_size=bs, sampler=train_sampler, drop_last=True, **dl_kwargs)\n",
    "        dataloaders = [train_dl]\n",
    "        for ds in datasets[1:]:\n",
    "            lengths = [len(t) for t in ds.x.items]\n",
    "            sampler = SortSampler(ds.x, key=lengths.__getitem__)\n",
    "            dataloaders.append(DataLoader(ds, batch_size=val_bs, sampler=sampler, **dl_kwargs))\n",
    "        return cls(*dataloaders, path=path, device=device, collate_fn=collate_fn, no_check=no_check)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqTextList(TextList):\n",
    "    _bunch = Seq2SeqDataBunch\n",
    "    _label_cls = TextList"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Refer to notebook 7-seq2seq-translation for the code we used to create, process, and save this data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = load_data(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos contenu introduction aux its saviez - vous que … ?</td>\n",
       "      <td>xxbos what 's inside introduction to xxunk did you know … ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos contenu introduction aux its saviez - vous que … ?</td>\n",
       "      <td>xxbos what 's inside introduction to xxunk did you know … ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos contenu introduction aux its saviez - vous que … ?</td>\n",
       "      <td>xxbos what 's inside introduction to xxunk did you know … ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos contenu introduction aux its saviez - vous que … ?</td>\n",
       "      <td>xxbos what 's inside introduction to xxunk did you know … ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos contenu introduction aux its saviez - vous que … ?</td>\n",
       "      <td>xxbos what 's inside introduction to xxunk did you know … ?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformer model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Transformer model](images/Transformer.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shifting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We add a transform to the dataloader that shifts the targets right and adds a padding at the beginning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = data.vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.stoi['xxpad']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def shift_tfm(b):\n",
    "    x,y = b\n",
    "    y = F.pad(y, (1, 0), value=1)\n",
    "    return [x,y[:,:-1]], y[:,1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.add_tfm(shift_tfm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The input and output embeddings are traditional PyTorch embeddings (and we can use pretrained vectors if we want to). The transformer model isn't a recurrent one, so it has no idea of the relative positions of the words. To help it with that, they had to the input embeddings a positional encoding which is cosine of a certain frequency:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.0000, 0.0667, 0.1333, 0.2000, 0.2667, 0.3333, 0.4000, 0.4667, 0.5333,\n",
       "        0.6000, 0.6667, 0.7333, 0.8000, 0.8667, 0.9333])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = 30\n",
    "torch.arange(0., d, 2.)/d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PositionalEncoding(nn.Module):\n",
    "    \"Encode the position with a sinusoid.\"\n",
    "    def __init__(self, d):\n",
    "        super().__init__()\n",
    "        self.register_buffer('freq', 1 / (10000 ** (torch.arange(0., d, 2.)/d)))\n",
    "    \n",
    "    def forward(self, pos):\n",
    "        inp = torch.ger(pos, self.freq)\n",
    "        enc = torch.cat([inp.sin(), inp.cos()], dim=-1)\n",
    "        return enc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOyddZhc1f243zO+7i5ZiRIXPEGSkiIJbsEpELxA228hOBQoFEpLiwYtHqwQJAmBEDzEnehK1m1WZ3ZH7++PO3d1fGZn01/2fZ48SebOvffMzD3ncz4uJElimGGGGWaYQxfVUA9gmGGGGWaYoWVYEAwzzDDDHOIMC4JhhhlmmEOcYUEwzDDDDHOIMywIhhlmmGEOcTRDPYBgSE1NlQoKCoZ6GMMMM8ww/1Ns2LChUZKktP6v/08KgoKCAtavXz/UwxhmmGGG+Z9CCFHu7vVh09AwwwwzzCHOsCAYZphhhjnEGRYEwwwzzDCHOMOCYJhhhhnmEGdYEAwzzDDDHOKERRAIIV4RQtQLIbZ7OC6EEP8SQuwTQmwVQkzrdexyIcRe15/LwzGeYYYZZphh/CdcGsFrwMlejp8CjHL9WQg8ByCESAbuA44EjgDuE0IkhWlMwwwzzDDD+EFY8ggkSfpOCFHg5S1nAK9Lcs3rNUKIRCFEFnACsFKSJCOAEGIlskB5JxzjCpUWs5Vvdtdz5pQchBAhXOgA7F0JaWNhxDEQyrUGEafkpLqjmr3NezF2GTHZTJjtZrQqLUmGJBL1ieTE5lCcWIxG9T+ZgjJ4dDZDex1YTWDvguypoIse1FtKkoTTZMbR0oKzvQ1HWzvOjnac5k6cnWakri4kmw3JZkey2fqerFYhNFqERoPQ61BFRaOKMqCKiUEVG4c6LhZVQgKapCR2Npix2p1Mzf/f3aO1mm0cMJqZmJsQ2oVsnbDjvxCTDvlHgj4uPAMcYiI1m3OAil7/r3S95un1AQghFiJrE+Tn5w/OKPvxxJe7eXPNAfKTY5g+IsBJIEmw/mXY/DZUbeh5Pe9ImPkHGP3biAiEDeXN3P7hVp6/ZBoj0wc+tKWtpayuWM13ld/xq/FXTDaTz2sa1AbGJo9lWsY05o6Yy2Eph4UmKP/X2bIElt4EDmvPaykj4fzXuf8XKEyN4fJjCgK6pCRJOJqasFZUYKuuxlZVjb22Blt9Pfb6BuwNDTiMRiSLJbyfxQ1duihaDHGUjhmBLiMDTUY62uxstFnZaHNz0OXloYqKGvRxhMIf3tvM9/saWX/3b4g3aAO/gNMJ2z+Arx6Atkr5NaGCrMkw514onh3eAbuhtrWLxd+VsOjUsWjV4XXvRkoQuFslJC+vD3xRkhYDiwFmzJgx6N10mk1WPtgg/+CfbK4KXBB8+xis/itkToI598GYU6Hse/jxKXjnAph0AZz1wqAKg/ImE9e8vh6jycqybbXcPEcWBCabiU/2fcKS3UsoaS0BYEzSGE4vPp3RSaMZlTSKjOgMojRRRGujsTlstFpaMVqMlLaWsqNxBzuadvD6jtd5Zfsr5MTmcGrhqSwYu4C06AHZ60EhSRLvra/ggNFMe5edLpuDq2cVMTrjINqBOZ2w+hH47nEomAUzrgRdLFjaYcWdSC/OoaPzMu5zHI/dKXHVzMIBl5Dsdqzl5Vj27sOyfx/W/SVYykqxlR/AaeorlFUJCWjT09Gkp6MvKkKdkowmKQl1UhKq+HjUcfGoYmNRRUejio5C6PWodDqEVgtabbewliQJHA4ku13+Y7Hg7OxC6jTjNJlwtHfIGkZrK9VlNXz13U4SLe3EtZhJrt2CvbZ2gIahychAN2IEuqJC9EXF6EcWox8zBk1KyuB9/37y0/5Gvt5VD8Dq3Q2cPjk7sAuYjfDWeVC1Xl74z3hafr38J9jxEbx9IVz8PhQdH+aR97B8ey13fLQVq93JWVNzQtds+hEpQVAJ5PX6fy5Q7Xr9hH6vr47QmLzy9toDdNmcTMpN4NMt1dwz7zD/pfDG12UhMOViOOOZnsU+fSxMv0I+9v3fIWsKHH3DoIy/2WTlylfXIUkSeclR/LS/iQXHJPHytpf5777/YrKZmJQ6iTuPvJMTck8gKzbL47W0Ki3R2miyYrMYnzKeeUXzAGi1tLLqwCpWlK3gpW0v8dqO15hfPJ/Lx19OUUJRSOPfV9/B7R9uQ60SxBk0tHXaiNZpuP/08SFdN2w47PDfhbD9Q5h6CZz2D9Doeo4XHkfjfy7lCfsLTE0wcddnEI2DMxO66Nqxnc7t27H8ugvLvn09u3oh0ObkoCsqJHr6DHT5+WjzctHl5KDNzkYVExOWoQshQKNBaFzTPzbW43ufX/YrL7ePZXRGHK932lj9fyegEeBoasJWXY21ohJbxQGs5QewlpXR9sUynG1t3eerU1MxjBmD4bDDMEyYgGH8eLQ52RHTIJ1OiUe++JWcxCgsdgdf7qgNTBBIEnx8PdRuhTOehckLQOVaB4pPhKOuh9dOg3cWwKUfQf5RYR1/l83BA5/u5J21B5iUm8BTF06lMDU8z0FvIiUIlgI3CSHeRXYMt0qSVCOEWAE80stBPBdYFKExecRid/DaT2UcNzqNy48ewVX/Wc93exqYMy7D98l7voRPb4XiOTD/qYE7frUWZt8DDbvhy7shaxIUzAz7+Be+sZ7Klk7evvpIPttWzpI9b3HaR99jdViZWzCXS8ZdwsS0iSHdJ0GfwFmjzuKsUWdxoO0Ar+98nY/3fczH+z7m/NHnc9PUm0jQB7dz2VrZCsDyW2YxKiOO857/ia2VLSGNN6ysXSwLgdn3wKw/DvydY9N5PnYRszbcx5ymT8lv203C0lrKnA4A1AkJ6A8bR9KCBejHjsEwejS6wsKDysTidEp8tqWGmaNSufQoeR58trWas6bmoklLQ5OWRtTkyX3OkSQJR2Mjln376Nq9G8vuPXTt2kXTq6+C3Q7IwiFq8mSiJk8mevo0DBMnotLp3A0hZD7ZUsX2qjb+ccFkfikx8tnWGix2B3qN2r8LrHkO9iyHkx+DqRcPPB6dDJd+DK+dKmsNV3wmaw1h4r31Fbyz9gDXHl/EH08ag04zOBH/YREEQoh3kHf2qUKISuRIIC2AJEnPA18ApwL7ADNwpeuYUQjxF2Cd61IPKo7joeSzLTU0tFv4+3mFHF2cQlK0lv9uqvItCFoq4P0rIHMCnP8fedF3hxBw5nPw4mz5/dd+B/EBqqte+GZXPevKmvn7eZMxa7bzdcf9aFIbGBU/k4eOu52ChIKw3UshPz6fu4+6mxum3MBzm5/jvT3vsaxsGTdPuZnzxpyHSgT2AG+raiVap6YoTd6tTspN5M015dgdTjRhto8GTFs1fPMwjPxNHyFgb2jAtOYXTL+soXPdes4tl+t7taijycuo49PxJ9JSMJrbbzk7orviYNlU0UxVSyd/nDuaE8ekMyYjjudW7+eMyTmoVO7HLoToFhIxRx/d/brTYsGyZw+d27bRtWUrnVu20PH11/I5Oh1RkyYRfeSRxBx1JIbJk8MiGLpsDp5YsYcJOfGcMTmHhCgt766r4Of9TZwwJt33Bao2wsp7YcxpcOS1nt8XlwGXfQIvnQSf3AQLv+3RGkJka2UraXF6Fp0yLizX80S4ooYW+DguATd6OPYK8Eo4xhEOJEnipR9KGZMRx6xRqQghmD85myXrKmjvshHnzdH07aPgtMEFb/qOJjDEy+97aQ4svRku+TBsn+HXmnaEuosNpuf59OtPKEoYSd2+s5mU8ZtBEQK9STYkc9dRd3HemPN4bO1jPPTLQ3x14CseOvYhMmL80KhcbKtqZXx2PGrXgjMpNwGL3cmeug4Oy44frOH7x/I7wGnHOedhzD/+hOmHHzD9+AOWvfsAZHv95Cm8FDuRGfNP5MKpnYj//o6ijGjuN47jnly38RAHHUs3V6PXqDjpsAxUKsF1JxRx25ItfLO73j/tuBcqvZ6oiROJmjgRLroIAHtzM50bN2JevwHzunU0Pvccjc88gzAYiJ4xg5iZxxI7cya64uKghOaba8qpaunk8XMnoVIJjilOJVqnZuXOOt+CwNIOH/wOYjNkn4Cv+yfkwkkPwkdXw7b3YPKFAY/XHbtq2xibOfh+seHM4n78vL+JX2vauGpWYffDd+bUHCx2J8u313o+sWGPHCF0+NWQ6GdUU/pYOO5PsO8rqAxfWe0NdRuJL36Kz8s+5eqJV/P+/CVMTJ3Gj/ubwnYPX4xOGs1Lc1/i3qPvZUvDFs5eejZfln3p17l2h5Od1W1MyOkxK010/Xtb1dCah2xrPqB56UoqtkxmzykLqLj6aprffhtNWjrpf/ojBR98wOiff2L7Dffy4agTmHjSsYjJ58C4+fy24VViO8poNll932iIsTucfL6thtlj07s3P/MmZZOTGMVL35eG5R6apCTi5swh4/Y/U/jB+4xe8zO5zz5D4rnnYquqov7RxyiZN5/9vzmJ2r88RMf3P+C0+v/d/by/idEZsRwzMhUAg1bN8aPTWLmzDqfTR7zJ2hehuRTOXiybf/xhwjmy3+/rv8hhpiFid8gbn3FZg7/xGRYE/Vi9pwGdRtXHoTQ1L5ERKdF8vLnK84nfPATaaNlUEAiHXw1RSfDt34IccQ+SJPHe7vfYYn8UrVrDG6e8wS3TbkGn1nFscQrbKlto67L5vlCYEEJw3ujzeG/ee+TH5fPHb//I39f/HYfLTu6J/Q0mOm0OJvWKjChIiSHOoGGLy3cQSSwlJTQ+/wKl55zDvivuoXZ9Ihajg8RzziFv8QuM/mUN+a+8TMrVVxM1YTxCrWZNSRPxBk3PJD71CSS1njs1b7Gnrj3inyFQ1pQYaeywMr/XPNCqZe1gS2WLHHkUZtTx8cTNnk3m3XdRvOwLRn79FZkPPIB+zBhaPvyQimuuYe9RR1N52220fv45jo4Or9craTQxMr2vI3zu+Azq2y1s8eZvsnXCmmflkNCCY/3/ACoVzH1IDi/95Xn/z/NAWZOcvxEJjWA4K6gfpY0mClKiMWh7nElCCM6cksO/Vu2lvr2L9DhD35OqN8HOT+D42yEmNbAb6uPg6Bth1UPydbKnBjVuq8PKI788wod7P8RhGsMZhXcwKW1S9/Gji1P516p9/FJi5KTDAlPrQ6UgoYDXT32dv639G6/teI29LXv523F/I17nfqezrUpe7Cf20ghUKsHEnAS2RUgQWEpKaFu2jPbly7tNPobiLNImtRF33WPoTrzYq7ni55ImjixK6TZtEZeJZfpC5vz8d5aWbB/UUMNwsHRLFTE6NbPH9jWhFKXFYLY6qG+3kBFv8HB2eNDm5JB0wfkkXXA+zq4uzL/8QvvXq2hftYr2ZcsROh0xM2cSf/JviZ09G3Wv6Cer3ckBo5l5k/pGw80ek4FaJfhyZ53nBLlNb4KpIfBNHUDhLBh9Mnz/JEy9DGKCD5/dVStHX40ZNg1FnrJGEwUpA8OzjipKQZJgT62bXcjXD8q7+qPdukF8c8RCMCTAt48HdXqHtYMbvrqBD/d+yDlFl2OuuJyJWX0nwLQRieg1Kn7a3xjcGENEq9Jy11F3cc9R9/BL9S9c/PnFVHW417C2VbYQrVNTmNp3NzcpN5FdtW1Y7N41imCx1dTQ9PLLlJx5FiWnnkbj08+gSkgg4667GPnNKgrnW0k9sQC9DyFQ3dJJeZOZo4r6LgIxx16DQ6jI2PX6oIw/nKwra2bmqNQ+GyKge26UNvpOPAwnKoOB2OOPJ+vBBxj17WpGvP0WSQsupGvHDqr/fDt7j51J5S230v7VVzitVg4YTTicEkVpfedyQrSWo4qSWbmzzv2NHHb46V+QewSMCEAb6M1vHgBrB/zwZHDnu9hV045aJQZoNYPBsCDohdMpUW40u43TzU+RywWUG/tNgKoNsH8VzLxNXsyDwZAAR90Iuz+Hmq0BndrY2cjvVvyODXUbeGTmI8xIuAhQMSqj78Oj16g5vCCZn/ZFzk/gjvPHnM+Lc1/E2GXksmWXUdJSMuA926pamZCd0LObdjEpNwGbQ2JXTfhMK06TiZaPP6b88ivYN3sO9Y8/gdDryLhzESNXr6bgzTdJvvQStNYyqN0GR1zj03H4s8sXc3Q/QSDiMvnRcAJTmz6DzoMoFLYfDqdEZbN5gCAGuudGpAVBb4RaTfS0aWQsWsTIb1Yx4u23SDz3XMzr1lF5083snXUcxocfYqyxnCI3m7rjR6exr74DoztfzfYP5ZIws/4QfLJn+lg47ExZs7B1BXcNZI2gOC3G/1DXEBgWBL2obu3EandS4EYQZMYb0KlVHDCa+x7Y+AZoomD6laHd/MhrQR8f0C6iqqOKy5ddTmlrKf+a/S/mF89nT10HKgHFaQMn8TEjU9hd105D++CXJfDGjMwZvPLbV3A4HVy+/HJ2NO3oPmZ3ONlZ09dRrKD4DLZWhWYekiQJ88ZNVN91F3tmHUfNHYuwVVeTeuONFK9YTuGSJSRfdhnajF5mkbWLZYE98Tyf1/+5pImkaK1b2+623IswSF1Im94I6TMMJjWtndgcEvnJA2slZSdGoVOrKBtCQdAboVIRPW0amffczahvV5P34mJiZ80ietUK/vHdv4leeBFNL7+MvbFHE1a0mor+c9nplOdf+mEw6rehDWz65dDVAr8uDfoSv9a0MzYzMhFyw4KgF2WN8oMxImXgBFCrBLnJURxo6vXwWM3yDuKwM+Rw0FCISoSpl8Kvn8kp7T6o6ajhqhVX0WJp4cW5LzIrdxYAe+vayU+OHqDSQ88OdV3ZkKdqMCZ5DK+f8jrRmmiuWnEVWxq2ALCvoaM7o7s/OYlRJMfo2FoR3G7a0dKC8T//oWT+fMovuoi2ZcuJP+VkRrz1JsVfriDtphvRjRgx8MS2GnlCT70UdL6zOteUNHFkYYrbWPu4wun84hyLc80LshniIETZ7LgTBGqVYERKNCUHiSDojdBqiZ01i5wnHuftPz/HS0ddhDY5mfrHn2DvCSdSefPv6fjhR/ISZd9GRXM/QbD/a2jYJWv3oeYBFBwHSQVylYEgaOuyUdXSydisyJRUGRYEvShtkh9uTync+cnRlPcWBLs+A0ubXGIgHEy+QM5D2PGR17fVmeq46suraLO0sXjuYqakT+k+tqeunVEe6vEor/f5DENIfnw+/znlPyQbkrl+5fX82vRrtzPYnUYghGBSbkK3M9lfOrdsofqORew9/gTq/voo6phYsh76C6O++47shx8mevp073HqG14DpwNm/M7nvTosdiqbO5mU595MODojjlfsJ6Nuq4DdXwT0OSKFslN2tyECKEiNOWg0Ak/s7pCoPPo3FLz9FkVffE7yZZdhXr+eiquvRn35eZy79xtqD/QLB9/yLkQlw/izQh+ASgXTLpPrizXtD/j0PbWy+XPcsEYQecoaTRi0KjL6RwW5GJEcTYXR3BM6t+kNWeoH61TqT+YkWS3dssTjWxo7G7n6y6sxdhl5/qTnGZ/SU3vHYndQ1mRmdIZ751KsXkNClJaqloNDEABkxmTy0tyXiNHFcO3Ka/nxwA5idGqKPAjjSTkJ7Klrx2z1vpt2Wq20fPwxpeedT9kFF9L+5ZcknHUmhR//l4Il75J47rmoY/2o2WK3woZXYdRJkFLs8+1VzXL8eG6S+0V0dGYcK50zaDdkw7qXfN9/CDhgNKNWCbIS3M+DotQYypvMOHzF4g8hJQ0d3Vnp+qIiMv78f4z8djXZTzyBPiuLq3Z8zpF3XEH1nXfRuWMHWDpkwTz+LM8VAQJlysUg1EFpBb+6BMGwRjAEKBFDntLn81NiaLfYaTbboLkMSr+Tf+wwpZMjhFyVtHKt212EyWbihq9uoM5cx7Nznu0THiqPX56c3ip05iRGUdkcerJLOMmOzealuS+hEipWtz7EqFybx99gUm4iTgl2Vre5PW5vaqLh6WfYd+Jsau5YhNNkIuOeuxn53Xdk3X8/hrFjAxvc3hXQUQeHX+PX2xUhm5PovmZQaqyepBgDa+NPkneLHfWBjScClDeZyUmM8ljKoyA1BqvDSXXLwfUcKTSbrDSbbRT3ixhS6XQkzDuNEW+8zpML7mfr+Jm0LVtG2TnnUn7hubSVOpHGnxu+gcRlyqGkm98GR2D5O7tq2og3aMgc5BBdhWFB0IvSJvehowqKzbS8yQSb3wGEXI0wnEw8T77u1r5agc1p44+r/8ie5j08ecKTTMuYNuBUJVFplJu+Awq5SVHdu9aDiRHxI3j+Ny9gl6zURz9Nq8W9+UfxHfRPLLOUlFB9993sO3E2jU8/jWHCePJefomizz8j+eKL/dv9u2PHx7K5wM9681UtcpRIbpLn4nGjMmJZaj8SJGdIzsTBosJo9mgWgoMjcsgbJY1yiHf/0NHe6EaO4qXDz2fUt6tJv/12bNXVVP2QzP5r7sP45ls4zWHSmqdfDqZ62L0soNN21bYzNis+YvWohgWBC7vDSYXR7DZiSEGZHAeaOmDzW3IZ2sQ8j+8PioQcOdlo6xK5BC5ylMuDPz/Ij9U/ct/R9zEzx3210r117aiE9wmQkxRFVUvnoGSGhorank1n5aWYnfXctvo2bG52UenxBpKitexvkCe7eeNGKm64kZJTT6Pt089IOPssir74nPwXXiD22GNDm0i2Lrny5Lh5oPYv97KquROtWpAWq/f4njEZcXzdlIKUOkYWNAcZB4xm8tw4ihUUQVDWdHAKgv318riK3IS/KuQlR1PZbIbYOFLOO43ikyvJuepYNCkp1D30EPtOnE39U09hN4YYWFE8B+Ky5fXCT5xOid217YyLQCKZwrAgcFHd0oXNIVGY6nkC5Lnsvo6SH6C1QjYLDQaTLpRNTxW/APDC1hf4eN/HXD/5es4a5dmRtaeug4KUGLcRQwo5iVGYrQ5azJErNeEvVS2dOMxFXDX2DtbVruO+n+5zK7Cy4g1EbVpL2SWXUH7RxXRu3EjqjTcy8ptVZN1/P/qi0HohdLN/lZwYdNgZAX2GrIQoj6YtkJ32HRYH7cXzoOwHucXlQUJbl41ms81txJBCepyeaJ2akoaDVBA0dqBVC69aWV5yFDaHRF1bF+z4CCEcxF/+RwqWvMuIt98iasYMmp57nn2z51D78CPYqquDG4xaA+PPhP3fyH4IP6hq6aTDYmdsBGoMKQwLAhdKxJA301CUTk16nJ7Uqq9ArYcxpwzOYMbNl+sWbXmHVQdW8czmZ5hfNJ/rJ1/v9bQ99e0+sxCVyVF1ENp3a1tls8o5o0/npik38WnJp7y0rcehKjmdtK1cyZ//+wjnvvcEtsoqMu68k5Grvibt5pvQJPtZHMxffl0KhkQo9L8cRHVLp0f/gIJSMuDX5DmAdFCZh5TwaG+CQAhBQUrMQasRlDTIJl5v5cqVTV2F0Qxb34OMiZAul3qOnjaNvGeepujzz4g/+WSa33mHfXN/S/Xdd2M9cCDwAY09DRwWOTzVD36tkf1fkagxpDAsCFwo4XC+uv+MSI5iTOsPUHSCXzHlQaGPhbGnsW/PZyz6fhETUiZw3zH3eTVzWOwOypvMPls55iTKE+BgcxgDVLd2oRKQFqdn4aSFnFp4Kv/e9G9+qPiOtmXLKD3jTKpu/j1RdgsvHLmAkV+uIPmyS1FFD0KTeLsVdn0hT+IAokiqmjvJ8bITBRjt8uFs6sqEtHFyM/SDhAovOQS9KUyLOXh9BA0dXs2jQLfpy1i5S25BOWlgoqC+uJjsR//KyBXLSbrgAtqWfsr+U06l+vY7sJaV+T+gvKNkP9Ouz/16+956WXOIZFvWYUHgorTRRIxOTVqcZ9suwBEx9WQ4amHsqYM6ntbiE7glQUuUSsM/TvwHerX3cZU0yLVV+peW6E/OQa0RdJIWp0erViGE4L6j7uXMykw6L76eqtv+gORwkP3439jwwPN8nDUdixjE1PvSb8HSGpBZyGp3UtfeRbYPjSAhWktGvF527o8/S+5921YT6ojDQncymRdnMUBhSgyVzXIm/sGE3SEXmytyk1nfm+xEA0JA7H5XLseEczy+V5uTQ+Y9d1P81UqSL7mEthUr2H/aPKrvWOSfhqDWyNaDPcv9ih6qaukkOUZHjD5yNUHDIgiEECcLIXYLIfYJIe5wc/wfQojNrj97hBAtvY45eh0bMh25rMnEiJQYn87FYx1rAegqPGnQxuKUnCyq/5ZqjYZ/JkwnMybT5zn7XLsIbxFDAEnRWqK06oMycqimtYvMhCgkSaJ99WpqL7iUBW9UoHbAkgXZZH70Lgnz55OZHNP9/kFj58dyyY+iE/w+pba1C0mCXB+CAGRHZlmjSbYfH0TmoXKjmcRoLfHeGjAha84OpzQwO3eIqWiWy2N4ykNR0GvUZMYbyG74Xs7fScj1eW1tejoZi+5g5MovZYGwbBn7TzmVmnvuwVbjQ5CPORW6WqH8R5/3qW/rGvTKrv0JWRAIIdTAM8ApwGHAAiHEYb3fI0nSbZIkTZEkaQrwb6B36mynckySpNNDHU+wlDWa/GoKPbb1ezY7i6m0D54j57Udr/F9zRr+LCUzpXyDX+fUtMoLe16y90VICOGKHDq4JjDIC+mM5hLKF1xE5XXX4zSZyP7bYxjefYH/Fjby8Lq/ApDpSnRSPnPYcdhkNX7MKaDxron1RtGyfJmGQP4MdW0WSBsjJxEeJNFDFUYzI3yYhYDu6LqDLcO4pEEJHfVdsXNMgpMC8zYYNTege2jS0shYdAfFK78kacECWj/+hP1zf0vtI49gb/JQ1LF4tlyTzA/zUG1bF5nx/j934SAcGsERwD5JkkokSbIC7wLe9OkFwDthuG/YsDmcVDR3UuAlYgiA9lqSW7ax0jF90Mo0bK7fzL82/ou5I+Zy4ahzoG673AvZB3VtFqJ1amL9UCdzEqMOOtNQ16+/8rul/+TcNx/BVlND5gMPUPz5ZyScfjrH5M3kuknX8WnJp3yy7xOyEuSFtnawNIKyH6CzOSCzEPQIAl+mIYD0eD317V1yVNSYU+UIsa7IN93pj6/QUYWigzSXQIlk6p9M5o4TdTtQ4wxYECho09PlJjorlhN/xuk0v/kW+0+aS8O/n8bR0e970UXLwmDX591h4Z6obR38Xg/9Cf3RDysAACAASURBVIcgyAF6r1SVrtcGIIQYARQCq3q9bBBCrBdCrBFCnOnpJkKIha73rW9oaAjDsHsNuLkTh1PyGjEEyDY+4CvntEERBC1dLfzp2z+RFZPF/cfcjxhzsnxgr+8Wj3UuddKfuPmcgyipzFpZRdX//ZnSs86muKmcknN+R/GK5SRdcD5C22OeWDhpIYdnHs7DvzyMBVkNHzTT0D5XVFjRiQGdpnynnkoz9CYjzoDNIclZ6iPngOSAkm+DGm64sDucVDV3+nQUAyTF6EiI0h58gqCxg+QYHYnROp/vPdy6lmYpFktmcM2gFLTZ2WQ/9BBFn31GzMyZND7zDPvnzsX45ltItl4+gbGnQVsV1Gz2eC2bw0mT6X9TELhbeTyJvAuBDyRJ6t1ZJF+SpBnARcA/hRBuC7pIkrRYkqQZkiTNSEtLC23E/fA3Yojdy5AS86nUFgwsRx0ikiRxz4/3YOwy8sQJTxCni4PU0XItoz0rfJ5f32Yh3YejWyEnMYpms81nvZ7BxNHWRt3jj1Nyyim0f/klqosv58qT7sR+3kWoDAMngVql5tFZj2JQG7jrp9tJjBlE09D+VTDiaHkXFwBVLWbS4vRe8zgUFPNWXVsX5B4Ouji/wwsHi5rWLuxO9+Wn3VGYevBFDlU2d/ql0eB0UtSyhu+ck6huC09Ojb6okNx/PUXBe0vQjxxJ3UMPUTL/dNpWrpQ1v9Eng1B5NQ81tFuQpJ7nI1KEQxBUAr3Ta3MBT9kXF9LPLCRJUrXr7xJgNRCaeA4C5WH2llWM1QQlqxFjTiM/JTbsguCDvR+wunI1t02/raeQnBByXfTSb+WS116oa/ffwdSdSzAEWoFks2F84032nzQX4yuvEn/aaRSvWE79hVdh0kV1m33ckR6dzkMzH2JP8x4MGcsGxzTUVgP1O/0uKdGb6pYunzkEChkuG3BdW5ccnlp0POxb5dNsMJj4GzGkUHgQViGtb7OQ4c+GqGYzBmsT3zimhH0uR02aRP5/XiP3+edArabq5t9TfumldJbWyqGku5d7PLe2TX6mI1VjSCEcgmAdMEoIUSiE0CEv9gNCIIQQY4Ak4OderyUJIfSuf6cCxwI7wzCmgDhgNBOjU5MS40WdLFkN9i4YczL5yVFyvaEwUd5WzuPrHueorKO4eFy/bOXRv5XvW/a9x/MlSXKZhvzXCAAqI+gnkCSJjm+/peSMM6l7+GH0h42j8KMPyX70r2gzM7vNPL7MKsflHscl4y7BpP+Wko5N4R/ofpfVMghBUOVHMpmC0ve6vs3VJGjkHGg9AI17A75vuPDWh8Ad2YkG6totB1UV0vr2LtL9mQd7VyIh+NY5aWCDmjAghCDuhBMo+uRjMu+/D2tJKWXnnkv19xpsZTvA5L5lbJ1rHvj1GcJIyIJAkiQ7cBOwAvgVeE+SpB1CiAeFEL2jgBYA70p9awaMA9YLIbYA3wCPSpIUcUFQ395FRoIP+3rJt7LXP/8YRqTEUNHciTMME8DutHPn93eiVWl56NiHUIl+P0nBTNDGdPsn3NHWZafL5vRbI8iJsEZgKSmhYuG1VFx7HTid5D77LPmvvIJh3Lju99QGMAFumXYLsapsGgyveyxOFzT7V0FMOqSP9/3eXjidkiwI/IgYgp7PWefaAVI8x3X/oTMPHTCa0aqFV62sN+lxBhxOyX3LxyHAanfSbLZ1C1mv7F0BOTMwqRMHNQRWaDQkXXghxSuWk3L1VbStL2P/5+k0PvEATsvAToF1/8MaAZIkfSFJ0mhJkoolSXrY9dq9kiQt7fWe+yVJuqPfeT9JkjRRkqTJrr9fDsd4AqWh3eK1SBggm2dGHA0aHfnJ0d3JQ6Hy4tYX2dq4lXuPvpeMmIyBb9Do5eJ2e770aDaob1MWUf8envQ4AxqVGPTIIUd7O3WPPkbJ6WfQuWkT6XfcTtHST4ibfeIAoVvT2kVqrM6v/qwGjYG5abciqdt5aM0j4Ruw0wkl38jfd4ClxZtMVqx2p98agV6jJila220KIGkEpIyEfUMoCJrM5CZFD+gV7Qkl+XKoW58qNHTI4/DpK+togKqNiNFzyUmKotI4+BsidVwc6X/6E0WfLiU220nDO19Rcto82r/6qk89rdo2Czq1imRv1olBYDizGPlB9rqIttfJLexcNWcUZ1RFiA/QLuMuFm9dzLyiefy2wEuP1KIToK0SjAMbvYMcOgr4ZxtFbjeYlWgYNI1Acjpp+fhj9p9yKsb//IfEs86Ud0RXXIHQuX/Aa1s7A3KQTUybiLVxNsvLvmBFmW9nul/UbgVzU9BmIfDch8AdGfGG7t8OkLWCsh9CangeCv6GjiooC259GDZE4aBnQ+RjHuz/GpBg1EnkJUdHNClOV1BI7pWHkz9PjSoqisqbbqbimoVYSkoBWSNIj9dHrPy0wrAgAOp9aQSl38l/Fx4H9Dj6QtkJ2Zw27vnxHhINidxxxIBk7L647uvJT6Cok4GEnA1WLkHXr79SfvEl1NyxCG1ONgXvvUfWX/6CJiXF63k1rV1kxvu/iGYnGLA2nkhB7Fge+eURmruaQx16j3+g6ISAT1WEqj85BAoZ8Ya+i+jI34C9Ew78FPD9w0FNayc5if4/QwebRlDvGkdarI/PUPqdXPsnczJ5SVGD4iPwStEJxMRWUPjyk2TceSedmzdTcsYZ1P/97zQ1tkY8dBSGBQEmix2z1eG9xlDpt2BIgKzJAN1CoyGEndAr215hl3EXdx91Nwl69/1tu0kdDbEZ8m7RDYqJKhAHU05idFg1AkdHB7WPPELpOediLS8n6+GHKXjnHaImTvDr/JrWLr/i7xVk7UHNadm30GZt49G1jwY58l7sXwUZE+TOUgHS3ZnMTx8ByBuKbh8BQMGxoNYNiXnI7nDSZLKS5o993UW3IOg4uASBz3lQ9r3se1OpyEuOptlso70rgmXZi2TLgqj4geTLLqV4+TIS5s2j6cWXuPaVRRxTvS3i/UIOeUGg7Ga82hVLv4WCWaCS7ddJ0TrUKhH0BNjbvJfntz7PKQWnMCd/ju8ThJAf3NLv3foJ6tssxBk0ROv8L1KVmxRFXXtXyEXDJEmibdkySk45leY33iTxgvMpXvYFieecjfDTzm622mnttAVkGlIcmo6uTBZOXMgXpV/wbUUICVlWExxYI/sHgqC6pYs4V09of8mIN9DQO+pGFwP5R/doJhHEaLIiSZAW679tOlqnIVav6Yl8GmIa2roQAu/Rf83l0HJAns/0RKn1EciDTepoiMvqTiDUpKaS/ddHGPH2W7RrDJz6/j+puO46rJWVERvSIS8IutVJT4KguUx+cHrVpFepBCkxuqBUYofTwb0/3ku8Lp5FRy7y/8SCWdBRC037Bhyqb+/yO5lMIScpCkkKrUyDtaKCimsWUnXbH9CkpVHw3hKy7rsPdYIPDacftX6GjvYmSqcmMVpLbWsXV0+8mlFJo3hwzYO0W9sDunc3ZT+C09YTvRMglX6Un+5PerwBpwRNvTcURcfLeQwmDzVrBgllU+Or+m5/0uP0B5VGkBKj99qHoLvoW4Hc5a9Hu49g5JMQ8npS+q0coODCcdhEbjz+VkrOu4rOdespmTefxsUv9s1OHiQOeUHQ4EudVNL+FTu9i7Q4PY0dgT887+5+l+1N27njiDtIMiT5f6Jyf8Vf0Yu6tsBT0nO7cwkCt49KNhuNLyymZN58OjdtIuPOOyl4/z2iJk4M+FrQIwgCzabMjDdQ09qFVq3lwWMepLGzkSc3PBnUGCj7TjbL5B8V1OlVLZ0B+Qegx7nfx2E84lj57wM/uzlj8FDmQaqv6Ll+pMbpaThINIL6dj+y68t+gOgUSBsLyOMHaIy0MCs6QQ5MqN/R/VJdWxcOlRr7OQso+vwzYmYeS8OTT1J69jmYNw1CzkwvhgWBy77u0Vlc+p1sn08b0+fltDh9wBpBramWf2/6N8fmHMvJBScHNtDkIrn3qRuHcV0QZWuV3WugDWo6t2yh9JxzafjHP4g9/niKvvic5MsuRaiD7w3Qk0wW2EKalWCgtk0e/4TUCVwy7hI+2PMBm+s913LxSPlPkDMdtIGNQcGfzmT9UX6zPmaJ7KmgMcjjiSDKpuZ/WyPwI5ms7HtZ2LrMlsq8j7wgcFkYSlZ3v9Qd/RdvQJuVRd7TT5P77DM4Ojoov+hiah/8C44O/9pdBsohLwjq2y1oVIIkd0WqJEkWBIXHyepcL1JjAxcEj619DIfTwd1H3h14eJgQUDhL3tH08hNIkiTXGQowE1FZhPz9DE6TidqHH6HswgU4WlvJffYZcv/1FNoMN7kPARJsWn1WYhQ1LT2L6I1TbiQjOoMH1zyIzRmAOm3pgJotMOKYgO6vYLLIPo6ANQLX563tLQg0ern2kB9168NJsBpBWpy+O2xzqPFZb6uffwAgIUqLRiUiH/kUny37CnoJgm7NuNc8iJs9m6JPPyXpkktofucdSk6bR9fu3WEfziEvCBraLaTG6t03G2/YBaZ6tz1r0+L0NJksfmcXf3PgG7468BXXTb6O3DjfTTDcUjALTA3yuFy0mG1YHU4yAoj2ADBo1cTpNX5NgI4ffqRk/uk0v/kmSQtktTVuduCx9p6oae0kMVpLlC4wrSIr3kCTyUqXTa5hGK2NZtGRi9jbvJc3d77p/4Uq14HTDvnBCQK/Ag7ckBqrQyUYuJCOOFbOaYhgWerGDrmMeaBdsdLjDJisDkyWoStgCOBwSjR2WLxnFffzD4DL3xeri7xGAPIG88AacMjfXa2HMHB1bAyZd91JwZJ3MYwfjy4/P+xDGRYEHRbP6nCpywxTOGvAobRYPTaHRGun752n2WbmkbWPMDJxJJeNvyz4wSrjKO0xDymho8HEHqfE6mjyUh7A0dpK9aI7qbj6aoRez4i33iTz3ntQx/pu+hEIta1dQaXUKz6F3lErc/LncELeCTy35TmqOzzVPuzHgZ/lqpB5RwQ8BugxK6QGKAg0ahWpsfq+PgKQNRPJCRVrgxpPMDS0e5kHXjhYcgmaTBacko/Q0X7+AYXU2OD8fSGTfzRYO7r9BHVtXcQbNB43RFGTJpH37DOoooIzX3rjkBcE9W1eJsCBnyE+BxJHDDiUGkAM9YvbXqTWVMu9R9+LVuV/eOEAkgogIV92bLrosSsGPolTY/U0epjA7atWUTJvPq1Ll5Jy7bUUfvxfoqdNC2rYvgg0h0BB8SlU9ytHvegIORrrr2v/6t+Fyn+CzIlgCK7rnLKIeA1b9EBGvGFgqZLcw0Gliah5SNGMAyX9IMklUDYDXrWyfv4BBVkQDMH4lcCEA2sAWRBEuvy0wiEvCBo6vNgVK36BvCMH+Aegl5PJx06orLWM13a8xunFpzM1PQwVtgtnuUId5bCzYLKKFWSNoO/47c3NVP3p/6i84UbUKSkUvv8e6bfdiko/eNUQa129igMly5UF2z8ENjs2m2snXcvqitV8X+m5aisAdqtsGgrSLAQ9GkEwO2o5qazfM6SLhuxpEXUYN3b4UW/LDcpnHupcgobuMHAP88CNf0AhLc7zhmhQSciFhLzuCLHaIKL/wsUhLQgcTokmT6ahlgq5m5CHcEJ/siolSeLRtXIzldum3xaWMTPiGOg0ducTKPblYBah/ipx+9dfy400li8n9aabKHxvCYbDDvNyhdDpsjloMlmD0ggUc5K7TmWXHnYpBfEFPLbuMawOL2p/zWa5zHeQjmKAJtd3GEyhsPR4g3tn64hjoGqjzz4U4aKhw0JqXBDj7zYNDa3DWCnV4XFT58Y/oKDMg0hn8wLy+nJgDUgSda2Rb1qvcEgLAsWu6HYRrfhF/jvvSLfn9iSieBYEqypW8WP1j9w45UZSo1JDHm+f8bjGV9dmITFa61dXrP6kxuppNluxGJupvv12Km+8CU1qKoUfvE/aTTd6LBAXTpSdZDAqcYxeQ7xB47ZTmU6t444j7qC8rZzXd77u+SLKrjv/6IDvr9Bkkn8DrbdEJg9kxBm6K5f2YcSxcoJb1fqgx+UvVruTFrPNd40eNyRF69CoRHdi5lChPEceN0TlP0JU0gD/AMhOe6vDSVvnEDi884+C9hoczeU0dFgiXn5a4ZAWBF6jPQ6skfsAZLivlRMfpUGnVnnUCLrsXfxt7d8YlTSKC8deGLYxkzJSfqAreuyKgUYMKaTG6phau5uyM8+k9bPPSb3helkLGDtwsgwWyiIe7ARId5VpcMexOccyO282i7cuptZU6/4C5T+5ajkF3/60scMSlH8AIDPBg2aZf6TswI6AeUgxDwajEahUIqhQ6nBT324hIcrLhqhirbyJclP2ZEhrJuXJFoeOPT/gcEpkDPsIIk+Dt/ISFWsgdwao3YfTCSG8JpW9tuM1qk3VLDpiERpVYCF5XhFCfqBdESV17YHnEICcFzD67Wd5+OcXcUTFULBkCWm//31EtIDeBJvIpJAS4z3y6f8O/z+ckpMn17vJOHY6ZIEfgjYA8mcIxtEKPT0kBpT6MCTIDmwPhQbDSaOrvEIwPgJw5RIMuSDwUmbFbITGPR6jwlKHKqkMIH0c6BOwl8kC/39aIxBCnCyE2C2E2CeEGFBTWQhxhRCiQQix2fXn6l7HLhdC7HX9uTwc4/GX7mqF/XfUXW1Qt8PnApEa677eUJ2pjle2v8JJI07i8MzDwzbebvKOkB9ss5H6ILKKOzdvpuTss4n76jM+HHk8TU8uJmpCYB25woXRtRtNCaDYWW9SY/V9a/X0IzculysnXMmysmUDM47rd4KlNST/AMgLSLCCIKO7ZaUbG3v+MVC5HhyDW2umoUO+d6DhrwrpQWTZh5t6b+GvlS7zWu5BKAhUasg7An21vLELJvovLMMI9QJCCDXwDHAKcBiwQAjhzsO4RJKkKa4/L7nOTQbuA44EjgDuE0IEUIAnNDxmU1auk+O48937BxQ81Rt6auNT2J12/jD9D2Ebax9cfgLngbXUt1v8fngkm436p56i7KKLwWZH99RzvDRhPo1D2GlQ+f7cZnb7QbIPjQDgyvFXkh6VzmNrH8Mp9bLFl7vq+YQoCJo6rKQGKcgy+res7E3eEXJ/grrtoQzPJ/9faATesoor14JQQ4778Ochz4XIP4rYtr0k0PE/rREcAeyTJKlEkiQr8C5whp/n/hZYKUmSUZKkZmAlEGARnuBpaLcQp3eTwFHxi2yfzfW+m3dnGtrasJVPSz7lssMuCz6D2BfZ00ClobP0Z9mu6MfDYykppWzBRTQ99zwJp59O4dJPSJslL4BNQ5FM4yIURyvImkSL2Ybd4bmcdrQ2mlun38r2pu18XvJ5z4HKtXL9psTgMzWtdietnTZSglxEk6J1aNWCOneLkPL8VawLenz+EGzlUYX0OD1G09A1sZckSQ4D9zQPKn6BzAlymW83JEZpUavE0GgE0G15OFy9N+jnKFTCIQhygIpe/690vdafc4QQW4UQHwgh8gI8FyHEQiHEeiHE+oaGhjAM25VN6W43fWANZIwHfZzX81Nj+04ASZJ4bN1jpEalcs2ka8IyRrfooiFzUnciire0ekmSaH73XUrPPhtbRQU5/3qK7Ef/ijo2lniDd4d3JDCarEE7WoHuiWM0exdmpxWdxviU8fxz4z8x21whmZXrZD9QCCiN24M1balUgvQ4g3uNICFXrltfOciCwLUhCibyDCDNXTntCNLWacdqd7rXCBx2qNzgMfoPesrKN0ayFHVvcqZhFxpm6vf53S863IRDELgbef+twadAgSRJk4CvgP8EcK78oiQtliRphiRJM9LSgo/w6I3bpvUOu2xTzPNdjjgtTi9PAJede3nZcrY2bOX3U39PjNb97iNs5B2JoX4zGuweTUP2piYqr7+B2vsfIHr6dAqXLiV+7tzu40LIdVaGUiNo7LCSEhP8LkgRIr4+g0qouP2I26k31/PajtfkBubNZT61Pl90l5cIYSeXHq93n5AlhCyoKge31ITXMit+oMyhoTIPKTkEbj9D/U6wmbwKAhjC7GIAbRTlulHMUIW/mJy/hEMQVAJ5vf6fC/Qp8iJJUpMkScq3/CIw3d9zBxO3E6Buu/zg+FGXvie72IrVYeWpjU8xJmkMpxefPhjD7UveEagdXYwTB9yahjq+/56SM87E9NNPZNy5iLwXF6PNSB/wviGdALg0giB309AjCIw+/AQAU9OncnLByby6/VXqSl3tIMMmCIL/DOlxes8N4HOPkAVWR3i0YHcEW15CQYlaGyrN0mPQB/TkA/n4nVOHuJz2dtU4Rtv3ypnuQ0A4BME6YJQQolAIoQMuBJb2foMQIqvXf08HfnX9ewUwVwiR5HISz3W9FhHq27oGPjw+Esl60zv++J1d71DVUcUfZ/wRtSr42vx+4xrfdNWePgup02ql7q9/peKahWiSkih4/32SL7vMY9vI1CHWCJo6LKEJggAjPm6Zdgt2yc4zu96S6/m4+lAHi/LdhbKQJsfoPQsyZQEbxMSyxjBpBEPVoKbeW8/uynUQm+nTD5QaqxuaMhMuNjhHosMGdduG5P4hCwJJkuzATcgL+K/Ae5Ik7RBCPCiEULbGvxdC7BBCbAF+D1zhOtcI/AVZmKwDHnS9NuiYLHZM7prWV66T7bKJee5P7IUy+cubG3hh6wvMzJnJ0dmhxaT7TUIOLboMjtTsQ6+RBY9l/37Kzr8A439eJ+niiyl4/z0MY0Z7vUzKEGoEdoeTZrON5AiYhhRy43JZMHYBH5vL2JM5Vva3hIDy3YXi5EuJ0dFstrkvaZ49RRZYg1iJVNYIghfGQ93E3mvBuYpfIO9wt/XCeqNEAA5JmQngp64C+R+VG4bk/mHJI5Ak6QtJkkZLklQsSdLDrtfulSRpqevfiyRJGi9J0mRJkk6UJGlXr3NfkSRppOvPq+EYjz94LBRWtUHuVOUHyrlfVr+JyWYavHBRD+zXj2eaag+SJNHywQeUnnse9tpacp99lsx77kZl8B1NJMfhD80EaDbbXGMIfhFKcEV89C+e541rJ1xNrNPJP0K4r0KTyYpBqyImwF4KvUmJ1eFweihpro2SE8sGyWHcZXPQ3mUPSSMwaNXEGzRD1qCmvt1ClFZNbP9eCh31slnNH+0+Vi+XmeiKfJkJi93BPksCJm1KREqKuOOQzSyud1dewmwEY4nfkSQxeg3R0S1sa/uCs0aexaikUYMxVI/sUI0h1dJE9c03UHP3PURNnkzhJ58QN/tEv6/RXWdlCCaAsngHU6xNQaUSJMfo/PIRKCS01bCwuZUfbI2sqVkT9L1Brj6bEqMPvONcL5TP7zEfIvcIuQCdI/y/UTic3eAq9TGEPoL0eDe/gaJF+SEIhjKprNlkAwTGpEk9yW8R5pAVBG7LS1RtlP/2UyMAMKSvBNTcMOWGMI7OPyprYyhdkUbbqu9Iu+028l9+ya1D2BtDOQGM3XX8Q1uEUmJ0gTUWqVzHgvZ2sqPS+Pv6v/dNMguQRpM16IxcBeXzewy/zD1cDmBo+NX98RDwWmYlANJiPUQ+RYAGT+UlKn4Btc4vP5AyD4YiqUyZe53pU8G4X96QRphDXhD0eYCq1gNCbiDuBzubdmKL2kCK4zekRwe2AIeC5HTS9PLLnL70XSQEI249kdRrFwbVQF5x1A6Fw7jRpDhaQzPRyCGwAUzgynXoDcncPP0P7DLuYlnpsqDv3dhuITUEjQZ6NALPDmOXhjoIfoJQaz0ppMcPXdRNk6cQ5KqNsllN4/uzKQX3hmRD5PrdncoGtHpjxMdwSAsCdf+m9VUb5DK1PhLJFJ7a+BQaYhCt/ptiQsVuNFJx3XXUP/4Ea7InEH1WEtH68qCvN5QagbJ4h2IaAnlHHYhpiMr1kDuDU4tOZUzSGJ7e9DS2IOv5NJlCC72EXsLY02dIKoCYtEExGwTbtL4/Q6kRGE1Wkvo/Q04HVG/yW7vvngdDoBEoz64ufzoghsRhfMgKgvr2Lrl5uJLJJ0muBcK/B2dNzRp+qv6JsYazaGqPzNdoXreO0jPPwrzmFxLuvIuHZlxKS8YkqNrU3bEsUJQJMBRZoUaTFZWAxCDrDCkkxwQQAtvZAg27IPdwVELFLdNuobKjkvf3vB/wfZ1OSd6NhqjRKJsRj8JMCNk8NAiJZT1RT6F9htQ4PZ02B2ZrZH1NTqdEs9lNdnrjHtmc5qcgSIrWucpMRF4zVjYAKUmp8kZ0CBzGh6wgkEsb9NoFNZfKnb/8eHCckpN/bPgHWTFZzEieR4vZNrCxSBiRnE4an3+e8suvQBUVRcGSd7GeehYIgTltMljboWlvUNdOitYiBDQMhWmow9o9AUMhNVZHu8WOxe7w/WZF7XaZW2bmzGRGxgxe2PpCT+kJP2nrsmF3SiHvpnUaFXEGjXetJvdwuStdmO3HDa46/koIcrAkRw+NibG104ZTcqNVVrl21dn+9dlWu4IOhsY0ZEGjEsRHaeSNaOV6eWMaQQ5ZQdBksvZ9eLodxb4jhlaWr2Rn005unHIjWfFxrusNzgNkNxqpuGYhDf98ivhTTqHgww8xjBvXvYtwKv6MquDUSY1aRXJ0gDb2MGE0hZZMptBdb8gf81Clyw/kEvhCCG6dfivGLqP3TmZu6G5aH47P4KuKqlI5s2az5/cEQajJZArKXGr2UfMp3CjfmVtBoI+XGzn5yVA12GnqkE1bQgh5/ek0yhvTCHLICoLm/oKgcj1ooiDde49eu9PO05ueZmTiSOYVzet2dA7GA2Rev142Ba1bR+YDD5D9xOOoY+UaRsqiF5U5FnRxQQsCkBeyofERhFZnSCE5kKSyqg2QOkpu/OJictpk5uTP4bUdr9Hc1ez3fcMVegm4QmC9/AZZU+S/q8LrSAw1mUwh2ZefY5BQBM9AQbDRlYzn/xKXOlTzoHfhRSUwIMJ+gkNWEAzUCDbID46HjmQKn+7/lLK2Mm6aehNqlXpQaplLTieNxFMjhAAAIABJREFUL75I+eVXIKIMFCx5l6QLzu8TJ60sGslxUfK4QxAESlJZpGkyWbsXkFBQFjKfk1iSZAeiG3PBzVNvptPeySvbX/H7vuEoL6GQHOPjN4hKhORiefxhRNYIQq+Br5iGmiMsCJTvrM9ctnXJNcMCCAMH2eE9FD4CY++1KG0caKMj7ic4JAWBzeGkvcve8+XbrVCzxeeDY3VYeW7Lc0xImcDsvNlATwx4QFErXnC0tFB5w400/P1J4k46iUKXKag/3Q6mGL087trt8gQIgqEqM9HUEXroJQTwG7RVQ0ed2wYlxYnFzCuaxzu73qHeXO/XfcPlaAXZNORz/DnTDnqNIFzzwF+M7kxDddvBaffbP6CQ5io8F+kse7nwomszodbI2l+EE8sOSUHQ3P/hqd8BDotPQfD+nvepMdXw+2m/796dh3MCdG7bRunZ59Dx449k3H03Of94EnVsrNv3GjusRGnVclOdnOngtAXdyWooCs9Z7XI2czgacST7mwuh7KY95IlcN/k6HE4Hi7cu9uu+TR0WVCL47mq9SY7V0Wz2Ueojexq0V0N7bcj3A7m8hMnqCKkfhEKcXoNWLYZAELgJQVa04wA1gtRYPVa7k3ZLZCOfmjosfX+D3OlQuzWilUgPSUEwwMHU3dPUs6PYbDOzeOtijsg8gqOyekpUx+jU6NQqn41RvCFJEsa336b8oouRkCh4602SL7nYa9mCPuqk8sAHaR5KjdXTbrHTZfMj6iZMeLTtBkGcXm6w0+jLYV+9US7gljnR7eG8uDzOGX0OH+75kIr2Crfv6U1Dh/wbhKOZSEqMDptD8l7qQ9FkwqQV9PwGoQtjIeScnMgLAhsxOnXfpjpVGyA2A+KzA7qWklQWSYexsiHqMw9ypoPDKvdSiBCHpCAYoBFUb4LoVEjwXHH07V1vY+wycvPUm/ss0EK4at0EuaN2mkxU/9+fqXvwL0QfczRFH31E1KRJPs9r6l3HPz5bLrUb5AKROgSOvnDU8VdQGuz4/A2qN0H6OLmQmwcWTlqIWqXm+S3P+7xvUwhN6/vjM7sYZAEmVGHzE/SYVbRhuZ4//aPDjdFkGehnqtooL6YB1n8Kt5nXH9xuiBSNNcz+IG8ckoJggEZQvUnebXl4cNqt7by6/VWOyz2OKelTBhxPitEFFTZnKSmh9IILaPviC9JuvYW8555DnZjo17l9NALhCocMQSOAyGZV9jj5wreQel2Euh3F3suHpEens2DsAj7d/yn7W/Z7fW9jiL0UetMjCLz8BroY2ZkYphIEcrGz8Ji2QP4MEXcWm6zdjmpAThhs2huwfwD8FMZhpqm73lavz5A4AqKShgXBYNNHCltNcqaplwXijZ1v0GZt48YpN7o9nhyjDfjhaVu+grJzz8PRZCT/pRdJve46j81j3GHsH/WUM02eAJ0tAY0DAm/uEg5C7fXbn5RYvfdciOYy6Gz2a4H43YTfEaWJ4tnNz3p9X5PJGjaNoKfwnC+H8VR5xxsGh6aS+xJOYRZp01Czud88UPIs3AQE+KI7FyKCn6FnHvR6joSr3tn/miAQQpwshNgthNgnhLjDzfE/CCF2uprXfy2EGNHrmEMIsdn1Z2n/cwcDZbIlRmnlaBvJ6VEQtHS18PrO1zlpxEkcluI+x8Brh6l+SDYbdY8+RtWtt6IfNYrC/35EzDHHBP4ZTP0cTCEkHKUOQeG5btNQmDSCVF8agbKL9qOgYJIhiYvHXcyX5V+y2+i5j2y48iAggKCD7GlywlFL8PWlFJQFL5waQSi+smAwdlj7apXdGcX+FY7sjfI9RNK85bEUe/ZUV7/lyPR4CFkQCCHUwDPAKcBhwAIhRP8VcxMww9W8/gPgb72OdUqSNMX1JwLNfuVdRGK0Fo26l701a6DJB+C1Ha9htpm5YbLnMtPJ0f5pBPaGBsqvvBLja6+RdPHFjHjjdbSZmQGP32y102Vz9p0AyvirgxEEke8wZTRZe9Lqw4DPekPVm0Ct95kwqHD5+MuJ08bxzOZn3B7vsjnosNi7HYyh0t1pzZ8QUgiLw9hotiHCUOtJITlGR4vZht0xeOVWeiNJkisfqJePo2ojJBVCdHLA14vSqYnSqiOqEbg1DYEsCJx2qNsRkXGEQyM4AtgnSVKJJElW4F3gjN5vkCTpG0mSlEIua5Cb1A8ZfeyK1ZtkR2t81oD3NXY28vautzml8BRGJnlOVU+O0dPWZcfmZQKYN2yg9Oxz6Nqxk+zHHyfznrsRuuAmoNuHJzpZ7ssahDpp0KqJ0UV+AiQrafVhICXWR9Gzqk2QOQE0/n3nCfoELht/Gd9UfMOOxoGTsTuHIAxRTyD/BtE6te8NRfp4ucZ+GPwERpOFRFeHt3Cg7Gpb3HVaGwQ6bQ4s9n4bourNQZmFFCKt1RhNVtQqQUJUP4d9t8M4MiWpwyEIcoDesXaVrtc8cRXQuwC8QQixXgixRghxpqeThBALXe9b39DQENKA+5SX8OJAfHX7q1gcFq6ffL3X6yk7EncOY0mSML7+ulwwLjqagiXvkjB/Xkjjd5tEA/LnCLIWTVKE7btNJktYQkcVvPZVcDrl7yVAB+Il4y4hQZ/A05ufHnCs27YbJtMQKFqND61Mo4OMCbJgC5Fmk21g+eYQiLSzdcCGyNQIbZUetXt/iLSfo8kkF15U9RfG8Tly6fEgNPxgCIcgcLedcOvJEkJcAswAHu/1cr4kSTOAi4B/CiGK3Z0rSdJiSZJmSJI0Iy0tLaQBd9cvt3TI5WrdCIIGcwNLdi9hXtE8ChIKvF4vqdvJ1Hcn5DSb5dDQR/5K7PHHU/DB+xhGe28m7+/4gYFhc1lTepyiARLpnVA4Ha3gw7TStBesHQHvFGN1sVw5/kp+qPqBzf+PvfMOc6ysF//nTTLJTJLpvezO9t536aD0osCiwgo2pFu4oHj1yk+vBUFBuHJREARBsVyVXkSkuQtSd5fdZSvb2/Rekswkk+T9/XFyMpmZlJPknMwsm8/zzJOZ0/JmTt7zfb+9beSEVO+Bng/ShIXnVGqXKYItmF7eR9foiJs0yXQF0jH3QH1o1qQuCIozHPnUNdrXp5Jhh7EegqABiAzArwOaRh8khDgT+B5woZQyvOyRUjaFXvcBa4DkvTxJEi7y1LIZkFG/OI9sfQR/0M9XFn0l4fWGe84Or+Z8Bw9y4NLL6HvhBcq/8Q3qfvVLzPnaGt5oGT/EsCtCSquIYntmJ4AedfwjKY3XVyFBRnE8LptzGSW5Jdz/wf0jtuuZEKeieTVas0wRbJ170nq/MZFnaaIuTDJVgbRr9D1oVv19iVtTxqLEnpNx01DMe1CzVGlP6kuuPHoq6CEI1gEzhRBThRBW4FJgRPSPEGIp8BsUIdAWsb1YCGEL/V4GnAQYmk4npaRb1QhiOIrbPG08tvMxLpx+IZMKYieZqZSM0gj616xh/8WX4G9tZdJDD1H2leuSCg1NRNS0ehieACmYh8bDNqqraSieRtC4AXIcUJa8NmbPsXPF/Ct4u+ltNrYNr866QvdaX0GgMfqsJvXAgEi6RodepklJvHtgAF2jTUNNm5TCfBGVZZOlxGEbo9kbSacrTuHFmqVKRGPLFsPHkfbTSUrpB64HXgJ2AI9JKbcJIW4RQqhRQHcCTuDxUWGic4H1QogPgNXA7VJKQwVB36Aff1AqX56mTYotLr9yxDG/3fJbgjLItYuu1XRNVSXucg3Sfu99NHzlq+RMqmPKk0/iPPkk3T9Dp9uH1WzCaRsVcWMvUdoapqBOKslAmZkA4YgbPU1D8XwEzZugehGYUmu+smr2KkpyS0ZEEHWHnHwFufpEPUGo97I7Qb0hgLLZYMlNqzeBuiDSUxAUZ7gC6RgTadOmtMxCoPj7XFqbHOnAiBLUowlHAhpvHtLlWyyl/Afwj1HbfhDx+5kxznsbiF74xSBGlJeI4ihucbfwxK4nWDljJXX52oKbih1W7EMD1P/ih3RsXUfhygup+vGPMeWmX943Gl3xIm6ql6QsCNQJkG63qkTENG2lgd1qIS/HPNY0FAwoK6pll6d+7Rw7Vy64krvW38X7re+zvHJ52MmnV9QTKP8Pnz+I2xcYK+QjMVsUh3HzBym/l7og0lMQ5JhNFCTqtKYjnW4fOWZBvs0S4Si+Lq1rRvr7qgqNnQdDgSC9A0OxAw4KqiG/OiOC4KjLLFYfQmVWr+JEHGUWenjLw0gpuWbRNZqvGdy3l1+98UtKt2+g8vvfp/r22w0TApDIrrhESTZKsqXh8GrOeK2gK1oNeR1QV9Qj6NgFQ560V4qrZq+iNLc0nG3cPTp+XQfCUTdanK01S6B5c8q9qvVOJlPJZNRNd6Qw1sFRDBHafQY+Q9jPFM9XliGH8VEnCNQJUOvZpWyI0Aha3C08uftJLpp5EbXOeBGww/S99DL7P3spDr+XZ7/0vYRVQ/VgRMG50aifJ0mzgfpQy8QE6AiXNtDPNAQxom7UB0QaIYUAeZY8rlp4FWtb1rKuZR1dHp/uD9GweUtL29PqJUqv6q749ZBi0TnarKITmRQEI5pLNaXvKIbMttzs0qIZVy9RFjPefkPHctQJgvA/v2+HsiFiBRHWBhYm1gZkIEDbL+6m8cYbyZ05k/sv+2+2l00zZMyjiasRqBMhSUdiWCPIxAQwSCOIGvrXvEnp+FQ2M+3rXzLrEsryynjggwd0t6/DcAG+TDiMwyZSAzSCjDmLI3NRmtN3FENmHd5Ru6uNpmYJIA13GB99giD0oHN2bVHKTjvKgJHaQI0zfh3zQE8Ph6/7Cp0PPkjRqlVM/uMfsFRUZmwlFFcQ5BUrKfZJawTjoBLr/SCNVg+/aZNSvjlFR3EkuZZcrlxwJWtb1tIZ2KG/aSuZh1D5HKVkRooO4zGhlzqRyQqk3Z6hCI0gfUcxRPoIMiAItGoEkJY/SAtHnyBw+7BZTJhbN49QI7VqA4M7d7H/klW433uPqh//mOpbfozJas2YSuz1KxE3cb88Nck7jDOtEusdcQNRyoGrjuI0zUKRXDLrEkpzSxl0/NOQhyhoFMbmHKVkRooaQczs9DRRQ2Az0e4x3NlLh4xilaK8HITIzIKoy6XBRJpfqZTAMTjD+KgUBJPsfkTnnvAKQqs20Pfiixy49FLk4CBT/vgHij+7KrxPjcM3egIMT+A4X56apdBzKCmHcWFoAmQiK7Tbo3/EDSj3wOMLDHda69wDQ25dVooquZZcLpt9OWbHXtxit27XBbBbzdgsJu0PoeolykoxBYdxt9uH1WLCbtU3MqbEkYMvoEQ+GclQQOnsVayGgUNKCYOjsZhNFOYlX1Y+FbrcPkwiVAU5HjVL0goV1sJRKQiW2RqUP0IrCFUbuHrh1VHPkYEAbf/zPzR+8yZy58xhypNPkLdk5MOlOBT65zF4AmiyK6YQf6xOgExpBHqGjqqM0Wp0chSP5mNVFxD0O3m/72+6XlcIoTi8tQrjmpDDuHt/0u+llpfQXxiH/BwGLyjUe1w6IjE0cWc/LZTYM5NcGbPO0GhUh7HPbdhYjjpB0On2sdgcmjjVi2l1t/Lk7idZOWNl1EihsD/god9S9NnPUv/o78mpqBhzXKZs7JoauqgTomVzUteOamM3AKXYmb6hlzDs8A5/huZNYMlLKaM4Hh6vCV/nx9nn2sSGVn2rQ5Y4rfG7lEWSRsKR3pndKuHoM4MfpCM0Y50cxSqZqjek+R5ULw5lGG81bCxHnSDodvuYFdwH+TXgrOCRrY/E1AYGd+1i/6rPKv6AW35M9Y9/FLN0dKbijzXZdvOKQxnGSUYOpdhyM1n0Lm2gMkYYN21S7OhmfX0RXe4hhrqPoyCnWFNv42RIpsmR4jC2pmQ2MO4eqJFPxva2UDWOYkeOYh7T0fyXKX+fZkFQY7zD+KgTBF1uH1N8u6F6Me2edp7Y9QQXTL9gTBZx38svc+DSywgOeKh/9FGKV62KcUUFzR2m0kRzVm714qS/OMoEyEBCmVv/GHwYlQsRDCoakc5mIQhFlEgrq2Z9kXea3xlTmTQdNFcghVBJ6vkpORLD9bZ0Jhz5ZLBpSNU4Kkxu6D2s633OmGasVRjnVyslqQ30ExxVgsDrDxDwuigbPAg1S3hk6yMEZGBEFrEMBmm75x4ab7gR28wZTH3iSezLEjuhMqcReEMRNwlMK9WLFdtxEj2MSzJQgTQQlPQYtBodUeuma69SoVPHlaKK+hD64rzLKLYV88Bm/bSCpMMvq0MZxkkGKcStcZMGxaP9NAahzrMyVygfKM1EskhUzTgTgR+ahLEQw4EBBnFUCYJu9xBzxSEEko7SaTy+63HOn3Y+k/KVCqOB/n4avvZ1Ou9/gMLPfJr6P/6RnMqx/oBoZG4CDGl3MEFSfgK1OY2RE6BvYIig1L+0AQxHPnV5hgxzFIMygXNzTJTYnVw+/3LeanyLLe36JPyUOKy4IyOfElGzBLy90LVP83sMBYL0D/oNuQcOqxmrxWR4QpaqceR3h7rH6eQoBkWrGQpIXN4Y3e50IBiUSh6E1ntQvRjadsDQgCHjOaoEQZfbx0KT4ij+fd8OhoJDYW3Au28/B1Z9Ftebbyr1gm69FVMSrSQLci1YTMLwCaBkU2pwtKaQiJKJ0D+jEpkgIvLJ7VPUaLMNymfr/j6RDV0um3MZRbaiMf0KUiXpDO8U7rOmGjcpIoTIiGbZ5fZRmJeDueUDKKpX/GI6UZyBwI/+QT+BoNRunqtZAjJgWA/jo04QLBD7abGX8diBv/PJqZ+kvqCe/tWrObBqFYHeXiY/8nBK9YKEEBmJNugOaQQJcZQqmdNJ2I8zUUZ4RPVXAwj3VWj+IOQo1j86KdK+bs+xc/n8y/l347+j9jZOlqRrPlXMBVNOUoIgHHBggEYAmXG2dnlCpi2dHcWQmbpbwwsijd/PNHqNaOHoEgQeHwtM+3m0vAZvwMfVC66i4/77afja17FOnszUJx7HceyxKV+/NEMTQPNDNEmHcSZCYI3KaFUpsVvpcQ0qn9sAsxCMvQeXzr6UAmuBLr4CNepGcxVYiw0q56UkCIwI4YUMCQKXj0l5XqU1q47+AchM3S1NiaGRFE6CvBLDMoyPKkHQ29tLmbmZJ839nF91JtYf3EP7Pb+k4Pzzqf+/P5NTE7/GUCKKMxBtkFSxs+rFSnatxsqFYUFg4ARQJ5cRESvqda39h8Dbp/sDQmX0PXBanXxp3pdYc3gNOzp3pHVtdYWoqQKpSvViZaWo0bejCpmYdfDTJBOCoNvjY6HlkPKHzve5NBwCa1wEXdJamRApRQJqRRdBIIQ4VwixUwixRwjx3Sj7bUKIv4X2vyeEmBKx7+bQ9p1CiHP0GE8szO3b+UuRg8KuAF+8Zxv9r71GxXe+Q83P79Clf4DR7R4VB1MygiC5yoVJ1cNPkU6jzRJ2K9WencofBgmCaOGvn5v7OfJz8vnN5t+kde2UzHPVi2GgWwmj1IAa42+kRmC4s9jtY64MOch11vyKw6Yh43IhulPRymqWKA5jv/7jSlsQCCHMwH3AecA84DIhxLxRh10FdEspZwB3A3eEzp2H0uN4PnAu8OvQ9QzB1L2RTZ1O7vwDiI4eJj30IKVXXqFbmr3RK6HeZCNuwnZFbauITEQ+dYcibvJ0rnGjUuywMsW3G2nKgYrRX8P0UWvcjBbG+dZ8vjDvC7x26DV2du1M+fpFdutw5JNWknQYqytdI6KGQJkH/YN+hgKpNc1JhNpmc+rQHiioC1cQ1gunzUKOWRirEaQSNFG9GIJDijDQGT00gmOBPVLKfVJKH/BXYOWoY1YCj4Z+fwI4QyhP35XAX6WUXinlfmBP6Hq6I6XE+97LfOtxia2qlqlPPI7zJH37CRc7rPQODOE3aAIk/eVRKxdqfEDk25TIJ2N9BEmEzKVAiSOHuewnWD5XSbjSmXimrc/P/TzOHCcPbn4w5eubTYIiNfJJK5XzQZg124+7PT4Kci3kmI2xDIdrPhn0PVLbbNYM7DJE6xNCUGxw5FN3qApyXk4SC6IZZ8G3dhqSG6PHN6EWiNRJG0Lboh4TanbfC5RqPBcAIcS1Qoj1Qoj17e3tSQ9SCIG9z8+e6RZmP/4k1kmTkr5GIkrsOUiprNyNIKWIm5olmh8Q4cgng30ERvkHAIrzclhgOsBA6QJDrq/a16MJs0JbIZ+b+zleOfgKe7r3pPwexclqljl5SrkJjQK/Mxk/UwoY7WvqdvtwMECh56AhD0Uw3syrFl5Myhphc0J+lSHj0UMQRPsko71WsY7Rcq6yUcoHpZQrpJQrysvLkxyiwsV/Xssnn92IyeFI6fxElDiT6DCVAp2pCILqxdCxE3weTYcbHflkVLEzlVpTB8XCRXfRfEOunyji5otzv0ieJS8tX0FKJQ6ScBgbVV5CxWhfU6fbxzxxEIE0zA9ktJlXc1ZxhtBDEDQAkcvrOqAp1jFCCAtQCHRpPFc3hMWCRecCZJEYXWZi2MGUpCCQQWjVVrlQUYmNs40m5exOgSq3Yp9vdeifSAaR5Y+jR9wU5RZx2ZzLeOnAS+zr1Z7tG0lKWlnNEnC3Q39LwkONKi+hkgmNYIFpuIKwERidE5RK0b+gDNLqbjVkPHoIgnXATCHEVCGEFcX5+9yoY54DLg/9fjHwL6nUMXgOuDQUVTQVmAms1WFM44LRXb7CPoJkbOxJOhKViA/joiWMKjinUtq3A780cThnqiHX1xKD/6X5XyLXkpuyryAlrSyJhKNug++B0fkoaj5QwFFpmKmk1GDTUCr34NWDr3LeU+fpkrg4mrQFQcjmfz3wErADeExKuU0IcYsQ4sLQYQ8DpUKIPcBNwHdD524DHgO2A/8Evi6lNLazi4EY3fi62+0jL8ecXMRNQQ3YyzT7CYodOXQnE7GSBD6/UuPGSI3A3rWN3bKWjkFjHKFhQRBnEpfklnDp7Et5cf+LHOg9kPR7pFT0rHIBIBIKfCmlYSWoVdSOW4YJArePBeIAskq/+kKjKbZb6fEYGPiRpIk0KIM8sPkB6vLrmFMyR/fx6DJbpJT/kFLOklJOl1LeFtr2Aynlc6HfB6WUl0gpZ0gpj5VS7os497bQebOllC/qMZ7xIhx/bKBtNOkJLESo1Z1GjcBupcfjIxDUv/Bcj8HJZEiJpfUDtslphq3mutw+8jVE3Hxp/pewmqw8tOWhpN+jxK4UPetPpuiZzak04Elwn92+AD5/0FD7tMVsoshuXLtHV18vM0Qj5lpjHMUwvKjrMSDwI1YIcjxWH1rN7u7dXLvoWswm/UOvj6rMYqOxWcw4bRZDNYKUkoCqF0P7DhgaTHhoscNKUCpVQvUmJdNWMvQ3I9zt7M+ZYZh9V6uPoyyvjEtmX8IL+17gcJ+2RC+VcD5HKuahBJpfl5ZWpzpgZE1/W9d2zEIidOhRHIuU74EGekIat1ZhLKXkgc0PUF9Qz7lTztV9PJAVBLpTYmD4ZZdHY8G50VQvgaAf2hLbFo109Bld40ZdDTflzTLULKH1Hly54EosJkvSWkFpqjb26sXQ3wSutpiHqP4fI53FYGzUTUmv2oPAOI2g1EAzb7LlJdYcXsOHXR9yzcJrsJiMCXbJCgKdMXICdKca7aHGWmvwExiZDBSOwTfqIdS0CRB0588xTBh3e7Tfg7K8Mi6edTHP732ehv4Gze+RchlkDZnk3alktKZA0rkQSVDl+ZBeU6Hi/zKIMf2vdSSZBZGUkvs/uJ86Zx2fnPZJ3ceikhUEOlPqsBrWpi/l2OPCSUq9dg0RJeoEMGQlZPRDqPkDKJuFI7/AQGE8lNQ9uHLBlZiEid9u+a3mc1IOQ1abs8QR+Op306iCcypG5qNM8e6mMW+24v8yiFKncfMgGWH8RsMb7OjawbWLrjVMG4CsINAdo1ZCXn8Al9efmn09iVZ3xmoEiSNu0qJ5E1QvNjTyqdPtTUqQVdgr+Mysz/DsnmdpdDVqOkddKSat1eQWQsn0uAI/bJYwoClNJIa1exwaoF4epiN/rr7XHYWRvTm0lmJXtYFaZy3nTz9f93FEkhUEOqPGH+s9AVQHU8oTuHoxtG5PWLnQaB+BloiblOhvhf5mqFliWOTTgC/A4FAwaUF25YIrEUJo1gqcNgtWsym1omcJSop0uX1YzSYcBhX9U1HbPSYV+aSBoaYtWAjSX2xM5riK1WIiP9diyKJO64Lo343/ZlvnNq5ZeA05JoP8aiGygkBnShxWfH792z2qKn3KETc1SzRVLszNMWO3mg0JgTW0vIS6Cq5eEo580rvmU9JdpUJUOar49MxP88yeZ2h2NSc8Xqn5lJNaGeTqJdDXAO6OqLvVe6BXxd1YGLWiHjz4PgDeCuNyCFRKDSqn3alhQSSl5IEPHqDWWcuF0y+MeZxeZAWBzoQdfTo/SNNu6BLOMNbmJzBCI+j2GJjRGnIUU73IsMzWdExbVy+8GkBzBJHS5ChFjQBiagVG13pSKTHIxh5o2kSXdJJbWq/rdaOhBH7on2WvJQT5zcY32dKxhasXXk2OAe1WR5MVBDozHHam7xco7RaPxVMUG7KGyKFSpzEOb8M1grKZYMs3rNVghysUepmCea7KUcVnZn6Gp/c8rUkrSDkMORw5tDHq7k63L6XxJ0uJQRqBpXUzW4NTKTbY2Q1KG0mj5kG8xYSqDVQ7qlk5fXRFf2PICgKdMareUNphf0m0ujMq4sPQGjdNG8Naj1EagXq9VCNuVK1Ai68g5aJnuYVQMi2uRmBknSEVQ8qtDA1i79nFVjk1I1qNYfMgQQjyW01vsbljc8a0AcgKAt1RHxJ6ryTU66l1XFKiejG0boNAfJNDicNmTPy0x6DVaISjGIyLfEo34kbVCp7a8xRNrvhFdtMqehYnQixjpiEj7kHbdkzSz5ZgZgRBidOYyKd4IchSSu7fdD9Vjio+NeNTur5vPLKCQGeG+53qrxEU5uVgSSfipnoJBLzQ/mG5CEVrAAAgAElEQVTcw0qdVjpcXl0nQKoRN5qIcBRDRDKQ7qYhHzlmQb4t9XhurVpBWkXPapYo/YvdnSM2qyHIRmcVA9itZmwWk77zIHSft8ipFNmNXymrkU99g/pGPsULQVa1gWsWXpMxbQCygkB3wqF/Oj+EdFnJqbVZmqLbj1VKHFa8/iAeHSOfVJ9JshE3mohwFAPkWc3k5ZgN0Ai8lDpsaUXcRPoK4mkFaRU9CwcGjLzP4czuDPgIhBD6Z9k3bcJjzqffVmNYm81IjMgujrcgklLy602/psZRk1FtALKCQHfCE8CAqKG0BUHxVLAldhinXOsmDt1GNkxv3gSlM8CWH95UYkDon15mFS0RRGkVPVMdxqPuc6bqDKnoLgiaP+CQbWa4E6DRqAJTz8iheCHIaqTQNYsyqw1AVhAYghH1hjpdOjj5TCaoWZxQIzAivd7Q8hJNm8b0ri12JNkAXgMdLn18HKpW8MzuZ2JmG6cljPOKlCixUaHCWnop6ImufX/9Xmjdxm7TdIozYBaCiAhAHRd1w33HRwozVRuoddZmLFIokqwgMAAjVqOKRqDDBKhZqrSt9MceX0nY4a3fSiilNpta6G9VKm6OqkRphMNbbTiuB1cvvBohBA9tjq4VpG2WqF4CTSMdxuGopwyYhkDnBVHrNggO8UFw+piHqFEYEX02HAY+ci7/u/HfbO3cmnHfgEpagkAIUSKEeEUIsTv0WhzlmCVCiHeEENuEEJuFEJ+N2Pd7IcR+IcSm0I9xdWUziN6lqKWUSRc7i0nNUgj44pakNqIEr3ot3XsRqKveUbXpywwzDenzEKpyVHHJrEt4Zs8zUfsVpF3qo2YJ9B4CT1d4U1eM1ahRFOvZk6BpAwDrh+qN8TNFIRwBaIAgiNTKpJTct+k+JYt4hvFZxNFIVyP4LvCalHIm8Fro79F4gC9JKecD5wL/K4Qoitj/bSnlktCPtn6KExy9fQRuXwBfIKjPalSDw7jUqf9KqNPlxWISFKYT/hqNUY5iFb2T4gaHQhE3Oq6mr1p4FRaThd9s/s2YfeHCc+loBDDiPne5fZgE+t+DGJQ6rPQP+vH5dWj32LQRmVfCNk9hxgSZGnRgjEYw/D1afXg12zu385XFXzG8plAs0hUEK4FHQ78/Clw0+gAp5S4p5e7Q701AG1Ce5vtOaEodVvq9frx+faJuVKGii223qF4pSR1HENitFnJz9A3963QpjlaTSecaN1EcxQClThsDQwE8Pn1C/4aTyfQTBBX2Ci6ZdQl/3/d3DvYdHLFP7XaXUpkJiCg1MXyfO90+iuxWzHrfgxioGmyPHtpx40YC1UsZChgUeRYDvf193R5FGBfkKp8hKIP8etOvmZw/mfOnGVthNB7pCoJKKWUzQOi1It7BQohjASuwN2LzbSGT0d1CiJiiXghxrRBivRBifXt7e5rDNpbhiA99ip7p6mgVQtEKEjmMHbZwSQU96HR7KTUi2iOKoxj0d/SlXeIjBlctvIocUw4PfPDAmH0pF54DRdiXTBupEbgyk0ymopuJ0eeB9h24SxcCmXN2g6JZ6q0RFNuHF0SvHnyVnd07+crirxjabyARCQWBEOJVIcTWKD9JubaFENXAH4ErpJSqrngzMAc4BigB/ivW+VLKB6WUK6SUK8rLJ7ZCoXf4pe6O1pqlShXSoYGYh+g9ATpcPsr0dlL2NSuO4pplY3aVhYSOXsIsnTpD8SjLK+PSOZfyj/3/YF/PvhH7SuxWutLpq1CzDBo3hP/MVFaxim59f1u2gAzSUaiUns6UsxuM0QjUexAIBvj1pl8ztXAqn5j6Cd3eIxUSCgIp5ZlSygVRfp4FWkMPePVBH7VZqhCiAHgB+L6U8t2IazdLBS/wO+BYPT7UeKN3tIHuZomapUoP49bYDmO9J0Cn2xt+OOtGyIFI7VhBEA6B1VkjMKKz1xULriDXnMt9m+4bsT3lekMqtcsVQdnfAoRKfByJGkHoPjc75gCZ1Qj0ngcdruEugy8deIm9vXv52uKvYTYZ2x8iEemahp4DLg/9fjnw7OgDhBBW4GngD1LKx0ftU4WIQPEvbE1zPBOC4YJb+qxGu3TXCEIPzjjmoRKdW252ugx4CDVuAGGGqrG16VUzlN73wIis3JLcEr4w7wu8fPBlPuwaLv+R9kNIFZAhrWDcNIJ0fQRNGyG/muaAEpRodJvNSJSeBPqZSDtcXsqdNvxBP/d/cD8zimZw9pSzdbt+qqQrCG4HzhJC7AbOCv2NEGKFEEItprIK+Bjw5Shhon8WQmwBtgBlwK1pjmdCoLtG4PFhMaVX42YEBTXgqBhhNhiNnhPA4/Pj8QX09xE0bYCKeWC1j9mlCp0OnYSZHnWG4nH5/MvJt+Zz38ZhraAk3fDLqkWKoGzaQCAo9clOT4KivByE0EEra9wANUsjenJk0llsY3AoqFvQQWfIRPr83uc50HeA65dej0mMfzpXWt9qKWUncEaU7euBq0O//wn4U4zzT0/n/ScqRXYrQuhXebErpE7q1lVKg8O41Dk8AezW9B5+4Ybpeq6mpVQeEPOix13n5ihRN/qZhtKvMxSPAmsBX57/ZX618Vdsbt/MovJF4cgnt9ePIxUBZLVDxVxo3ECPx4eUBmV2x8BiNlGYl5OeRjDYB527YdEqutxD5JgFToOEcTQigw7sJem9r88fpHdgiCKHifs/uJ8FpQs4fdLEeASOvyj6CGI2CYrt+iU0dbgMsK/XLIWOneB1Rd1d4tDPxq46WnV1Fnftg8EexQ4eg1KnflpNJswqn5/7eYptxfxq46+A4f9XWg7vmqXQtIHucNG/zAkC9f3SmgdqOe2aZXSFqnYa3WYzkmIdtXv1u3jYv5pmdzP/sfQ/MvpZ4pEVBAZRbM/RzTSkCAKdJ3DNUpBBJSIjCnpmF4c1Aj1tu6o2EyViSKVURz+HXnWG4uHIcXDVwqt4t/ld1javpSxfh8in2mUw0I2rRYnYzqR9HRTzVlqasRoQULOULvdQRh3FoK+Zt9PlA+Hj3a7HWF65nBNqTkj7mnqRFQQGUeqw6agR+Cg3QiOA4Yk2iuEJkP6KOlz1Us8HaeP7YMlTTB8xKHXqlwuhZ52heHx29mepsFdwz4Z7KAu9X3t/Gt+jkKCUIX/QeGgEaT1EmzZC0WRwlCrmuQyGjoK+C6J2lxdr8Tv0D3Vxw9IbJow2AFlBYBgl6Yb+hZBS0u7yUp6vsyDIr4SCOmhYH3W3aorSxzRkgEbQuEEpKxGnQFeZUz/znJ51huKRa8nlq4u/yuaOzex2vQekqRFUzgezDVubUr3liBMEIUcxQLdnHDQCHUtRN/R0Yi1bw/LyE1hWGVuTHQ+ygsAgSnRKyOr3KrVadPcRANStgMbogkDPnrOdLh8Oq5k8q06x0gG/YjuOYxYCRfB0uX0Eg+l1WjOizlA8LppxEfUF9fxx5wNAMD1BYM6BqoUUdimR2ZmMuIHhAowpdbtzd0DPwfB97nB5M5oHAZBvs5BjFrrMg9eaH0OYB/jG8ht1GJm+ZAWBQZTYlQmQ7kOooz/kaM03YALUrYCeQ+AamweoZ6vBTrc3bO/WhfYd4B+I6ygGxRQVCEp6U+nyFYERdYbiYTFZuH7J9ezp2UNB+db0zVu1yyh37aDQZsJmyWziUkmo3WO/N4Xwy8b3lde6YxgcCtA/6NdfM06AXo2m2j3tbOp9nmD/EpZUztdpdPqRFQQGUeKwEpQpthqMoF0VBEZoBLUrlNco5iEhBGVOmy6mId2TyRpjZxRHoldSmVF1huJx9pSzmVMyB1PxS7T1edK7WM0yrMEBltijJv4bSlrRZw3rlDyImiVhYZhpQQD69Lb4zebfEMRPoXf8CsvFIysIDEKvUs6qfd0QQVC9WJloccxDethGO1w6F5xr2gC5hUpRtTjo4mzFuDpD8TAJEzcuu5GAuZO93lfSu1hIYC637EtwoP6kVfOpYR1UzgOrI7wgGg9BUJpmCOzhvsM8uetJSoKnUJFbp+PI9CMrCAxCr7Cz4Rh8AyaA1a44E2M4jPXqtKZ7wbnGDYrdOEHUhd4aQaZDL0+qOYkiMY92ywu4fNHzPTRROpN+4WCR3KPf4DRSUaD8z9QHuWaCQeU+1x0z4vxyZ66u49NCug7vX236FRaThZy+s42ZxzqQFQQGMdxqML2HUIfLi0kYaJaoO0YJ0QuObR6iRxx+MCjDWbm64PNA2/aEZiHQr/CckXWG4iGEYJnz80iTi99v+33qFzKZ2MpMZg3t0G1sWlHDnpMWBB27wNsXNl+2uwz0lSUgnQjAbZ3beHH/i3xx3hfp7s+jfBzGr4WsIDAIvRrAd7iUbErDmonUrVAmXMeuMbv0KEXdMzBEUOpoVmnaqFROnXRcwkOLQ6U+0u29bHSdoXjMKp7HUO8iHt32KO2e1PpwSClZ759OlXc/ePt1HmF8ikONcNr6B5M7UTVXhjSCjv7x0cqU90yt0ZSUkl+s/wXFtmIun/dlfRdEOpMVBAahV4mG9n6fseqk6jCO4icocaTf5aszbF/X6TMcVmLr1QdEPMwmQYndSkeawszoOkPxKHNa8bafzVBwKGrzGi24vH7WBWZgIjgciZMhTCZBmdOavEbQsE7xA5XOAKDdNUixPQerJfOPLFUTTLbR1JuNb7K2ZS3XLb4Ov99GUOpcZkVHsoLAIGwWM4V5OcmvhEZhSJ2hSEpngK1QmXijd+kgzMLObr1MWw3rlDHbSzQdrvQuTt9HkOlELJUypw05VMZpNSt5cveT7OtN3uHb4fKxKag8UKPdZ6Mpz7elIAjWK+HBJuUR1d5vQFKlRoazi7V/hkAwwC/e/wWT8iexataq4XkwTp8hEVlBYCAV+Tba+tL3ERi6ijCZFHt7w9iVoh4O7+HyEjpMACnh8Fqo096/qNSRfghsJuoMxUL9v32s/DJyLbncvf7upK/R1jdIHw48BdPhcOYFQUV+btjGrwmvS/EDRWh94ykI1IzyZObBc3ufY0/PHm5cdiM55pxhzThrGjr6qCzIpS3ZlVAEUkrjNQJQJlzbNvC5R2zWIwS2Mxz+qsODtHs/eDpgUhKCQIcyE5mqMxQN9f/m9dq5euHVrGlYw9rmtUldozX0HfTVrFA0glSyfNOg3JmkRtC0USmIGCkIMjEPYqAuiLSGwHqGPNy76V4Wli3k7Hql6Ux7OA/iI2gaEkKUCCFeEULsDr0WxzguENGU5rmI7VOFEO+Fzv9bqJvZR4aKVFTiCNy+AINDQePVyboVysRr2jRis7p6SSeztTMU9VSkR42Yw6EHYBKCoEyHwnOdLm9G6gxFIzIO/wtzv0C1o5q71t9FUI6N8opFW59inrTWHwcDXdC515CxxqI830aHy0dAa5a96q8KZY5LKenoN6DwokYqQyGwWrX7R7c9Spunjf9c8Z9hv5Kh+UA6kK5G8F3gNSnlTOC10N/RGJBSLgn9RHYSuQO4O3R+N3BVmuOZUJQXKIIgpTorRJSXMPrLE84wHmk2KNFBI+gI2dd1iXo6vBas+VA+R/MppQ4r/YPJR3yoDA4FcPsC42Yays0xk2+z0OHykWvJ5YZlN7Cjawd/3/d3zddo7/ditZjImxYqe5xhP0F5vi3cIU0TDeuhZHrYD+T2BRgYCoybachps2C3mjVp963uVn637XecXX/2iMJynS4vFpOgIDeztZ60kq4gWAk8Gvr9UZS+w5oI9Sk+HXgilfOPBCryc/EFgvR4UiszYUhDl2g4SpWJp0bkqJutZqxp1hvqdOkYMtewFuqWQxKNvlUbe6qfIdN1hqJRlm8LmxY+MfUTzC+dzz0b7mHAP6Dp/LZ+pU+uKJ8DtgLl/5hB1Ae4Ju1YSkVQjfIPRF4n0wghqMi30dqXOPDjlxt/iT/o5xvLvzFiu5Jdb8VkVBh4mqQrCCqllM0AodeKGMflCiHWCyHeFUKoD/tSoEdKqcYmNgC1aY5nQlER+uKm6icwNKt4NPUnwKF3RySWCSEoc1jT6vurm6PV64LWbZryByJJN6lMPW+8ooZAWQio2qFJmPj2Md+mzdPG77b+TtP5rX2DinnDZFLMLRl2GFckIwh6DoKrVTFXhhhvQQBQUZCb0DS0rXMbz+19ji/M+wKT8ieN2Kdk109MsxBoEARCiFeFEFuj/KxM4n0mSylXAJ8D/lcIMR2IJhpj2lCEENeGhMn69vbUEmsyzbAgSC2EtD30EKrIxASYfELIfrx7xObyfFtaIbCdetUZanw/5EDU7h+A9Ns9GtJUJ0lG+zmWVy7n3Cnn8sjWR2h2NSc8v63fS0V+qDTDpGOVwIAMJpaVJ7MgOviO8jp5uHuXoYUXNVJZkEtrnHkgpeTna39OSW4J1y68dsx+3eaBQSQUBFLKM6WUC6L8PAu0CiGqAUKvUcsbSimbQq/7gDXAUqADKBJCqOmadUBTnHE8KKVcIaVcUV5ensRHHD8qCpTJl2oIaUe/F2FkeYlI1Il36J0RmysLcjWpxLHQrfKoas6oi196ejSqWSpVjUD97OEH6TigCIKR479p+U0A/OL9XyQ8v61vMFzzh7pjFYHaGL0znRGUJVNm4tDbSiJZxbzwpvbQA3g8NYLKUCh4LH/fSwdeYkPbBq5fej1Oq3PMft3rbelMuqah54DLQ79fDjw7+gAhRLEQwhb6vQw4Cdgulf/oauDieOcfyagr+XgriXh0uLwU261YzBmI8i2ZBo7y4RVZiKrCXFp6Uxv/4FCAfq9fnwlweB2UzYa8qIFpMRku9ZGaMG7pVc6rLBhfQdA7MITPP2y2q3ZWc+WCK/nngX/yfmvsbOHBoQB9g/5hrVIVpIcz5ydw2Cw4rGaNguBdmHR8OJEMlIeo2SQy3p0sksqCXAZC3+fReIY83LX+LuaWzOXTMz49Zn/GwsDTIN0nzO3AWUKI3cBZob8RQqwQQvw2dMxcYL0Q4gOUB//tUsrtoX3/BdwkhNiD4jN4OM3xTCgcNgtOmyV1jcDoZLJIhFC0gigaQd+gnwFf8lE3YUdruhNAdSAmETaq4rRZsFpMKWsELX2DlDmt41LaQEUttDZamF2x4Aoq7ZXcsfYOAsHo90f97qnaKXnFymr70NvGDTgK5REO75i4O5SaV/Ujm7q39yudyQyrt6WBinAI6dhF0cNbH6bV08rNx92MOUogg8vrx+sPfnQ1Aillp5TyDCnlzNBrV2j7einl1aHf35ZSLpRSLg69Phxx/j4p5bFSyhlSykuklPp0Gp9ApJNL0N6f4VXE5BMUZ13fsIWuKvQAaUnBPNSpV+x0+07FfzH5+KRPTdfhrThax08bgIhcglF9FfIseXxrxbfY0bWDJ3c/GfVc1b8zws9UfxIceg8C6TVNSgalzESC75C6CJl84ojNhvTsThLVNNg6alF3uP8wv9/6ez457ZMsrVga9dyJnkMA2cxiw0nH2ZrxSAP1QXvo3fCmqsKQIEjBPNShl6P1wL+V1/qTUjq91GlL2TTU3DsYFobjRbzmLudOOZdjqo7hng330DXYNWa/6qAd4eOYchIMuZW+zxmiIl9Dlv3Bd8CSG25WrzKe5SVU1KSy0f6yO9fdidlk5pvLvhnzXN0LLxpAVhAYTEUaZSYyblesWgQ5jhHmIXU1nIowC2sE6eYRHHwLCuqgeEpKp5c6rSlHDbX2DVJZOL6CIFzTP8pnEELwveO+h2fIwz0b7hmzX31wqQ8yYHjFffAt/QcbA02F5w69rSQ3WkYuHDKuGUchHPgR8RneaHiD1YdXc+2ia6l0VMY8N2P5QGmQFQQGU5Eg2iAWHp8fjy+Q2UYcZosSvx0hCNLSCEITIK2GLlLCgTeVVWyKZaBTLf7n9QfocvvGXyPIjx8CO71oOl+c90We2v0UH7SPXOW39SsZrSMcrfmVUDoTDmRWEPQP+hkciuFr8rqgefMY/0AwKOl0j79G4Az5+1TBOuAf4Kfv/ZRphdO4fN7lcc9Vw8DHq0SGFrKCwGAq8pWa/q4o0QbxUO3BGV8J1Z+oJG4N9gLDEyAVH0FL7yD5ofNTpmM3uNthyskpX6K6MI92l3dE1I0WVOEx3oLAblVKHMRzeF+3+Doq7BXc9u5tIxzHbX3KQ3RMRuuUk0IJhKmV3kiWhJ3KGtaCDIzIHwDoHRhiKCAnxEM0ckHx0OaHaHQ18v3jv0+OOX7ZCNU0VDyOSYmJyAoCgwlHGyRpHgpXK8z0BJh8vBJnHlGPprJAW3r9aJp6BqguSvMhmqZ/AKCmKBcpx9p3E6EKv/E2DUHi4nmOHAffPubb7OjawV8+/Et4e1v/YPSExPqTwNsLrVuNGO4YEiaVHXwHhGlMZNhw1c4JIAhC82Bf7z5+t+13XDDtAo6pStwgSQkDzyEnE2HgKTJxR/YRQXXSJWuayGh5iUhqV4Awj8gnSDWXoKl3gJqivPTGc/AtyK9R8hxSRB1DU4+22jwq6meunhCCILGf45z6czip9iR+ufGX4Yzjtj7vcOhoJKpgzZB5KGG9oUPvKD4qW/6IzROhvISKml3803d/Go7Y0kKH0V0GdSArCAwm1TITYUGQ6frlNqfSqGb/G+FNlfm5Y8LmtNDcM0h1YRqCQAf/ABAeQ3OSwmzY0ToRBIFtTPjoaIQQfP+47wNw23u3IaWMrREU1irO9ww5jMP1hqIJM79P0UDrTxyzayKUl1CpLMilQ77Fey3v8Y1l36A0r1TTeWrBuYlMVhAYjLoaSzaXYDybdTPtVKW2T8hPUFmolJkIaq0nj5LR2un2UZuOaahzr1KALA3/ACimIVA0lGRo7h0kL8dMQW7mm9aPpixfW1+Fuvw6vr7k67ze8Dov7n+Zbs9Q7PIY9SfBwbdHFBo0ihKHFSFizIPG9eAfHOMfgImlETjyBjCXP8+isiVcPOvixCeEaO5Nc0GUAbKCwGAKci3YLKak7dMdLi+FeePTrJtppymOuwNvAoqz1B+USXX6UlffaZmGwv6B9ASB3WqhMC8nedNQ3yBVhbnj0rR+NGVOG10eH/5A4of25+d+nrklc7l97c/ANDAydDSS+pOURL32D3Ue7VgsZhOljhhN7PeuVvwDUz82ZleHS+mlMBGE8Xt9D4PwcdWc72AS2ualPxCkpW+Q2nRNpAaTFQQGI4SgosCWtLO4rX9w/FZBdcdAjh32rQGGTSPJCDP1oZvWSujAm+CsgtLpqV8jRE1RHs09SZqGegdjP0QzTHWh4vDWEr1lMVn44Yk/pMfbja3yheGCc6OZEvITZMg8VOaMkV28b7VSHjuvaMyudrWXwjgL49WHVrOl5w18HWeQE6zSfF5L3yCBoKSuOCsIjnoq8hPXMh9NQ/fA+H15LFZltRgSBKnkEjSGBEHKKyEplQdUmv4BlZrCXJqS9BG09I1/VrHKpGI7oHwvtDC/dD6nVq3CWrSeJu/G6AcV1UPhpPB9NpqKgtyxGsFAt2KGnHZa1HMmQnmJfl8/t757K1Pyp+Pr/HhSc1m9X7VZQZClIoUyE4e7POO7iph2qlIArLcxpXpD6uq7sjDFSdyxG/qb0/YPqFQX5dKchI9ASklbn5eqCWLbVb8Lh7s8ms9Z6lxFwFvBwx/eSZ+vb+wBQsCMM2Df6xmpOxS1if3+fyvhytNPj3rORCgvcfva2+kc7OSHJ/wYMCdVTbixO80FUYbICoIMoAgC7auI3oEh+gb94VXguDDtVOV1/+uUOa2YRPKmofJ8GzaL9raSI9jzivI6/YzUzh9FTVEePZ4hPD5tiX1dbh++QJCqCWIaqinKQwjtGgFAlzuIr/kSurwd3LXurugHzTgTfP0ZKUutViAdkWW/b7XShzqiI1kk411e4l+H/sVze5/jqoVXsaJ6Mfm5yVUTVjXjtMOoDSYrCDJARUFu/PT6UTR0K6u+SSXjKAgq5yv9CfatwWI2UZ5vS8o01NQ7QE068fe7X1H6DxTXp36NCGoK1VwCbZ9BdXZXTYAcAgCrxURVQS6Hu7VrBK19g5RYZnDF/Ct4es/TvNHwxtiDpn4MTBbY86qOo41Oeb6NoYAc2cN7778UrS9Kdq7XH6DL4xs3P03XYBc/fufHzCmZw1cWfQVAc+9ilYZuD+X5NnJzUlwQZYisIMgA4axKjSuJw13KKmJcNQIhFK1g3xqQkqqC3KRMQ009aSST+dyKf2DmWamdHwU1KUyreWgi5RCo1BXnJaURtPV7qSiw8bUlX2NG0Qx+8NYP6BzoHHlQbqHSBzpDggAicgm69kP3gZhmocNdA0gJk8dhQSSl5NZ3b6Xf189tJ98WLiNRmWQRycaecfT1JUFWEGSAZCt4qhrBuH+Bpp2qxPG37UiqZaWUkubewdQFwf43IOBTzBY6kWx2sSr0JopGAMrCoDEZQdDnpTI/F6vZyh0fu4N+Xz8/fPuHYwsgzjgDWjZDf6vOIx6JKoxVcwn7Viuv06M7ig91uQGoL3UYOq5oPL/veV45+ApfW/I1ZhXPCm9PtnVrY/fAhPcPQJqCQAhRIoR4RQixO/Q6po+gEOI0IcSmiJ9BIcRFoX2/F0Lsj9i3JJ3xTFQqEtVZGUVD9wBOm4Uie/xiVoYz9ePK6741SZWZ6B0YwuMLpF6aYfcrSjnsKJmmqaLkA2g3DbX2DmISE6tiZF1xHs29AwxpyCWAYY0AYFbxLL65/Ju83vA6j+96fOSBqsDd+y89hzuG+lJlZX+wwz38fgV1UDoj6vEHOz0jzssUB3oPcOu7t7K8cjlXzL9ixL6KAu3VhINBSVPP4ISPGIL0NYLvAq9JKWcCr4X+HoGUcrWUcomUcglwOuABXo445NvqfinlpjTHMyEJCwKNKwk1Ymi8Y6cpmqSUK97zSlItK9WHbUorISkVQTDt42DR7yGcYzZR7rRpNg0pLSptmekXrZG6EjtBiaZ8CH8gGCrfPCyMPzf3c5xYcyJ3rruTfT37hg+uXAiOCsPNQ+VOGw6rmQOdHiCwwKMAABrKSURBVKXq6f43FG0gxvf8YKcHh9VMaQardvoCPr7zxnewmq3cfsrtY1pPVubn4gsER/o5YtDu8uILBKkbTxOvRtL9lq8EHg39/ihwUYLjLwZelFJq93h9BCi2W8kxC1q0+gi6PePrKI5kzidg/xtMylOyirX4CcLJZKkIgo5d0HtIV7OQSnVRnuZ6Qy193glRbC6ScAipBodxh8uHlCNbVJqEiVtPulUpmPb6txjwh4SiyaSYh/b+y9Cy1EII6ksdHOh0Q8N6pYRJDLMQwMFON5NLHRldEN2z4R52dO3glhNvocoxNnFM1bC0hJCGTbwfddMQUCmlbAYIvVYkOP5S4C+jtt0mhNgshLhbCBFzCSiEuFYIsV4Isb69vT29UWcYk0kwqcTOAVUljoOUksNdA+PrKI5kzgUQ9DO7X2lfqcU+qtb0qUmlztDuUNiojo5ildqi3GH7dAKUrOKJJQiGk8oSC4LGHuWY0feg3F7O7afczt6evdz27m3DO2acqZSbaDJWKZ9a5lDmwY7nwJQTV+Af7PJQn8EF0euHX+cP2//ApbMv5fTJ0R3YYX+fhkWd6tgfd1+fBhIKAiHEq0KIrVF+VibzRkKIamAh8FLE5puBOcAxQAnwX7HOl1I+KKVcIaVcUV5ensxbTwimlzvZ2+5KeFyX28fAUGDifHlql4OzitoWxWygSRD0DJJjFqm1qNz9MpTPgaLJyZ+bgOpCpcyEFvtuc+/AhHIUg+JsNZuEpsihvW3KomN6uXPMvhNrT+TaRdfy7N5neXr308rGaacBYjh/wyCmlNlp6PYgdzynaAO5hVGPCwQlDV0DGfMPHOo7xM3/vpm5JXPjlpeuzNeeXBnOrp8oczkOCQWBlPJMKeWCKD/PAq2hB7z6oG+Lc6lVwNNSyrBxTUrZLBW8wO+AY2OefYQzvdzJgU53wqJhh0OTfMKYhkwmmPMJHIfWYMOnyWHc1DNAdWHe2K5YifC6lLr0BpiFQHmQDgwF6B2Ib98d8AXoG/RPOI3AYg7lEmjILt7b7sJqNsW0T3918Vc5ruo4bnvvNnZ27QRHqdIUZsff9R72COpLHcyWBxA9h2DuBTGPa+kbxBcIZiRiaMA/wDfXfBMhBL849RfkWmLf96rCXCwmoUm7b+geoNieg906/gXzEpGuaeg5QG3YeTnwbJxjL2OUWShCiAgU/0Jm2iWNA9PKHQwFZMLVnDrJJ5VMoFXEnPMRQ27OtG3XtBJq7h1I0Sz0khI2OuvcFAaZGNV5ncg8FA4dnWCCAJTvhSaNoN3F1DIH5hjC2Gwyc/vHbqfAWsCNq2+ke7Ab5n8KWrdA+y69hx1mapmDc81rkcIMsz8Z87iDnWroqLELIiklt7xzC7u7d3P7KbdTl18X93irxcSUMge72xJr943dA0eEoxjSFwS3A2cJIXYDZ4X+RgixQgjxW/UgIcQUYBLw+qjz/yyE2AJsAcqAW9Mcz4RFVdETmYcOh3MIJtAXaMopYCvk/Jz3NZuGalKp0bPlCciv1jVsNBLVeZ0o6qZlgmUVR1JXbNfkLN7T5mJGxVizUCRleWX872n/S7unnW+9/i2G5pwPCNj2lE6jHUt9qZ3zTGtpLl6uaCExOBQKHTU6mezPO/7M3/f9na8u+Sqn1J2i6ZyZFU72aBEEPUdGDgGkKQiklJ1SyjOklDNDr12h7eullFdHHHdASlkrpQyOOv90KeXCkKnpC1LKxP/dI5Tp5YqKm1AQdCnqZFoN3/XGYoVZ53BycB0H2qIUL4sgEJS09KWQTDbQrTiK538aTMak49dozC5Wo54moiCYVGyntc+L1x87usfrD3CoyxP+zsVjUfkifnTij1jXso47PnxUEcJbn1LCeA2gfGA/M0xNbHTEf+ge7PKQYxaG1uh5o+EN7lx/J6dPOp3rFl2n+byZlfkc7HTHLRkjpaShe5wLRybBxAmS/ohTZLdS6rCyrz2+bbFhIoWORjL3fPKDfRR3vo/PH9vP0dav1F9Pumn99ucgOAQLtXd+SpYyp40cs0hYjnpnaz9WiymjEStaUR8s8RLjDnZ6CEqYnkAjULlg+gVcseAK/rbzb/ylZjp07IS27bqMdzQi5IN4JRi9yJzKoU4PdcX2mKatdNnZtZNvv/5tZhfP5men/ExzoxlQNIKghP1x/ARdbh+DQ8EjwlEMWUGQUbREDjV0T6DQ0Uimn0HAZOVM1sb9DE2pVlvc+oTSoL5maTqjjIvJJKgqzE1YZmJHcx+zK/MnVDKZipZy1KrZIlrEUCxuXHojp9adys9aXudVhx22PpneQGOx41n25s7ng9743/EDnW7DzELtnna+/trXcVqd3HvGvdhzknufmZXK/zWen2A4dHQCzuUoTLxv+keY6RUO9sbRCIJBqTiYJpKjWMXmZKD+DC4wv8OHjR0xD1NXqkn5CPqalbr0Cy/RpQlNPNQQ0lhIKdne1Mfc6nxDx5EqqrYYz2G8N/SAmqbBNKRiNpn5+cd/zsLyhfxXRTnrdzypv3moaz+0bOFg5Zkc7vLEjKCTUnKo02OIo7jX28t1r15Hn6+Pe0+/lwp7otSnsUwtc2ASsLu1P+YxaTdmyjBZQZBBppU56XL76I7R+7e1XwmZm5AaAZB3/JWUiT5knBDDYY0gCdPQtqcBCQuMMwup1BTGTypr6/fS6fYxr7rA8LGkQmVBLjlmEddhvLfdRW1RXtJhi3mWPO47/T5qbMXcYPeza9fz6Q53JBv/BMKEZ8b5+IMy5n3o9gzR7/XrHjrqGfLwtde+xoHeA9xz2j3MLZ2b0nVsFjNTSh3sbo2tETQeIZ3JVLKCIINMr1C+2Ps6on+BJnomonnmGbSaKpjdENtssL/DTZE9h/zcJArmbXkcqhZC+azEx6bJzMp8GnsG6PFEF8bbmxVn+NwJKgjMJsWBGlcjaHcnpQ1EUpRbxG/OuJ88Kbl27S0jaxKlg98HGx6FWedSUacUmYtlYw+HjupoGvIFfHxj9TfY2rGVOz92JyfUnJDW9WZUONndFlsjaOj2kJ9roTBvnAtHaiQrCDJIOIS0LfoEGM4hmJgaASYzG8ouZL53E3TujXrIhkPdLJ00tgl5TDr3QtMGxSyUAZZOVsa28VBP1P3bm0KCoGZiCgJQ+xJE1wiCQcne9sSho/GoKZ/HQ3lzYGiAK1+6Qh9hsOM5cLfDMVcxpSxUhbQz+mc41KVv1VFvwMtNa27ineZ3uOXEWzijPv2ud7Mq8znQ6YkZOHEkhY5CVhBklLpiO1azib0xNAK1Ic1E/gL1zF6FX5pwv/PwmH29niF2tbpYXj+mGnls3r1fqTmzcJWOo4zN4roizCbBhkPdUffvaO6jrjiPgmQ0mgxTV2QPf1dG09I3iMcXSMpRHI1pJ3yTR5pbwD/IlS9dmb4wWPcwFE+FaaeHq5DG1gj0WxB5hjxc/9r1vN7wOv99/H+zckZSlXFiMrPSSSAolQJ6UWg4gpLJICsIMorZJJhSZo+tEXR7qCyY2G3t6qdM59XgcnK2/B/4RxbeUh+uy+tLtF3M3aHYjRd9Fgqq9R5qVBw2C3Oq8mMKgu3NfRPWP6AyqSSPDpc3aklwNaIrXUHAlJOZVjafR3qVHs9f/ueX2dK+JbVrtW6DQ2/DiivBZApXIT0Y4yF6sNNDVUFu2vPA5XPx1Ve/ytqWtdx28m2smq3fYkPVuHZFcRj3DQ6xu801YQMOopEVBBlmermTfTHCL3e3ucalLV8yzKsu4P8Cp2P1dsOHI53G6w92YTYJFk+KXkhsDGsfBP8AnHSDASONzbLJxWw61EMgODIqxuPzs7/DzbwJbBYCxc8B8EHDWPOWGjGUjmkIUKK3TrieaW17+P2ca7Dn2Lnq5at4/fDo4gAaWP8ImG2w9AvhTVPK7Epfgigc6nIzOU2zUIu7hS//88tsbt/Mzz/2cy6cfmFa1xvN9HInQhDVYfzu3k4CQcnJM8p0fU8jyQqCDDO93MnBrrG2xQ6Xl80NPZw0wb88RXYre5zH0JlTBe89OCLE8P2D3cyvKdAWreJzK4Jg9iegfLaBIx7Lsvoi3L4AO1tGruZ2tvQj5cR1FKucNKOMHLNg9YdjazzuaXdRkGuhzKlDM5d5F0F+DVM2/Y0/feJPTC2cyg2rb+CxnY9pv4a3Hz74Kyz4DNiHNcUppY6YIaQHOj1MSUMQbO3YymUvXEajq5FfnfErzplyTsrXikVujpnJJfaopSbe3NOB3Wpm6eQkTKTjTFYQZJhp5Q4CQRl2iKms/rANKeHMuZXjNDLtzK4u5P/MF8Hhd8NawVAgyKbDPdr9Axv/pJSVOOlGA0caneWTlQfSaPOQGjE00U1DTpuFY6eWsHrnWEGwt83N9AqnPs1cLFY47lrY/zplvc387pzfcWLNifzk3Z/ww7d/yKBfQ5Of934DPhccc/WIzVNKHVFDSHs8Ptr7vSmHjr64/0Wu+OcV2Mw2/njeHzm59uSUrqOFmRX5USOH3tzTwXFTS7BajpzH65Ez0o8IsYrPvbajjaqCXOZPcLMEKCvme/tOJlg2G17+b/B72d7Ux+BQUJsgCAzB2/fCpONh8vHGD3gUk0ryKHNa2XBwpCDY0dxHfq5lwobvRnLa7Ap2tbrGRA/tbXel7x+IZPmXlf7Rb92DPcfOvaffyzULr+Gp3U/xpRe/xOH+w7HP7W2Af/8PzDkf6paP2DU1FN66o3lk7arnNzcD8PFZyfUc8Qx5+NHbP+I7b3yHuaVz+fMn/syM4ui9kPViZqWT/R3uET2km3oG2NfunvCa/WiygiDDTItSfG5wKMAbu9s5Y27F+Pcp1sDc6gK8QROHj/kedO+HtQ/yfuihukKLo3j9I0o7ynHQBkBpmbh0cvFYjaCpj7nVBUfEPThtjpIRG2ke6hscoq3fm75/IJK8YjjuOiXXY89rmE1mblh2A/edcR+NrkYuef4SHtv5GEEZJYzype+BDMI5Px2za3FdEeX5Nv66bqQgefL9BuZU5Se1IPqw60MufeFSntr9FNcsvIaHz3mY0rzYlU31YmaFk6GAHBEG++YeJev+lJlHVvOsrCDIMPm5OdQW5fH6zvZwp6x393Xi8QWOCLMQEI6GWGdZrjSRef1Odu7dT21RXuKKnW074JUfwIyzYPZ5GRhtdJZNLuZAp4dOlxL5FAxKPmzpn/BmIZVpZQ7qS+38K0IQbArlRuiqEQB8/L+gbBY89x9Kn2HgY3Uf47ELHmNB2QJ+8u5PuPrlqzncF/FQ37cGtj8Dp3wLiuvHXNJqMfG5YyezZmd7OIx0T1s/mw73cPHyOk3C2OVz8fN1P+fSv1+Ky+fiwbMf5IZlN5Bjykzo78wKZR5Elpp4c3cH5fk2ZlXqfA8MJisIxoHrPj6N9/Z38c+tLYBiFsrLMXPCdONXMXowpdSBzWJS1Pqzb0P6XBx38P7EZiG/F568GqxOuOjXhtcVioc61g2hh+fBLg8eX+CIEQRCCE6bXcHbezsZHAowFAhy2ws7qCnM5US9v0c5uXDR/dDfrKzyQ9Q6a3norIf40Qk/YkfnDi569iLuXHcn3e42+Md3oHgKnBg7Iuzzx03GYhL88Z2DADzxfiNmk2Dlktq4wwkEAzy/93kufOZC/rT9T3xq5qd46sKnOL46s2bG6RUOLCbBcx80IaUkGJS8taeDk2eUHRFaZSRZQTAOfO7YycypyufWF3Yw4Avw2o5WTp5ZNqHzByKxmE3Mrsrn9V3tNFrrcS2+kk8HX+aKwBPxT3ztFmjdCivvA2fyxb70ZFFdIZZQYpmUkle2K0J5okcMRXLanAq8/iDv7O3kwTf2sbO1n1tWLsBhRC+LuhWKKW/jH5W+ESGEEHxm1md4ZuUzfHLaJ/nTjj9x3hNnc5+/hY7Tv6cIkRhUFORy3sJqHl9/mL7BIZ7e2MCps8opz4/e63ooMMTTu59m5bMr+X9v/j/K8sr48yf+zA9P+CFFuUlks+uE3Wrhm2fN4sWtLTz85n4+bOmn0+074vwDkKYgEEJcIoTYJoQICiFiFhgXQpwrhNgphNgjhPhuxPapQoj3hBC7hRB/E0LoEPM28bGYTfzowvk09gxw02ObaOod5My54/tgTJarT5lGQ7eHM//ndb7VczFPB05i6Z57Yc0dYw8OBuCdX8M79yrRI7ONaUWZDLk5ZubVFPDS1hYuuPdNfvqPD5lTlc+sqiNHpT9uagl5OWb+8M4Bfvnabs5bUMWZ8ww0L556M5TPgSeuhM0jQ0grHZXccuzNPG2ZwQnufh4oLuSsjT/jpjU38XbT2wwFo/eJ/vKJ9fR7/fznYx/Q2ufl4uUjW0VKKfmw60PuWncX5zx5Dj94+wfYLXbuPvVu/nr+X1lYvtCwj6uFr506nXPmV/KzFz/knteUFp9HUv6ASrpLh63Ap4HfxDpACGEG7kNpZdkArBNCPCel3A7cAdwtpfyrEOIB4Crg/jTHdERw/LRSLlhcw/MfNAHDzr8jhQsX17B0UhE/+ft2Xt7eyjvW61m5aBKmNT9VasrMPR+qFkHPQXj+G9C8CaafAWf9ZLyHHmZFfQmPvLWf+lI7P794EZ9aWkvOBOxBEIvcHDMnzSjj1R2t5Nss/OjC+ca+ocUGn39cMe89dQ3segnOvhWGPOBqhdduYdqhd7n7vDs4MOccntj1BM/sfYZXDr6CI8fBcVXHcULNCcwsnkl9QT2luaUsm1zMgtoCXt7eSpE9h+NnONnbs5ctHVvY1LaJ91vf50DfASwmC6fUnsLFsy7mlNpTJozpRQjBXZcs5qL73uKlba3MqHBOyM52iRBSh5rjQog1wH9KKddH2XcC8CMp5Tmhv28O7bodaAeqpJT+0cfFY8WKFXL9+jFvdcTR3DvA6Xe9zqyqfJ79+knjPZyUeWNXO0OBIGfMLoMXboL3fz/yAGclnPszpQ3lBJnAAN1uH5sO93DKzLIJ2YRGC39Ze4ibn9rCT1bO54snTMnMmwb88ObdsOZnICPKXJit8KnfwIJPhzd5A17ebHiTt5re4q3Gt2hyN4X3OXIc2C12fH5BtzuA1erBz3BeQb41nyXlS/h43cc5Z8o542L+0cqeNhefuu8tPnfcZG7+RGrlrTOBEOJ9KeUY600mBMHFwLlqD2MhxBeB44AfAe9KKWeEtk8CXpRSLojxHtcC1wJMnjx5+cGDB9Me90Rg7f4uCvIszKk6cmzTCXF3QMsW5SfghWOvhVyNZSeyJIXPH+T1Xe2cMacCk0FtHWPStEmJDnJWKMK+fA4Uxnb0Silpcjexv3c/B3oP0OBqYNA/yKDfy5bGbpbW1jK1uIYKewVzS+YyrWhaUi0kx5tutw+HzTKhE8liCYKEpiEhxKtAVZRd35NSPqvlvaNsk3G2R0VK+SDwICgagYb3PSI4dqrGAm1HEo4ymH6a8pPFUKwWE2cZ6ReIR80S5UcjQghqnbXUOmsNzfgdL4odR66LM6EgkFKemeZ7NACTIv6uA5qADqBICGGRUvojtmfJkiVLlgySCR1mHTAzFCFkBS4FnpOKTWo1oPYnvBzQomFkyZIlSxYdSTd89FNCiAbgBOAFIcRLoe01Qoh/AIRW+9cDLwE7gMeklNtCl/gv4CYhxB6gFBjb7SRLlixZshiKLs7iTPNRiRrKkiVLlkwSy1k8cd3bWbJkyZIlI2QFQZYsWbIc5WQFQZYsWbIc5WQFQZYsWbIc5RyRzmIhRDuQampxGUoOw9HG0fi5j8bPDEfn585+Zm3USynHdM05IgVBOggh1kfzmn/UORo/99H4meHo/NzZz5weWdNQlixZshzlZAVBlixZshzlHI2C4MHxHsA4cTR+7qPxM8PR+bmznzkNjjofQZYsWbJkGcnRqBFkyZIlS5YIsoIgS5YsWY5yjipBIIQ4VwixUwixRwjx3fEejxEIISYJIVYLIXYIIbYJIW4MbS8RQrwihNgdei0e77HqjRDCLITYKIT4e+jvqUKI90Kf+W+hMugfKYQQRUKIJ4QQH4bu+Qkf9XsthPhm6Lu9VQjxFyFE7kfxXgshHhFCtAkhtkZsi3pvhcIvQ8+2zUKIZcm811EjCIQQZuA+4DxgHnCZEGLe+I7KEPzAt6SUc4Hjga+HPud3gdeklDOB10J/f9S4EaXUucodwN2hz9wNXDUuozKWe4B/SinnAItRPv9H9l4LIWqBG4AVoba2ZpQeJx/Fe/174NxR22Ld2/OAmaGfa4H7k3mjo0YQAMcCe6SU+/5/e2fPGlUQheHnQDSYiPgBiiZCDIitsRIVkWgVxVjYCabwD1gJYmUvYiM2EYkiCsagwVYFK6NGREXFDxSzGk1AEsUmAV+LmYUl7GIguV6YOQ9c9s7swD2H9zLv3nOHHUkzwHWgt+SYFh1J45KexvNfhImhjZDrQBw2ABwqJ8JiMLN2YD/QH9sGdAODcUiKOa8AdhP38ZA0I2mKxLUm7Ky4zMyagBZgnAS1lvQA+DGnu5G2vcBlBR4Sdn9cP99r5WQEbcBYTbsS+5LFzDqALmAEWCdpHIJZAGvLi6wQzgEngD+xvQaYihsjQZp6dwKTwKVYEus3s1YS1lrSF+AM8JlgANPAKOlrXaWRtgua33IyAqvTl+zaWTNbDtwEjkv6WXY8RWJmB4AJSaO13XWGpqZ3E7ANuCCpC/hNQmWgesSaeC+wCdgAtBLKInNJTet/saD7PScjqAAba9rtwNeSYikUM1tCMIGrkoZi9/fqo2L8nCgrvgLYCRw0s0+Ekl834QlhZSwfQJp6V4CKpJHYHiQYQ8pa7wM+SpqUNAsMATtIX+sqjbRd0PyWkxE8BjbH1QVLCS+YhkuOadGJtfGLwGtJZ2u+Ggb64nkfcPt/x1YUkk5KapfUQdD1nqQjwH3gcByWVM4Akr4BY2a2JXbtBV6RsNaEktB2M2uJ93o156S1rqGRtsPA0bh6aDswXS0hzQtJ2RxAD/AW+ACcKjuegnLcRXgkfA48i0cPoWZ+F3gXP1eXHWtB+e8B7sTzTuAR8B64ATSXHV8B+W4FnkS9bwGrUtcaOA28AV4CV4DmFLUGrhHeg8wSfvEfa6QtoTR0Ps5tLwirquZ9Lf+LCcdxnMzJqTTkOI7j1MGNwHEcJ3PcCBzHcTLHjcBxHCdz3Agcx3Eyx43AcRwnc9wIHMdxMucvV4TN1HmQJtkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "tst_encoding = PositionalEncoding(20)\n",
    "res = tst_encoding(torch.arange(0,100).float())\n",
    "_, ax = plt.subplots(1,1)\n",
    "for i in range(1,5): ax.plot(res[:,i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000],\n",
       "        [ 0.8415,  0.3877,  0.1578,  0.0631,  0.0251,  0.0100],\n",
       "        [ 0.9093,  0.7147,  0.3117,  0.1259,  0.0502,  0.0200],\n",
       "        [ 0.1411,  0.9300,  0.4578,  0.1882,  0.0753,  0.0300],\n",
       "        [-0.7568,  0.9998,  0.5923,  0.2497,  0.1003,  0.0400],\n",
       "        [-0.9589,  0.9132,  0.7121,  0.3103,  0.1253,  0.0500]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res[:6,:6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransformerEmbedding(nn.Module):\n",
    "    \"Embedding + positional encoding + dropout\"\n",
    "    def __init__(self, vocab_sz, emb_sz, inp_p=0.):\n",
    "        super().__init__()\n",
    "        self.emb_sz = emb_sz\n",
    "        self.embed = embedding(vocab_sz, emb_sz)\n",
    "        self.pos_enc = PositionalEncoding(emb_sz)\n",
    "        self.drop = nn.Dropout(inp_p)\n",
    "    \n",
    "    def forward(self, inp): \n",
    "        pos = torch.arange(0, inp.size(1), device=inp.device).float()\n",
    "        return self.drop(self.embed(inp) * math.sqrt(self.emb_sz) + self.pos_enc(pos))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Feed forward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The feed forward cell is easy: it's just two linear layers with a skip connection and a LayerNorm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def feed_forward(d_model, d_ff, ff_p=0., double_drop=True):\n",
    "    layers = [nn.Linear(d_model, d_ff), nn.ReLU()]\n",
    "    if double_drop: layers.append(nn.Dropout(ff_p))\n",
    "    return SequentialEx(*layers, nn.Linear(d_ff, d_model), nn.Dropout(ff_p), MergeLayer(), nn.LayerNorm(d_model))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-head attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Multi head attention](images/attention.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadAttention(nn.Module):\n",
    "    def __init__(self, n_heads, d_model, d_head=None, p=0., bias=True, scale=True):\n",
    "        super().__init__()\n",
    "        d_head = ifnone(d_head, d_model//n_heads)\n",
    "        self.n_heads,self.d_head,self.scale = n_heads,d_head,scale\n",
    "        self.q_wgt,self.k_wgt,self.v_wgt = [nn.Linear(\n",
    "            d_model, n_heads * d_head, bias=bias) for o in range(3)]\n",
    "        self.out = nn.Linear(n_heads * d_head, d_model, bias=bias)\n",
    "        self.drop_att,self.drop_res = nn.Dropout(p),nn.Dropout(p)\n",
    "        self.ln = nn.LayerNorm(d_model)\n",
    "        \n",
    "    def forward(self, q, kv, mask=None):\n",
    "        return self.ln(q + self.drop_res(self.out(self._apply_attention(q, kv, mask=mask))))\n",
    "    \n",
    "    def create_attn_mat(self, x, layer, bs):\n",
    "        return layer(x).view(bs, x.size(1), self.n_heads, self.d_head\n",
    "                            ).permute(0, 2, 1, 3)\n",
    "    \n",
    "    def _apply_attention(self, q, kv, mask=None):\n",
    "        bs,seq_len = q.size(0),q.size(1)\n",
    "        wq,wk,wv = map(lambda o: self.create_attn_mat(*o,bs),\n",
    "                       zip((q,kv,kv),(self.q_wgt,self.k_wgt,self.v_wgt)))\n",
    "        attn_score = wq @ wk.transpose(2,3)\n",
    "        if self.scale: attn_score /= math.sqrt(self.d_head)\n",
    "        if mask is not None: \n",
    "            attn_score = attn_score.float().masked_fill(mask, -float('inf')).type_as(attn_score)\n",
    "        attn_prob = self.drop_att(F.softmax(attn_score, dim=-1))\n",
    "        attn_vec = attn_prob @ wv\n",
    "        return attn_vec.permute(0, 2, 1, 3).contiguous().view(bs, seq_len, -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Masking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The attention layer uses a mask to avoid paying attention to certain timesteps. The first thing is that we don't really want the network to pay attention to the padding, so we're going to mask it. The second thing is that since this model isn't recurrent, we need to mask (in the output) all the tokens we're not supposed to see yet (otherwise it would be cheating)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def get_output_mask(inp, pad_idx=1):\n",
    "    return torch.triu(inp.new_ones(inp.size(1),inp.size(1)), diagonal=1)[None,None].byte()\n",
    "#     return ((inp == pad_idx)[:,None,:,None].long() + torch.triu(inp.new_ones(inp.size(1),inp.size(1)), diagonal=1)[None,None] != 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Example of mask for the future tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 1, 1, 1, 1, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 1, 1, 1, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 1, 1, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 1, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 0, 0, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 0, 0, 0, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=torch.uint8)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.triu(torch.ones(10,10), diagonal=1).byte()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoder and decoder blocks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to regroup these layers in the blocks we add in the model picture:\n",
    "\n",
    "![Transformer model](images/Transformer.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EncoderBlock(nn.Module):\n",
    "    \"Encoder block of a Transformer model.\"\n",
    "    #Can't use Sequential directly cause more than one input...\n",
    "    def __init__(self, n_heads, d_model, d_head, d_inner, p=0., bias=True, scale=True, double_drop=True):\n",
    "        super().__init__()\n",
    "        self.mha = MultiHeadAttention(n_heads, d_model, d_head, p=p, bias=bias, scale=scale)\n",
    "        self.ff  = feed_forward(d_model, d_inner, ff_p=p, double_drop=double_drop)\n",
    "    \n",
    "    def forward(self, x, mask=None): return self.ff(self.mha(x, x, mask=mask))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DecoderBlock(nn.Module):\n",
    "    \"Decoder block of a Transformer model.\"\n",
    "    #Can't use Sequential directly cause more than one input...\n",
    "    def __init__(self, n_heads, d_model, d_head, d_inner, p=0., bias=True, scale=True, double_drop=True):\n",
    "        super().__init__()\n",
    "        self.mha1 = MultiHeadAttention(n_heads, d_model, d_head, p=p, bias=bias, scale=scale)\n",
    "        self.mha2 = MultiHeadAttention(n_heads, d_model, d_head, p=p, bias=bias, scale=scale)\n",
    "        self.ff   = feed_forward(d_model, d_inner, ff_p=p, double_drop=double_drop)\n",
    "    \n",
    "    def forward(self, x, enc, mask_out=None): return self.ff(self.mha2(self.mha1(x, x, mask_out), enc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### The whole model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class Transformer(Module):\n",
    "    def __init__(self, inp_vsz, out_vsz, n_layers=6, n_heads=8, d_model=256, d_head=32, \n",
    "                 d_inner=1024, p=0.1, bias=True, scale=True, double_drop=True, pad_idx=1):\n",
    "        self.enc_emb = TransformerEmbedding(inp_vsz, d_model, p)\n",
    "        self.dec_emb = TransformerEmbedding(out_vsz, d_model, 0.)\n",
    "        args = (n_heads, d_model, d_head, d_inner, p, bias, scale, double_drop)\n",
    "        self.encoder = nn.ModuleList([EncoderBlock(*args) for _ in range(n_layers)])\n",
    "        self.decoder = nn.ModuleList([DecoderBlock(*args) for _ in range(n_layers)])\n",
    "        self.out = nn.Linear(d_model, out_vsz)\n",
    "        self.out.weight = self.dec_emb.embed.weight\n",
    "        self.pad_idx = pad_idx\n",
    "        \n",
    "    def forward(self, inp, out):\n",
    "        mask_out = get_output_mask(out, self.pad_idx)\n",
    "        enc,out = self.enc_emb(inp),self.dec_emb(out)\n",
    "        enc = compose(self.encoder)(enc)\n",
    "        out = compose(self.decoder)(out, enc, mask_out)\n",
    "        return self.out(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### Bleu metric (see dedicated notebook)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class NGram():\n",
    "    def __init__(self, ngram, max_n=5000): self.ngram,self.max_n = ngram,max_n\n",
    "    def __eq__(self, other):\n",
    "        if len(self.ngram) != len(other.ngram): return False\n",
    "        return np.all(np.array(self.ngram) == np.array(other.ngram))\n",
    "    def __hash__(self): return int(sum([o * self.max_n**i for i,o in enumerate(self.ngram)]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def get_grams(x, n, max_n=5000):\n",
    "    return x if n==1 else [NGram(x[i:i+n], max_n=max_n) for i in range(len(x)-n+1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def get_correct_ngrams(pred, targ, n, max_n=5000):\n",
    "    pred_grams,targ_grams = get_grams(pred, n, max_n=max_n),get_grams(targ, n, max_n=max_n)\n",
    "    pred_cnt,targ_cnt = Counter(pred_grams),Counter(targ_grams)\n",
    "    return sum([min(c, targ_cnt[g]) for g,c in pred_cnt.items()]),len(pred_grams)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class CorpusBLEU(Callback):\n",
    "    def __init__(self, vocab_sz):\n",
    "        self.vocab_sz = vocab_sz\n",
    "        self.name = 'bleu'\n",
    "    \n",
    "    def on_epoch_begin(self, **kwargs):\n",
    "        self.pred_len,self.targ_len,self.corrects,self.counts = 0,0,[0]*4,[0]*4\n",
    "    \n",
    "    def on_batch_end(self, last_output, last_target, **kwargs):\n",
    "        last_output = last_output.argmax(dim=-1)\n",
    "        for pred,targ in zip(last_output.cpu().numpy(),last_target.cpu().numpy()):\n",
    "            self.pred_len += len(pred)\n",
    "            self.targ_len += len(targ)\n",
    "            for i in range(4):\n",
    "                c,t = get_correct_ngrams(pred, targ, i+1, max_n=self.vocab_sz)\n",
    "                self.corrects[i] += c\n",
    "                self.counts[i]   += t\n",
    "    \n",
    "    def on_epoch_end(self, last_metrics, **kwargs):\n",
    "        precs = [c/t for c,t in zip(self.corrects,self.counts)]\n",
    "        len_penalty = exp(1 - self.targ_len/self.pred_len) if self.pred_len < self.targ_len else 1\n",
    "        bleu = len_penalty * ((precs[0]*precs[1]*precs[2]*precs[3]) ** 0.25)\n",
    "        return add_metrics(last_metrics, bleu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_x_vocab,n_y_vocab = len(data.train_ds.x.vocab.itos), len(data.train_ds.y.vocab.itos)\n",
    "\n",
    "model = Transformer(n_x_vocab, n_y_vocab, d_model=256)\n",
    "learn = Learner(data, model, metrics=[accuracy, CorpusBLEU(n_y_vocab)], loss_func = CrossEntropyFlat())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    },
    {
     "ename": "BrokenPipeError",
     "evalue": "[Errno 32] Broken pipe",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mBrokenPipeError\u001b[0m                           Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-31-c7a9c29f9dd1>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mlearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlr_find\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0mlearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrecorder\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mplot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\site-packages\\fastai\\train.py\u001b[0m in \u001b[0;36mlr_find\u001b[1;34m(learn, start_lr, end_lr, num_it, stop_div, wd)\u001b[0m\n\u001b[0;32m     39\u001b[0m     \u001b[0mcb\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mLRFinder\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlearn\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstart_lr\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mend_lr\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnum_it\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstop_div\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     40\u001b[0m     \u001b[0mepochs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mceil\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnum_it\u001b[0m\u001b[1;33m/\u001b[0m\u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtrain_dl\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 41\u001b[1;33m     \u001b[0mlearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mepochs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstart_lr\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mcb\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mwd\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mwd\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     42\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     43\u001b[0m def to_fp16(learn:Learner, loss_scale:float=None, max_noskip:int=1000, dynamic:bool=True, clip:float=None,\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\site-packages\\fastai\\basic_train.py\u001b[0m in \u001b[0;36mfit\u001b[1;34m(self, epochs, lr, wd, callbacks)\u001b[0m\n\u001b[0;32m    198\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mopt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlr\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mopt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwd\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlr\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mwd\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    199\u001b[0m         \u001b[0mcallbacks\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mcb\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mcb\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcallback_fns\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mlistify\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdefaults\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mextra_callback_fns\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mlistify\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 200\u001b[1;33m         \u001b[0mfit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mepochs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmetrics\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmetrics\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[1;33m+\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    201\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    202\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mcreate_opt\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlr\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mFloats\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mwd\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mFloats\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m0.\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m->\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\site-packages\\fastai\\basic_train.py\u001b[0m in \u001b[0;36mfit\u001b[1;34m(epochs, learn, callbacks, metrics)\u001b[0m\n\u001b[0;32m     97\u001b[0m             \u001b[0mcb_handler\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mset_dl\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtrain_dl\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     98\u001b[0m             \u001b[0mcb_handler\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mon_epoch_begin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 99\u001b[1;33m             \u001b[1;32mfor\u001b[0m \u001b[0mxb\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0myb\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mprogress_bar\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtrain_dl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mparent\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mpbar\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    100\u001b[0m                 \u001b[0mxb\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0myb\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcb_handler\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mon_batch_begin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mxb\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0myb\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    101\u001b[0m                 \u001b[0mloss\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mloss_batch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mxb\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0myb\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mloss_func\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mopt\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcb_handler\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\site-packages\\fastprogress\\fastprogress.py\u001b[0m in \u001b[0;36m__iter__\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m     70\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     71\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 72\u001b[1;33m             \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mo\u001b[0m \u001b[1;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_gen\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     73\u001b[0m                 \u001b[1;32mif\u001b[0m \u001b[0mi\u001b[0m \u001b[1;33m>=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtotal\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;32mbreak\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     74\u001b[0m                 \u001b[1;32myield\u001b[0m \u001b[0mo\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\site-packages\\fastai\\basic_data.py\u001b[0m in \u001b[0;36m__iter__\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m     73\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m__iter__\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     74\u001b[0m         \u001b[1;34m\"Process and returns items from `DataLoader`.\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 75\u001b[1;33m         \u001b[1;32mfor\u001b[0m \u001b[0mb\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdl\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;32myield\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mproc_batch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mb\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     76\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     77\u001b[0m     \u001b[1;33m@\u001b[0m\u001b[0mclassmethod\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\site-packages\\torch\\utils\\data\\dataloader.py\u001b[0m in \u001b[0;36m__iter__\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    277\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0m_SingleProcessDataLoaderIter\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    278\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 279\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0m_MultiProcessingDataLoaderIter\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    280\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    281\u001b[0m     \u001b[1;33m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\site-packages\\torch\\utils\\data\\dataloader.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, loader)\u001b[0m\n\u001b[0;32m    717\u001b[0m             \u001b[1;31m#     before it starts, and __del__ tries to join but will get:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    718\u001b[0m             \u001b[1;31m#     AssertionError: can only join a started process.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 719\u001b[1;33m             \u001b[0mw\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    720\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_index_queues\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mindex_queue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    721\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_workers\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mw\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\multiprocessing\\process.py\u001b[0m in \u001b[0;36mstart\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    110\u001b[0m                \u001b[1;34m'daemonic processes are not allowed to have children'\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    111\u001b[0m         \u001b[0m_cleanup\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 112\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_popen\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_Popen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    113\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_sentinel\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_popen\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msentinel\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    114\u001b[0m         \u001b[1;31m# Avoid a refcycle if the target function holds an indirect\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\multiprocessing\\context.py\u001b[0m in \u001b[0;36m_Popen\u001b[1;34m(process_obj)\u001b[0m\n\u001b[0;32m    221\u001b[0m     \u001b[1;33m@\u001b[0m\u001b[0mstaticmethod\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    222\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_Popen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mprocess_obj\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 223\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0m_default_context\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_context\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mProcess\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_Popen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mprocess_obj\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    224\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    225\u001b[0m \u001b[1;32mclass\u001b[0m \u001b[0mDefaultContext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mBaseContext\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\multiprocessing\\context.py\u001b[0m in \u001b[0;36m_Popen\u001b[1;34m(process_obj)\u001b[0m\n\u001b[0;32m    320\u001b[0m         \u001b[1;32mdef\u001b[0m \u001b[0m_Popen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mprocess_obj\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    321\u001b[0m             \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m\u001b[0mpopen_spawn_win32\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mPopen\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 322\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mPopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mprocess_obj\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    323\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    324\u001b[0m     \u001b[1;32mclass\u001b[0m \u001b[0mSpawnContext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mBaseContext\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\multiprocessing\\popen_spawn_win32.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, process_obj)\u001b[0m\n\u001b[0;32m     87\u001b[0m             \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     88\u001b[0m                 \u001b[0mreduction\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdump\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mprep_data\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mto_child\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 89\u001b[1;33m                 \u001b[0mreduction\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdump\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mprocess_obj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mto_child\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     90\u001b[0m             \u001b[1;32mfinally\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     91\u001b[0m                 \u001b[0mset_spawning_popen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\multiprocessing\\reduction.py\u001b[0m in \u001b[0;36mdump\u001b[1;34m(obj, file, protocol)\u001b[0m\n\u001b[0;32m     58\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mdump\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfile\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mprotocol\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     59\u001b[0m     \u001b[1;34m'''Replacement for pickle.dump() using ForkingPickler.'''\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 60\u001b[1;33m     \u001b[0mForkingPickler\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfile\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mprotocol\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdump\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     61\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     62\u001b[0m \u001b[1;31m#\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mBrokenPipeError\u001b[0m: [Errno 32] Broken pipe"
     ]
    }
   ],
   "source": [
    "learn.lr_find()\n",
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>bleu</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2.388738</td>\n",
       "      <td>2.521896</td>\n",
       "      <td>0.608056</td>\n",
       "      <td>0.440603</td>\n",
       "      <td>02:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1.901857</td>\n",
       "      <td>1.958518</td>\n",
       "      <td>0.687528</td>\n",
       "      <td>0.504890</td>\n",
       "      <td>01:55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1.565219</td>\n",
       "      <td>1.652850</td>\n",
       "      <td>0.723345</td>\n",
       "      <td>0.536689</td>\n",
       "      <td>02:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.306575</td>\n",
       "      <td>1.455323</td>\n",
       "      <td>0.747950</td>\n",
       "      <td>0.565299</td>\n",
       "      <td>02:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.137455</td>\n",
       "      <td>1.342423</td>\n",
       "      <td>0.764108</td>\n",
       "      <td>0.587329</td>\n",
       "      <td>02:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.949592</td>\n",
       "      <td>1.282115</td>\n",
       "      <td>0.774509</td>\n",
       "      <td>0.601252</td>\n",
       "      <td>02:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.808173</td>\n",
       "      <td>1.269965</td>\n",
       "      <td>0.778810</td>\n",
       "      <td>0.607137</td>\n",
       "      <td>01:58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.767959</td>\n",
       "      <td>1.274921</td>\n",
       "      <td>0.778822</td>\n",
       "      <td>0.607116</td>\n",
       "      <td>01:51</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(8, 5e-4, div_factor=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_predictions(learn, ds_type=DatasetType.Valid):\n",
    "    learn.model.eval()\n",
    "    inputs, targets, outputs = [],[],[]\n",
    "    with torch.no_grad():\n",
    "        for xb,yb in progress_bar(learn.dl(ds_type)):\n",
    "            out = learn.model(*xb)\n",
    "            for x,y,z in zip(xb[0],xb[1],out):\n",
    "                inputs.append(learn.data.train_ds.x.reconstruct(x))\n",
    "                targets.append(learn.data.train_ds.y.reconstruct(y))\n",
    "                outputs.append(learn.data.train_ds.y.reconstruct(z.argmax(1)))\n",
    "    return inputs, targets, outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='149' class='' max='149', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [149/149 00:28<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "inputs, targets, outputs = get_predictions(learn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj pendant que xxunk les activités requises pour maintenir mon xxunk physique , est - ce que je xxunk de la protection d’un régime d’assurance ou de pension ?,\n",
       " Text xxbos xxmaj while i go about maintaining this high degree of fitness , am i protected under an insurance or pension plan ?,\n",
       " Text xxbos xxmaj while i do to the my physical physical of physical , do i aware by the pension plan service plan ?)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[10],targets[10],outputs[10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quelles sont les conséquences sur la recherche , la mise en pratique et les politiques en ce qui a trait à l'ac ?,\n",
       " Text xxbos xxmaj what are the xxunk for xxup kt research , practice / policy ?,\n",
       " Text xxbos xxmaj what are the implications implications research kt , , policy and policies in)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[700],targets[700],outputs[700]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quelle est la position des xxmaj états - xxmaj unis , du xxmaj canada et de la xxup xxunk à ce propos ?,\n",
       " Text xxbos xxmaj where do the xxup us , xxmaj canada and xxup xxunk stand ?,\n",
       " Text xxbos xxmaj what is xxmaj xxup us xxmaj xxmaj united and the xxunk fit in)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[701],targets[701],outputs[701]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quels sont les atouts particuliers du xxmaj canada en recherche sur l'obésité sur la scène internationale ?,\n",
       " Text xxbos xxmaj what are the unique xxmaj canadian strengths in obesity research that set xxmaj canada apart on an international front ?,\n",
       " Text xxbos xxmaj what are xxmaj specific strengths canada strengths in obesity - ? are up canada ? from international international stage ?)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[2500],targets[2500],outputs[2500]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quelles sont les répercussions politiques à long terme de cette révolution scientifique mondiale ?,\n",
       " Text xxbos xxmaj what are some of the long - term policy implications of this global knowledge revolution ?,\n",
       " Text xxbos xxmaj what are the long the long - term policies implications of this global scientific ? ?)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[4002],targets[4002],outputs[4002]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Label smoothing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They point out in the paper that using label smoothing helped getting a better BLEU/accuracy, even if it made the loss worse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Transformer(len(data.train_ds.x.vocab.itos), len(data.train_ds.y.vocab.itos), d_model=256)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, model, metrics=[accuracy, CorpusBLEU(len(data.train_ds.y.vocab.itos))], \n",
    "                loss_func=FlattenedLoss(LabelSmoothingCrossEntropy, axis=-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>bleu</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.281034</td>\n",
       "      <td>3.357356</td>\n",
       "      <td>0.621848</td>\n",
       "      <td>0.458009</td>\n",
       "      <td>01:45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.872045</td>\n",
       "      <td>2.923340</td>\n",
       "      <td>0.690921</td>\n",
       "      <td>0.510376</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.598603</td>\n",
       "      <td>2.653438</td>\n",
       "      <td>0.729291</td>\n",
       "      <td>0.545735</td>\n",
       "      <td>01:49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.407944</td>\n",
       "      <td>2.514847</td>\n",
       "      <td>0.748187</td>\n",
       "      <td>0.567057</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.195246</td>\n",
       "      <td>2.403729</td>\n",
       "      <td>0.766409</td>\n",
       "      <td>0.592165</td>\n",
       "      <td>01:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.095695</td>\n",
       "      <td>2.362098</td>\n",
       "      <td>0.776127</td>\n",
       "      <td>0.604666</td>\n",
       "      <td>01:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.999303</td>\n",
       "      <td>2.358647</td>\n",
       "      <td>0.779535</td>\n",
       "      <td>0.609675</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.923621</td>\n",
       "      <td>2.359421</td>\n",
       "      <td>0.780211</td>\n",
       "      <td>0.610871</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(8, 5e-4, div_factor=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit_one_cycle(8, 5e-4, div_factor=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Quels sont les atouts particuliers du Canada en recherche sur l'obésité sur la scène internationale ?\n",
      "What are Specific strengths canada strengths in obesity - ? are up canada ? from international international stage ?\n",
      "Quelles sont les répercussions politiques à long terme de cette révolution scientifique mondiale ?\n",
      "What are the long the long - term policies implications of this global scientific ? ?\n"
     ]
    }
   ],
   "source": [
    "print(\"Quels sont les atouts particuliers du Canada en recherche sur l'obésité sur la scène internationale ?\")\n",
    "print(\"What are Specific strengths canada strengths in obesity - ? are up canada ? from international international stage ?\")\n",
    "print(\"Quelles sont les répercussions politiques à long terme de cette révolution scientifique mondiale ?\")\n",
    "print(\"What are the long the long - term policies implications of this global scientific ? ?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quelle distance y a - t - il entre le point le plus rapproché de la surface à xxunk et la position d’utilisation habituelle du tube radiogène ?,\n",
       " Text xxbos xxmaj what is the distance between the nearest point of the area to be shielded and the usual operational position of the x - ray tube ?,\n",
       " Text xxbos xxmaj what is the xxmaj between the xxmaj and of the xxmaj ? the ? and the most ? ? of the xxmaj - ray tube ?)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[10],targets[10],outputs[10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quels types de présentations xxmaj santé xxmaj canada xxunk - t - il dans le format ectd à compter du 1er septembre ?,\n",
       " Text xxbos xxmaj what kind of submission types will xxmaj health xxmaj canada accept on xxmaj september 1 , 2004 in ectd format ?,\n",
       " Text xxbos xxmaj what is of information is of be canadian xxmaj canada take ? the canadian ? , and ? the format ?)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[700],targets[700],outputs[700]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quelles sont les trois caractéristiques qui vous incitent le plus à investir dans votre région ( xxup nommez - xxup les ) ?,\n",
       " Text xxbos xxmaj what are the three most attractive features about investing in your region ( xxup name xxup it ) ?,\n",
       " Text xxbos xxmaj what is the main main important concerns of the in the country ? xxup xxunk , xxunk ) ?)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[701],targets[701],outputs[701]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Text xxbos xxmaj quelles actions avez - vous prises et quel en a été le résultat ?,\n",
       " Text xxbos xxmaj what were your actions and the outcomes ?,\n",
       " Text xxbos xxmaj what is the targets ? how main of)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs[4001],targets[4001],outputs[4001]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test leakage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we change a token in the targets at position n, it shouldn't impact the predictions before that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.model.eval();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xb,yb = data.one_batch(cpu=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inp1,out1 = xb[0][:1],xb[1][:1]\n",
    "inp2,out2 = inp1.clone(),out1.clone()\n",
    "out2[0,15] = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y1 = learn.model(inp1, out1)\n",
    "y2 = learn.model(inp2, out2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0., device='cuda:0', grad_fn=<MeanBackward1>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(y1[0,:15] - y2[0,:15]).abs().mean()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}