7-seq2seq-translation_jcat.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summary of my results:\n",
    "\n",
    "model            | train_loss | valid_loss | seq2seq_acc | bleu\n",
    "-------------------|----------|----------|----------|----------\n",
    "seq2seq            | 3.355085 | 4.272877 | 0.382089 | 0.291899\n",
    "\\+ teacher forcing | 3.154585 |\t4.022432 | 0.407792 | 0.310715\n",
    "\\+ attention       | 1.452292 | 3.420485 | 0.498205 | 0.413232\n",
    "transformer        | 1.913152 | 2.349686 | 0.781749 | 0.612880"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Translation with an RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is modified from [this one](https://github.com/fastai/fastai_docs/blob/master/dev_course/dl2/translation.ipynb) created by Sylvain Gugger.\n",
    "\n",
    "Today we will be tackling the task of translation. We will be translating from French to English, and to keep our task a manageable size, we will limit ourselves to translating questions.\n",
    "\n",
    "This task is an example of sequence to sequence (seq2seq). Seq2seq can be more challenging than classification, since the output is of variable length (and typically different from the length of the input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "French/English parallel texts from http://www.statmt.org/wmt15/translation-task.html .  It was created by Chris Callison-Burch, who crawled millions of web pages and then used *a set of simple heuristics to transform French URLs onto English URLs (i.e. replacing \"fr\" with \"en\" and about 40 other hand-written rules), and assume that these documents are translations of each other*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Translation is much tougher in straight PyTorch: https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download and preprocess data from the `giga-fren` dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start by reducing the original `giga-fren` dataset to questions. Fastai has archived it on `Amazon Web Server (AWS)`. \n",
    "\n",
    "The dataset can be downloaded [here](https://s3.amazonaws.com/fast-ai-nlp/giga-fren.tgz).\n",
    "You only need to do this once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = Config().data_path()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For windows 10 machines, extract manually by executing the command\n",
    "`wget https://s3.amazonaws.com/fast-ai-nlp/giga-fren.tgz`\n",
    "\n",
    "in the `~/.fastai/data/` folder\n",
    "\n",
    "Then use `7-zip` to extract the .tgz file\n",
    "\n",
    "### For Linux, execute the following 2 lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! wget https://s3.amazonaws.com/fast-ai-nlp/giga-fren.tgz -P {path}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! tar xf {path}/giga-fren.tgz -C {path} "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/cc.en.300.bin'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/cc.fr.300.bin'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/data_save.pkl'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/giga-fren.release2.fixed.en'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/giga-fren.release2.fixed.fr'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/models'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/questions_easy.csv')]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = Config().data_path()/'giga-fren'\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load English and French texts (this takes a few minutes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add the encoding=\"utf8\" switch \n",
    "# with open(path/'giga-fren.release2.fixed.fr') as f: fr = f.read().split('\\n')\n",
    "with open(path/'giga-fren.release2.fixed.fr', encoding=\"utf8\") as f: \n",
    "    fr = f.read().split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add the encoding=\"utf8\" switch \n",
    "# with open(path/'giga-fren.release2.fixed.en') as f: en = f.read().split('\\n')\n",
    "with open(path/'giga-fren.release2.fixed.en', encoding=\"utf8\") as f:\n",
    "    en = f.read().split('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use regex to pick out questions by finding the strings in the English dataset that start with \"Wh\" and end with a question mark.  You only need to run these lines once:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# English question: first two character are Wh, followed by 1 or more characters that are not \"?\",\".\",or\"!\", followed by \"?\"\n",
    "re_eq = re.compile('^(Wh[^?.!]+\\?)')\n",
    "# French question\n",
    "re_fq = re.compile('^([^?.!]+\\?)')\n",
    "# paths to en and fr databases\n",
    "en_fname = path/'giga-fren.release2.fixed.en'\n",
    "fr_fname = path/'giga-fren.release2.fixed.fr'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract tuples of (english questions,french translations) via `regex` search:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "lines = ((re_eq.search(eq), re_fq.search(fq)) \n",
    "         for eq, fq in zip(open(en_fname, encoding='utf-8'), open(fr_fname, encoding='utf-8')))\n",
    "qs = [(e.group(), f.group()) for e,f in lines if e and f]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Put english questions,french translations into a dataframe, save to .csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "qs = [(q1,q2) for q1,q2 in qs]\n",
    "df = pd.DataFrame({'fr': [q[1] for q in qs], 'en': [q[0] for q in qs]}, columns = ['en', 'fr'])\n",
    "df.to_csv(path/'questions_easy.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/cc.en.300.bin'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/cc.fr.300.bin'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/data_save.pkl'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/giga-fren.release2.fixed.en'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/giga-fren.release2.fixed.fr'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/models'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren/questions_easy.csv')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load our data into a DataBunch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our questions look like this now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>en</th>\n",
       "      <th>fr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>What is light ?</td>\n",
       "      <td>Qu’est-ce que la lumière?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>Who are we?</td>\n",
       "      <td>Où sommes-nous?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Where did we come from?</td>\n",
       "      <td>D'où venons-nous?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>What would we do without it?</td>\n",
       "      <td>Que ferions-nous sans elle ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>What is the absolute location (latitude and lo...</td>\n",
       "      <td>Quelle sont les coordonnées (latitude et longi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  en  \\\n",
       "0                                    What is light ?   \n",
       "1                                        Who are we?   \n",
       "2                            Where did we come from?   \n",
       "3                       What would we do without it?   \n",
       "4  What is the absolute location (latitude and lo...   \n",
       "\n",
       "                                                  fr  \n",
       "0                          Qu’est-ce que la lumière?  \n",
       "1                                    Où sommes-nous?  \n",
       "2                                  D'où venons-nous?  \n",
       "3                       Que ferions-nous sans elle ?  \n",
       "4  Quelle sont les coordonnées (latitude et longi...  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(path/'questions_easy.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make it simple, we lowercase everything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['en'] = df['en'].apply(lambda x:x.lower())\n",
    "df['fr'] = df['fr'].apply(lambda x:x.lower())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>en</th>\n",
       "      <th>fr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>what is light ?</td>\n",
       "      <td>qu’est-ce que la lumière?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>who are we?</td>\n",
       "      <td>où sommes-nous?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>where did we come from?</td>\n",
       "      <td>d'où venons-nous?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>what would we do without it?</td>\n",
       "      <td>que ferions-nous sans elle ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>what is the absolute location (latitude and lo...</td>\n",
       "      <td>quelle sont les coordonnées (latitude et longi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>52326</td>\n",
       "      <td>what are the mechanisms for increasing women's...</td>\n",
       "      <td>quels sont les mécanismes susceptibles d’améli...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>52327</td>\n",
       "      <td>why is it still pushed aside, silenced, or the...</td>\n",
       "      <td>pourquoi, aujourd'hui encore, écarte-t-on cett...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>52328</td>\n",
       "      <td>why should they not demote a judge who rules a...</td>\n",
       "      <td>pourquoi ne devraient-ils pas destituer un jug...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>52329</td>\n",
       "      <td>why is it that we have achieved judicial indep...</td>\n",
       "      <td>pourquoi avons-nous réussi à atteindre l'indép...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>52330</td>\n",
       "      <td>what's inside introduction to sti did you know…?</td>\n",
       "      <td>contenu introduction aux its saviez-vous que…?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>52331 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                      en  \\\n",
       "0                                        what is light ?   \n",
       "1                                            who are we?   \n",
       "2                                where did we come from?   \n",
       "3                           what would we do without it?   \n",
       "4      what is the absolute location (latitude and lo...   \n",
       "...                                                  ...   \n",
       "52326  what are the mechanisms for increasing women's...   \n",
       "52327  why is it still pushed aside, silenced, or the...   \n",
       "52328  why should they not demote a judge who rules a...   \n",
       "52329  why is it that we have achieved judicial indep...   \n",
       "52330   what's inside introduction to sti did you know…?   \n",
       "\n",
       "                                                      fr  \n",
       "0                              qu’est-ce que la lumière?  \n",
       "1                                        où sommes-nous?  \n",
       "2                                      d'où venons-nous?  \n",
       "3                           que ferions-nous sans elle ?  \n",
       "4      quelle sont les coordonnées (latitude et longi...  \n",
       "...                                                  ...  \n",
       "52326  quels sont les mécanismes susceptibles d’améli...  \n",
       "52327  pourquoi, aujourd'hui encore, écarte-t-on cett...  \n",
       "52328  pourquoi ne devraient-ils pas destituer un jug...  \n",
       "52329  pourquoi avons-nous réussi à atteindre l'indép...  \n",
       "52330     contenu introduction aux its saviez-vous que…?  \n",
       "\n",
       "[52331 rows x 2 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing is that we will need to collate inputs and targets in a batch: they have different lengths so we need to add padding to make the sequence length the same;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_data??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def seq2seq_collate(samples, pad_idx=1, pad_first=True, backwards=False):\n",
    "    \"Function that collect samples and adds padding. Flips token order if needed\"\n",
    "    samples = to_data(samples)\n",
    "    # maximum question length\n",
    "    max_len_x,max_len_y = max([len(s[0]) for s in samples]),max([len(s[1]) for s in samples])\n",
    "    # initialize batch tensors to zero\n",
    "    # add pad_idx = 1 to each tensor via broadcase (could have used torch.ones() instead of torch.zeros() )\n",
    "    res_x = torch.zeros(len(samples), max_len_x).long() + pad_idx\n",
    "    res_y = torch.zeros(len(samples), max_len_y).long() + pad_idx\n",
    "    if backwards: pad_first = not pad_first\n",
    "    for i,s in enumerate(samples):\n",
    "        if pad_first: \n",
    "            res_x[i,-len(s[0]):],res_y[i,-len(s[1]):] = LongTensor(s[0]),LongTensor(s[1])\n",
    "        else:         \n",
    "            res_x[i,:len(s[0]):],res_y[i,:len(s[1]):] = LongTensor(s[0]),LongTensor(s[1])\n",
    "    if backwards: res_x,res_y = res_x.flip(1),res_y.flip(1)\n",
    "    return res_x,res_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we create a special `DataBunch` that uses this collate function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc(Dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc(DataLoader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "doc(DataBunch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqDataBunch(TextDataBunch):\n",
    "    \"Create a `TextDataBunch` suitable for training an RNN classifier.\"\n",
    "    @classmethod\n",
    "    def create(cls, train_ds, valid_ds, test_ds=None, path:PathOrStr='.', bs:int=32, val_bs:int=None, pad_idx=1,\n",
    "               dl_tfms=None, pad_first=False, device:torch.device=None, no_check:bool=False, backwards:bool=False, **dl_kwargs) -> DataBunch:\n",
    "        \"Function that transform the `datasets` in a `DataBunch` for classification. Passes `**dl_kwargs` on to `DataLoader()`\"\n",
    "        datasets = cls._init_ds(train_ds, valid_ds, test_ds)\n",
    "        val_bs = ifnone(val_bs, bs)\n",
    "        collate_fn = partial(seq2seq_collate, pad_idx=pad_idx, pad_first=pad_first, backwards=backwards)\n",
    "        train_sampler = SortishSampler(datasets[0].x, key=lambda t: len(datasets[0][t][0].data), bs=bs//2)\n",
    "        train_dl = DataLoader(datasets[0], batch_size=bs, sampler=train_sampler, drop_last=True, **dl_kwargs)\n",
    "        dataloaders = [train_dl]\n",
    "        for ds in datasets[1:]:\n",
    "            lengths = [len(t) for t in ds.x.items]\n",
    "            sampler = SortSampler(ds.x, key=lengths.__getitem__)\n",
    "            dataloaders.append(DataLoader(ds, batch_size=val_bs, sampler=sampler, **dl_kwargs))\n",
    "        #return cls(*dataloaders, path=path, device=device, collate_fn=collate_fn, no_check=no_check)\n",
    "        return cls(*dataloaders, path=path, device=torch.device(type='cpu'), collate_fn=collate_fn, no_check=no_check)\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "SortishSampler??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And a subclass of `TextList` that will use this `DataBunch` class in the call `.databunch` and will use `TextList` to label (since our targets are other texts)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['__annotations__',\n",
       " '__call__',\n",
       " '__class__',\n",
       " '__closure__',\n",
       " '__code__',\n",
       " '__defaults__',\n",
       " '__delattr__',\n",
       " '__dict__',\n",
       " '__dir__',\n",
       " '__doc__',\n",
       " '__eq__',\n",
       " '__format__',\n",
       " '__ge__',\n",
       " '__get__',\n",
       " '__getattribute__',\n",
       " '__globals__',\n",
       " '__gt__',\n",
       " '__hash__',\n",
       " '__init__',\n",
       " '__init_subclass__',\n",
       " '__kwdefaults__',\n",
       " '__le__',\n",
       " '__lt__',\n",
       " '__module__',\n",
       " '__name__',\n",
       " '__ne__',\n",
       " '__new__',\n",
       " '__qualname__',\n",
       " '__reduce__',\n",
       " '__reduce_ex__',\n",
       " '__repr__',\n",
       " '__setattr__',\n",
       " '__sizeof__',\n",
       " '__str__',\n",
       " '__subclasshook__']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir(TextList.databunch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqTextList(TextList):\n",
    "    _bunch = Seq2SeqDataBunch\n",
    "    _label_cls = TextList"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thats all we need to use the data block API!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For Windows 10: deal with `BrokenProcessPool` Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "failure count is 90\n",
      "\n",
      "Wall time: 58min 37s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# throws `BrokenProcessPool' Error sometimes. Keep trying `till it works!\n",
    "count = 0\n",
    "error = True\n",
    "while error:\n",
    "    try: \n",
    "        # Preprocessing steps\n",
    "        src = Seq2SeqTextList.from_df(df, path = path, cols='fr')\n",
    "        src = src.split_by_rand_pct(seed=42)\n",
    "        src = src.label_from_df(cols='en', label_cls=TextList)\n",
    "        error = False\n",
    "        print(f'failure count is {count}\\n')    \n",
    "    except: # catch *all* exceptions\n",
    "        # accumulate failure count\n",
    "        count = count + 1\n",
    "        print(f'failure count is {count}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.0"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.percentile([len(o) for o in src.train.x.items] + [len(o) for o in src.valid.x.items], 90)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23.0"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.percentile([len(o) for o in src.train.y.items] + [len(o) for o in src.valid.y.items], 90)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We remove the items where one of the target is more than 30 tokens long."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "src = src.filter_by_func(lambda x,y: len(x) > 30 or len(y) > 30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "52331"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(src.train) + len(src.valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# avoid the BrokenPipeError?\n",
    "data = src.databunch(num_workers = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Seq2SeqDataBunch;\n",
       "\n",
       "Train: LabelList (41865 items)\n",
       "x: Seq2SeqTextList\n",
       "xxbos contenu introduction aux its saviez - vous que … ?,xxbos contenu introduction aux its saviez - vous que … ?,xxbos contenu introduction aux its saviez - vous que … ?,xxbos contenu introduction aux its saviez - vous que … ?,xxbos contenu introduction aux its saviez - vous que … ?\n",
       "y: TextList\n",
       "xxbos what 's inside introduction to xxunk did you know … ?,xxbos what 's inside introduction to xxunk did you know … ?,xxbos what 's inside introduction to xxunk did you know … ?,xxbos what 's inside introduction to xxunk did you know … ?,xxbos what 's inside introduction to xxunk did you know … ?\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\giga-fren;\n",
       "\n",
       "Valid: LabelList (10466 items)\n",
       "x: Seq2SeqTextList\n",
       "xxbos quand demander une protection à l’étranger ?,xxbos quand demander une protection à l’étranger ?,xxbos quand demander une protection à l’étranger ?,xxbos quand demander une protection à l’étranger ?,xxbos quand demander une protection à l’étranger ?\n",
       "y: TextList\n",
       "xxbos when to apply for protection abroad ?,xxbos when to apply for protection abroad ?,xxbos when to apply for protection abroad ?,xxbos when to apply for protection abroad ?,xxbos when to apply for protection abroad ?\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\giga-fren;\n",
       "\n",
       "Test: None"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "WindowsPath('C:/Users/cross-entropy/.fastai/data/giga-fren')"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = load_data(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos pourquoi avons - nous réussi à atteindre l'indépendance de la xxunk alors que d'autres ont encore de la difficulté à comprendre ce concept ?</td>\n",
       "      <td>xxbos why is it that we have achieved judicial independence when others have a hard time even understanding it ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos pourquoi avons - nous réussi à atteindre l'indépendance de la xxunk alors que d'autres ont encore de la difficulté à comprendre ce concept ?</td>\n",
       "      <td>xxbos why is it that we have achieved judicial independence when others have a hard time even understanding it ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos pourquoi avons - nous réussi à atteindre l'indépendance de la xxunk alors que d'autres ont encore de la difficulté à comprendre ce concept ?</td>\n",
       "      <td>xxbos why is it that we have achieved judicial independence when others have a hard time even understanding it ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos pourquoi avons - nous réussi à atteindre l'indépendance de la xxunk alors que d'autres ont encore de la difficulté à comprendre ce concept ?</td>\n",
       "      <td>xxbos why is it that we have achieved judicial independence when others have a hard time even understanding it ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos pourquoi avons - nous réussi à atteindre l'indépendance de la xxunk alors que d'autres ont encore de la difficulté à comprendre ce concept ?</td>\n",
       "      <td>xxbos why is it that we have achieved judicial independence when others have a hard time even understanding it ?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.show_batch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos pourquoi avons - nous réussi à atteindre l'indépendance de la xxunk alors que d'autres ont encore de la difficulté à comprendre ce concept ?</td>\n",
       "      <td>xxbos why is it that we have achieved judicial independence when others have a hard time even understanding it ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos pourquoi avons - nous réussi à atteindre l'indépendance de la xxunk alors que d'autres ont encore de la difficulté à comprendre ce concept ?</td>\n",
       "      <td>xxbos why is it that we have achieved judicial independence when others have a hard time even understanding it ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos pourquoi avons - nous réussi à atteindre l'indépendance de la xxunk alors que d'autres ont encore de la difficulté à comprendre ce concept ?</td>\n",
       "      <td>xxbos why is it that we have achieved judicial independence when others have a hard time even understanding it ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos pourquoi avons - nous réussi à atteindre l'indépendance de la xxunk alors que d'autres ont encore de la difficulté à comprendre ce concept ?</td>\n",
       "      <td>xxbos why is it that we have achieved judicial independence when others have a hard time even understanding it ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos pourquoi avons - nous réussi à atteindre l'indépendance de la xxunk alors que d'autres ont encore de la difficulté à comprendre ce concept ?</td>\n",
       "      <td>xxbos why is it that we have achieved judicial independence when others have a hard time even understanding it ?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.show_batch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['__class__',\n",
       " '__delattr__',\n",
       " '__dict__',\n",
       " '__dir__',\n",
       " '__doc__',\n",
       " '__eq__',\n",
       " '__format__',\n",
       " '__ge__',\n",
       " '__getattr__',\n",
       " '__getattribute__',\n",
       " '__gt__',\n",
       " '__hash__',\n",
       " '__init__',\n",
       " '__init_subclass__',\n",
       " '__le__',\n",
       " '__lt__',\n",
       " '__module__',\n",
       " '__ne__',\n",
       " '__new__',\n",
       " '__reduce__',\n",
       " '__reduce_ex__',\n",
       " '__repr__',\n",
       " '__setattr__',\n",
       " '__setstate__',\n",
       " '__sizeof__',\n",
       " '__str__',\n",
       " '__subclasshook__',\n",
       " '__weakref__',\n",
       " '_grab_dataset',\n",
       " '_init_ds',\n",
       " 'add_test',\n",
       " 'add_tfm',\n",
       " 'batch_size',\n",
       " 'create',\n",
       " 'device',\n",
       " 'dl',\n",
       " 'dl_tfms',\n",
       " 'dls',\n",
       " 'empty_val',\n",
       " 'export',\n",
       " 'fix_dl',\n",
       " 'fix_ds',\n",
       " 'from_csv',\n",
       " 'from_df',\n",
       " 'from_folder',\n",
       " 'from_ids',\n",
       " 'from_tokens',\n",
       " 'is_empty',\n",
       " 'label_list',\n",
       " 'load',\n",
       " 'load_empty',\n",
       " 'loss_func',\n",
       " 'one_batch',\n",
       " 'one_item',\n",
       " 'path',\n",
       " 'remove_tfm',\n",
       " 'sanity_check',\n",
       " 'save',\n",
       " 'show_batch',\n",
       " 'single_dl',\n",
       " 'single_ds',\n",
       " 'test_dl',\n",
       " 'test_ds',\n",
       " 'train_dl',\n",
       " 'train_ds',\n",
       " 'valid_dl',\n",
       " 'valid_ds']"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create our Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pretrained embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need to download the word embeddings (crawl vectors) from the fastText docs. FastText has [pre-trained word vectors](https://fasttext.cc/docs/en/crawl-vectors.html) for 157 languages, trained on Common Crawl and Wikipedia. These models were trained using CBOW.\n",
    "\n",
    "If you need a refresher on word embeddings, you can check out my gentle intro in [this word embedding workshop](https://www.youtube.com/watch?v=25nC0n9ERq4&list=PLtmWHNX-gukLQlMvtRJ19s7-8MrnRV6h6&index=10&t=0s) with accompanying [github repo](https://github.com/fastai/word-embeddings-workshop). \n",
    "\n",
    "More reading on CBOW (Continuous Bag of Words vs. Skip-grams):\n",
    "\n",
    "- [fastText tutorial](https://fasttext.cc/docs/en/unsupervised-tutorial.html#advanced-readers-skipgram-versus-cbow)\n",
    "- [StackOverflow](https://stackoverflow.com/questions/38287772/cbow-v-s-skip-gram-why-invert-context-and-target-words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To install fastText (Linux or Mac):\n",
    "```\n",
    "# clone into ~/\n",
    "$ git clone https://github.com/facebookresearch/fastText.git\n",
    "$ cd fastText\n",
    "$ pip install .\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fasttext as ft"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['__doc__',\n",
       " '__file__',\n",
       " '__loader__',\n",
       " '__name__',\n",
       " '__package__',\n",
       " '__path__',\n",
       " '__spec__']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir(ft)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the fastText pretrained word vector objects `cc.fr.300.bin` and  `cc.fr.300.bin`\n",
    "The lines to download the word vectors only need to be run once:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For Windows 10: in an anaconda terminal window, in the directory `~/.fastai/data/giga-fren/` execute the following two lines manually, then extract the `.gz` files with `7-zip`:\n",
    "\n",
    "`wget https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.en.300.bin.gz`\n",
    "\n",
    "`wget https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.fr.300.bin.gz`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! wget https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.en.300.bin.gz -P {path}\n",
    "# ! wget https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.fr.300.bin.gz -P {path}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# gunzip {path} / cc.en.300.bin.gz\n",
    "# gunzip {path} / cc.fr.300.bin.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the fastText pretrained word vector objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "module 'fasttext' has no attribute 'load_model'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-42-9ba2598ed0b4>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mfr_vecs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mft\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mload_model\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[1;33m/\u001b[0m\u001b[1;34m'cc.fr.300.bin'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0men_vecs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mft\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mload_model\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[1;33m/\u001b[0m\u001b[1;34m'cc.en.300.bin'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mAttributeError\u001b[0m: module 'fasttext' has no attribute 'load_model'"
     ]
    }
   ],
   "source": [
    "fr_vecs = ft.load_model(str((path/'cc.fr.300.bin')))\n",
    "en_vecs = ft.load_model(str((path/'cc.en.300.bin')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create embeddings for English and French sentences.\n",
    "We create an embedding module with the pretrained vectors and random data for the missing parts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_emb(vecs, itos, em_sz=300, mult=1.):\n",
    "    # input fastText object, itos for vocabulary, size of embedding, mult\n",
    "    emb = nn.Embedding(len(itos), em_sz, padding_idx=1)\n",
    "    wgts = emb.weight.data\n",
    "    # {word:vector} dictionary\n",
    "    vec_dic = {w:vecs.get_word_vector(w) for w in vecs.get_words()}\n",
    "    miss = []\n",
    "    for i,w in enumerate(itos):\n",
    "        try: wgts[i] = tensor(vec_dic[w])\n",
    "        except: miss.append(w)\n",
    "    return emb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(fr_vecs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dir(fr_vecs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fr_vecs.words[:12]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fr_vecs.labels[:12]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## embedding encoder and decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "emb_enc = create_emb(fr_vecs, data.x.vocab.itos)\n",
    "emb_dec = create_emb(en_vecs, data.y.vocab.itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dir(emb_enc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "emb_enc.cpu()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "emb_enc.cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "emb_enc.weight.size(), emb_dec.weight.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_path = Config().model_path()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(emb_enc, model_path/'fr_emb.pth')\n",
    "torch.save(emb_dec, model_path/'en_emb.pth')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "emb_enc = torch.load(model_path/'fr_emb.pth')\n",
    "emb_dec = torch.load(model_path/'en_emb.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Review Question: What are the two types of numbers in deep learning?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoders & Decoders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model in itself consists in an **encoder** and a **decoder**\n",
    "\n",
    "![Seq2seq model](images/seq2seq.png)\n",
    "\n",
    "<center><i>Diagram from Smerity's <a href=\"https://smerity.com/articles/2016/google_nmt_arch.html\">Peeking into the neural network architecture used for Google's Neural Machine Translation</a></i></center>\n",
    "\n",
    "The **encoder** is a recurrent neural net (RNN). Given an **input sentence**, it produces:\n",
    "\n",
    "* an **output**, which we discard,\n",
    "* a **hidden state** -- the set of activations produced by the input sentence\n",
    "\n",
    "\n",
    "The **hidden state** is then passed into the **decoder** (another RNN).\n",
    "\n",
    "Given a **hidden state** decoder produces\n",
    "* **outputs** -- which are softmaxed to produce the **translation of the input sentence**\n",
    "* a **hidden state**, which we discard\n",
    "\n",
    "We loop until the **decoder** produces a padding token (or at 30 iterations to make sure it's not an infinite loop at the beginning of training). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use a GRU for our encoder and a separate GRU for our decoder. Other options are to use LSTMs or QRNNs (see here).  GRUs, LSTMs, and QRNNs all solve the problem of how RNNs can lack long-term memory.\n",
    "\n",
    "Links:\n",
    "- [Illustrated Guide to LSTM’s and GRU’s: A step by step explanation](https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21)\n",
    "- [fast.ai implementation of seq2seq with QRNNs](https://github.com/fastai/fastai_docs/blob/master/dev_course/dl2/translation.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqRNN(nn.Module):\n",
    "    def __init__(self, emb_enc, emb_dec, \n",
    "                    nh, out_sl, \n",
    "                    nl=2, bos_idx=0, pad_idx=1):\n",
    "        super().__init__()\n",
    "        self.nl,self.nh,self.out_sl = nl,nh,out_sl\n",
    "        self.bos_idx,self.pad_idx = bos_idx,pad_idx\n",
    "        self.em_sz_enc = emb_enc.embedding_dim\n",
    "        self.em_sz_dec = emb_dec.embedding_dim\n",
    "        self.voc_sz_dec = emb_dec.num_embeddings\n",
    "                 \n",
    "        self.emb_enc = emb_enc\n",
    "        self.emb_enc_drop = nn.Dropout(0.15)\n",
    "        self.gru_enc = nn.GRU(self.em_sz_enc, nh, num_layers=nl,\n",
    "                              dropout=0.25, batch_first=True)\n",
    "        self.out_enc = nn.Linear(nh, self.em_sz_dec, bias=False)\n",
    "        \n",
    "        self.emb_dec = emb_dec\n",
    "        self.gru_dec = nn.GRU(self.em_sz_dec, self.em_sz_dec, num_layers=nl,\n",
    "                              dropout=0.1, batch_first=True)\n",
    "        self.out_drop = nn.Dropout(0.35)\n",
    "        self.out = nn.Linear(self.em_sz_dec, self.voc_sz_dec)\n",
    "        self.out.weight.data = self.emb_dec.weight.data\n",
    "        \n",
    "    def encoder(self, bs, inp):\n",
    "        h = self.initHidden(bs)\n",
    "        emb = self.emb_enc_drop(self.emb_enc(inp))\n",
    "        _, h = self.gru_enc(emb, h)\n",
    "        h = self.out_enc(h)\n",
    "        return h\n",
    "    \n",
    "    def decoder(self, dec_inp, h):\n",
    "        emb = self.emb_dec(dec_inp).unsqueeze(1)\n",
    "        outp, h = self.gru_dec(emb, h)\n",
    "        outp = self.out(self.out_drop(outp[:,0]))\n",
    "        return h, outp\n",
    "        \n",
    "    def forward(self, inp):\n",
    "        bs, sl = inp.size()\n",
    "        h = self.encoder(bs, inp)\n",
    "        dec_inp = inp.new_zeros(bs).long() + self.bos_idx\n",
    "        \n",
    "        res = []\n",
    "        for i in range(self.out_sl):\n",
    "            h, outp = self.decoder(dec_inp, h)\n",
    "            dec_inp = outp.max(1)[1]\n",
    "            res.append(outp)\n",
    "            if (dec_inp==self.pad_idx).all(): break\n",
    "        return torch.stack(res, dim=1)\n",
    "    \n",
    "    def initHidden(self, bs): return one_param(self).new_zeros(self.nl, bs, self.nh)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.valid_dl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(data.device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dir(torch.device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Throws BrokenPipeError!\n",
    "xb,yb = next(iter(data.valid_dl))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xb.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnn = Seq2SeqRNN(emb_enc, emb_dec, 256, 30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "rnn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(xb[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "h = rnn.encoder(64, xb.cpu())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "h.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss pads output and target so that they are of the same size before using the usual flattened version of cross entropy. We do the same for accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def seq2seq_loss(out, targ, pad_idx=1):\n",
    "    bs,targ_len = targ.size()\n",
    "    _,out_len,vs = out.size()\n",
    "    if targ_len>out_len: out  = F.pad(out,  (0,0,0,targ_len-out_len,0,0), value=pad_idx)\n",
    "    if out_len>targ_len: targ = F.pad(targ, (0,out_len-targ_len,0,0), value=pad_idx)\n",
    "    return CrossEntropyFlat()(out, targ)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, rnn, loss_func=seq2seq_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit_one_cycle(4, 1e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's free up some RAM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del fr_vecs\n",
    "del en_vecs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As loss is not very interpretable, let's also look at the accuracy.  Again, we will add padding so that the output and target are of the same length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def seq2seq_acc(out, targ, pad_idx=1):\n",
    "    bs,targ_len = targ.size()\n",
    "    _,out_len,vs = out.size()\n",
    "    if targ_len>out_len: out  = F.pad(out,  (0,0,0,targ_len-out_len,0,0), value=pad_idx)\n",
    "    if out_len>targ_len: targ = F.pad(targ, (0,out_len-targ_len,0,0), value=pad_idx)\n",
    "    out = out.argmax(2)\n",
    "    return (out==targ).float().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bleu metric (see dedicated notebook)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In translation, the metric usually used is BLEU.\n",
    "\n",
    "A great post by Rachael Tatman: [Evaluating Text Output in NLP: BLEU at your own risk](https://towardsdatascience.com/evaluating-text-output-in-nlp-bleu-at-your-own-risk-e8609665a213)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NGram():\n",
    "    def __init__(self, ngram, max_n=5000): self.ngram,self.max_n = ngram,max_n\n",
    "    def __eq__(self, other):\n",
    "        if len(self.ngram) != len(other.ngram): return False\n",
    "        return np.all(np.array(self.ngram) == np.array(other.ngram))\n",
    "    def __hash__(self): return int(sum([o * self.max_n**i for i,o in enumerate(self.ngram)]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_grams(x, n, max_n=5000):\n",
    "    return x if n==1 else [NGram(x[i:i+n], max_n=max_n) for i in range(len(x)-n+1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_correct_ngrams(pred, targ, n, max_n=5000):\n",
    "    pred_grams,targ_grams = get_grams(pred, n, max_n=max_n),get_grams(targ, n, max_n=max_n)\n",
    "    pred_cnt,targ_cnt = Counter(pred_grams),Counter(targ_grams)\n",
    "    return sum([min(c, targ_cnt[g]) for g,c in pred_cnt.items()]),len(pred_grams)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CorpusBLEU(Callback):\n",
    "    def __init__(self, vocab_sz):\n",
    "        self.vocab_sz = vocab_sz\n",
    "        self.name = 'bleu'\n",
    "    \n",
    "    def on_epoch_begin(self, **kwargs):\n",
    "        self.pred_len,self.targ_len,self.corrects,self.counts = 0,0,[0]*4,[0]*4\n",
    "    \n",
    "    def on_batch_end(self, last_output, last_target, **kwargs):\n",
    "        last_output = last_output.argmax(dim=-1)\n",
    "        for pred,targ in zip(last_output.cpu().numpy(),last_target.cpu().numpy()):\n",
    "            self.pred_len += len(pred)\n",
    "            self.targ_len += len(targ)\n",
    "            for i in range(4):\n",
    "                c,t = get_correct_ngrams(pred, targ, i+1, max_n=self.vocab_sz)\n",
    "                self.corrects[i] += c\n",
    "                self.counts[i]   += t\n",
    "    \n",
    "    def on_epoch_end(self, last_metrics, **kwargs):\n",
    "        precs = [c/t for c,t in zip(self.corrects,self.counts)]\n",
    "        len_penalty = exp(1 - self.targ_len/self.pred_len) if self.pred_len < self.targ_len else 1\n",
    "        bleu = len_penalty * ((precs[0]*precs[1]*precs[2]*precs[3]) ** 0.25)\n",
    "        return add_metrics(last_metrics, bleu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training with metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "learn = Learner(data, rnn, loss_func=seq2seq_loss, metrics=[seq2seq_acc, CorpusBLEU(len(data.y.vocab.itos))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit_one_cycle(4, 1e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "learn.fit_one_cycle(4, 1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So how good is our model? Let's see a few predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_predictions(learn, ds_type=DatasetType.Valid):\n",
    "    learn.model.eval()\n",
    "    inputs, targets, outputs = [],[],[]\n",
    "    with torch.no_grad():\n",
    "        for xb,yb in progress_bar(learn.dl(ds_type)):\n",
    "            out = learn.model(xb)\n",
    "            for x,y,z in zip(xb,yb,out):\n",
    "                inputs.append(learn.data.train_ds.x.reconstruct(x))\n",
    "                targets.append(learn.data.train_ds.y.reconstruct(y))\n",
    "                outputs.append(learn.data.train_ds.y.reconstruct(z.argmax(1)))\n",
    "    return inputs, targets, outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs, targets, outputs = get_predictions(learn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs[700], targets[700], outputs[700]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs[701], targets[701], outputs[701]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs[2513], targets[2513], outputs[2513]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs[4000], targets[4000], outputs[4000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's usually beginning well, but falls into repeated words at the end of the question."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Teacher forcing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way to help training is to help the decoder by feeding it the real targets instead of its predictions (if it starts with wrong words, it's very unlikely to give us the right translation). We do that all the time at the beginning, then progressively reduce the amount of teacher forcing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TeacherForcing(LearnerCallback):\n",
    "    \n",
    "    def __init__(self, learn, end_epoch):\n",
    "        super().__init__(learn)\n",
    "        self.end_epoch = end_epoch\n",
    "    \n",
    "    def on_batch_begin(self, last_input, last_target, train, **kwargs):\n",
    "        if train: return {'last_input': [last_input, last_target]}\n",
    "    \n",
    "    def on_epoch_begin(self, epoch, **kwargs):\n",
    "        self.learn.model.pr_force = 1 - epoch/self.end_epoch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will add the following code to our `forward` method:\n",
    "\n",
    "```\n",
    "    if (targ is not None) and (random.random()<self.pr_force):\n",
    "        if i>=targ.shape[1]: break\n",
    "        dec_inp = targ[:,i]\n",
    "```\n",
    "Additionally, `forward` will take an additional argument of `target`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqRNN_tf(nn.Module):\n",
    "    def __init__(self, emb_enc, emb_dec, nh, out_sl, nl=2, bos_idx=0, pad_idx=1):\n",
    "        super().__init__()\n",
    "        self.nl,self.nh,self.out_sl = nl,nh,out_sl\n",
    "        self.bos_idx,self.pad_idx = bos_idx,pad_idx\n",
    "        self.em_sz_enc = emb_enc.embedding_dim\n",
    "        self.em_sz_dec = emb_dec.embedding_dim\n",
    "        self.voc_sz_dec = emb_dec.num_embeddings\n",
    "                 \n",
    "        self.emb_enc = emb_enc\n",
    "        self.emb_enc_drop = nn.Dropout(0.15)\n",
    "        self.gru_enc = nn.GRU(self.em_sz_enc, nh, num_layers=nl,\n",
    "                              dropout=0.25, batch_first=True)\n",
    "        self.out_enc = nn.Linear(nh, self.em_sz_dec, bias=False)\n",
    "        \n",
    "        self.emb_dec = emb_dec\n",
    "        self.gru_dec = nn.GRU(self.em_sz_dec, self.em_sz_dec, num_layers=nl,\n",
    "                              dropout=0.1, batch_first=True)\n",
    "        self.out_drop = nn.Dropout(0.35)\n",
    "        self.out = nn.Linear(self.em_sz_dec, self.voc_sz_dec)\n",
    "        self.out.weight.data = self.emb_dec.weight.data\n",
    "        self.pr_force = 0.\n",
    "        \n",
    "    def encoder(self, bs, inp):\n",
    "        h = self.initHidden(bs)\n",
    "        emb = self.emb_enc_drop(self.emb_enc(inp))\n",
    "        _, h = self.gru_enc(emb, h)\n",
    "        h = self.out_enc(h)\n",
    "        return h\n",
    "    \n",
    "    def decoder(self, dec_inp, h):\n",
    "        emb = self.emb_dec(dec_inp).unsqueeze(1)\n",
    "        outp, h = self.gru_dec(emb, h)\n",
    "        outp = self.out(self.out_drop(outp[:,0]))\n",
    "        return h, outp\n",
    "            \n",
    "    def forward(self, inp, targ=None):\n",
    "        bs, sl = inp.size()\n",
    "        h = self.encoder(bs, inp)\n",
    "        dec_inp = inp.new_zeros(bs).long() + self.bos_idx\n",
    "        \n",
    "        res = []\n",
    "        for i in range(self.out_sl):\n",
    "            h, outp = self.decoder(dec_inp, h)\n",
    "            res.append(outp)\n",
    "            dec_inp = outp.max(1)[1]\n",
    "            if (dec_inp==self.pad_idx).all(): break\n",
    "            if (targ is not None) and (random.random()<self.pr_force):\n",
    "                if i>=targ.shape[1]: continue\n",
    "                dec_inp = targ[:,i]\n",
    "        return torch.stack(res, dim=1)\n",
    "\n",
    "    def initHidden(self, bs): return one_param(self).new_zeros(self.nl, bs, self.nh)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "emb_enc = torch.load(model_path/'fr_emb.pth')\n",
    "emb_dec = torch.load(model_path/'en_emb.pth')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnn_tf = Seq2SeqRNN_tf(emb_enc, emb_dec, 256, 30)\n",
    "\n",
    "learn = Learner(data, rnn_tf, loss_func=seq2seq_loss, metrics=[seq2seq_acc, CorpusBLEU(len(data.y.vocab.itos))],\n",
    "               callback_fns=partial(TeacherForcing, end_epoch=3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "learn.fit_one_cycle(6, 3e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs, targets, outputs = get_predictions(learn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs[700],targets[700],outputs[700]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs[2513], targets[2513], outputs[2513]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs[4000], targets[4000], outputs[4000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}