5-nn-imdb_orig.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This lesson was adapted from the end of [lesson 3](https://course.fast.ai/videos/?lesson=3) and beginning of [lesson 4](https://course.fast.ai/videos/?lesson=4) of the latest fast.ai Practical Deep Learning for Coders course.  We will cover all the material you need here in this notebook, so no need to have taken the Deep Learning course.  Even if you have taken the DL class, we will go slower and get into more detail here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Language Modeling & Sentiment Analysis of IMDB movie reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be looking at IMDB movie reviews.  We want to determine if a review is negative or positive, based on the text.  In order to do this, we will be using **transfer learning**.\n",
    "\n",
    "Transfer learning has been widely used with great success in computer vision for several years, but only in the last year or so has it been successfully applied to NLP (beginning with ULMFit, which we will use here, which was built upon by BERT and GPT-2).\n",
    "\n",
    "As Sebastian Ruder wrote in [The Gradient](https://thegradient.pub/) last summer, [NLP's ImageNet moment has arrived](https://thegradient.pub/nlp-imagenet/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Language Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Language modeling can be a fun creative form. Research scientist [Janelle Shane blogs](https://aiweirdness.com/) & [tweets](https://twitter.com/JanelleCShane) about her creative AI explorations, which often involve text.  For instance, see her:\n",
    "\n",
    "- [Why did the neural network cross the road?](https://aiweirdness.com/post/174691534037/why-did-the-neural-network-cross-the-road)\n",
    "- [Try these neural network-generated recipes at your own risk.](https://aiweirdness.com/post/163878889437/try-these-neural-network-generated-recipes-at-your)\n",
    "- [D&D character bios - now making slightly more sense](https://aiweirdness.com/post/183471928977/dd-character-bios-now-making-slightly-more)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need to have the fastai library installed for this lesson, and you will want to use a GPU to train your neural net.  If you don't have a GPU you can use in your computer (currently, only Nvidia GPUs are fully supported by the main deep learning libraries), no worries!  There are a number of cloud options you can consider:\n",
    "\n",
    "[GPU Cloud Options](https://course.fast.ai/#using-a-gpu)\n",
    "\n",
    "**Reminder: If you are using a cloud GPU, always be sure to shut it down when you are done!!! Otherwise, you could end up with an expensive bill!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import fastai.utils.collect_env\n",
    "\n",
    "# fastai.utils.collect_env.show_install()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that language models can use a lot of GPU, so you may need to decrease batchsize here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# bs=48\n",
    "# bs=24\n",
    "bs=192"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.cuda.set_device(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Preparing the data (on a sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "First let's download the dataset we are going to study. The [dataset](http://ai.stanford.edu/~amaas/data/sentiment/) has been curated by Andrew Maas et al. and contains a total of 100,000 reviews on IMDB. 25,000 of them are labelled as positive and negative for training, another 25,000 are labelled for testing (in both cases they are highly polarized). The remaning 50,000 is an additional unlabelled data (but we will find a use for it nonetheless).\n",
    "\n",
    "We'll begin with a sample we've prepared for you, so that things run quickly before going over the full dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jhoward/.fastai/data/imdb_sample/data_lm_export.pkl'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb_sample/data_clas_export.pkl'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb_sample/models'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb_sample/texts.csv')]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.IMDB_SAMPLE)\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "It only contains one csv file, let's have a look at it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "It contains one line per review, with the label ('negative' or 'positive'), the text and a flag to determine if it should be part of the validation set or the training set. If we ignore this flag, we can create a DataBunch containing this data in one line of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data_lm = TextDataBunch.from_csv(path, 'texts.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "By executing this line a process was launched that took a bit of time. Let's dig a bit into it. Images could be fed (almost) directly into a model because they're just a big array of pixel values that are floats between 0 and 1. A text is composed of words, and we can't apply mathematical functions to them directly. We first have to convert them to numbers. This is done in two differents steps: tokenization and numericalization. A `TextDataBunch` does all of that behind the scenes for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Before we delve into the explanations, let's take the time to save the things that were calculated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The first step of processing we make texts go through is to split the raw sentences into words, or more exactly tokens. The easiest way to do this would be to split the string on spaces, but we can be smarter:\n",
    "\n",
    "- we need to take care of punctuation\n",
    "- some words are contractions of two different words, like isn't or don't\n",
    "- we may need to clean some parts of our texts, if there's HTML code for instance\n",
    "\n",
    "To see what the tokenizer had done behind the scenes, let's have a look at a few texts in a batch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The texts are truncated at 100 tokens for more readability. We can see that it did more than just split on space and punctuation symbols: \n",
    "- the \"'s\" are grouped together in one token\n",
    "- the contractions are separated like his: \"did\", \"n't\"\n",
    "- content has been cleaned for any HTML symbol and lower cased\n",
    "- there are several special tokens (all those that begin by xx), to replace unkown tokens (see below) or to introduce different text fields (here we only have one)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Numericalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Once we have extracted tokens from our texts, we convert to integers by creating a list of all the words used. We only keep the ones that appear at list twice with a maximum vocabulary size of 60,000 (by default) and replace the ones that don't make the cut by the unknown token `UNK`.\n",
    "\n",
    "The correspondance from ids tokens is stored in the `vocab` attribute of our datasets, in a dictionary called `itos` (for int to string)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxunk',\n",
       " 'xxpad',\n",
       " 'xxbos',\n",
       " 'xxeos',\n",
       " 'xxfld',\n",
       " 'xxmaj',\n",
       " 'xxup',\n",
       " 'xxrep',\n",
       " 'xxwrep',\n",
       " 'the']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.vocab.itos[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "And if we look at what a what's in our datasets, we'll see the tokenized text as a representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text xxbos i remember when i first saw this short , i was really laughing so hard , that like with a lot of other films that i have seen , no sound came out ! xxmaj curly is really great at \" singing \" opera in this one , i am surprised that he did not consider a career as a professional singer , because he was really good ! \n",
       " \n",
       "  xxmaj if you noticed , this was filmed near the end of xxmaj curly 's career as a xxmaj xxunk , you could really tell he had changed , because he had lost weight and was xxunk , his voice was xxunk , his face was getting lined with wrinkles , though he still could pull it off , he looked like he was fifty at the age of forty . xxmaj this was because he was suffering many minor strokes before his big one that ended his career . xxmaj be he still managed to pull it off in his last ones ! \n",
       " \n",
       "  xxmaj if you do n't mind the fact that xxmaj curly was really getting very ill at this point , this is actually one of their funniest xxunk . i know that i did n't mind the fact that xxmaj curly was really changing , because i still thought that he was great ! \n",
       " \n",
       "  10 / 10"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.train_ds[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "But the underlying data is all numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  2,  19, 428,  73,  19, 105, 239,  20, 460,  11])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.train_ds[0][0].data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Alternative apporach: with the data block API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We can use the data block API with NLP and have a lot more flexibility than what the default factory methods offer. In the previous example for instance, the data was randomly split between train and validation instead of reading the third column of the csv.\n",
    "\n",
    "With the data block API though, we have to manually call the tokenize and numericalize steps. This allows more flexibility, and if you're not using the defaults from fastai, the various arguments to pass will appear in the step they're revelant, so it'll be more readable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data = (TextList.from_csv(path, 'texts.csv', cols='text')\n",
    "                .split_from_df(col=2)\n",
    "                .label_from_df(cols=0)\n",
    "                .databunch())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's grab the full dataset for what follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jhoward/.fastai/data/imdb/README'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/lm_databunch_bwd'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/test'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/tmp_clas'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/imdb.vocab'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/lm_databunch'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/models'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/tmp_lm'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/imdb_textlist_class'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/unsup'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/train')]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.IMDB)\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jhoward/.fastai/data/imdb/train/labeledBow.feat'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/train/neg'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/train/unsupBow.feat'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/train/pos')]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(path/'train').ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reviews are in a training and test set following an imagenet structure. The only difference is that there is an `unsup` folder in `train` that contains the unlabelled data.\n",
    "\n",
    "We're not going to train a model that classifies the reviews from scratch. Like in computer vision, we'll use a model pretrained on a bigger dataset (a cleaned subset of wikipeia called [wikitext-103](https://einstein.ai/research/blog/the-wikitext-long-term-dependency-language-modeling-dataset)). That model has been trained to guess what the next word, its input being all the previous words. It has a recurrent structure and a hidden state that is updated each time it sees a new word. This hidden state thus contains information about the sentence up to that point.\n",
    "\n",
    "We are going to use that 'knowledge' of the English language to build our classifier, but first, like for computer vision, we need to fine-tune the pretrained model to our particular dataset. Because the English of the reviews left by people on IMDB isn't the same as the English of wikipedia, we'll need to adjust a little bit the parameters of our model. Plus there might be some words extremely common in that dataset that were barely present in wikipedia, and therefore might no be part of the vocabulary the model was trained on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### More about WikiText-103"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We will be using the [WikiText-103](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/) dataset created by [Stephen Merity](https://smerity.com/) to pre-train a language model.\n",
    "\n",
    "To quote [Stephen's post](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/):\n",
    "\n",
    "*The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia. The dataset is available under the Creative Commons Attribution-ShareAlike License.*\n",
    "\n",
    "*Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over 110 times larger. The WikiText dataset also features a far larger vocabulary and retains the original case, punctuation and numbers - all of which are removed in PTB. As it is composed of full articles, the dataset is well suited for models that can take advantage of long term dependencies.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Creating the TextLMDataBunch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "This is where the unlabelled data is going to be useful to us, as we can use it to fine-tune our model. Let's create our data object with the data block API (next line takes a few minutes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jhoward/.fastai/data/imdb/README'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/lm_databunch_bwd'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/test'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/tmp_clas'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/imdb.vocab'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/lm_databunch'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/models'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/tmp_lm'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/imdb_textlist_class'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/unsup'),\n",
       " PosixPath('/home/jhoward/.fastai/data/imdb/train')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data_lm = (TextList.from_folder(path)\n",
    "           #Inputs: all the text files in path\n",
    "            .filter_by_folder(include=['train', 'test', 'unsup']) \n",
    "           #We may have other temp folders that contain text files so we only keep what's in train and test\n",
    "            .split_by_rand_pct(0.1, seed=42)\n",
    "           #We randomly split and keep 10% (10,000 reviews) for validation\n",
    "            .label_for_lm()           \n",
    "           #We want to do a language model so we label accordingly\n",
    "            .databunch(bs=bs, num_workers=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60004, 90000)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data_lm.vocab.itos),len(data_lm.train_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We have to use a special kind of `TextDataBunch` for the language model, that ignores the labels (that's why we put 0 everywhere), will shuffle the texts at each epoch before concatenating them all together (only for training, we don't shuffle for the validation set) and will send batches that read that text in order with targets that are the next word in the sentence.\n",
    "\n",
    "The line before being a bit long, we want to load quickly the final ids by using the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>xxmaj xxunk are too busy dealing with the fairly uninteresting lives of xxmaj stan and xxmaj daniel rather than focusing on xxmaj richie . xxmaj one flashback in a music store , where xxmaj richie has a crush on an employee stands out and really shows the viewer where this film could have gone . xxmaj but in the end , not much . xxmaj two many drawn out scenes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>guy . i mean , xxup how bad can a xxup bad guy xxup be ? ( xxmaj ask xxmaj richard xxmaj johnson ) . xxmaj joseph xxmaj cotten tries hard not to look embarrassed as he staggers through his cameo appearance . xxmaj in the name of all that 's holy , do n't rent this darned bomb . xxbos xxmaj this is quite possibly the worst sequel ever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>made to appeal to the type of person who 's willing to kneel at the alter of either firemen specifically , \" xxmaj heroes \" or gods in general , i suppose . i just do n't have enough bad things to say about this film . xxmaj it could n't have been cheap , either , with all the stars that were in it plus all those special effects</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>, this performance reminds me of the man xxmaj maurice xxmaj xxunk played in xxup gigi ( but without the xxunk cute older man who loves the ladies . i strongly doubt the real - life xxmaj marquis de xxmaj sade would have recognized this character at all ! \\n \\n  xxmaj the film , surprisingly , does n't have a lot of nudity , though what it does</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>even his friend gives money to his father , but the butterfly effect completely fails to return to its cause . \\n \\n  xxmaj this is a very dark film as each character that gets affected , suffers never ending depression . xxmaj none of the normal avenues for relief seem to be effective and the only thing the authorities seem to do is give out quantities of stress</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Let's save our databunch for next time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data_lm.save('lm_databunch')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading saved data, and creating the language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the future we can load the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = load_data(path, 'lm_databunch', bs=bs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then put this in a learner object very easily with a model loaded with the pretrained weights. They'll be downloaded the first time you'll execute the following line and stored in `~/.fastai/models/` (or elsewhere if you specified different paths in your config file)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "learn_lm = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "wiki_itos = pickle.load(open(Config().model_path()/'wt103-1/itos_wt103.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxunk', 'xxpad', 'xxmaj', 'the', ',', '.', 'of', 'and', 'in', 'to']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wiki_itos[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = data_lm.vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30136"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.stoi[\"stingray\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'stingray'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.itos[vocab.stoi[\"stingray\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xxunk'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.itos[vocab.stoi[\"mobula\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "awd = learn_lm.model[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.spatial.distance import cosine as dist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "enc = learn_lm.model[0].encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([49906, 400])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enc.weight.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Difference in vocabulary between IMDB and Wikipedia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We are going to load wiki_itos, which can be downloaded along with wikitext-103.  We will compare the vocabulary from wikitext with the vocabulary in IMDB.  It is to be expected that the two sets have some different vocabulary words, and that is no problem for transfer learning!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60001"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(wiki_itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "49906"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocab.itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "i, unks = 0, []\n",
    "while len(unks) < 50:\n",
    "    if data_lm.vocab.itos[i] not in wiki_itos: unks.append((i,data_lm.vocab.itos[i]))\n",
    "    i += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "wiki_words = set(wiki_itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "imdb_words = set(vocab.itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "wiki_not_imbdb = wiki_words.difference(imdb_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "imdb_not_wiki = imdb_words.difference(wiki_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "wiki_not_imdb_list = []\n",
    "\n",
    "for i in range(100):\n",
    "    word = wiki_not_imbdb.pop()\n",
    "    wiki_not_imdb_list.append(word)\n",
    "    wiki_not_imbdb.add(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['49ers',\n",
       " 'bans',\n",
       " 'christianization',\n",
       " 'severin',\n",
       " 'binaural',\n",
       " '91st',\n",
       " 'graphite',\n",
       " 'campaigners',\n",
       " 'phaser',\n",
       " 'polynomial',\n",
       " 'diamandis',\n",
       " '384',\n",
       " 'irritability',\n",
       " 'crenellated',\n",
       " 'plenipotentiary']"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wiki_not_imdb_list[:15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "imdb_not_wiki_list = []\n",
    "\n",
    "for i in range(100):\n",
    "    word = imdb_not_wiki.pop()\n",
    "    imdb_not_wiki_list.append(word)\n",
    "    imdb_not_wiki.add(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['mannerist',\n",
       " 'mihic',\n",
       " 'drea',\n",
       " \"d'arcy\",\n",
       " 'hohl',\n",
       " 'be-',\n",
       " 'castorini',\n",
       " 'cuties',\n",
       " 'solondz',\n",
       " 'upends',\n",
       " 'babbar',\n",
       " 'heero',\n",
       " 'pronouncement',\n",
       " 'starrett',\n",
       " 'hocus']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imdb_not_wiki_list[:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "All words that appear in the IMDB vocab, but not the wikitext-103 vocab, will be initialized to the same random vector in a model.  As the model trains, we will learn these weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.stoi[\"modernisation\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"modernisation\" in wiki_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24319"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.stoi[\"30-something\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, True)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"30-something\" in wiki_words, \"30-something\" in imdb_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18065"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.stoi[\"linklater\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, True)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"linklater\" in wiki_words, \"linklater\" in imdb_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"house\" in wiki_words, \"house\" in imdb_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(enc.weight[vocab.stoi[\"30-something\"], :], \n",
    "            enc.weight[vocab.stoi[\"linklater\"], :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(enc.weight[vocab.stoi[\"30-something\"], :], \n",
    "            enc.weight[vocab.stoi[\"house\"], :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "new_word_vec = enc.weight[vocab.stoi[\"linklater\"], :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Generating fake movie reviews (using wiki-text model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "TEXT = \"The color of the sky is\"\n",
    "N_WORDS = 40\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The color of the sky is often Black in color , and the color is as black as white . \n",
      " \n",
      "  = = History = = \n",
      " \n",
      "  The Black Hills of New York City , in 1911 ,\n",
      "The color of the sky is the Blue Betty , a double - as in English and Italian , and the Latin word for \" grey \" in English . \n",
      " \n",
      "  = = = Philosophy = = = \n",
      " \n",
      " \n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "TEXT = \"I hated this movie\"\n",
    "N_WORDS = 30\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I hated this movie . The German language magazine Die Saturday News ( German ) , writing on the themes of the German Jews , wrote\n",
      "I hated this movie The Devil Wears Prada . \n",
      " \n",
      "  = = = Cultural references = = = \n",
      " \n",
      "  The film opens with the film Wounded\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I hated this movie , but rather \" in the same way \" , and when he heard Hugh Grant discussing the influences he found , he was \" very strong about\n",
      "I hated this movie , but after seeing the film The Lord of the Rings , they decided to shoot it in Heaven . \n",
      " \n",
      "  = = Production =\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "doc(LanguageLearner.predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Lowering `temperature` will make the texts less randomized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I hated this movie , but it was a popular film . The film was released on DVD in November 2006 . The film was released on DVD and\n",
      "I hated this movie , but the film was a commercial success . It was released on DVD in September 2005 . \n",
      " \n",
      "  = = = Critical reception = =\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.10) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "doc(LanguageLearner.predict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I hated this movie , but the film was a commercial success . The film was released on DVD and Blu - ray in November 2008 . The film\n",
      "I hated this movie , but the film was shot in Los Angeles , California . The film was released on September 15 , 2006 , and was released\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.10) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we want to choose a good learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_lm.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ4AAAEKCAYAAAAiizNaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd8ldX9wPHPN4MECAkjYQYIIHvIiAxRQcRRURBcICiIlVrrah0ttVWLWrW21VqtPxFFBdyj4qoDAQcihD2UHVYgC5KQvb6/P+4TuEASAuTOfN+v133x3PM8z73fw03uN+c85zlHVBVjjDHGW0J8HYAxxpi6xRKPMcYYr7LEY4wxxqss8RhjjPEqSzzGGGO8yhKPMcYYr7LEY4wxxqss8RhjjPEqSzzGGGO8KszXAXhDbGysJiQk+DoMY4wJKCtWrMhQ1bjaft06kXgSEhJISkrydRjGGBNQRGSnJ17XutqMMcZ4lSUeY4wxXmWJxxhjjFdZ4jHGGONVlniMMcZ4lSUeY4wxXmWJxxhjjFd5PPGISKiIrBKRjyvZ115EFojIWhFZJCLxTvn5IrLa7VEoIlc4+14RkR1u+/p6ug7GGBNoNqce4p9fbibtUKGvQzmON1o8dwI/VbHv78BrqtoHmAE8BqCqC1W1r6r2BUYA+cAXbufdW7FfVVd7MHZjjAlIa3Zn8cyCLRQWl/s6lON4NPE4LZhRwKwqDukBLHC2FwJjKjnmKuAzVc2v/QiNMSY4pea4WjrNoyN8HMnxPN3ieRq4D6gq5a4BrnS2xwKNRKTZMceMB944puxRp3vuKRGp9H9VRKaJSJKIJKWnp59i+MYYE5hSc4qIqR9OZHior0M5jscSj4hcBqSp6opqDrsHGCYiq4BhwF6g1O01WgG9gc/dzpkOdAPOApoCv6/shVV1pqomqmpiXFytz3FnjDF+LTWnkJbRkb4Oo1KenCR0KDBaRC4FIoFoEZmrqpMqDlDVFGAcgIhEAVeqarbba1wDfKCqJW7n7HM2i0RkNq7kZYwxxk1qTqFfdrOBB1s8qjpdVeNVNQFXd9nX7kkHQERiRaQihunAy8e8zASO6WZzWkGIiABXAOs9EL4xxgS01Jwiv23xeP0+HhGZISKjnafDgU0ishloATzqdlwC0BZYfMxLzBORdcA6IBZ4xMMhG2NMQCkrV9Jzi2jhp4nHK+vxqOoiYJGz/YBb+bvAu1Wckwy0qaR8hCdiNMaYYJGZW0RZudKirnW1GWOM8Y3UnCIAv23xWOIxxpggs9+5h8cSjzHGGK9ItcRjjDHGm9JyCgkRiI2q5+tQKmWJxxhjgsz+nEJioyIIC/XPr3j/jMoYY8wpS80pomWMf3azgSUeY4wJOqk5hTRvZInHGGOMl6TmFPrtPTxgiccYY4JKUWkZB/NL/Ha6HLDEY4wxQSXNz28eBUs8xhgTVPx5AbgKlniMMSaIVEyXY6PajDHGeMXh6XJsVJsxxhhvSMsppF5YCI0bhPs6lCpZ4jHGmCBSMZTatVamf7LEY4wxQWR/TqFfd7OBJR5jjAkqaTlFtPDjgQVgiccYY4KGqlqLxxhjjPfkFpWSX1zm19PlgBcSj4iEisgqEfm4kn3tRWSBiKwVkUUiEu+2r0xEVjuP+W7lHUTkRxHZIiJviYh/LjhhjDFeFgj38IB3Wjx3Aj9Vse/vwGuq2geYATzmtq9AVfs6j9Fu5U8AT6lqZ+AgcJMngjbGmEBzeNaCutzV5rRgRgGzqjikB7DA2V4IjDnB6wkwAnjXKXoVuOL0IzXGmMBXkXjqeovnaeA+oLyK/WuAK53tsUAjEWnmPI8UkSQRWSoiFcmlGZClqqXO8z1AGw/EbYwxAaeiq615ozp6jUdELgPSVHVFNYfdAwwTkVXAMGAvUJFU2qlqInAd8LSIdAIquyNKq3j/aU7iSkpPTz/lehhjTKBIzSmkUUQYDSPCfB1KtTzZ4hkKjBaRZOBNYISIzHU/QFVTVHWcqvYD7nfKsiv2Of9uBxYB/YAMoLGIVPyvxgMplb25qs5U1URVTYyLi6vtuhljjN9JzSn0+3t4wIOJR1Wnq2q8qiYA44GvVXWS+zEiEisiFTFMB152ypuISETFMbiS2EZVVVzXgq5yzpkMfOipOhhjTCDZ7+crj1bw+n08IjJDRCpGqQ0HNonIZqAF8KhT3h1IEpE1uBLN46q60dn3e+B3IrIV1zWfl7wWvDHG+LG0nCK/v3kUwCsdgaq6CFd3Gar6gFv5uxwZoeZ+/BKgdxWvtR0Y6Ik4jTEmUJWXK2mH6nhXmzHGGO85kF9MSZnSws9HtIElHmOMCQoV9/C0iLYWjzHGGC9Ic+7hsa42Y4wxXrHfWjzGGGO8aV92ISIQF2XXeIwxxnjBvqwC4qIiqBfm/1/r/h+hMcaYE9qXXUirxvV9HUaNWOIxxpggkJJdQOsAGFgAlniMMSbgqSr7sgppFWMtHmOMMV6QXVBCQUkZrRtbi8cYY4wXpGS5hlK3tms8xhhjvGFfdgEArewajzHGGG9IybYWjzHGGC/al1VAWIgQGwA3j4IlHmOMCXgpWQW0iI4kNER8HUqNWOIxxpgAl5JdGDAj2sASjzHGBLx92QUBcw8PWOIxxpiAVl6u7M8upJW1eIwxxnhDRl4RJWVKa2vxGGOM8YZ9AXbzKHgh8YhIqIisEpGPK9nXXkQWiMhaEVkkIvFOeV8R+UFENjj7rnU75xUR2SEiq51HX0/XwRhj/FWg3TwKEOaF97gT+AmIrmTf34HXVPVVERkBPAZcD+QDN6jqFhFpDawQkc9VNcs5715VfdcLsRtjjF8LtOlywMMtHqcFMwqYVcUhPYAFzvZCYAyAqm5W1S3OdgqQBsR5MlZjjAlE+7ILiAgLoUmDcF+HUmOe7mp7GrgPKK9i/xrgSmd7LNBIRJq5HyAiA4F6wDa34kedLrinRCQwbtU1xhgPcN3DUx+RwLh5FDyYeETkMiBNVVdUc9g9wDARWQUMA/YCpW6v0QqYA9yoqhXJazrQDTgLaAr8vor3nyYiSSKSlJ6eftr1McYYf5SSVRBQ13fAsy2eocBoEUkG3gRGiMhc9wNUNUVVx6lqP+B+pywbQESigU+AP6nqUrdz9qlLETAbGFjZm6vqTFVNVNXEuDjrpTPGBKdAWgCugscSj6pOV9V4VU0AxgNfq+ok92NEJFZEKmKYDrzslNcDPsA18OCdY85p5fwrwBXAek/VwRhj/FlpWTlphwJruhzwwX08IjJDREY7T4cDm0RkM9ACeNQpvwY4D5hSybDpeSKyDlgHxAKPeC96Y4zxH6mHiihXAq7F443h1KjqImCRs/2AW/m7wHHDolV1LjD32HJn3wiPBGmMMQFmX5brHh5r8RhjjPGKQFsAroIlHmOMCVAVLR4b1WaMMcYr9mUX0igijEaRgXPzKFjiMcaYgJWSVRBQyyFUsMRjjDEBKiXAFoCrYInHGGMC1L6swLuHByzxGGNMQCosKSMzr9haPMYYY7xjvzOUOtBGtIElHmOMCUgp2RU3j1qLxxhjjBcE4pLXFSzxGGNMAArEJa8rWOIxxpgAtDergGYN6xEZHurrUE6aJR5jjAlAOzLyaN+sga/DOCWWeIwxJgAlZ+STENvQ12GcEks8xhgTYAqKy9ifU0iHZpZ4jDHGeEFyZh6AtXiMMcZ4R3KGK/F0sMRjjDHGG3ZYi8cYY4w3JWfkERsVQVREmK9DOSWWeIwxJsAkZ+TTITYwh1KDFxKPiISKyCoR+biSfe1FZIGIrBWRRSIS77ZvsohscR6T3coHiMg6EdkqIs+IiHi6DsYY4092ZOaREKAj2sA7LZ47gZ+q2Pd34DVV7QPMAB4DEJGmwIPAIGAg8KCINHHOeR6YBnR2Hpd4LnRjjPEvuUWlpB8qCtjrO+DhxOO0YEYBs6o4pAewwNleCIxxti8GvlTVA6p6EPgSuEREWgHRqvqDqirwGnCFxypgjDF+JtBHtIHnWzxPA/cB5VXsXwNc6WyPBRqJSDOgDbDb7bg9TlkbZ/vY8uOIyDQRSRKRpPT09FOvgTHG+JHD9/BYV9vxROQyIE1VV1Rz2D3AMBFZBQwD9gKlQGXXbbSa8uMLVWeqaqKqJsbFxZ1c8MYY46cqWjwJwT64QEQ6iUiEsz1cRO4QkcYnOG0oMFpEkoE3gREiMtf9AFVNUdVxqtoPuN8py8bVkmnrdmg8kOKUx1dSbowxdcKOjHxaREfQoF5gDqWGmrd43gPKROQM4CWgA/B6dSeo6nRVjVfVBGA88LWqTnI/RkRiRaQihunAy87258BFItLEGVRwEfC5qu4DDonIYGc02w3AhzWsgzHGBLzkAB/RBjVPPOWqWorrOszTqvpboNWpvKGIzBCR0c7T4cAmEdkMtAAeBVDVA8DDwHLnMcMpA/g1rsEKW4FtwGenEocxxgSi5Iy8gB5YAFDTtlqJiEwAJgOXO2XhNX0TVV0ELHK2H3Arfxd4t4pzXuZIC8i9PAnoVdP3NsaYYJFTWEJmXnFAD6WGmrd4bgSGAI+q6g4R6QDMPcE5xhhjatHhgQUB3tVWoxaPqm4E7gBwrrk0UtXHPRmYMcaYo+0Ignt4oOaj2haJSLQzo8AaYLaI/NOzoRljjHGXnJEPELBLXleoaVdbjKrmAOOA2ao6ABjpubCMMcYcKzkzj9YxkUSGh/o6lNNS08QT5kxXcw1w3GSfxhhjPG9HRl7ADyyAmieeGbjurdmmqstFpCOwxXNhGWOMOVZyZnAknpoOLngHeMft+XaOzLFmjDHGw7Lyi8nKL6FDgI9og5oPLogXkQ9EJE1EUkXkPfe1c4wxxnjWjozAXu7aXU272mYD84HWuGaD/sgpM8YY4wUVs1IH8sqjFWqaeOJUdbaqljqPVwCb8tkYY7xkR0Y+IQJtm9adxJMhIpOcZaxDRWQSkOnJwIwxxhyRnJFH68b1iQgL7KHUUPPEMxXXUOr9wD7gKlzT6BhjjPGCrWm5nNE8ytdh1IoaJR5V3aWqo1U1TlWbq+oVuG4mNcYY42Fl5cq29Fw616XEU4Xf1VoUxhhjqrTrQD5FpeV0btHI16HUitNJPJUtQ22MMaaWbUk9BEAXSzxorUVhjDGmSlvScgGC5hpPtTMXiMghKk8wAtT3SETGGGOOsjn1EG0a1ycqoqZrd/q3amuhqsHRrjPGmAC2JTWXzi2Co7UDp9fVZowxxsOCbUQbWOIxxhi/Fmwj2sALiceZ6WCViBy3jo+ItBORhc7+tSJyqVM+UURWuz3KRaSvs2+RiGxy29fc03UwxhhfCbYRbVDDZRFO053AT0B0Jfv+BLytqs+LSA/gUyBBVecB8wBEpDfwoaqudjtvoqomeThuY4zxuWAb0QYebvE4SyeMAmZVcYhyJCHFACmVHDMBeKP2ozPGGP8XbCPawPNdbU8D9wHlVex/CJgkIntwtXZur+SYazk+8cx2utn+LCKV3sgqItNEJElEktLT008temOM8bHNQTaiDTyYeETkMiBNVVdUc9gE4BVVjQcuBeaIyOGYRGQQkK+q693OmaiqvYFzncf1lb2wqs5U1URVTYyLsxUcjDGBJxhHtIFnWzxDgdEikgy8CYwQkbnHHHMT8DaAqv4ARAKxbvvHc0xrR1X3Ov8eAl4HBnoieGOM8bVdB/IpDrIRbeDBxKOq01U1XlUTcCWQr1V10jGH7QIuABCR7rgST7rzPAS4GlfSwikLE5FYZzscuAxYjzHGBKHNQTiiDbwzqu0oIjIDSFLV+cDdwIsi8ltcAw2mqGrFFD3nAXtUdbvb6RHA507SCQW+Al70XvTGGOM9W4NwRBt4KfGo6iJgkbP9gFv5RlxdclWdM/iYsjxggIfCNMYYvxKMI9rAZi4wxhi/tTk1eFYddWeJxxhj/FDFiLYuQTaUGizxGGOMXwrWEW1giccYY/xSxYi2YLuHByzxGGOMX6oY0WYtHmOMMV4RrCPawBKPMcb4pZ/25QTdHG0VLPEYY4yfycovZnNqLgPaNfF1KB5hiccYY/zMip0HATirQ1MfR+IZlnhMrSspKycjt8jXYRgTsJYlHyA8VOjbtrGvQ/EISzymVpWVK1NfWc55f1t4eMleY8zJWb7jAH3iGxMZHurrUDzCEo+pVf/8chPfbskA4Ja5K8gtKvVxRMYElsKSMtbtzSYxITiv74AlHlOLvtiwn+cWbmPCwLbMmpzIjow8fv/eWo5MOG6MOZFVu7IoKVMGJgTn9R2wxFPnqCr7sgtqPRnsyMjj7rfX0Cc+hgcv78nZnWK59+JufLJ2H68sSa7V9zImmC1PPoAIJLa3xGMCQGFJGZ+u28esb7eTU1hy3P68olLufnsNQx77mic/31RrySe/uJRb5qwgLFT4z8T+h/ulbxnWkQt7tODRT35ixc4DtfJexgS75ckH6NqiETENwn0discE3y2xdUxZufLDtkz+u3ovn6/fzyHnmsr/Ld7O9F90Y2y/NoSECJv2H+LWeSvYnpHHgPZN+M+ibYSIcPdFXRCRU37/wpIy7nhjNZvTDvHa1IHEN2lweJ+I8Perz2T0s99xw0vL6BgXRdOG9WgWVY/2TRsy7byO1K8XnBdPjTkVpWXlrNx5kHH9430dikdZ4glwf3x/HW8l7SYqIoxLerXkir5tiIoM46H5G7j7nTXM+3EnI3u04JkFW4iKCGfeTYMY3LEZf/xgHc8u3EpIiPC7C7scfj1VZc/BAlrFRBIWWn2DOLeolGmvJbFkWyYPj+nJuZ3jjjsmpn44L085i5mLt5N2qJADecVsTcvlg1V7+WF7Bi9NPouGQTgliDGnYuO+HPKKy4L2/p0K9hsfwBZuSuOtpN3cODSB31/S7aihl+//+mzeW7mHxz/7mb/9bxNDz2jGU9f2pXmjSAD+OrY35ao8s2ALqkq3ltEs3pzG4s3ppOYUMbBDU168IZGY+pU39w/kFXPj7GWsT8nhqWvPZGy/qv9C6xQXxRNX9Tmq7MPVe/nd22uYMnsZs28cGJTzURlzspbtcHVJB/PAArDEE7AOFZZw//vrOKN5FH/4RTciwo7usgoJEa5ObMtFPVuSlHyA4V2bExoiR+1/fFwfyhX+/fVWAKIjwzi3Sxyd4qJ4ftFWrn3hB16bOpDm0ZFHvfa+7AKuf2kZuw7k88KkAYzs0eKk4x/Ttw1hISHc+eYqrn/pR16dOpDoyKr7tFX1tLoEjQkESckHadu0Pi1jIk98cADzeOIRkVAgCdirqpcds68d8CrQGAgF/qCqn4pIAvATsMk5dKmq3uKcMwB4BagPfArcqXVwvO4T//uZfTmFvHvL2cclHXcx9cO5oHvliSEkRHjiyj6M7N6cuEYRnBnf+HD32sCEpvxqThLjnl/CnJsG0SG2IWv3ZDFv6S7mr0khNER4bepABndsdsp1GNWnFaEhwu1vrGTiiz8ycVA72jVrQEKzhjRvFMFP+w7x3dYMvt+awfLkA/x6eCfuGtml0tf6amMqIlRZV2P8naqyPPkAw7oe32UdbLzR4rkTVxKJrmTfn4C3VfV5EemBK5EkOPu2qWrfSs55HpgGLHWOvwT4rLaD9mdLt2cyd+kupg7twID2p3eTWWiIcEmvVseVn9M5ljemDebG2cu56vkltGocyfq9OdQPD2X0ma25+byOtbIW/CW9WvL8xAHc9dZq/vD+usPlIlDx50S3lo3o0TqaZxZs4bwucfQ/ZuLElbsOcsvcFZSWK38a1Z1fntvxtOMKFKVl5YSGiLUGg8D2jDwy84qDvpsNPJx4RCQeGAU8CvyukkOUIwkpBkg5weu1AqJV9Qfn+WvAFQRx4vl5fw47M/NJaNaQ9s0aoAp/eG8t7Zo24J6LK//rv7b0iW/MO7cMYeoryyktUx4e05Mx/dpU2yV2Kkb2aMGaBy8iJauAXQfySc7MIyWrgC4tGjGkUzOaN4rkUGEJlzz9LXe/vYZP7jiHBvVcP7rZ+SXc/voqWsZE0qt1DI988hOZecXcd3HXoP8y/mpjKr99azXlqnRqHkWnuCg6xTVkXP94Wjeu7+vwzEla7lzfCfaBBeD5Fs/TwH1AVUvoPQR8ISK3Aw2BkW77OojIKiAH+JOqfgu0Afa4HbPHKQtKxaXlXP/SMtIPHZlwM6Z+ONkFJbz+y0GHv3w9qWNcFAvvGe7xL/HQEKFt0wa0bdqAoWfEHre/UWQ4T17dh+te/JHHP/uZGWN6oarc++4a0g4V8s4tZ9O7TQwPfLie5xdt42BeMY9c0euEI/MC1dylO3ngw/X0aB1NYvumbEvP5cftmXywai/zftzFm9MG075ZQ1+HaU7CsuQDxEbVo2Ns8H9uHvvmEpHLgDRVXSEiw6s4bALwiqr+Q0SGAHNEpBewD2inqpnONZ3/ikhPoLJvv0qv74jINFxdcrRr1+40a+Mbn63fR/qhIh4e05Po+uHszMwnOSOPbq0acXYlX86e4i8th7M7xTJ1aAde/n4HF/Zowba0XL7YmMqfRnU/PIvvI1f0olnDejzz9VZyCkt4Zny/gEs+qsraPdl8uDqFFbsOMrhjUy7v05qeraNRhSe/2MTzi7Yxoltznr2u31F/gGxIyWbirB+ZMHMpb04bQrtmDap5J+MvVJVlOw6Q2L6p3/y+eZJ46rq8iDwGXA+UApG4utTeV9VJbsdsAC5R1d3O8+3AYFVNO+a1FgH3AHuBharazSmfAAxX1V9VF0tiYqImJSXVVtW8Zux/vicrv4QFvxtGSEjw/zDWRGFJGaOe+ZbsglJyCko4t3MssyYnHvfL+uI323n0058Yf1ZbHhvXOyB+mbPyi3llSTIfrk5hR0Ye9UJD6N46mg17syktVxKaNaBNk/p8vzWTCQPb8fCYnpUm1Yrk07BeGG9OG0zbppZ8/N329FxG/GMxD4/pyfVDEnwdzmEiskJVE2v7dT32p6CqTlfVeFVNAMYDX7snHccu4AIAEemOK0Gli0icMxoOEekIdAa2q+o+4JCIDBbXN8kNwIeeqoMvrd2TxapdWVw/uL0lHTeR4aE8dW1fDuYXExtVj79ffWalSeXm8zpy2/ln8Oby3fzji80+iPTkJCUf4Bf/+pZ/LdhCi+gIHh/Xm+X3j+TD3wxl+f0jeWxcb9o0qc+qXVnce3FX/jq26m7Enq1jmHvTIHKLShk/cyk7MvK8XBtzshZvTgdgWJfmPo7EO7x+H4+IzACSVHU+cDfwooj8FleX2RRVVRE5D5ghIqVAGXCLqlZM9vVrjgyn/owgHVjw6pKdNKgXylWJwT11xqnoE9+Yeb8cRKuYSJo0rFflcXdf1IXMvCKeXbiVZlH1uHFoB8A148K3m9PZc7CAsf3bEBsV4a3Qj1Nerjy/eBv//HIzbRrX58PfDKVP/NGLfzVpWI8JA9sxYWC7Gt/P1KtNDPN+OYiJs35k5D8X84teLbnpnA70C9KllAPdok3pdIxtWGe6Rj3W1eZPAq2rLTO3iCGPf821iW15+Ipevg4noJWVK7fOW8HnG1K5+dwObE7N5YdtmRSXlQMQGR7CpEHtmTas4+FZHbwlM7eIu95azbdbMhjVpxWPjetd6yMGU7IKeGVJMm8s28WhwlL6t2vMoI7NOJhXTEZuMQfyiggNEXq2jqFn62h6tYnhjOZRhAfYdbFAVlhSxpl/+YLrBrXjwct7+jqco3iqq81mLvCx3QfyiWsUcdR0N28u301xaTmTz27vw8iCQ2iI8K/x/ZgyexkvfruDhGYNmHx2e0Z2b0HThvV4ftE2Xv5+B3OW7mTioPbce3FXr0xcWlJWzs2vJbE+JYe/ju3NhIFtPXIdqnXj+vzx0u7ccUFn3k3azStLkpn5zXbXZK0N6xEbFUFhSRlvJ+0mv7gMgKYN63HfxV25JrGtdfN6wdLtmRSVljO8a93oZgNr8fjU7gP5jPjHIlrGRPLwmF4M79qc0rJyzv3bQjrFRTH3l4N8HWLQKCotIzW7iLZN6x/3BZ+ckcezC7fy3so99GkTw4uTE0+p9VNerojUbBTgwx9v5KXvdvDvCf24/MzWJ/1ep0pVUeW4hFJWriRn5rF+bzbzlu5iWfIB+rZtzMNjetE7PsZr8dVFD83fwBvLdrHmwYv8bqlra/EEoRe+2QZAeGgIU2YvZ1SfVgxMaMq+7EL+Mtq/mtyBLiIstMr+84TYhvz96jO5uGdL7nhjFWOfW8LLU86ia8uqbj873vq92dw6byUdYhvy3MT+1U56+tm6fbz03Q6mnJ3g1aQDrqRYWV4MDRHnBtQoRp/Zmv+u3sujn/zM6Oe+Y8yZrYlv0oCIsBAiwkOIigjnnDNi68z1CE/7ZnM6Qzo187uk40nW4vGR9ENFDH3ia8b1a8NfxvRk5uLt/HvhVopLy4lvUp/F955/1KSexjvW7cnmpleXU1BcxnMT+3NelxPPm/Xuij388YN1xNQP50BeMT1bR/PylLMqHbSwIyOP0f/+jk7No3j7V0OoF+a/11JyCkt46svNvJu0h7ziUsqP+aro0SqaX/RqyS96t+SM5jVP0uaIXZn5nPfkQh68vMfhwS/+xFMtHks8PvLE/37mhcXbWHD3cDo4dyonZ+Txzy83c0mvllza+/j504x3pGQVMPWV5WxJy2Xm9QOqnHi0uLSchz/eyJylOzm7UzP+PaEfq3dn8ZvXV9Iqpj6vTR141D00hSVlXPHc9+zPKeSTO86lTYBNa1NaVk5RaTlph4r4amMq/9uwnxU7DwL47Renv3vth2Qe+HADC+858j3gTyzxnAZ/Szw5hSUMfexrzusax3PX9fd1OKYShwpLuO7FH9menssHvxlKlxZH/0WflV/Mza8lsTz5INPO68h9F3c9fF/Nip0HmPpKEvXCQrh1eCd2Hyhga3oum/bnkHaoiNlTzgqaC8mpOYX86b/r+eqnVGbdkGizg5+km5w/cBbf6/lpqU5FwN1Aaqo254edHCoq5dfDOvk6FFOFRpHhzLxhAA0iwvjlq0kczCs+vC8tp5BrX1jKmt3ZPDOhH3+8tPtRN3MOaN+Ud28ZQliI8JePNvLGsl1k5hYxuGMznp3QP2iSDkCL6EieGd+PXq1juP2NVWxMyfF1SAGjsKQTPr52AAAVaElEQVSMJdsyGd41zi+TjidZi8fLCkvKOOeJr+nROobXpg70dTjmBFbuOsj4mUsZ0K4Jr900kH1ZhUx66Ucycot48YbESic0rVBQXEZmXhGtY+oH/bDk1JxCrnjuewD++5uhtIgO7oXMasO3W9K5/qVlvDwlkRHd/LOlaC0eP5RdUMK6Pdkndc47SbvJyC3m1uHW2gkE/ds14bGxvflheya/fWs1V/3fEnIKS3j95sHVJh2A+vVCiW/SIOiTDrhaPi9NPovsghJuenU52QUlvg7J7y3elE690JDTWkwxUFniOUVfbkzlwn8u5vJnv+P/Fm+r0TmlZeW88M12193jdWDNjWBx5YB4pp3XkY/X7kME3v7VkMOzYZsjerSO5t8T+rExJYeBj37Fba+vZOHPaZQ6s0SYoy3anM6gjk29sryJv6l7NT5NB/KKeWj+BuavSaFby0b0bduYxz/7mQN5xfzhkm7V/nX7zoo97DlYwF9G96xzfbqB7veXdKN9swYM6xJHfBO7f6UqF3RvwfzbzuGt5bv5aG0KH6/dR6wzT94vz+1Q7TLtdcm29Fy2puUyYWBgLtlyuizxnIQlWzO4481VZOWXcNfIztw6/AznAvIGZn6znczcYh6/snel81wVlpTx9Feb6d+uMSO6Bc/F5boiNESYOMimMKqJXm1i6NUmhj9f1oNFm9J4Y9kunvx8E++u2MODl/cIqsEVp2r+6hREYFQdvW3CEk8N7crM55a5K2geHcmcmwbRvVX04X0Pje5J04YRPPXVZrLyi/nPpP7H/WX36pJkUnOKeGZ8P2vtmDqhXlgIF/VsyUU9W7J4czoPzd/AlNnLuahHCx4a3bPOLs+tqny0JoVBHZrSMqZuDsKwazw1UFhSxq2vrwBg9pSzjko64JqG5M6RnXn4il4s+DmNP7y3DvfRgtkFJfxn0TbO7xrHoDp4IdGYYV3i+N9d53LfJV35dksG42cuJSu/+MQnBqH1e3PYnpHHmL5tfB2Kz1jiqYEZH29k/d4c/nFN32pXc7x+cHvuuagLH6zay1NfbTlc/sLibWQXlHDvxd28Ea4xfikiLJRbh5/BvJsHsT+7kNteX1UnBx7MX7OX8FDhF71a+joUn7HEcwLvr9zD6z/u4pZhnbiwx4nH2v/m/DO4JjGeZxZs4Z2k3aTlFPLy9zsY07c1PVpHn/B8Y4Jd/3ZNeGRsL77bmsHjn/3s63C8qrxc+WjNPoZ1iaNxg6oXMQx2do2nGpv2H+L+D9YzsENT7rmoS43OEREeHdublKxCpr+/jgHtm1Baptx9YVcPR2tM4LgmsS0bU3KY9d0OureK5soBdWOl3WXJB9ifU8j0S+t274e1eKqgqvzxg3U0jAjj2Qn9qlzfvjLhoSH8Z1J/OsY15McdB7huUDubQt6YY9w/qjtDOjZj+gfrWLM7y9fheMX8NSnUDw+tUe9JMLPEUwUR4V/j+zLzhgE0P4XpP6Ijw5l940BuHJrAXSNr1loypi4JDw3huYn9ad4oglvnrQz62Q6KS8v5dN0+LuzRok7eNOrOEk814ps0oH+7Jqd8fpvG9Xnw8p40bVh3+3KNqU7ThvV47rr+pOYUcv8HR48GrVBYUsaW1EM+iK52fbc1naz8Esb09e7if/7I44lHREJFZJWIfFzJvnYistDZv1ZELnXKLxSRFSKyzvl3hNs5i0Rkk4isdh52N5oxAezMto357YVd+HjtPt5bufeofZm5RVzzwg9c+NQ33PXmKtIPFfkoytP34eoUYuqHc27nEy8uGOy80eK5E/ipin1/At5W1X7AeOA/TnkGcLmq9gYmA3OOOW+iqvZ1HmmeCNoY4z23DOvEoA5NefDD9SRn5AGwN6uAq1/4gU37DzFhYFs+XbefEf9YxJwfkik7djlUP5dXVMqXG1O5tHcrv1511ls8+j8gIvHAKGBWFYcoUDHGOAZIAVDVVaqa4pRvACJF5Ph1hI0xQSE0RHjq2r6Ehgh3vrWan/fncNXzS0g/VMScmwbx2Lg+fHbXufSJj+HPH27gqv9bwqHCwLkmNGfpTvKLy7gmsW6M3jsRT6fep4H7gKruEnsImCQie4BPgdsrOeZKYJWqurexZzvdbH8Wm3/GmKDQunF9HhvXhzW7sxj1zHeUlClvTRvCQGcm905xUcy9aRD/uPpMVu/O4snPN/k44prJLSrlhcXbGN41jn6ncc04mHgs8YjIZUCaqq6o5rAJwCuqGg9cCswRkcMxiUhP4AngV27nTHS64M51HtdX8f7TRCRJRJLS09NPszbGGG8Y1acVU85OoENsQ9779ZDjbroWEa4cEM/kIQnMWbqTFTsP+CjSmnt1STIH80v4rY1uPcxjK5CKyGO4kkIpEImrS+19VZ3kdswG4BJV3e083w4MVtU0p5vua+BGVf2+iveYAiSq6m3VxeJPK5AaY05MVaudTDe3qJSLn/qG+vVC+eSOc/x2uYWcwhLOfWIhZyU0Ydbks3wdzkkLuBVIVXW6qsaragKugQNfuycdxy7gAgAR6Y4rQaWLSGPgE2C6e9IRkTARiXW2w4HLgPWeqoMxxjdO1IMeFRHGI2N7sTUtl+cX1WwhRl+Y/V0y2QUldi/fMbw+vEJEZojIaOfp3cDNIrIGeAOYoq4m2G3AGcCfjxk2HQF8LiJrgdXAXuBFb9fBGON753dtzpi+rXlu4Va/vM8nO7+EWd9t5+KeLejVJsbX4fgVj3W1+RPrajMmOGXkFjHyn4vpFBfF6zcP8qsut39+sYlnvt7KZ3eee9xSKoEi4LrajDHG02KjInjgsh6s2HmQc55YyLNfb+Fgnu/X+cnMLeLl75MZ1btVwCYdT6rbEwYZYwLeuP7xNG8Uycxvt/P3Lzbz7MKtXD2gLRMHt6NbS+9/6asqv39vLcWl5dw1srPX3z8QWOIxxgS8czrHck7nWDbtP8Ssb7fz1vLdzFm6kz7xMVyd2JbRfVoT0yDcK7G8siSZr35K48HLe9C5RSOvvGegsWs8xpigcyCvmP+u2svbSbv5ef8h6oWF8MgVvbgmsa1H33f93mzG/WcJ53WJ5cUbEk84Os/feeoaj7V4jDFBp2nDekw9pwM3Dk1gQ0oOj37yE9PfX0ermEiPTdKZW1TK7W+somnDejx51ZkBn3Q8yQYXGGOClojQq00MM28YQOfmUdw6dyWbPTT0+oH/rmdnZh7/Gt+XJrYUSrUs8Rhjgl6jyHBemnIWkfVCuXH28lpfXuGLDft5f9Ve7rygC4M6NqvV1w5GlniMMXVCm8b1eWlyIpl5Rdz8WhKFJWW19tozv9lOu6YNuG3EGbX2msHMEo8xps7oE9+Yp6/tx5o9WQx+bAG3zlvBvB93sjMzr9LVT2tize4sknYeZMrZCYSG2HWdmrDBBcaYOuWSXi159caBfLQmhe+2ZvDpuv0AjOzegucm9jvp2Q9e/n4HURFhXG1r7dSYJR5jTJ1zXpc4zusSh6qyIyOP+WtSePqrLdw6dyXPTxpQ41VC92cX8snafUw+O4FGkd65TygYWFebMabOEhE6xkVx18guPHJFLxb8nMZvXl9JSVlVa1cebc7SZMpVmXJ2gmcDDTKWeIwxBpg0uD1/Gd2TLzemcscbq06YfAqKy5j34y4u7NGCtk0beCnK4GCJxxhjHJPPTuDPl/Xgs/X7uevN1dUmnw9W7SUrv4SbzunoxQiDg13jMcYYNzed0wFV5ZFPfqK4rJxnrzt+wIGq8vL3O+jVJpqzEpr4KNLAZS0eY4w5xi/P7ciMMa5ut1/NWXHUPT9FpWW89N0OtqblMnVoB5sa5xRYi8cYYypxw5AEwkND+OMH65j6ynKeuLIP76/cy5ylO8nILaJfu8aM6tPK12EGJEs8xhhThQkD2xERFsI976zh3L8tBOD8rnFMPacD55wRa62dU2SJxxhjqjGufzyNIsNZuj2T6wa1o1NclK9DCniWeIwx5gQu7NGCC3u08HUYQcPjgwtEJFREVonIx5XsayciC539a0XkUrd900Vkq4hsEpGL3covccq2isgfPB2/McaY2uWNFs+dwE9AZYuf/wl4W1WfF5EewKdAgrM9HugJtAa+EpEuzjnPARcCe4DlIjJfVTd6uhLGGGNqh0dbPCISD4wCZlVxiHIkIcUAKc72GOBNVS1S1R3AVmCg89iqqttVtRh40znWGGNMgPB0V9vTwH1AVbf/PgRMEpE9uFo7tzvlbYDdbsftccqqKj+OiEwTkSQRSUpPTz/lChhjjKldHks8InIZkKaqK6o5bALwiqrGA5cCc0QkBKhsjKJWU358oepMVU1U1cS4OM+ssW6MMebkefIaz1BgtDNgIBKIFpG5qjrJ7ZibgEsAVPUHEYkEYnG1ZNq6HRfPkW64qsqNMcYEAI+1eFR1uqrGq2oCroECXx+TdAB2ARcAiEh3XAkqHZgPjBeRCBHpAHQGlgHLgc4i0kFE6jmvO99TdTDGGFP7vH4fj4jMAJJUdT5wN/CiiPwWV5fZFHWtP7tBRN4GNgKlwG9Utcw5/zbgcyAUeFlVN3i7DsYYY06dnOo644FERNKBnccUxwDZJyhzf36i7Vgg4zTCrCyemh5zsnU59nnFtnvZ6dTndOpS1T77bOyzOZk4a3KcP9Wlujhrcoynfs4aqmrtXyRX1Tr5AGaeqMz9+Ym2cbXiajWemh5zsnWppg7uZadcn9Opi3029tnUZl1q87Oxn7Pae9TlZRE+qkHZRye5Xdvx1PSYk63Lsc8/quKYU3U6dalqn302tSOYPpuavkZtfTb2c1ZL6kRXmzeISJKqJvo6jtoSTPUJprpAcNXH6uK/PFmfutziqW0zfR1ALQum+gRTXSC46mN18V8eq4+1eIwxxniVtXiMMcZ4lSWeSojIyyKSJiLrT+HcASKyzlm24RlxW6JQRG53lnTYICJ/q92oq4yn1usiIg+JyF4RWe08Lj3Ra9UWT302zv57RERFJLb2Iq42Hk98Ng87S4ysFpEvRKR17UdeZUyeqM+TIvKzU6cPRKRx7UdeaTyeqMvVzu9+uYh4/FrQ6dShitebLCJbnMdkt/Jqf68q5anhcoH8AM4D+gPrT+HcZcAQXPPKfQb8wik/H/gKiHCeNw/gujwE3BMsn42zry2uG5N3ArGBWhcg2u2YO4D/C+TPBrgICHO2nwCeCOC6dAe6AouARH+tgxNfwjFlTYHtzr9NnO0m1dW3uoe1eCqhqt8AB9zLRKSTiPxPRFaIyLci0u3Y80SkFa5f/B/U9Ym8Blzh7P418LiqFjnvkebZWrh4qC4+48H6PIVrJnWvXfT0RF1UNcft0IYEfn2+UNVS59CluOZn9DgP1eUnVd3kjfid9zulOlThYuBLVT2gqgeBL4FLTvV7whJPzc0EblfVAcA9wH8qOaYNrglOK7gv29AFOFdEfhSRxSJylkejrd7p1gXgNqf742URaeK5UGvktOojIqOBvaq6xtOB1sBpfzYi8qiI7AYmAg94MNaaqI2ftQpTcf1F7Su1WRdfqUkdKlPdUjUnXV+vz9UWiEQkCjgbeMet+zKiskMrKav4izMMVxN1MHAW8LaIdHT+SvCaWqrL88DDzvOHgX/g+lLwutOtj4g0AO7H1aXjU7X02aCq9wP3i8h04DbgwVoOtUZqqz7Oa92Pa97GebUZY03VZl18pbo6iMiNuFaLBjgD+FREioEdqjqWWliqxp0lnpoJAbJUta97oYiEAhXrDc3H9YXs3hXgvmzDHuB9J9EsE5FyXHMheXuVutOui6qmup33IvCxJwM+gdOtTyegA7DG+WWMB1aKyEBV3e/h2I9VGz9n7l4HPsFHiYdaqo9zIfsy4AJv/6HmprY/G1+otA4AqjobmA0gIotwTdic7HbIHmC42/N4XNeC9nAq9fX0Ba5AfQAJuF2UA5YAVzvbApxZxXnLcbVqKi60XeqU3wLMcLa74Gq2SoDWpZXbMb/FtUx5wH42xxyTjJcGF3jos+nsdsztwLuB/NngWq9rIxDnzXp48ucMLw0uONU6UPXggh24em2aONtNa1LfSuPy9ocZCA/gDWAfUIIro9+E66/i/wFrnF+EB6o4NxFYD2wDnuXITbr1gLnOvpXAiACuyxxgHbAW1195rbxRF0/V55hjkvHeqDZPfDbvOeVrcc271SaQPxtgK64/0lY7D6+M0vNQXcY6r1UEpAKf+2MdqCTxOOVTnc9jK3Djiepb3cNmLjDGGONVNqrNGGOMV1niMcYY41WWeIwxxniVJR5jjDFeZYnHGGOMV1niMXWSiOR6+f1miUiPWnqtMnHNPr1eRD460YzNItJYRG6tjfc2pjbYcGpTJ4lIrqpG1eLrhemRySw9yj12EXkV2Kyqj1ZzfALwsar28kZ8xpyItXiMcYhInIi8JyLLncdQp3ygiCwRkVXOv12d8iki8o6IfAR8ISLDRWSRiLwrrjVk5lWsTeKUJzrbuc5EnmtEZKmItHDKOznPl4vIjBq2yn7gyGSnUSKyQERWimt9lDHOMY8DnZxW0pPOsfc677NWRP5Si/+NxpyQJR5jjvgX8JSqngVcCcxyyn8GzlPVfrhme/6r2zlDgMmqOsJ53g+4C+gBdASGVvI+DYGlqnom8A1ws9v7/8t5/xPOd+XME3YBrtkjAAqBsaraH9f6T/9wEt8fgG2q2ldV7xWRi4DOwECgLzBARM470fsZU1tsklBjjhgJ9HCbuTdaRBoBMcCrItIZ18y74W7nfKmq7mueLFPVPQAishrXXFnfHfM+xRyZWHUFcKGzPYQja5m8Dvy9ijjru732Clxro4Brrqy/OkmkHFdLqEUl51/kPFY5z6NwJaJvqng/Y2qVJR5jjggBhqhqgXuhiPwbWKiqY53rJYvcducd8xpFbttlVP47VqJHLq5WdUx1ClS1r4jE4EpgvwGewbX+ThwwQFVLRCQZiKzkfAEeU9UXTvJ9jakV1tVmzBFf4Fq/BgARqZg+PgbY62xP8eD7L8XVxQcw/kQHq2o2ruWt7xGRcFxxpjlJ53ygvXPoIaCR26mfA1Od9VkQkTYi0ryW6mDMCVniMXVVAxHZ4/b4Ha4v8UTngvtGXEtZAPwNeExEvgdCPRjTXcDvRGQZ0ArIPtEJqroK10zD43EtkpYoIkm4Wj8/O8dkAt87w6+fVNUvcHXl/SAi64B3OToxGeNRNpzaGD/hrIZaoKoqIuOBCao65kTnGRNo7BqPMf5jAPCsMxItCx8tJ26Mp1mLxxhjjFfZNR5jjDFeZYnHGGOMV1niMcYY41WWeIwxxniVJR5jjDFeZYnHGGOMV/0/0nhBsT9rvDQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.recorder.plot(skip_end=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-3\n",
    "lr *= bs/48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.to_fp16();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.102362</td>\n",
       "      <td>3.967245</td>\n",
       "      <td>0.296941</td>\n",
       "      <td>15:21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(1, lr*10, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this is relatively slow to train, we will save our weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save('fit_1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.load('fit_1');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To complete the fine-tuning, we can then unfreeze and launch a new training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.872039</td>\n",
       "      <td>3.803822</td>\n",
       "      <td>0.315553</td>\n",
       "      <td>16:43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.794908</td>\n",
       "      <td>3.739284</td>\n",
       "      <td>0.321787</td>\n",
       "      <td>17:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.736880</td>\n",
       "      <td>3.705275</td>\n",
       "      <td>0.325063</td>\n",
       "      <td>17:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.673460</td>\n",
       "      <td>3.661957</td>\n",
       "      <td>0.329320</td>\n",
       "      <td>17:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.613378</td>\n",
       "      <td>3.630191</td>\n",
       "      <td>0.333327</td>\n",
       "      <td>17:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.531500</td>\n",
       "      <td>3.602567</td>\n",
       "      <td>0.336381</td>\n",
       "      <td>17:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.458650</td>\n",
       "      <td>3.574097</td>\n",
       "      <td>0.340086</td>\n",
       "      <td>17:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.365858</td>\n",
       "      <td>3.561914</td>\n",
       "      <td>0.341489</td>\n",
       "      <td>17:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.288830</td>\n",
       "      <td>3.558446</td>\n",
       "      <td>0.342408</td>\n",
       "      <td>17:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.250091</td>\n",
       "      <td>3.561159</td>\n",
       "      <td>0.342382</td>\n",
       "      <td>17:03</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(10, lr, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save('fine_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to save not just the model but also it's encoder, the part that's responsible for creating and updating the hidden state. For the next part, we don't care about the part that tries to guess the next word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save_encoder('fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Loading our saved weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_lm.load('fine_tuned');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Now that we've trained our model, different representations have been learned for the words that were in IMDB but not wiki (remember that at the beginning we had initialized them all to the same thing):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "enc = learn_lm.model[0].encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(enc.weight[vocab.stoi[\"30-something\"], :], \n",
    "            enc.weight[vocab.stoi[\"linklater\"], :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(enc.weight[vocab.stoi[\"30-something\"], :], new_word_vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### More generated movie reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "How good is our model? Well let's try to see what it predicts after a few given words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "TEXT = \"i liked this movie because\"\n",
    "N_WORDS = 40\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i liked this movie because it was about Jack the Ripper and his family and friends . This was a good movie , but not as touching , when it was over . The friends ( who were great in their\n",
      "i liked this movie because of the very good performances of Sandra Bullock . She has the nice - looking look and personality of Sandra Bullock . Sandra Bullock is hopeless as the little girl who grew up\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "TEXT = \"This movie was\"\n",
    "N_WORDS = 30\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This movie was ... awful . The plot was groundbreaking , the cinematography was alright , but the acting was terrible . The acting was pathetic . The story was\n",
      "This movie was so bad i had to fast forward through most of it . My friends and i are talking about movies that are so bad they 're good . i\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "TEXT = \"I hated this movie\"\n",
    "N_WORDS = 40\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I hated this movie . Just watch it and you 'll know who was better . The Architect , i think , was an idiot , and the whole movie was a waste of time . i found it interesting ,\n",
      "I hated this movie . i watched it with a friend , and after watching it i was constantly looking at the cover of the film , thinking , \" This is a great movie , not a Blockbuster \" , but\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Risks of language models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We will talk about ethical concerns raised by very accurate language models in lesson 7, but here are a few brief notes:\n",
    "\n",
    "In reference to [OpenAI's GPT-2](https://www.theverge.com/2019/2/14/18224704/ai-machine-learning-language-models-read-write-openai-gpt2): Jeremy Howard said, *I’ve been trying to warn people about this for a while. We have the technology to totally fill Twitter, email, and the web up with reasonable-sounding, context-appropriate prose, which would drown out all other speech and be impossible to filter.*\n",
    "\n",
    "For a small example, consider when completely incorrect (but reasonable sounding) ML generated answers were [posted to StackOverflow](https://meta.stackoverflow.com/questions/384596/completely-incorrect-machine-learning-generated-answers?stw=2):\n",
    "\n",
    "<img src=\"images/robot-overflow.png\" alt=\"Roboflow\" style=\"width: 80%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<img src=\"images/husain-tweet.png\" alt=\"Roboflow\" style=\"width: 60%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll create a new data object that only grabs the labelled data and keeps those labels. Again, this line takes a bit of time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs=48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = (TextList.from_folder(path, vocab=data_lm.vocab)\n",
    "             #grab all the text files in path\n",
    "             .split_by_folder(valid='test')\n",
    "             #split by train and valid folder (that only keeps 'train' and 'test' so no need to filter)\n",
    "             .label_from_folder(classes=['neg', 'pos'])\n",
    "             #label them all with their folders\n",
    "             .databunch(bs=bs, num_workers=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas.save('imdb_textlist_class')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = load_data(path, 'imdb_textlist_class', bs=bs, num_workers=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj match 1 : xxmaj tag xxmaj team xxmaj table xxmaj match xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley vs xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley started things off with a xxmaj tag xxmaj team xxmaj table xxmaj match against xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit . xxmaj according to the rules</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj by now you 've probably heard a bit about the new xxmaj disney dub of xxmaj miyazaki 's classic film , xxmaj laputa : xxmaj castle xxmaj in xxmaj the xxmaj sky . xxmaj during late summer of 1998 , xxmaj disney released \" xxmaj kiki 's xxmaj delivery xxmaj service \" on video which included a preview of the xxmaj laputa dub saying it was due out</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos 8 xxmaj simple xxmaj rules for xxmaj dating xxmaj my xxmaj teenage xxmaj daughter had an auspicious start . xxmaj the supremely - talented xxmaj tom xxmaj shadyac was involved in the project . xxmaj this meant that the comedy would be nothing less of spectacular , and that 's exactly what happened : the show remains one of the freshest , funniest , wittiest shows made in a</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos i have n't liked many xxup tv shows post 1990 , but xxup that xxup 70s xxup show is great . xxmaj never seeing it during it 's first run , thinking a gimmicky period piece , i was wrong ! i started watching in reruns and the more i watched , the more i liked ! xxmaj now , it is the only show xxunk xxunk that i</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos * xxmaj some spoilers * \\n \\n  xxmaj this movie is sometimes subtitled \" xxmaj life xxmaj everlasting . \" xxmaj that 's often taken as reference to the final scene , but more accurately describes how dead and buried this once - estimable series is after this sloppy and illogical send - off . \\n \\n  xxmaj there 's a \" hey kids , let 's</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_clas.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then create a model to classify those reviews and load the encoder we saved before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.3) #.to_fp16()\n",
    "learn_c.load_encoder('fine_tuned_enc')\n",
    "learn_c.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_c.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd4HNW9xvHvT12WLEuy5CY3uVewsWyK6cUYQ6gXAqRAuEAILQESAiGBXKoTQgIEUgiBkIQaTDHV9A62ZXCvsmwjuchFLpJs9XP/2DUsiuyVrd2d3dX7eZ59vDN7Zud3tFq9npkzM+acQ0REZG8SvC5ARESin8JCRESCUliIiEhQCgsREQlKYSEiIkEpLEREJCiFhYiIBKWwEBGRoBQWIiISVJLXBYRKXl6e69+/v9dliIjElDlz5mx2zuUHaxc3YdG/f3+Ki4u9LkNEJKaY2Zq2tNNuKBERCUphISIiQSksREQkKIWFiIgEpbAQEZGgFBYiIhJUWMPCzCab2TIzKzGzG1p5/Q9mNtf/WG5m2wJeawp4bXo46xQRkb0L23kWZpYIPAicAJQDs81sunNu8e42zrlrAtpfBYwNeItdzrkx4aovHpRsrOKjFZs5bnh3+uR28rocEYlj4TwpbwJQ4pwrBTCzp4DTgMV7aH8ecEsY64kLDU3NvLm4gn99uoZPS7cAcNsrSzj1wF5cdtRAhvbo/FXb2oYm6puayUpL9qpcEYkT4QyLAqAsYLocOLi1hmbWDygE3gmYnWZmxUAjMNU590K4Co0FDU3NPFNcxh/fLmHDjloKstP5+eRhHDMsn2lzynl85pc8/8VaxvfPobahmfXbd7G5uh4zGN8vl8mjenDiqB4UZKd73RURiUHmnAvPG5udDZzonLvYP/09YIJz7qpW2v4c6B34mpn1cs6tM7MB+ELkOOfcyhbLXQpcCtC3b99xa9a06az1/7Jw7XZG9MwiIcH2a/n2WLh2O8s2VJGQAAlmmBn5makU5mXQPSsVgNcXbuDuGcso3VxDUb8cfnT0QI4e2o3EgHq31tTz2KereXvJRnIzUuiVnUbPLuk0NjXzxuIKlm6oAuCwgV35/Tlj6NElrU31bdtZz9yybcwt28a8sm1U7mygvrGZ+kbfVktzMzjncMDuXyUzMCAtJZEfHNaf8w/u941aRSR6mNkc51xR0HZhDItDgV875070T98I4Jy7q5W2XwBXOOc+2cN7/QN42Tn37J7WV1RU5Pbn2lAlG6s56b4P+NaBvfjNWQeQnBi5AWIvz1/Hj5+aS1Nz659BenIiOZ2SWbe9lkHdMvn55GEcP7wbZvv+h3fV5hpeXbCeP71bQkZqEn/93jjG9s356vU1W2q4543lLFi7nYamZhqbHA1NzWypqQd8ATCkW2e6d0kjJTGB1KQEUpIS/AEHCQaG4XA4B86/zjlrtjKiZxa3njaSov65+/VzEpHwiYawSAKWA8cBa4HZwPnOuUUt2g0FZgCFzl+MmeUAO51zdWaWB3wKnBZ4cLyl/Q0L5xwPvFPCPW8u5/jh3Xjg/INIS07c5/fZVy/OXcs1T89lXL8cpp51AIlmNDtHU7Njw45aVm+uYdXmnazbtotjhuVz1kG9SQpBkC3bUMUl/yxmw45a7jpjNMcP784f31nBY5+uJikhgWOG5ZOWlEhSopGUmEBBdjpj+2ZzQO9sMlP3ba+lc45XF2zg9lcWs357LZNH9qAgJ53EBF/A4KCqrpHq2kaqahvIy0zlhpOG0TUztd39FJG28Tws/EVMAe4FEoFHnHN3mNmtQLFzbrq/za+BNOfcDQHLHQb8FWjGN7z3Xufc3/e2rv0Ni93+9elqbp6+iAn9c3n4giI6h/Gg8LQ55fzs2XlMKMzl7xeMJ2Mf/wi319aaeq544nM+WbmFjJREdjY0cfa43lw3aSjds9q2e2pf7Kxv5E/vruTxmWuoa2ym2TmamwGDzqlJdE5LonNaMssqqsjplMwfzzuICYXaChGJhKgIi0hqb1iA73/71z0zj6E9OnPVsYMo6p9LXsD/cusbm1leUcXyiirKt+5i7dZdrN22i9qGJg7ql8OE/rmM759Ll06tB01DUzOPfbKaO15dwsSBefzt+0Wkp4R/K2ZPtdw9Yxmlm2q49oQhjOiV5UkdgRat284Vj39O2dZdXHvCEH501EBPjiOJdCQKi/307rKNXP3EF1TVNQIwIC+D4T2zWLW5hhUbq2ho+vrnlZeZSkFOOkkJxoLy7dQ3NWMGI3tlcfLoXpxyQE/65Haiqdnxwhdrue/tFXxZuZPjhnXjwe9EZndXrKmqbeDG5xbw8vz1jCrI4rCBeRzYO5sD+3ShIDt9v47XiMieKSzaob6xmQVrtzN7dSXFqytZVlFF/64ZjOzVhZG9shjeszO9czp94499bUMT88q2MXNVJe8s3cjcMt/J6GP6ZFNV28DKTTWM7JXFdZOGcMzQ/TtI3VE453h6dhlPF5exaN0O6hubARjWozO3fGskhw7s6nGFIvFDYeGxssqdvDx/Pa8sWAfAFUcP4sSRPbRbZR/VNzazbEMVc9ZU8rcPV7F22y5OPqAnN00ZTi+dMyLSbgoLiTu1DU385f2V/Pm9lSSYcdKoHvTPy6Bf1070ze3EyF5dSEnStTFF9oXCQuJWWeVO7p6xjFmrKtmwo/ar+QPyM7jrjNEcPEC7qUTaSmEhHUJtQxNllTtZvH4Hv3tjGWWVuzhvQh9uOGk4XdJ1TSyRYNoaFpEd4C8SYmnJiQzu3pnB3Ttzwoju3PvWCh7+sJS3lmzksqMGcsbYAnIzUrwuUyTmactC4s7Ctdu5Zfoi5qzZSkpiAieM6M454/tw5OA8jUITaUG7oaTDW7phB0/PLuP5L9aybWcDZx3Um9+cNTokl00RiRdtDQt9ayRuDeuRxS3fGsnMXxzH1ccOYtrn5fz4qblfnbchIm2nYxYS91KTErl20lA6pyVzx6tLqG1o0hn0IvtIWxbSYVxy5ABuP30Uby/dyEX/mM07Syso37qTeNkVKxJO2rKQDuW7h/QjPTmRG56bzycrfbelzUxNYkj3TMb0yWFcvxwO6pdNzy46O1wkkA5wS4e0fVcDKyqqWFZRxfINVSxev4P55dup8x/P6Ne1E5ccMYBzivrorHCJaxoNJbKP6hubWbJ+B3PWbOWl+ev44sttFGSnc8Uxg/ifcb0VGhKXFBYi7eCc44MVm/nDm8uZW7aNwrwMHr6giIH5mV6XJhJSGjor0g5mxlFD8nn+8sN45MIiduxq4Kw/f0Lx6kqvSxPxhMJCZC/MjGOHdee5yw8jp1MK5z88k9cWrPe6LJGIU1iItEG/rhlM+9FhjOqVxeVPfM6f3ivRyX3SoSgsRNooNyOFJy45hJNG9eC3ry/j+N+/z/NflNPUHB/H/UT2RmEhsg/SkhN58PyDePQH48lMTeKap+cx5b4PeX/5Jq9LEwkrhYXIPjIzjhnajZevOpwHzh9LQ1MzFzwyi7teW0Jjk3ZNSXxSWIjsp4QE45QDevHqj4/gOwf35a/vl3L+wzPZGHD3PpF4obAQaae05ETuOGM09357DAvKtzPl/g+ZWbrF67JEQkphIRIip48tYPqVE8lKT+aCR2cpMCSuKCxEQmhw984888ND6Z3TiYv+MZs5a7Z6XZJISCgsREIsLzOVxy8+mPzOqVz46CwWlG/3uiSRdlNYiIRB96w0Hr/kELLSkvneIzNZumGH1yWJtIvCQiRMCrLTefKSQ0hNSuCiR2ezqarO65JE9pvCQiSM+nbtxN8vGE/lznou+/cc6hqbvC5JZL8oLETCbFRBF3539oHMWbOVXz6/ULdxlZik26qKRMApB/Ri+YYq7n+nhKE9OnPxEQO8Lklkn2jLQiRCfnL8EE4c2Z07X13CRys2e12OyD5RWIhESEKC8ftzxlCYl8GNz89nV72OX0jsUFiIRFBGahK3nz6asspdPPhuidfliLSZwkIkwg4d2JUzxxbw1w9WUrKxyutyRNpEYSHigV+cPJz05ER++YJGR0lsCGtYmNlkM1tmZiVmdkMrr//BzOb6H8vNbFvAaxeY2Qr/44Jw1ikSaXmZqVw/eRiflVbywty1XpcjElTYwsLMEoEHgZOAEcB5ZjYisI1z7hrn3Bjn3Bjgj8Bz/mVzgVuAg4EJwC1mlhOuWkW8cP6EvhzYJ5s7XlnC9p0NXpcjslfh3LKYAJQ450qdc/XAU8Bpe2l/HvCk//mJwJvOuUrn3FbgTWByGGsVibiEBOOO00dRWVPP795Y5nU5InsVzrAoAMoCpsv98/6LmfUDCoF39nVZkVg2qqAL3zukH4/PXMPCtbo6rUSvcIaFtTJvT0fyzgWedc7tHnjepmXN7FIzKzaz4k2bNu1nmSLeunbSUHI6pfCrFxfS3KyD3RKdwhkW5UCfgOnewLo9tD2Xr3dBtXlZ59xDzrki51xRfn5+O8sV8UaX9GRunDKcL77cxrNzyr0uR6RV4QyL2cBgMys0sxR8gTC9ZSMzGwrkAJ8GzJ4BTDKzHP+B7Un+eSJx6cyxBRT1y2Hq60t1sFuiUtjCwjnXCFyJ74/8EuAZ59wiM7vVzE4NaHoe8JQLGGzunKsEbsMXOLOBW/3zROJSQoJx62mj2LZTB7slOlm8nBBUVFTkiouLvS5DpF1+PX0Rj326mulXHM7o3l28Lkc6ADOb45wrCtZOZ3CLRJFrJw2ha0aqDnZL1FFYiESRrLRkbjp5GHPLtvF0cVnwBUQiRGEhEmVOH1PAwYW5/Ob1pVTW1HtdjgigsBCJOmbGbaePorq2kd+8ttTrckQAhYVIVBrSvTMXHV7I08VlzFmz1etyRBQWItHqx8cNpkdWGr96YSGNTc1elyMdnMJCJEplpCbxq1NGsHj9Dp4p1pnd4i2FhUgUmzK6BxP65/L7N5dRVaszu8U7CguRKGZm/PKU4WyurufP7630uhzpwBQWIlHugN7ZnDm2gIc/WkX51p1elyMdlMJCJAb89MShJBj89nVdN0q8obAQiQG9stO55IgBTJ+3js+/1FBaiTyFhUiMuOyogeR3TuX2lxcTLxcAldihsBCJERmpSfxs0lA+/3Ibz+i6URJhCguRGPI/43ozoTCX219ZQsWOWq/LkQ5EYSESQxISjKlnjqa+sZmbX1zodTnSgSgsRGLMgPxMfnL8EGYsquC1Beu9Lkc6CIWFSAy65IhCRhVkcfP0Rbpnt0SEwkIkBiUlJvCbsw6gsqae219Z7HU50gEoLERi1MheXbjkiAH8Z045yzZUeV2OxDmFhUgM++GRA0hPTuShD0q9LkXinMJCJIblZKTw7fF9eHHuWtZv3+V1ORLHFBYiMe5/Dy/EAY9+vNrrUiSOKSxEYlyf3E6cPLonT8z8ku27NDJKwkNhIRIHLj1yANV1jTwx80uvS5E4pbAQiQOjCrpw+KA8Hv14FXWNTV6XI3FIYSESJ3541AA2VtXx4hfrvC5F4pDCQiROHD4ojxE9s/jLBytpbGr2uhyJMwoLkThhZlx93GBKN9XwnznlXpcjcUZhIRJHThzZnaJ+OdzzxnJq6hq9LkfiiMJCJI6YGTdOGc7m6jqd1S0hpbAQiTPj+uUwZXQPHvqglI26QVLc+09xGY/PXBP29SgsROLQ9ScOo7G5mT+8tdzrUiTMnppdxvS54R8Bp7AQiUP98zL4zsH9eHp2GcsrdEXaeLZqcw0D8jPCvh6FhUicuvq4wWSkJHHnq0u8LkXCZNvOeipr6hmQlxn2dSksROJUbkYKVx83mPeWbeLtJRVelyNhsHJTDQCFedqyEJF2uHBifwZ1y+T/XlpMbYMuAxJvVm32hUXU7IYys4Fmlup/frSZXW1m2W1YbrKZLTOzEjO7YQ9tzjGzxWa2yMyeCJjfZGZz/Y/pbe2QiHwtOTGBX39rJF9W7tRQ2jhUuqmapASjT26nsK+rrVsW04AmMxsE/B0oBJ7Y2wJmlgg8CJwEjADOM7MRLdoMBm4EJjrnRgI/CXh5l3NujP9xahvrFJEWDh+cx5TRPXjw3RLKKnd6XY6E0KrNNfTN7URyYvh3ErV1Dc3OuUbgDOBe59w1QM8gy0wASpxzpc65euAp4LQWbS4BHnTObQVwzm1se+ki0lY3nTyCBDNuf2Wx16VICJVuisxIKGh7WDSY2XnABcDL/nnJQZYpAMoCpsv98wINAYaY2cdm9pmZTQ54Lc3Miv3zT29jnSLSioLsdK48dhAzFlXw/vJNXpcjIdDc7Fi1pSYiB7eh7WHxA+BQ4A7n3CozKwT+HWQZa2WeazGdBAwGjgbOAx4OOBbS1zlXBJwP3GtmA/9rBWaX+gOleNMmfQFE9ubiIwopzMvg6ie/4JOVm70uR9pp7bZd1Dc2MyA//MNmoY1h4Zxb7Jy72jn3pJnlAJ2dc1ODLFYO9AmY7g20PM2wHHjROdfgnFsFLMMXHjjn1vn/LQXeA8a2UtdDzrki51xRfn5+W7oi0mGlJiXyz4sm0K1zKt//+yyemV0WfCGJWrtHQkXVloWZvWdmWWaWC8wDHjWz3wdZbDYw2MwKzSwFOBdoOarpBeAY/zry8O2WKjWznIDRV3nAREA7W0XaqU9uJ6ZdfhiHDuzK9dPmc9drS2hubrnBL7GgdFM1EJlhs9D23VBdnHM7gDOBR51z44Dj97aA/4D4lcAMYAnwjHNukZndama7RzfNALaY2WLgXeBnzrktwHCg2Mzm+edPdc4pLERCICstmUcvHM93Du7LX98v5daX9dWKRas215CZmkR+ZmpE1pfU1nZm1hM4B7iprW/unHsVeLXFvJsDnjvgWv8jsM0nwOi2rkdE9k1SYgK3nz6K5MQE/vHJaqaM7smEwlyvy5J9UOq/JpRZa4eHQ6+tWxa34tsKWOmcm21mA4AV4StLRMLNzLh+8lB656Rzw7T5OsM7xpRuitxIKGj7Ae7/OOcOcM79yD9d6pw7K7yliUi4dUpJ4s4zRlO6uYYH3inxuhxpo9qGJtZt3xWRCwju1tYD3L3N7Hkz22hmFWY2zcx6h7s4EQm/I4fkc+ZBBfzl/ZUsWb/D63KkDVZvqcE5KIzQwW1o+26oR/GNZOqF78S6l/zzRCQO/OrkEXRJT+aGafNp0uioqFfqv9rsgGjbDQXkO+cedc41+h//AHRig0icyMlI4ZZTRzKvfDv//HS11+VIEJE+xwLaHhabzey7Zpbof3wX2BLOwkQksr51QE8OH5THvW+tYPvOBq/Lkb1YuamaHllpZKS2dUBr+7U1LC7CN2x2A7Ae+B98lwARkThhZvxiynB21DbwwLsa7BjNIj0SCto+GupL59ypzrl851w359zp+E7QE5E4MqJXFmeP681jn6zhyy26nHk0cs5Ruqk6Ymdu79aei6BfG7yJiMSaa08YSmKC8ZsZS70uRVpRWVPPjtrG6Nyy2IPInDYoIhHVo0salxw5gFfmr2fOmq1elyMt7D64PTBCV5vdrT1hofF1InHqh0cOIC8zlTtfXYLvqjwSLb4aNhtNu6HMrMrMdrTyqMJ3zoWIxKGM1CSumzSEOWu28pQuZR5VSjfXkJxoFGSnR3S9ex135ZzrHKlCRCS6nFPUh5fnr+Om5xeQnZ7MSaOD3UlZImHttl30yk4nKQL33Q4U2bWJSMxITDAe+l4RY/vmcPVTX/Du0o1elyRAZU0dXTNSIr5ehYWI7FFGahKP/mA8w3pk8cN/z+GTEt2O1WuVNQ3kKixEJNpkpSXzz4smUNg1g4v/Wcwzs8t0dz0PVdbUKSxEJDrlZKTwr4snMLxnFtdPm8+3HviImaW64k+kOefYWtNAjsJCRKJVt85pPHvZodx37hi21tTz7Yc+4/LH57C5us7r0jqM6rpG6puadcxCRKKbmXHamALevu5orj1hCG8v2ciU+z5k1qpKr0vrELbW+C7wmNNJYSEiMSA9JZGrjxvM85dPJCM1ifP+9hl/eX+ljmWE2ZYa31Zc10yFhYjEkBG9sph+5UROHNmdqa8t5dJ/zaG+sdnrsuLW1p31gLYsRCQGdU5L5sHzD+KXJw/nrSUVPPrxKq9Liltbqn1h0TUjNeLrVliISLuZGRcfMYDjhnXj/rdXULGj1uuS4tLuLYtc7YYSkVj2q1NG0NDkmPqaLm8eDltq6klJTCAjJTHi61ZYiEjI9M/L4JIjC3n+i7UUr9YIqVDbWlNPbkYKZpG/Q4TCQkRC6opjBtGzSxo3v7iIJo2OCqnKmnpPTsgDhYWIhFinlCR+MWU4i9fv4MlZX3pdTlyprKn35IQ8UFiISBicckBPDhmQy+/eWKaD3SGkLQsRiStmxh1njKauoZlrnp6r3VEhoi0LEYk7A/Mz+b9TR/LJyi385f2VXpcT8xqamtlR2+jJCXmgsBCRMDq7qDenHNCT37+5nDlrtnpdTkzbWuPdORagsBCRMDIz7jxzNL2y07j6yS/YvqvB65JiVuXuE/K0ZSEi8SgrLZn7zx1LxY5afvnCQq/LiVmV/kt9eHHjI1BYiEgEjO2bw+VHD+Sleeso2VjldTkx6astC4WFiMSz7x/Wn+RE49+f6dyL/VFZo7AQkQ4gLzOVKaN7Mu3zcnbWN3pdTszZHRbZnZI9Wb/CQkQi5ruH9KOqtpHpc9d5XUrMqaypp0t6MsmJ3vzZDutazWyymS0zsxIzu2EPbc4xs8VmtsjMngiYf4GZrfA/LghnnSISGUX9chjWozP/+mwNzulEvX1R6b+IoFfCFhZmlgg8CJwEjADOM7MRLdoMBm4EJjrnRgI/8c/PBW4BDgYmALeYWU64ahWRyDAzvntIPxat28Hcsm1elxNT4jYs8P2RL3HOlTrn6oGngNNatLkEeNA5txXAObfRP/9E4E3nXKX/tTeByWGsVUQi5PSxBWSkJPKvz9Z4XUpMieewKADKAqbL/fMCDQGGmNnHZvaZmU3eh2VFJAZlpiZx5kG9eXn++q/OSpbgKmvqPTshD8IbFq3dnaPlTsokYDBwNHAe8LCZZbdxWczsUjMrNrPiTZs2tbNcEYmU7x7Sj/rGZv4zpyx4Y8E5x9ad9Z5d6gPCGxblQJ+A6d5AyyEQ5cCLzrkG59wqYBm+8GjLsjjnHnLOFTnnivLz80NavIiEz9AenZnQP5dHPlpNdZ2G0QZTVddIQ5OL2y2L2cBgMys0sxTgXGB6izYvAMcAmFkevt1SpcAMYJKZ5fgPbE/yzxOROPHzk4ZRUVXLb1/X/bqD2erxCXkQxrBwzjUCV+L7I78EeMY5t8jMbjWzU/3NZgBbzGwx8C7wM+fcFudcJXAbvsCZDdzqnycicWJcvxwuPKw///x0DbNW6eu9N1uiICySwvnmzrlXgVdbzLs54LkDrvU/Wi77CPBIOOsTEW/9dNJQ3lxcwQ3T5vPqj48gLTnR65KiUlxvWYiIBJORmsTUMw+gdHMN97+9wutyolY0bFkoLETEU4cPzuOcot789YNSFq7d7nU5UUlbFiIiwE1TRpCbkcIP/zWHFRW6hHlLlTX1pCQl0CnFu910CgsR8VyXTsk8csF46puaOfNPn/DRis1elxRVKmvq6ZqRgllrp6BFhsJCRKLC6N5deOGKifTKTufCR2fx1Czd92K3ypp6cjw8xwIUFiISRQqy03n2R4dy2KA8bnhuAQ99sNLrkqJC5c56unp49jYoLEQkynROS+aRC4o4cWR3fjdjOas213hdkue0ZSEi0oqkxARuO20UqUkJ3Pziwg5/7wuvrzgLCgsRiVLdstK4btIQPlyxmVcXbPC6HM/UNzZTVduosBAR2ZPvHtKPET2zuPXlRR32goPbdnp/jgUoLEQkiiUlJnD7GaOo2FHHfW8t97ocT+w+e7urwkJEZM8O6pvDeRP68MjHq5lZusXrciJu99nbOQoLEZG9u/7EYeRmpPDthz7jtAc/5pnZZeys7xi7pbRlISLSRjkZKbx1zVHcfMoIauoauX7afA6+823eXbbR69LCbutObVmIiLRZl07JXHR4IW9ecyT/uexQenVJ5/pn57N9V4PXpYXV5mpfWGSnJ3tah8JCRGKKmTG+fy73nHMglTX1TH1tidclhdWaLTUUZKeTlOjtn2uFhYjEpFEFXbj4iEKenFXGpyvj98D3iopqBnfP9LoMhYWIxK6fHDeEfl078YvnF1Db0OR1OSHX1OxYuamawd0UFiIi+y09JZE7zxjNqji901751p3UNTYzuFtnr0tRWIhIbJs4KI+zx/nutLdk/Q6vywmpFRXVAAzSbigRkfa76eThZKYmcfeMZV6XElIrNvrDQruhRETaL7tTCpcdNZB3lm6keHWl1+WEzIqNVfTISiMrzdths6CwEJE4ceFh/cnvnMpvX18WN5c0L9kYHSOhQGEhInEiPSWRq48dxKzVlby/fJPX5bRbc7OjZGN1VOyCAoWFiMSRb4/vS5/cdO6esYzm5j1vXdTUNbJ4XXQfDF+3fRc765sUFiIioZaSlMA1xw9h0bodvLaw9RsmfVKymRPv/YAp938Y1ff43n1wOxqGzYLCQkTizGljChjSPZN73ljG0g07vjpZr7qukZueX8D5D88kOTGB44Z1485Xl3LPG9F5jKOkYndYRMeWRZLXBYiIhFJigvHzycP438eKmXzvh5hBn5xO1DY0sam6jkuOKOS6SUNJTkzgF88t4I/vlFBV28jNp4wgIcG8Lv8rKzZWkZeZ4vnVZndTWIhI3DlueHfeuvYoFq/fwcqN1azcVE11XSNXHTuYcf1yvmo39azRZKYl8fePVrGrvompZ43GLDoCY0UUHdwGhYWIxKlB3TKD/rE1M3558nDSkhN48N2VjCzI4vuH9o9MgXvhnKOkoprTxxZ4XcpXdMxCRDo0M+O6E4ZyzNB8bn95CQvXbve6JCp21FFV1xg151iAwkJEhIQE455zxpCbkcIVT3xOVa23N1RasbEKiI7LfOymsBARAXIzUvjj+WMp37qLG59b4OkIqRUV0TVsFhQWIiJfGd8/l2tPGMLL89fzxKwvPatjxcZqsjslk5cZHSOhQGEhIvINPzpqIEcOyefWlxZT4t8dFGklG6sY3C0zakZmgcJCROQbEhLw++8XAAANB0lEQVSM3519ABmpSfz4qbnUNzZHdP3OOZZXVDMoinZBgcJCROS/dOucxtQzR7No3Q7ufWt5SN+7rHInE6e+w8WPFTNtTjnbd37zYPrm6nq272qImjO3dwvreRZmNhm4D0gEHnbOTW3x+oXA3cBa/6wHnHMP+19rAhb453/pnDs1nLWKiASaNLIH547vw5/fX8nRQ7sxoTA3JO/79pIK1m7bRWNzM28tqSApwTh4QC7DemQxID+DXfW+y5NE07BZCGNYmFki8CBwAlAOzDaz6c65xS2aPu2cu7KVt9jlnBsTrvpERIL51Skj+LR0C9c8PZfXfnJESG5CNGt1JQXZ6Xz082OYV76d1xau54Plm3l85hpqG77e5TWke3TthgrnlsUEoMQ5VwpgZk8BpwEtw0JEJCplpCbxh2+P4ey/fMqN0xZw/3ljSWzH9aOcc8xaVcmRg/MxM8b0yWZMn2xuPMl3/4oNO2pZtbkGgO5ZaaHqRkiE85hFAVAWMF3un9fSWWY238yeNbM+AfPTzKzYzD4zs9PDWKeIyB4d1DeH608cyisL1vOL5xbs9T4ZwZRurmFzdX2ru7QSEoxe2elMHJTHxEF57Sk5LMK5ZdFa/Lb8Kb8EPOmcqzOzy4DHgGP9r/V1zq0zswHAO2a2wDn3jYvPm9mlwKUAffv2DW31IiJ+PzxqIDV1jdz/TgnJScZtp43ar2Gts1f57g8equMfkRTOLYtyIHBLoTewLrCBc26Lc67OP/k3YFzAa+v8/5YC7wFjW67AOfeQc67IOVeUn58f2upFRAJcc8IQfnjkAP792Zfc9vKS/TrDe9aqSvIyUynMywhDheEVzi2L2cBgMyvEN9rpXOD8wAZm1tM5t94/eSqwxD8/B9jp3+LIAyYCvw1jrSIie2Vm3HDSMOqbmnnk41XMK9/GoPxM+nbtRJ/cTkwc2JWumal7fY+Zqyo5uDA3qk62a6uwhYVzrtHMrgRm4Bs6+4hzbpGZ3QoUO+emA1eb2alAI1AJXOhffDjwVzNrxrf1M7WVUVQiIhFlZtx8yghyOqXw7rKNvL10I5urfTtHMlOT+NHRA7loYiHpKYn/tWz51p2s3baLS48cEOmyQ8Ki8XaC+6OoqMgVFxd7XYaIdDA76xtZUVHNA++W8ObiCnpkpXHdpCGcdVDvb9x57/kvyrnm6Xm8evURjOiV5WHF32Rmc5xzRcHa6QxuEZF26JSSxIF9svnb94t4+tJD6N4ljZ89O5+731j2jXazVlWSlZbE0B7Rdf5EWyksRERC5OABXXnh8sM466De/O2DUpZu2PHVazNXVTK+f267ztPwksJCRCSEdt+qtXNaEr98fiHNzY5NVXWUbqqJySGzuyksRERCLCcjhV9MGU7xmq08U1zG7NWxe37FbgoLEZEw+J9xvZlQmMtdry3ltYUbSE9OZFRBF6/L2m8KCxGRMDAz7jh9FDV1jbw0bx3j+uWQnBi7f3Jjt3IRkSg3uHvnr86riOVdUBDm+1mIiHR0Vx07mPrGZs4a19vrUtpFYSEiEkbpKYn88pQRXpfRbtoNJSIiQSksREQkKIWFiIgEpbAQEZGgFBYiIhKUwkJERIJSWIiISFAKCxERCSpu7pRnZpuANa281AXYHmRe4HRrz3f/mwds3s8SW6ujrW2C9WFP/WmtTTj7sLfX9/Yzbzkd7LkXfQjF71Hg8/3tQzh/j1pO7+27ANHZh7b0J9q+z22dDtd3oZ9zLj9oa+dcXD+Ah4LNC5xu7XnAv8WhrKOtbYL1YU/92UNfwtaHvb2+t595Wz4Dr/sQit+jUPQhnL9Hbaw7cF7U9aEt/Ym273NbpyP9XWj56Ai7oV5qw7yXgjxv7T1CUUdb2wTrw576s7c2+yPYe+zt9b39zFtOt+X5/trfPoTi96gt6w8mnL9HLafj6bsQ+Dza+tDW6Uh/F74hbnZDRYKZFbs23Ng8mqkP0UF98F6s1w+R7UNH2LIIpYe8LiAE1IfooD54L9brhwj2QVsWIiISlLYsREQkqA4bFmb2iJltNLOF+7HsODNbYGYlZna/mVnAa1eZ2TIzW2Rmvw1t1f9VR8j7YGa/NrO1ZjbX/5gS+sq/UUdYPgf/6z81M2dmeaGruNU6wvE53GZm8/2fwRtm1iv0lX9VQzjqv9vMlvr78LyZZYe+8m/UEY4+nO3/HjebWdiOC7Sn9j283wVmtsL/uCBg/l6/L0Ht77CrWH8ARwIHAQv3Y9lZwKGAAa8BJ/nnHwO8BaT6p7vFYB9+Dfw0lj8H/2t9gBn4zr3Ji7U+AFkBba4G/hJj9U8CkvzPfwP8JgY/g+HAUOA9oCjaavfX1b/FvFyg1P9vjv95zt762dZHh92ycM59AFQGzjOzgWb2upnNMbMPzWxYy+XMrCe+L/KnzvcJ/BM43f/yj4Cpzrk6/zo2xmAfIiqMffgDcD0Q9oNy4eiDc25HQNMMwtiPMNX/hnOu0d/0MyCs9xQNUx+WOOeWhbPu9tS+BycCbzrnKp1zW4E3gcmh+M532LDYg4eAq5xz44CfAn9qpU0BUB4wXe6fBzAEOMLMZprZ+2Y2PqzVtq69fQC40r/74BEzywlfqXvUrj6Y2anAWufcvHAXuhft/hzM7A4zKwO+A9wcxlpbE4rfo90uwvc/2UgLZR8irS21t6YAKAuY3t2fdvdT9+D2M7NM4DDgPwG78lJba9rKvN3/60vCt+l3CDAeeMbMBviTPOxC1Ic/A7f5p28D7sH3ZY+I9vbBzDoBN+HbDeKJEH0OOOduAm4ysxuBK4FbQlxqq0JVv/+9bgIagcdDWWMwoexDpO2tdjP7AfBj/7xBwKtmVg+scs6dwZ770+5+Kiy+lgBsc86NCZxpZonAHP/kdHx/TAM3qXsD6/zPy4Hn/OEwy8ya8V27ZVM4Cw/Q7j445yoClvsb8HI4C25Fe/swECgE5vm/aL2Bz81sgnNuQ5hr3y0Uv0uBngBeIUJhQYjq9x9cPQU4LlL/YQoQ6s8gklqtHcA59yjwKICZvQdc6JxbHdCkHDg6YLo3vmMb5bS3n+E6aBMLD6A/AQeVgE+As/3PDThwD8vNxrf1sPtA0RT//MuAW/3Ph+DbHLQY60PPgDbXAE/F2ufQos1qwnyAO0yfw+CANlcBz8ZY/ZOBxUB+uH/24f49IswHuPe3dvZ8gHsVvj0cOf7nuW3pZ9AaI/VBRtsDeBJYDzTgS93/xfc/0teBef5f9Jv3sGwRsBBYCTzA1yc3pgD/9r/2OXBsDPbhX8ACYD6+/3n1jLU+tGizmvCPhgrH5zDNP38+vmv4FMRY/SX4/rM01/8I22iuMPbhDP971QEVwIxoqp1WwsI//yL/z78E+MG+fF/29tAZ3CIiEpRGQ4mISFAKCxERCUphISIiQSksREQkKIWFiIgEpbCQuGZm1RFe38NmNiJE79VkvqvOLjSzl4JdudXMss3s8lCsW6QlDZ2VuGZm1c65zBC+X5L7+gJ5YRVYu5k9Bix3zt2xl/b9gZedc6MiUZ90LNqykA7HzPLNbJqZzfY/JvrnTzCzT8zsC/+/Q/3zLzSz/5jZS8AbZna0mb1nZs+a754Nj+++N4B/fpH/ebX/YoDzzOwzM+vunz/QPz3bzG5t49bPp3x9ocRMM3vbzD433/0JTvO3mQoM9G+N3O1v+zP/euab2f+F8McoHYzCQjqi+4A/OOfGA2cBD/vnLwWOdM6NxXeV1zsDljkUuMA5d6x/eizwE2AEMACY2Mp6MoDPnHMHAh8AlwSs/z7/+oNen8d/PaPj8J1RD1ALnOGcOwjfPVTu8YfVDcBK59wY59zPzGwSMBiYAIwBxpnZkcHWJ9IaXUhQOqLjgREBV/TMMrPOQBfgMTMbjO+KnMkBy7zpnAu858As51w5gJnNxXdtn49arKeery/EOAc4wf/8UL6+l8ATwO/2UGd6wHvPwXdvAvBd2+dO/x/+ZnxbHN1bWX6S//GFfzoTX3h8sIf1ieyRwkI6ogTgUOfcrsCZZvZH4F3n3Bn+/f/vBbxc0+I96gKeN9H6d6nBfX1QcE9t9maXc26MmXXBFzpXAPfju79FPjDOOddgZquBtFaWN+Au59xf93G9Iv9Fu6GkI3oD3/0hADCz3ZeC7gKs9T+/MIzr/wzf7i+Ac4M1ds5tx3dr1Z+aWTK+Ojf6g+IYoJ+/aRXQOWDRGcBF/vsjYGYFZtYtRH2QDkZhIfGuk5mVBzyuxfeHt8h/0HcxvkvLA/wWuMvMPgYSw1jTT4BrzWwW0BPYHmwB59wX+K5Aei6+GwkVmVkxvq2Mpf42W4CP/UNt73bOvYFvN9enZrYAeJZvholIm2norEiE+e/mt8s558zsXOA859xpwZYT8ZKOWYhE3jjgAf8Ipm1E8La1IvtLWxYiIhKUjlmIiEhQCgsREQlKYSEiIkEpLEREJCiFhYiIBKWwEBGRoP4fUqdtPfBIUscAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.246133</td>\n",
       "      <td>0.182420</td>\n",
       "      <td>0.931400</td>\n",
       "      <td>03:39</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(1, 2e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save('first')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.load('first');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.226397</td>\n",
       "      <td>0.164568</td>\n",
       "      <td>0.938320</td>\n",
       "      <td>04:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.freeze_to(-2)\n",
    "learn_c.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save('2nd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.181977</td>\n",
       "      <td>0.159952</td>\n",
       "      <td>0.939480</td>\n",
       "      <td>06:07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.freeze_to(-3)\n",
    "learn_c.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save('3rd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.167931</td>\n",
       "      <td>0.157721</td>\n",
       "      <td>0.939800</td>\n",
       "      <td>06:30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.147676</td>\n",
       "      <td>0.160450</td>\n",
       "      <td>0.940560</td>\n",
       "      <td>07:32</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.unfreeze()\n",
    "learn_c.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The state of the art for this dataset in 2017 was 94.1%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save('clas')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Category pos, tensor(1), tensor([2.0825e-04, 9.9979e-01]))"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.predict(\"I really loved that movie, it was awesome!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Category neg, tensor(0), tensor([0.9874, 0.0126]))"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.predict(\"I didn't really love that movie, and I didn't think it was awesome.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Language Model Zoo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "fast.ai alumni have applied ULMFit to dozens of different languages, and have beat the SOTA in Thai, Polish, German, Indonesian, Hindi, & Malay.\n",
    "\n",
    "They share tips and best practices in [this forum thread](https://forums.fast.ai/t/language-model-zoo-gorilla/14623) in case you are interested in getting involved!\n",
    "\n",
    "<img src=\"images/language_model_zoo.png\" alt=\"language model zoo\" style=\"width: 80%\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}