6.1-using-word-embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.0.8'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using word embeddings\n",
    "\n",
    "This notebook contains the second code sample found in Chapter 6, Section 1 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "Another popular and powerful way to associate a vector with a word is the use of dense \"word vectors\", also called \"word embeddings\". \n",
    "While the vectors obtained through one-hot encoding are binary, sparse (mostly made of zeros) and very high-dimensional (same dimensionality as the \n",
    "number of words in the vocabulary), \"word embeddings\" are low-dimensional floating point vectors \n",
    "(i.e. \"dense\" vectors, as opposed to sparse vectors). \n",
    "Unlike word vectors obtained via one-hot encoding, word embeddings are learned from data. \n",
    "It is common to see word embeddings that are 256-dimensional, 512-dimensional, or 1024-dimensional when dealing with very large vocabularies. \n",
    "On the other hand, one-hot encoding words generally leads to vectors that are 20,000-dimensional or higher (capturing a vocabulary of 20,000 \n",
    "token in this case). So, word embeddings pack more information into far fewer dimensions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![word embeddings vs. one hot encoding](https://s3.amazonaws.com/book.keras.io/img/ch6/word_embeddings.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two ways to obtain word embeddings:\n",
    "\n",
    "* Learn word embeddings jointly with the main task you care about (e.g. document classification or sentiment prediction). \n",
    "In this setup, you would start with random word vectors, then learn your word vectors in the same way that you learn the weights of a neural network.\n",
    "* Load into your model word embeddings that were pre-computed using a different machine learning task than the one you are trying to solve. \n",
    "These are called \"pre-trained word embeddings\". \n",
    "\n",
    "Let's take a look at both."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning word embeddings with the `Embedding` layer\n",
    "\n",
    "\n",
    "The simplest way to associate a dense vector to a word would be to pick the vector at random. The problem with this approach is that the \n",
    "resulting embedding space would have no structure: for instance, the words \"accurate\" and \"exact\" may end up with completely different \n",
    "embeddings, even though they are interchangeable in most sentences. It would be very difficult for a deep neural network to make sense of \n",
    "such a noisy, unstructured embedding space. \n",
    "\n",
    "To get a bit more abstract: the geometric relationships between word vectors should reflect the semantic relationships between these words. \n",
    "Word embeddings are meant to map human language into a geometric space. For instance, in a reasonable embedding space, we would expect \n",
    "synonyms to be embedded into similar word vectors, and in general we would expect the geometric distance (e.g. L2 distance) between any two \n",
    "word vectors to relate to the semantic distance of the associated words (words meaning very different things would be embedded to points \n",
    "far away from each other, while related words would be closer). Even beyond mere distance, we may want specific __directions__ in the \n",
    "embedding space to be meaningful. \n",
    "\n",
    "[...]\n",
    "\n",
    "\n",
    "In real-world word embedding spaces, common examples of meaningful geometric transformations are \"gender vectors\" and \"plural vector\". For \n",
    "instance, by adding a \"female vector\" to the vector \"king\", one obtain the vector \"queen\". By adding a \"plural vector\", one obtain \"kings\". \n",
    "Word embedding spaces typically feature thousands of such interpretable and potentially useful vectors.\n",
    "\n",
    "Is there some \"ideal\" word embedding space that would perfectly map human language and could be used for any natural language processing \n",
    "task? Possibly, but in any case, we have yet to compute anything of the sort. Also, there isn't such a thing as \"human language\", there are \n",
    "many different languages and they are not isomorphic, as a language is the reflection of a specific culture and a specific context. But more \n",
    "pragmatically, what makes a good word embedding space depends heavily on your task: the perfect word embedding space for an \n",
    "English-language movie review sentiment analysis model may look very different from the perfect embedding space for an English-language \n",
    "legal document classification model, because the importance of certain semantic relationships varies from task to task.\n",
    "\n",
    "It is thus reasonable to __learn__ a new embedding space with every new task. Thankfully, backpropagation makes this really easy, and Keras makes it \n",
    "even easier. It's just about learning the weights of a layer: the `Embedding` layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.layers import Embedding\n",
    "\n",
    "# The Embedding layer takes at least two arguments:\n",
    "# the number of possible tokens, here 1000 (1 + maximum word index),\n",
    "# and the dimensionality of the embeddings, here 64.\n",
    "embedding_layer = Embedding(1000, 64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The `Embedding` layer is best understood as a dictionary mapping integer indices (which stand for specific words) to dense vectors. It takes \n",
    "as input integers, it looks up these integers into an internal dictionary, and it returns the associated vectors. It's effectively a dictionary lookup."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The `Embedding` layer takes as input a 2D tensor of integers, of shape `(samples, sequence_length)`, where each entry is a sequence of \n",
    "integers. It can embed sequences of variable lengths, so for instance we could feed into our embedding layer above batches that could have \n",
    "shapes `(32, 10)` (batch of 32 sequences of length 10) or `(64, 15)` (batch of 64 sequences of length 15). All sequences in a batch must \n",
    "have the same length, though (since we need to pack them into a single tensor), so sequences that are shorter than others should be padded \n",
    "with zeros, and sequences that are longer should be truncated.\n",
    "\n",
    "This layer returns a 3D floating point tensor, of shape `(samples, sequence_length, embedding_dimensionality)`. Such a 3D tensor can then \n",
    "be processed by a RNN layer or a 1D convolution layer (both will be introduced in the next sections).\n",
    "\n",
    "When you instantiate an `Embedding` layer, its weights (its internal dictionary of token vectors) are initially random, just like with any \n",
    "other layer. During training, these word vectors will be gradually adjusted via backpropagation, structuring the space into something that the \n",
    "downstream model can exploit. Once fully trained, your embedding space will show a lot of structure -- a kind of structure specialized for \n",
    "the specific problem you were training your model for.\n",
    "\n",
    "Let's apply this idea to the IMDB movie review sentiment prediction task that you are already familiar with. Let's quickly prepare \n",
    "the data. We will restrict the movie reviews to the top 10,000 most common words (like we did the first time we worked with this dataset), \n",
    "and cut the reviews after only 20 words. Our network will simply learn 8-dimensional embeddings for each of the 10,000 words, turn the \n",
    "input integer sequences (2D integer tensor) into embedded sequences (3D float tensor), flatten the tensor to 2D, and train a single `Dense` \n",
    "layer on top for classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/text-datasets/imdb.npz\n",
      "12460032/17464789 [====================>.........] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "from keras.datasets import imdb\n",
    "from keras import preprocessing\n",
    "\n",
    "# Number of words to consider as features\n",
    "max_features = 10000\n",
    "# Cut texts after this number of words \n",
    "# (among top max_features most common words)\n",
    "maxlen = 20\n",
    "\n",
    "# Load the data as lists of integers.\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "\n",
    "# This turns our lists of integers\n",
    "# into a 2D integer tensor of shape `(samples, maxlen)`\n",
    "x_train = preprocessing.sequence.pad_sequences(x_train, maxlen=maxlen)\n",
    "x_test = preprocessing.sequence.pad_sequences(x_test, maxlen=maxlen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, 20, 8)             80000     \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 160)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 161       \n",
      "=================================================================\n",
      "Total params: 80,161\n",
      "Trainable params: 80,161\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 2s - loss: 0.6560 - acc: 0.6482 - val_loss: 0.5906 - val_acc: 0.7146\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 2s - loss: 0.5189 - acc: 0.7595 - val_loss: 0.5117 - val_acc: 0.7364\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 2s - loss: 0.4512 - acc: 0.7933 - val_loss: 0.4949 - val_acc: 0.7470\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 2s - loss: 0.4190 - acc: 0.8069 - val_loss: 0.4905 - val_acc: 0.7538\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 2s - loss: 0.3965 - acc: 0.8198 - val_loss: 0.4914 - val_acc: 0.7572\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 2s - loss: 0.3784 - acc: 0.8311 - val_loss: 0.4953 - val_acc: 0.7594\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 2s - loss: 0.3624 - acc: 0.8419 - val_loss: 0.5004 - val_acc: 0.7574\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 2s - loss: 0.3474 - acc: 0.8484 - val_loss: 0.5058 - val_acc: 0.7572\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 2s - loss: 0.3330 - acc: 0.8582 - val_loss: 0.5122 - val_acc: 0.7528\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 2s - loss: 0.3194 - acc: 0.8669 - val_loss: 0.5183 - val_acc: 0.7554\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "# We specify the maximum input length to our Embedding layer\n",
    "# so we can later flatten the embedded inputs\n",
    "model.add(Embedding(10000, 8, input_length=maxlen))\n",
    "# After the Embedding layer, \n",
    "# our activations have shape `(samples, maxlen, 8)`.\n",
    "\n",
    "# We flatten the 3D tensor of embeddings \n",
    "# into a 2D tensor of shape `(samples, maxlen * 8)`\n",
    "model.add(Flatten())\n",
    "\n",
    "# We add the classifier on top\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n",
    "model.summary()\n",
    "\n",
    "history = model.fit(x_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=32,\n",
    "                    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get to a validation accuracy of ~76%, which is pretty good considering that we only look at the first 20 words in every review. But \n",
    "note that merely flattening the embedded sequences and training a single `Dense` layer on top leads to a model that treats each word in the \n",
    "input sequence separately, without considering inter-word relationships and structure sentence (e.g. it would likely treat both _\"this movie \n",
    "is shit\"_ and _\"this movie is the shit\"_ as being negative \"reviews\"). It would be much better to add recurrent layers or 1D convolutional \n",
    "layers on top of the embedded sequences to learn features that take into account each sequence as a whole. That's what we will focus on in \n",
    "the next few sections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using pre-trained word embeddings\n",
    "\n",
    "\n",
    "Sometimes, you have so little training data available that could never use your data alone to learn an appropriate task-specific embedding \n",
    "of your vocabulary. What to do then?\n",
    "\n",
    "Instead of learning word embeddings jointly with the problem you want to solve, you could be loading embedding vectors from a pre-computed \n",
    "embedding space known to be highly structured and to exhibit useful properties -- that captures generic aspects of language structure. The \n",
    "rationale behind using pre-trained word embeddings in natural language processing is very much the same as for using pre-trained convnets \n",
    "in image classification: we don't have enough data available to learn truly powerful features on our own, but we expect the features that \n",
    "we need to be fairly generic, i.e. common visual features or semantic features. In this case it makes sense to reuse features learned on a \n",
    "different problem.\n",
    "\n",
    "Such word embeddings are generally computed using word occurrence statistics (observations about what words co-occur in sentences or \n",
    "documents), using a variety of techniques, some involving neural networks, others not. The idea of a dense, low-dimensional embedding space \n",
    "for words, computed in an unsupervised way, was initially explored by Bengio et al. in the early 2000s, but it only started really taking \n",
    "off in research and industry applications after the release of one of the most famous and successful word embedding scheme: the Word2Vec \n",
    "algorithm, developed by Mikolov at Google in 2013. Word2Vec dimensions capture specific semantic properties, e.g. gender.\n",
    "\n",
    "There are various pre-computed databases of word embeddings that can download and start using in a Keras `Embedding` layer. Word2Vec is one \n",
    "of them. Another popular one is called \"GloVe\", developed by Stanford researchers in 2014. It stands for \"Global Vectors for Word \n",
    "Representation\", and it is an embedding technique based on factorizing a matrix of word co-occurrence statistics. Its developers have made \n",
    "available pre-computed embeddings for millions of English tokens, obtained from Wikipedia data or from Common Crawl data.\n",
    "\n",
    "Let's take a look at how you can get started using GloVe embeddings in a Keras model. The same method will of course be valid for Word2Vec \n",
    "embeddings or any other word embedding database that you can download. We will also use this example to refresh the text tokenization \n",
    "techniques we introduced a few paragraphs ago: we will start from raw text, and work our way up."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting it all together: from raw text to word embeddings\n",
    "\n",
    "\n",
    "We will be using a model similar to the one we just went over -- embedding sentences in sequences of vectors, flattening them and training a \n",
    "`Dense` layer on top. But we will do it using pre-trained word embeddings, and instead of using the pre-tokenized IMDB data packaged in \n",
    "Keras, we will start from scratch, by downloading the original text data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the IMDB data as raw text\n",
    "\n",
    "\n",
    "First, head to `http://ai.stanford.edu/~amaas/data/sentiment/` and download the raw IMDB dataset (if the URL isn't working anymore, just \n",
    "Google \"IMDB dataset\"). Uncompress it.\n",
    "\n",
    "Now let's collect the individual training reviews into a list of strings, one string per review, and let's also collect the review labels \n",
    "(positive / negative) into a `labels` list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "imdb_dir = '/home/ubuntu/data/aclImdb'\n",
    "train_dir = os.path.join(imdb_dir, 'train')\n",
    "\n",
    "labels = []\n",
    "texts = []\n",
    "\n",
    "for label_type in ['neg', 'pos']:\n",
    "    dir_name = os.path.join(train_dir, label_type)\n",
    "    for fname in os.listdir(dir_name):\n",
    "        if fname[-4:] == '.txt':\n",
    "            f = open(os.path.join(dir_name, fname))\n",
    "            texts.append(f.read())\n",
    "            f.close()\n",
    "            if label_type == 'neg':\n",
    "                labels.append(0)\n",
    "            else:\n",
    "                labels.append(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenize the data\n",
    "\n",
    "\n",
    "Let's vectorize the texts we collected, and prepare a training and validation split.\n",
    "We will merely be using the concepts we introduced earlier in this section.\n",
    "\n",
    "Because pre-trained word embeddings are meant to be particularly useful on problems where little training data is available (otherwise, \n",
    "task-specific embeddings are likely to outperform them), we will add the following twist: we restrict the training data to its first 200 \n",
    "samples. So we will be learning to classify movie reviews after looking at just 200 examples...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 88582 unique tokens.\n",
      "Shape of data tensor: (25000, 100)\n",
      "Shape of label tensor: (25000,)\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "import numpy as np\n",
    "\n",
    "maxlen = 100  # We will cut reviews after 100 words\n",
    "training_samples = 200  # We will be training on 200 samples\n",
    "validation_samples = 10000  # We will be validating on 10000 samples\n",
    "max_words = 10000  # We will only consider the top 10,000 words in the dataset\n",
    "\n",
    "tokenizer = Tokenizer(num_words=max_words)\n",
    "tokenizer.fit_on_texts(texts)\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "print('Found %s unique tokens.' % len(word_index))\n",
    "\n",
    "data = pad_sequences(sequences, maxlen=maxlen)\n",
    "\n",
    "labels = np.asarray(labels)\n",
    "print('Shape of data tensor:', data.shape)\n",
    "print('Shape of label tensor:', labels.shape)\n",
    "\n",
    "# Split the data into a training set and a validation set\n",
    "# But first, shuffle the data, since we started from data\n",
    "# where sample are ordered (all negative first, then all positive).\n",
    "indices = np.arange(data.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "data = data[indices]\n",
    "labels = labels[indices]\n",
    "\n",
    "x_train = data[:training_samples]\n",
    "y_train = labels[:training_samples]\n",
    "x_val = data[training_samples: training_samples + validation_samples]\n",
    "y_val = labels[training_samples: training_samples + validation_samples]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the GloVe word embeddings\n",
    "\n",
    "\n",
    "Head to `https://nlp.stanford.edu/projects/glove/` (where you can learn more about the GloVe algorithm), and download the pre-computed \n",
    "embeddings from 2014 English Wikipedia. It's a 822MB zip file named `glove.6B.zip`, containing 100-dimensional embedding vectors for \n",
    "400,000 words (or non-word tokens). Un-zip it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-process the embeddings\n",
    "\n",
    "\n",
    "Let's parse the un-zipped file (it's a `txt` file) to build an index mapping words (as strings) to their vector representation (as number \n",
    "vectors)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 400000 word vectors.\n"
     ]
    }
   ],
   "source": [
    "glove_dir = '/home/ubuntu/data/'\n",
    "\n",
    "embeddings_index = {}\n",
    "f = open(os.path.join(glove_dir, 'glove.6B.100d.txt'))\n",
    "for line in f:\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = np.asarray(values[1:], dtype='float32')\n",
    "    embeddings_index[word] = coefs\n",
    "f.close()\n",
    "\n",
    "print('Found %s word vectors.' % len(embeddings_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Now let's build an embedding matrix that we will be able to load into an `Embedding` layer. It must be a matrix of shape `(max_words, \n",
    "embedding_dim)`, where each entry `i` contains the `embedding_dim`-dimensional vector for the word of index `i` in our reference word index \n",
    "(built during tokenization). Note that the index `0` is not supposed to stand for any word or token -- it's a placeholder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "embedding_dim = 100\n",
    "\n",
    "embedding_matrix = np.zeros((max_words, embedding_dim))\n",
    "for word, i in word_index.items():\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if i < max_words:\n",
    "        if embedding_vector is not None:\n",
    "            # Words not found in embedding index will be all-zeros.\n",
    "            embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a model\n",
    "\n",
    "We will be using the same model architecture as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_4 (Embedding)      (None, 100, 100)          1000000   \n",
      "_________________________________________________________________\n",
      "flatten_3 (Flatten)          (None, 10000)             0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 32)                320032    \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 1,320,065\n",
      "Trainable params: 1,320,065\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_words, embedding_dim, input_length=maxlen))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the GloVe embeddings in the model\n",
    "\n",
    "\n",
    "The `Embedding` layer has a single weight matrix: a 2D float matrix where each entry `i` is the word vector meant to be associated with \n",
    "index `i`. Simple enough. Let's just load the GloVe matrix we prepared into our `Embedding` layer, the first layer in our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.layers[0].set_weights([embedding_matrix])\n",
    "model.layers[0].trainable = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Additionally, we freeze the embedding layer (we set its `trainable` attribute to `False`), following the same rationale as what you are \n",
    "already familiar with in the context of pre-trained convnet features: when parts of a model are pre-trained (like our `Embedding` layer), \n",
    "and parts are randomly initialized (like our classifier), the pre-trained parts should not be updated during training to avoid forgetting \n",
    "what they already know. The large gradient update triggered by the randomly initialized layers would be very disruptive to the already \n",
    "learned features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train and evaluate\n",
    "\n",
    "Let's compile our model and train it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 200 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "200/200 [==============================] - 1s - loss: 1.9075 - acc: 0.5050 - val_loss: 0.7027 - val_acc: 0.5102\n",
      "Epoch 2/10\n",
      "200/200 [==============================] - 0s - loss: 0.7329 - acc: 0.7100 - val_loss: 0.8200 - val_acc: 0.5000\n",
      "Epoch 3/10\n",
      "200/200 [==============================] - 0s - loss: 0.4876 - acc: 0.7400 - val_loss: 0.6917 - val_acc: 0.5616\n",
      "Epoch 4/10\n",
      "200/200 [==============================] - 0s - loss: 0.3640 - acc: 0.8400 - val_loss: 0.7005 - val_acc: 0.5557\n",
      "Epoch 5/10\n",
      "200/200 [==============================] - 0s - loss: 0.2673 - acc: 0.8950 - val_loss: 1.2560 - val_acc: 0.4999\n",
      "Epoch 6/10\n",
      "200/200 [==============================] - 0s - loss: 0.1936 - acc: 0.9400 - val_loss: 0.7294 - val_acc: 0.5704\n",
      "Epoch 7/10\n",
      "200/200 [==============================] - 0s - loss: 0.2455 - acc: 0.8800 - val_loss: 0.7187 - val_acc: 0.5659\n",
      "Epoch 8/10\n",
      "200/200 [==============================] - 0s - loss: 0.0591 - acc: 0.9950 - val_loss: 0.7393 - val_acc: 0.5723\n",
      "Epoch 9/10\n",
      "200/200 [==============================] - 0s - loss: 0.0399 - acc: 1.0000 - val_loss: 0.8691 - val_acc: 0.5522\n",
      "Epoch 10/10\n",
      "200/200 [==============================] - 0s - loss: 0.0283 - acc: 1.0000 - val_loss: 0.9322 - val_acc: 0.5413\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['acc'])\n",
    "history = model.fit(x_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=32,\n",
    "                    validation_data=(x_val, y_val))\n",
    "model.save_weights('pre_trained_glove_model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot its performance over time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8VNW5//HPAwS537EqCKHWKtdAiKACKsULWoGq1ILY\nI1pBUbTV9lgrtHps0V+rVatSK3psa0UoxwtSRe1RaZVaFFABgSOgIAYQAbkIQSDy/P5YkzAJuUzC\nhJnZ+b5fr3nNzN5r9n5mT/LMmrXXXsvcHRERiZY6qQ5ARESST8ldRCSClNxFRCJIyV1EJIKU3EVE\nIkjJXUQkgpTcI8zM6prZTjPrkMyyqWRm3zCzpPffNbMzzWxN3PMPzGxAImWrsa9HzeyW6r5eJBH1\nUh2AHGBmO+OeNgL2AF/Fnl/l7lOrsj13/wpokuyytYG7n5CM7ZjZlcCl7n5G3LavTMa2RSqi5J5G\n3L04ucZqhle6+yvllTezeu5eeDhiE6mM/h7Ti5plMoiZ/crM/mpm08zsC+BSMzvFzOaZ2TYz22Bm\n95tZVqx8PTNzM8uOPX8itv5FM/vCzP5tZp2qWja2/lwzW2Fm283sATP7l5mNLifuRGK8ysxWmdlW\nM7s/7rV1zexeM9tiZh8Bgys4PhPMbHqpZZPN7J7Y4yvNbHns/XwYq1WXt618Mzsj9riRmf0lFttS\noHepshPN7KPYdpea2dDY8u7Ag8CAWJPX5rhje1vc66+OvfctZjbTzI5O5NhU5TgXxWNmr5jZ52b2\nqZndFLefn8eOyQ4zW2Bmx5TVBGZmc4s+59jxfD22n8+BiWZ2vJnNie1jc+y4NY97fcfYe9wUW/87\nM2sQi7lzXLmjzazAzFqX936lEu6uWxregDXAmaWW/QrYCwwhfDE3BE4C+hJ+hX0dWAGMj5WvBziQ\nHXv+BLAZyAOygL8CT1Sj7JHAF8Cw2LobgX3A6HLeSyIxPgc0B7KBz4veOzAeWAq0B1oDr4c/2zL3\n83VgJ9A4btufAXmx50NiZQz4FrAb6BFbdyawJm5b+cAZscd3A/8AWgIdgWWlyl4MHB37TC6JxfC1\n2LorgX+UivMJ4LbY47NjMfYEGgC/B15L5NhU8Tg3BzYCPwSOAJoBfWLrfgYsAo6PvYeeQCvgG6WP\nNTC36HOOvbdCYBxQl/D3+E1gEFA/9nfyL+DuuPfzfux4No6V7xdbNwWYFLefHwPPpvr/MJNvKQ9A\nt3I+mPKT+2uVvO4nwP/EHpeVsP8QV3Yo8H41yl4BvBG3zoANlJPcE4zx5Lj1zwA/iT1+ndA8VbTu\nvNIJp9S25wGXxB6fC3xQQdnngWtjjytK7mvjPwvgmviyZWz3feDbsceVJfc/A3fErWtGOM/SvrJj\nU8Xj/H1gfjnlPiyKt9TyRJL7R5XEMLxov8AA4FOgbhnl+gGrAYs9fw+4MNn/V7XppmaZzPNJ/BMz\nO9HMXoj9zN4B3A60qeD1n8Y9LqDik6jllT0mPg4P/4355W0kwRgT2hfwcQXxAjwJjIw9viT2vCiO\n883srViTwTZCrbmiY1Xk6IpiMLPRZrYo1rSwDTgxwe1CeH/F23P3HcBWoF1cmYQ+s0qO87GEJF6W\nitZVpvTf41FmNsPM1sVi+FOpGNZ4OHlfgrv/i/AroL+ZdQM6AC9UMyZBbe6ZqHQ3wIcJNcVvuHsz\n4BeEmnRN2kCoWQJgZkbJZFTaocS4gZAUilTWVXMGcKaZtSM0Gz0Zi7Eh8BRwJ6HJpAXw9wTj+LS8\nGMzs68BDhKaJ1rHt/l/cdivrtrme0NRTtL2mhOafdQnEVVpFx/kT4LhyXlfeul2xmBrFLTuqVJnS\n7+/XhF5e3WMxjC4VQ0czq1tOHI8DlxJ+Zcxw9z3llJMEKLlnvqbAdmBX7ITUVYdhn88DuWY2xMzq\nEdpx29ZQjDOAH5lZu9jJtZ9WVNjdPyU0HfyJ0CSzMrbqCEI78CbgKzM7n9A2nGgMt5hZCwvXAYyP\nW9eEkOA2Eb7nxhBq7kU2Au3jT2yWMg34gZn1MLMjCF8+b7h7ub+EKlDRcZ4FdDCz8WZ2hJk1M7M+\nsXWPAr8ys+Ms6GlmrQhfap8STtzXNbOxxH0RVRDDLmC7mR1LaBoq8m9gC3CHhZPUDc2sX9z6vxCa\ncS4hJHo5BErume/HwGWEE5wPE0581ih33wh8D7iH8M96HPAuocaW7BgfAl4FlgDzCbXvyjxJaEMv\nbpJx923ADcCzhJOSwwlfUom4lfALYg3wInGJx90XAw8Ab8fKnAC8Fffa/wVWAhvNLL55pej1LxGa\nT56Nvb4DMCrBuEor9zi7+3bgLOAiwhfOCuD02Oq7gJmE47yDcHKzQay5bQxwC+Hk+jdKvbey3Ar0\nIXzJzAKejouhEDgf6Eyoxa8lfA5F69cQPuc97v5mFd+7lFJ08kKk2mI/s9cDw939jVTHI5nLzB4n\nnKS9LdWxZDpdxCTVYmaDCT1TdhO60u0j1F5FqiV2/mIY0D3VsUSBmmWkuvoDHxHams8BLtAJMKku\nM7uT0Nf+Dndfm+p4okDNMiIiEaSau4hIBKWszb1NmzaenZ2dqt2LiGSkhQsXbnb3iroeAylM7tnZ\n2SxYsCBVuxcRyUhmVtlV2oCaZUREIknJXUQkgpTcRUQiKK0uYtq3bx/5+fl8+eWXqQ5FKtCgQQPa\nt29PVlZ5w6WISKqlVXLPz8+nadOmZGdnEwYalHTj7mzZsoX8/Hw6depU+QtEJCUqbZYxs8fM7DMz\ne7+c9RabZmuVmS02s9zqBvPll1/SunVrJfY0Zma0bt1av64ko0ydCtnZUKdOuJ9apanmMzOORNrc\n/0QF81YSZrs5PnYbSxjFr9qU2NOfPiPJJFOnwtix8PHH4B7ux449/An+cMdRaXJ399cJQ6SWZxjw\nuAfzgBYWm+BXRGq3dKgxT5gABQUllxUUhOVRjiMZvWXaUXKqrXzKmZXHzMbGZlZfsGnTpiTsOrm2\nbNlCz5496dmzJ0cddRTt2rUrfr53796EtnH55ZfzwQcfVFhm8uTJTE3V70KRwyRdasxryxmGrLzl\nkYkjkYlWCbOuv1/OuueB/nHPXyU223xFt969e3tpy5YtO2hZRZ54wr1jR3ezcP/EE1V6eYVuvfVW\nv+uuuw5avn//fv/qq6+St6MMVdXPSmqfjh3dQ1oveevYUXEcShzAAj9ME2Svo+T8ku2p3vyPVXI4\nawWrVq2iS5cujBo1iq5du7JhwwbGjh1LXl4eXbt25fbbby8u279/f9577z0KCwtp0aIFN998Mzk5\nOZxyyil89tlnAEycOJH77ruvuPzNN99Mnz59OOGEE3jzzTABza5du7jooovo0qULw4cPJy8vj/fe\ne++g2G699VZOOukkunXrxtVXX130BcuKFSv41re+RU5ODrm5uaxZswaAO+64g+7du5OTk8OEw/27\nVGqVdKkxT5oEjRqVXNaoUVge6TgS+Qag4pr7twlTjxlwMvB2Its81Jp7TX8bx9fcV65c6Wbm8+fP\nL16/ZcsWd3fft2+f9+/f35cuXeru7v369fN3333X9+3b54DPnj3b3d1vuOEGv/POO93dfcKECX7v\nvfcWl7/pppvc3f25557zc845x93d77zzTr/mmmvc3f29997zOnXq+LvvvntQnEVx7N+/30eMGFG8\nv9zcXJ81a5a7u+/evdt37drls2bN8v79+3tBQUGJ11aHau5SmXSpMbvX7K/8wx0Hyaq5m9k0wsS2\nJ5hZvpn9wMyuNrOrY0VmEyZtWAU8AlyT1G+fchzuWsFxxx1HXl5e8fNp06aRm5tLbm4uy5cvZ9my\nZQe9pmHDhpx77rkA9O7du7j2XNqFF154UJm5c+cyYsQIAHJycujatWuZr3311Vfp06cPOTk5/POf\n/2Tp0qVs3bqVzZs3M2TIECBcdNSoUSNeeeUVrrjiCho2bAhAq1atqn4gRBKULjVmgFGjYM0a2L8/\n3I+q7iy1GRRHpRcxufvIStY7cG3SIkpQhw6hKaas5TWhcePGxY9XrlzJ7373O95++21atGjBpZde\nWma/7/r16xc/rlu3LoWFhWVu+4gjjqi0TFkKCgoYP34877zzDu3atWPixInqfy5poyhxTZgQKl0d\nOoTEnqrEWttk7NgyqawV7Nixg6ZNm9KsWTM2bNjAyy+/nPR99OvXjxkzZgCwZMmSMn8Z7N69mzp1\n6tCmTRu++OILnn46TDTfsmVL2rZty9/+9jcgXBxWUFDAWWedxWOPPcbu3bsB+Pzzinq4ihy6dKkx\n10ZpNfxAVaSyVpCbm0uXLl048cQT6dixI/369Uv6Pq677jr+4z/+gy5duhTfmjdvXqJM69atueyy\ny+jSpQtHH300ffv2LV43depUrrrqKiZMmED9+vV5+umnOf/881m0aBF5eXlkZWUxZMgQfvnLXyY9\ndhFJvZTNoZqXl+elJ+tYvnw5nTt3Tkk86aawsJDCwkIaNGjAypUrOfvss1m5ciX16qXH97E+K5HU\nMLOF7p5XWbn0yBRykJ07dzJo0CAKCwtxdx5++OG0Sewikv6ULdJUixYtWLhwYarDEJEMlbEnVEVE\npHxK7iIiEaTkLiISQUruIiIRpOQeZ+DAgQddkHTfffcxbty4Cl/XpEkTANavX8/w4cPLLHPGGWdQ\nuutnaffddx8FcQM+n3feeWzbti2R0EVESlByjzNy5EimT59eYtn06dMZObLCERiKHXPMMTz11FPV\n3n/p5D579mxatGhR7e2JSO2l5B5n+PDhvPDCC8UTc6xZs4b169czYMCA4n7nubm5dO/eneeee+6g\n169Zs4Zu3boBYWiAESNG0LlzZy644ILiS/4Bxo0bVzxc8K233grA/fffz/r16xk4cCADBw4EIDs7\nm82bNwNwzz330K1bN7p161Y8XPCaNWvo3LkzY8aMoWvXrpx99tkl9lPkb3/7G3379qVXr16ceeaZ\nbNy4EQh96S+//HK6d+9Ojx49iocveOmll8jNzSUnJ4dBgwYl5diKyOGVtv3cf/QjKGP48kPSsyfE\n8mKZWrVqRZ8+fXjxxRcZNmwY06dP5+KLL8bMaNCgAc8++yzNmjVj8+bNnHzyyQwdOrTc+UQfeugh\nGjVqxPLly1m8eDG5uQfmDZ80aRKtWrXiq6++YtCgQSxevJjrr7+ee+65hzlz5tCmTZsS21q4cCF/\n/OMfeeutt3B3+vbty+mnn07Lli1ZuXIl06ZN45FHHuHiiy/m6aef5tJLLy3x+v79+zNv3jzMjEcf\nfZTf/OY3/Pa3v+WXv/wlzZs3Z8mSJQBs3bqVTZs2MWbMGF5//XU6deqk8WdEMpRq7qXEN83EN8m4\nO7fccgs9evTgzDPPZN26dcU14LK8/vrrxUm2R48e9OjRo3jdjBkzyM3NpVevXixdurTMQcHizZ07\nlwsuuIDGjRvTpEkTLrzwQt544w0AOnXqRM+ePYHyhxXOz8/nnHPOoXv37tx1110sXboUgFdeeYVr\nrz0woGfLli2ZN28ep512Gp06dQI0LLBIpkrbmntFNeyaNGzYMG644QbeeecdCgoK6N27NxAG4tq0\naRMLFy4kKyuL7Ozsag2vu3r1au6++27mz59Py5YtGT169CEN01s0XDCEIYPLapa57rrruPHGGxk6\ndCj/+Mc/uO2226q9PxHJDKq5l9KkSRMGDhzIFVdcUeJE6vbt2znyyCPJyspizpw5fFzWYPJxTjvt\nNJ588kkA3n//fRYvXgyE4YIbN25M8+bN2bhxIy+++GLxa5o2bcoXX3xx0LYGDBjAzJkzKSgoYNeu\nXTz77LMMGDAg4fe0fft22rULc5b/+c9/Ll5+1llnMXny5OLnW7du5eSTT+b1119n9erVgIYFrqqp\nUyE7G+rUCfeaB11SRcm9DCNHjmTRokUlkvuoUaNYsGAB3bt35/HHH+fEE0+scBvjxo1j586ddO7c\nmV/84hfFvwBycnLo1asXJ554IpdcckmJ4YLHjh3L4MGDi0+oFsnNzWX06NH06dOHvn37cuWVV9Kr\nV6+E389tt93Gd7/7XXr37l2iPX/ixIls3bqVbt26kZOTw5w5c2jbti1TpkzhwgsvJCcnh+9973sJ\n76e2O5zz+opURkP+SrXoszpYdnbZs4N17BgmqhBJhkSH/FXNXSRJDve8viIVUXIXSZLy5u+tqXl9\nRSqSdsk9Vc1Ekjh9RmVL5by+IqWlVXJv0KABW7ZsUfJIY+7Oli1baNCgQapDSTujRsGUKaGN3Szc\nT5miSaElNdLqhOq+ffvIz88/pH7fUvMaNGhA+/btycrKSnUoIrVORs6hmpWVVXxlpIiIVF9aNcuI\niEhyKLmLiESQkruISAQpuYuIRJCSu4hIBCm5i4hEkJK7iEgEJZTczWywmX1gZqvM7OYy1nc0s1fN\nbLGZ/cPM2ic/VBERSVSlyd3M6gKTgXOBLsBIM+tSqtjdwOPu3gO4Hbgz2YGKiEjiEqm59wFWuftH\n7r4XmA4MK1WmC/Ba7PGcMtaL1DjNgiRyQCLJvR3wSdzz/NiyeIuAC2OPLwCamlnr0hsys7FmtsDM\nFmzatKk68YqUSbMgiZSUrBOqPwFON7N3gdOBdcBXpQu5+xR3z3P3vLZt2yZp1yIwYQIUFJRcVlAQ\nlovURokMHLYOODbuefvYsmLuvp5Yzd3MmgAXufu2ZAUpUhnNgiRSUiI19/nA8WbWyczqAyOAWfEF\nzKyNmRVt62fAY8kNU6RimgVJpKRKk7u7FwLjgZeB5cAMd19qZreb2dBYsTOAD8xsBfA1QHPPyGGl\nWZBESkqozd3dZ7v7N939OHefFFv2C3efFXv8lLsfHytzpbvvqcmgRUrTLEglqeeQpNVkHSKHYtSo\n2pvM4xX1HCo6wVzUcwh0fGoTDT8gEjHqOSSg5C4SOeo5JKDkLhI56jkkoOQuEjnqOSSg5C4SOeo5\nJKDeMiKRpJ5Dopq7iEgEKbmLiESQkruISAQpuYuIRJCSu4hIBCm5i4hEkJK7iEgEKbnLIdPwsiLp\nRxcxySHR8LIi6Uk1dzkkGl5WJD0pucsh0fCyIulJyV0OiYaXFUlPSu5ySDS8rEh6UnKXQ6LhZUXS\nk3rLyCHT8LIi6Uc1dxGRCFJyFxGJICV3EZEIUnIXEYkgJXcRkQhSchcRiSAldxGRCFJyFxGJoISS\nu5kNNrMPzGyVmd1cxvoOZjbHzN41s8Vmdl7yQxURkURVmtzNrC4wGTgX6AKMNLMupYpNBGa4ey9g\nBPD7ZAcqIiKJS6Tm3gdY5e4fufteYDowrFQZB5rFHjcH1icvRBERqapEkns74JO45/mxZfFuAy41\ns3xgNnBdWRsys7FmtsDMFmzatKka4YqISCKSdUJ1JPAnd28PnAf8xcwO2ra7T3H3PHfPa9u2bZJ2\nLSIipSWS3NcBx8Y9bx9bFu8HwAwAd/830ABok4wARUSk6hJJ7vOB482sk5nVJ5wwnVWqzFpgEICZ\ndSYkd7W7iIikSKXJ3d0LgfHAy8ByQq+YpWZ2u5kNjRX7MTDGzBYB04DR7u41FbSIiFQsock63H02\n4URp/LJfxD1eBvRLbmgiIlJdukJVRCSClNxFRCJIyV1EJIKU3EVEIkjJXUQkgpTcRUQiSMldRCSC\nlNxFRCJIyV1EJIKU3EVEIkjJXUQkgpTcRUQiSMldRCSClNxFRCJIyV1EJIKU3DPY1KmQnQ116oT7\nqVNTHZGIpIuEJuuQ9DN1KowdCwUF4fnHH4fnAKNGpS4uEUkPqrlnqAkTDiT2IgUFYbmIiJJ7hlq7\ntmrLRaR2UXLPUB06VG25iNQuSu4ZatIkaNSo5LJGjcJyEREl9ww1ahRMmQIdO4JZuJ8yRSdTRSRQ\nb5kMNmqUkrmIlE01dxGRCFJyFxGJICV3EZEIUnIXEYkgJfdq0JguIpLu1FumijSmi4hkAtXcq0hj\nuohIJlByryKN6SIimSCh5G5mg83sAzNbZWY3l7H+XjN7L3ZbYWbbkh9qetCYLiKSCSpN7mZWF5gM\nnAt0AUaaWZf4Mu5+g7v3dPeewAPAMzURbDrQmC4ikgkSqbn3AVa5+0fuvheYDgyroPxIYFoygktH\nGtNFRDJBIr1l2gGfxD3PB/qWVdDMOgKdgNfKWT8WGAvQIYPbMTSmi4iku2SfUB0BPOXuX5W10t2n\nuHueu+e1bds2ybsWEZEiiST3dcCxcc/bx5aVZQQRbpIREckUiST3+cDxZtbJzOoTEvis0oXM7ESg\nJfDv5IYoIiJVVWlyd/dCYDzwMrAcmOHuS83sdjMbGld0BDDd3b1mQhURkUQlNPyAu88GZpda9otS\nz29LXlgiInIodIWqiEgEKbmLiESQkruISAQpuYuIRJCSu4hIBCm5i4hEkJK7iEgEKbmLiESQkruI\nSAQpuYuIRJCSu4hIBCm5i4hEkJK7iEgEKbmLiESQkruISAQpuYuIRJCSu4hIBCm5i4hEkJK7iEgE\nKbmLiESQkruISAQpuYuIRJCSu4hIBCm5i4hEkJK7iEgEKbmLiESQkruISAQpuYuIRJCSu4hIBCm5\ni4hEUELJ3cwGm9kHZrbKzG4up8zFZrbMzJaa2ZPJDVNERKqiXmUFzKwuMBk4C8gH5pvZLHdfFlfm\neOBnQD9332pmR9ZUwCIiUrlEau59gFXu/pG77wWmA8NKlRkDTHb3rQDu/llywxQRkapIJLm3Az6J\ne54fWxbvm8A3zexfZjbPzAaXtSEzG2tmC8xswaZNm6oXsYiIVCpZJ1TrAccDZwAjgUfMrEXpQu4+\nxd3z3D2vbdu2Sdq1iIiUlkhyXwccG/e8fWxZvHxglrvvc/fVwApCshcRkRRIJLnPB443s05mVh8Y\nAcwqVWYmodaOmbUhNNN8lMQ4RUSkCipN7u5eCIwHXgaWAzPcfamZ3W5mQ2PFXga2mNkyYA7wn+6+\npaaCFhGRipm7p2THeXl5vmDBgpTsW0QkU5nZQnfPq6ycrlAVEYkgJXcRkQhSchcRiSAldxGRCKp0\nbBkRkUS4Q0EBbNkCmzeH+6Jb8+bQvz907AhmqY60dlByF0myf/8b7r0XsrKgaVNo1iyx+6ZNoV6a\n/Efu3w/btpWdqONvpdft2VPxdtu1C0l+wIBw360b1K17eN5TbZMmf0oi0fCnP8FVV4WaavPmsGMH\nfPEF7N6d2OsbNqzaF0IiXxR791ackMtavnVrSPBlqVsXWrWCNm2gdWv4+tfhpJPC49atDyyPv23c\nCHPnhtsbb8Bf/xq21awZnHpqSPT9+0OfPuEYyKFTP3eRJPjqK7jpJrjnHhg0CGbMCAmwyL59sHPn\ngWR/KPdffplYTA0bhkS8c2fFZSpKymUtb9780JtWPv74QLKfOxfefz8sz8qC3r0P1OxPPTXsXw5I\ntJ+7knuG++wz2LABWrYMyaRxY7VpHm7bt8OIEfDSS3DddfDb34YkVVP27QuJvuhW0RdBYWHFyTpd\nasmffx6as4pq9vPnh18cAJ07H6jZ9+8PnTrV7r9xJfeIc4fHH4drr4Vduw4sr1cvJPlWrQ4k/EQe\nt2yZPu29mWTlShgyBD78EH7/exgzJtURRcOXX8KCBQdq9v/6VzgHAHDMMSWTfY8etavdPtHkrn/n\nDLRtG4wbB9Onw2mnwfjxofa4dWuoARXdf/55qNUvXRoe79hR8XabNq38C6Cs9bX118L//i9cfHH4\nUnz11fBZSHI0aHAgeUNo/1+2LNTqixL+jBlhXdOmcMopB8r37QuNGqUu9nShmnuGefNNuOQSyM+H\n//ovuPnmxGsthYXhiyE++Vf2uOi+6CdyWerVCz/xL70Ubr89+v9Y7vDAA3DjjdClCzz3XGgqkMNr\n7dpQoy9K9kuWhM+mXr3Qbl+U7Pv1gyhNH6FmmYgpLIQ77gjJs0MHePJJOPnkw7Pvov7LFSX/lSvh\nqafguOPgv/8bTj/98MR2uO3dG5rCHn0Uhg2Dv/wl1Bwl9bZtC5WfomT/9tsHumaecEJI9N27hy/i\nTp0gOzszPzsl9whZuxZGjQp/sJdcEtp2mzdPdVQHmzMntDl/+CFcfTX8+tehq1tUbNoEF10UmgYm\nTAhftHV0jXfa2rMHFi4s2Stn69aSZVq3PpDo4+87dQoXXKXLCed4Su4R8T//A2PHhpr7Qw+Fpo90\nVlAAP/853HdfOPH18MNw3nmpjurQLV4MQ4eG/tp//GPoHSOZxT18Qa9ZA6tXH7gverxmzcHNj0cd\nVTLhxz8+9lioX/9wvwsl94y3axdcfz089li4sOPJJ0OTR6Z46y244opwEuz73w9XbLZuneqoqufZ\nZ8N7aNECZs6EvEr/rSQT7d8Pn35aMuHH369dG65nKFKnTrjitqzkn50N7dvXTC8eJfcM9s47MHJk\naMf+2c/gtttqtt90TdmzByZNgjvvDL1qJk+G4cNTHVXi3EP8P/95+IKdOROOPjrVUUmqFBbCunVl\n1/pXrw7r4tNpvXrh/FhZyb9z59DjrDqU3DPQ/v3hCsdbboEjj4QnnoAzzkh1VIdu0SL4wQ9C++eF\nF8KDD6Z/kiwoCL88/vrX0BT2yCOhe55IefbsgU8+KT/5b9x4oOyDD4YT89Whfu4ZZsMGuOyy0Hf6\nO98JvTEytRmjtJwcmDcvXLl5663w2muhmeayy9Kzf3x+fvgM3nknnBT+z/9MzzglvRxxBHzjG+FW\nloKCMOzC6tWhC21N07n+NPD88+Equ7lzwwnIZ56JTmIvUq8e/PSnoRbfrRtcfjmce274Y08n8+aF\nNvUVK2DWrDBejBK7JEOjRqE55rzzQvNMTVNyT6EvvwwnTYcMCT1LFiwIPWOinExOOAH++c9wEdDc\nuSHRT55c/giEh9Pjj4f++Y0bh3FOzj8/1RGJVJ+Se4osXRpO0j3wAPzoR6F3yeH4qZYO6tQJQya8\n/364bHz8+HBuYcWK1MRTNKLjZZeFqxnffhu6dk1NLCLJouR+mLmH/up5eeEEy+zZof25Np6sy86G\nl18O3T2XLAlt87/5TeiVcLhs3x76r991VzjB9fLL0WsSk9pJyf0w2rw5nKi75prw83/x4tDuXJuZ\nhfb3ZctF5CGNAAAG50lEQVRg8ODQLn/KKeHY1LRVq8K+/v738IX74IOZ2eVUpCxK7ofJa6+FmulL\nL4Wa+uzZ8LWvpTqq9HH00eFE8owZ4SRr796hZ01FA5YdildfDc1in30WeihdfXXN7EckVZTca9je\nvWHkxjPPDIMUzZsX2tg1JsnBzOC73w21+BEjwtgtubmhDTxZ3EMN/Zxzwknst9+OxrUEIqUpxdSg\nlSvDCbpf/zoMqLVwIfTqleqo0l+bNmG0xeefD23ip5wCP/lJ6Cd8KPbuDTX0664L3dHefDPM/ykS\nRUruNcAd/vznkMg//DAMhfvww6GLnSTu298OvYrGjAkXQOXkhG6U1bF5M5x9NkyZEoZ0mDkzWiNW\nipSm5J5k27aFYXlHjw49YhYtCsPESvU0awZ/+EM4Z+EemlDGjat8Vql4S5bASSeFJrEnngjj4qtZ\nTKIuo/7Ep04N3dTMQtfBU08N7dkPPQQvvBD6TVflnz7Z3nwTevYMw/T+6lfhpN2xx6YunigZODD0\noLnxxlD77tYNXnyx8tc991z4O9mzJ4zDPmpUzccqkg4yZuCwqVPD1Zvx7a5moQYWPwwnhKFZO3QI\ng+2XdX/UUcmtuaVylqTaKJHhhN3DaJQTJ4ZfUDNnhhOoIpkuqaNCmtlg4HdAXeBRd/9/pdaPBu4C\n1sUWPejuj1a0zaom9+zsssch6dAhXCq+dm1YX9Z90azpRbKyQo26Y8eyvwCOPTbxi4riZ0kaNSrM\nkqS23JpX0XDCu3eHUSinTQtNZI8+mp4z6ohUR9KSu5nVBVYAZwH5wHxgpLsviyszGshz9/GJBljV\n5F6nTsmxkg/su/JxSXbsKD/5f/wxrF9/8La/9rWKa/+tWoUTpWPHhl8Ov/99+s+SFEWlhxO+5ZbQ\nI2bhwvBr6qc/jfZYPVL7JHPI3z7AKnf/KLbh6cAwYFmFr0qyDh3Kr7lXplmz0EbbrVvZ6/ftC8O8\nlvUFsGRJaM/fvbvkaxo1Ck1EmThLUpSUHk74mWegSZPQ1j5kSKqjE0mdRJJ7O+CTuOf5QN8yyl1k\nZqcRavk3uPsnpQuY2VhgLECHRLJynEmTDm5zb9QoLD9UWVkHZkopi3voSlc6+bdrBz/8oS5ZT7Wi\n4YS/8x24//7Qm6a8L3KR2iKRZpnhwGB3vzL2/PtA3/gmGDNrDex09z1mdhXwPXf/VkXbrc5MTFOn\nhlnn164NNfZJk9T7QURql2Q2y6wD4jv0tefAiVMA3H1L3NNHgd8kEmRVjRqlZC4ikohEOgTOB443\ns05mVh8YAcyKL2Bm8TNiDgWWJy9EERGpqkpr7u5eaGbjgZcJXSEfc/elZnY7sMDdZwHXm9lQoBD4\nHBhdgzGLiEglMuYiJhERSbzNPaOGHxARkcQouYuIRJCSu4hIBCm5i4hEUMpOqJrZJqCMAQUS0gbY\nnMRwMp2OR0k6HgfoWJQUhePR0d3bVlYoZcn9UJjZgkTOFtcWOh4l6XgcoGNRUm06HmqWERGJICV3\nEZEIytTkPiXVAaQZHY+SdDwO0LEoqdYcj4xscxcRkYplas1dREQqoOQuIhJBGZfczWywmX1gZqvM\n7OZUx5MqZnasmc0xs2VmttTMfpjqmNKBmdU1s3fN7PlUx5JqZtbCzJ4ys/8zs+VmdkqqY0oVM7sh\n9n/yvplNM7MGqY6ppmVUco9N1j0ZOBfoAow0sy6pjSplCoEfu3sX4GTg2lp8LOL9EM0nUOR3wEvu\nfiKQQy09LmbWDrgeyHP3boShy0ekNqqal1HJnbjJut19L1A0WXet4+4b3P2d2OMvCP+47VIbVWqZ\nWXvg24TZwGo1M2sOnAb8N4C773X3bamNKqXqAQ3NrB7QCFif4nhqXKYl97Im667VCQ3AzLKBXsBb\nqY0k5e4DbgL2pzqQNNAJ2AT8MdZM9aiZNU51UKng7uuAu4G1wAZgu7v/PbVR1bxMS+5Sipk1AZ4G\nfuTuO1IdT6qY2fnAZ+6+MNWxpIl6QC7wkLv3AnYBtfIclZm1JPzC7wQcAzQ2s0tTG1XNy7TkXulk\n3bWJmWUREvtUd38m1fGkWD9gqJmtITTXfcvMnkhtSCmVD+S7e9GvuacIyb42OhNY7e6b3H0f8Axw\naopjqnGZltwrnay7tjAzI7SnLnf3e1IdT6q5+8/cvb27ZxP+Ll5z98jXzsrj7p8Cn5jZCbFFg4Bl\nKQwpldYCJ5tZo9j/zSBqwcnlSifITiflTdad4rBSpR/wfWCJmb0XW3aLu89OYUySXq4DpsYqQh8B\nl6c4npRw97fM7CngHUIvs3epBcMQaPgBEZEIyrRmGRERSYCSu4hIBCm5i4hEkJK7iEgEKbmLiESQ\nkruISAQpuYuIRND/B8sHA/B6JcgEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7ff6bd49d748>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEICAYAAABYoZ8gAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUVNW1x/HvZpZBZgdAAYcnNMhkBzRIEDUEo8JDiQEB\n0WhQFI0a84JoosGQoDGOMUZicAIlPhWDIxoljxgVaQhCBAmogA1EAQGZRJre749zG6qhh+ru6r5F\n1++zVq2quuOu6q59zz333HPM3RERkcxRI+4ARESkainxi4hkGCV+EZEMo8QvIpJhlPhFRDKMEr+I\nSIZR4pcyM7OaZrbNzI5O5bJxMrPjzCzlbZvN7EwzW5nwfpmZ9Ulm2XLs62EzG1/e9UvY7i/N7NFU\nb1fiUyvuAKTymdm2hLf1gV3Anuj95e4+rSzbc/c9QMNUL5sJ3P2EVGzHzC4DRrj7aQnbviwV25bq\nT4k/A7j73sQblSgvc/e/Fre8mdVy97yqiE1Eqp6qeqTgVP7PZvaUmW0FRpjZKWb2rpltNrN1Znaf\nmdWOlq9lZm5m7aL3U6P5r5jZVjN7x8zal3XZaP5ZZvZvM9tiZveb2T/M7OJi4k4mxsvNbIWZbTKz\n+xLWrWlmd5vZRjP7GBhQwvdzk5lN32/aA2Z2V/T6MjNbGn2ej6LSeHHbyjWz06LX9c3siSi2D4CT\n9lv2ZjP7ONruB2Y2MJp+IvA7oE9UjbYh4bu9NWH9K6LPvtHMnjezI5P5bkpjZoOjeDab2ZtmdkLC\nvPFmttbMvjSzDxM+68lmtiCa/pmZ/SbZ/UklcHc9MugBrATO3G/aL4GvgXMJhYFDgG8AvQhnhccA\n/wbGRsvXAhxoF72fCmwAsoHawJ+BqeVY9jBgKzAomnc9sBu4uJjPkkyMfwEaA+2ALwo+OzAW+ABo\nAzQH5oSfQ5H7OQbYBjRI2PbnQHb0/txoGQNOB3YCXaJ5ZwIrE7aVC5wWvb4T+BvQFGgLLNlv2QuA\nI6O/yYVRDIdH8y4D/rZfnFOBW6PX/aMYuwH1gN8Dbybz3RTx+X8JPBq97hjFcXr0NxoPLItedwJW\nAUdEy7YHjolezwOGRa8bAb3i/i1k8kMlfinwlru/4O757r7T3ee5+1x3z3P3j4HJQN8S1n/G3XPc\nfTcwjZBwyrrsOcBCd/9LNO9uwkGiSEnG+Gt33+LuKwlJtmBfFwB3u3uuu28EJpWwn4+BfxEOSADf\nBja5e040/wV3/9iDN4E3gCIv4O7nAuCX7r7J3VcRSvGJ+33a3ddFf5MnCQft7CS2CzAceNjdF7r7\nV8A4oK+ZtUlYprjvpiRDgZnu/mb0N5pEOHj0AvIIB5lOUXXhJ9F3B+EAfryZNXf3re4+N8nPIZVA\niV8KfJr4xsw6mNlLZvYfM/sSmAC0KGH9/yS83kHJF3SLW7ZVYhzu7oQScpGSjDGpfRFKqiV5EhgW\nvb4wel8QxzlmNtfMvjCzzYTSdknfVYEjS4rBzC42s/ejKpXNQIcktwvh8+3dnrt/CWwCWicsU5a/\nWXHbzSf8jVq7+zLgx4S/w+dR1eER0aKXAFnAMjN7z8y+m+TnkEqgxC8F9m/K+BChlHucux8K/JxQ\nlVGZ1hGqXgAwM6NwotpfRWJcBxyV8L605qZPA2eaWWtCyf/JKMZDgGeAXxOqYZoAryUZx3+Ki8HM\njgEeBMYAzaPtfpiw3dKanq4lVB8VbK8RoUppTRJxlWW7NQh/szUA7j7V3XsTqnlqEr4X3H2Zuw8l\nVOf9FnjWzOpVMBYpJyV+KU4jYAuw3cw6ApdXwT5fBHqY2blmVgv4EdCykmJ8GrjWzFqbWXPgpyUt\n7O7/Ad4CHgWWufvyaFZdoA6wHthjZucAZ5QhhvFm1sTCfQ5jE+Y1JCT39YRj4A8JJf4CnwFtCi5m\nF+Ep4FIz62JmdQkJ+O/uXuwZVBliHmhmp0X7/gnhusxcM+toZv2i/e2MHvmEDzDSzFpEZwhbos+W\nX8FYpJyU+KU4PwZGEX7UDxEuwlYqd/8M+D5wF7AROBb4J+G+g1TH+CChLn4x4cLjM0ms8yThYu3e\nah533wxcB8wgXCAdQjiAJeMWwpnHSuAV4PGE7S4C7gfei5Y5AUisF38dWA58ZmaJVTYF679KqHKZ\nEa1/NKHev0Lc/QPCd/4g4aA0ABgY1ffXBe4gXJf5D+EM46Zo1e8CSy20GrsT+L67f13ReKR8LFSj\niqQfM6tJqFoY4u5/jzsekepCJX5JK2Y2IKr6qAv8jNAa5L2YwxKpVkpN/GZ2lJnNNrMl0U0bPypi\nGbNw88wKM1tkZj0S5o0ys+XRY1SqP4BUO6cCHxOqEb4DDHb34qp6RKQcSq3qie72O9LdF0QtA+YD\n/+3uSxKW+S5wNaEerxdwr7v3MrNmQA6h7bFH657k7psq5dOIiEipSi3xRzeQLIhebwWWcmATu0HA\n49ENLO8CTaIDxneA1939iyjZv04Jt8aLiEjlK1MnbRb6W+lO4dYFEA4EiTei5EbTipteohYtWni7\ndu3KEpqISEabP3/+BncvqfnzXkknfjNrCDwLXBvdBZhSZjYaGA1w9NFHk5OTk+pdiIhUW2ZW2t3n\neyXVqie6UeNZYJq7P1fEImsofAdiwZ18xU0/gLtPdvdsd89u2TKpg5aIiJRDMq16DPgTsNTd7ypm\nsZnARVHrnpOBLe6+DpgF9DezpmbWlNCHyawUxS4iIuWQTFVPb2AksNjMFkbTxhP1K+LufwBeJrTo\nWUHo7OmSaN4XZnYb4c5IgAnu/kXqwhcRkbIqNfG7+1uU0uFU1IviVcXMmwJMKVd0IlIldu/eTW5u\nLl999VXcoUgp6tWrR5s2bahdu7humkqnoRdFhNzcXBo1akS7du0ItbuSjtydjRs3kpubS/v27Utf\noRjVpsuGadOgXTuoUSM8TyvT8OEime2rr76iefPmSvppzsxo3rx5hc/MqkWJf9o0GD0aduwI71et\nCu8Bhle4P0KRzKCkf3BIxd+pWpT4b7ppX9IvsGNHmC4iIoVVi8S/enXZpotI+ti4cSPdunWjW7du\nHHHEEbRu3Xrv+6+/Tq7L/ksuuYRly5aVuMwDDzzAtBTVAZ966qksXLiw9AXTVLWo6jn66FC9U9R0\nEUm9adPCGfXq1eF3NnFi+atVmzdvvjeJ3nrrrTRs2JAbbrih0DLujrtTo0bRZdVHHnmk1P1cdVWR\nDQ8zUrUo8U+cCPXrF55Wv36YLiKpVXBNbdUqcN93TS3VDSpWrFhBVlYWw4cPp1OnTqxbt47Ro0eT\nnZ1Np06dmDBhwt5lC0rgeXl5NGnShHHjxtG1a1dOOeUUPv/8cwBuvvlm7rnnnr3Ljxs3jp49e3LC\nCSfw9ttvA7B9+3bOP/98srKyGDJkCNnZ2aWW7KdOncqJJ55I586dGT9+PAB5eXmMHDly7/T77rsP\ngLvvvpusrCy6dOnCiBEjUvuFlUG1KPEXlDRSVQIRkeKVdE0t1b+5Dz/8kMcff5zs7GwAJk2aRLNm\nzcjLy6Nfv34MGTKErKysQuts2bKFvn37MmnSJK6//nqmTJnCuHHjDti2u/Pee+8xc+ZMJkyYwKuv\nvsr999/PEUccwbPPPsv7779Pjx49DlgvUW5uLjfffDM5OTk0btyYM888kxdffJGWLVuyYcMGFi9e\nDMDmzZsBuOOOO1i1ahV16tTZOy0O1aLED+EfbuVKyM8Pz0r6IpWjKq+pHXvssXuTPsBTTz1Fjx49\n6NGjB0uXLmXJkiUHrHPIIYdw1llnAXDSSSexcuXKIrd93nnnHbDMW2+9xdChQwHo2rUrnTp1KjG+\nuXPncvrpp9OiRQtq167NhRdeyJw5czjuuONYtmwZ11xzDbNmzaJx48YAdOrUiREjRjBt2rQK3YBV\nUdUm8YtI1Sju2lllXFNr0KDB3tfLly/n3nvv5c0332TRokUMGDCgyPbsderU2fu6Zs2a5OXlFbnt\nunXrlrpMeTVv3pxFixbRp08fHnjgAS6//HIAZs2axRVXXMG8efPo2bMne/bsSel+k6XELyJlEtc1\ntS+//JJGjRpx6KGHsm7dOmbNSn1/j7179+bpp58GYPHixUWeUSTq1asXs2fPZuPGjeTl5TF9+nT6\n9u3L+vXrcXe+973vMWHCBBYsWMCePXvIzc3l9NNP54477mDDhg3s2L/OrIpUizp+Eak6cV1T69Gj\nB1lZWXTo0IG2bdvSu3fvlO/j6quv5qKLLiIrK2vvo6Capiht2rThtttu47TTTsPdOffcczn77LNZ\nsGABl156Ke6OmXH77beTl5fHhRdeyNatW8nPz+eGG26gUaNGKf8MySh1zN04ZGdnuwZiEak6S5cu\npWPHjnGHEbu8vDzy8vKoV68ey5cvp3///ixfvpxatdKrjFzU38vM5rt7djGrFJJen0ZEJEbbtm3j\njDPOIC8vD3fnoYceSruknwrV7xOJiJRTkyZNmD9/ftxhVDpd3BURyTBK/CIiGUaJX0Qkw5Rax29m\nU4BzgM/dvXMR838CFDTkqgV0BFpG4+2uBLYCe4C8ZK84i4hI5UmmxP8oMKC4me7+G3fv5u7dgBuB\n/9tvQPV+0XwlfREpUr9+/Q64Ieuee+5hzJgxJa7XsGFDANauXcuQIUOKXOa0006jtObh99xzT6Gb\nqb773e+mpC+dW2+9lTvvvLPC20m1UhO/u88Bvihtucgw4KkKRSQiGWfYsGFMnz690LTp06czbNiw\npNZv1aoVzzzzTLn3v3/if/nll2nSpEm5t5fuUlbHb2b1CWcGzyZMduA1M5tvZqNLWX+0meWYWc76\n9etTFZaIHASGDBnCSy+9tHfglZUrV7J27Vr69Omzt219jx49OPHEE/nLX/5ywPorV66kc+dQE71z\n506GDh1Kx44dGTx4MDt37ty73JgxY/Z263zLLbcAcN9997F27Vr69etHv379AGjXrh0bNmwA4K67\n7qJz58507tx5b7fOK1eupGPHjvzwhz+kU6dO9O/fv9B+irJw4UJOPvlkunTpwuDBg9m0adPe/Rd0\n1VzQQdz//d//7R2Mpnv37mzdurXc321RUtmO/1zgH/tV85zq7mvM7DDgdTP7MDqDOIC7TwYmQ7hz\nN4VxiUgZXHstpHpwqW7dIMqZRWrWrBk9e/bklVdeYdCgQUyfPp0LLrgAM6NevXrMmDGDQw89lA0b\nNnDyySczcODAYseeffDBB6lfvz5Lly5l0aJFhbpWnjhxIs2aNWPPnj2cccYZLFq0iGuuuYa77rqL\n2bNn06JFi0Lbmj9/Po888ghz587F3enVqxd9+/aladOmLF++nKeeeoo//vGPXHDBBTz77LMl9rF/\n0UUXcf/999O3b19+/vOf84tf/IJ77rmHSZMm8cknn1C3bt291Ut33nknDzzwAL1792bbtm3Uq1ev\nDN926VLZqmco+1XzuPua6PlzYAbQM4X7E5FqJLG6J7Gax90ZP348Xbp04cwzz2TNmjV89tlnxW5n\nzpw5exNwly5d6NKly955Tz/9ND169KB79+588MEHpXbC9tZbbzF48GAaNGhAw4YNOe+88/j73/8O\nQPv27enWrRtQcvfPEMYI2Lx5M3379gVg1KhRzJkzZ2+Mw4cPZ+rUqXvvEu7duzfXX3899913H5s3\nb0753cMp2ZqZNQb6AiMSpjUAarj71uh1f2BCMZsQkTRRUsm8Mg0aNIjrrruOBQsWsGPHDk466SQA\npk2bxvr165k/fz61a9emXbt2RXbHXJpPPvmEO++8k3nz5tG0aVMuvvjicm2nQEG3zhC6di6tqqc4\nL730EnPmzOGFF15g4sSJLF68mHHjxnH22Wfz8ssv07t3b2bNmkWHDh3KHev+Si3xm9lTwDvACWaW\na2aXmtkVZnZFwmKDgdfcfXvCtMOBt8zsfeA94CV3fzVlkYtItdKwYUP69evHD37wg0IXdbds2cJh\nhx1G7dq1mT17NquKGmA7wbe+9S2efPJJAP71r3+xaNEiIHTr3KBBAxo3bsxnn33GK6+8snedRo0a\nFVmP3qdPH55//nl27NjB9u3bmTFjBn369CnzZ2vcuDFNmzbde7bwxBNP0LdvX/Lz8/n000/p168f\nt99+O1u2bGHbtm189NFHnHjiifz0pz/lG9/4Bh9++GGZ91mSUkv87l7qZXV3f5TQ7DNx2sdA1/IG\nJiKZZ9iwYQwePLhQC5/hw4dz7rnncuKJJ5KdnV1qyXfMmDFccskldOzYkY4dO+49c+jatSvdu3en\nQ4cOHHXUUYW6dR49ejQDBgygVatWzJ49e+/0Hj16cPHFF9OzZ6ilvuyyy+jevXuJ1TrFeeyxx7ji\niivYsWMHxxxzDI888gh79uxhxIgRbNmyBXfnmmuuoUmTJvzsZz9j9uzZ1KhRg06dOu0dUSxV1C2z\niKhb5oNMRbtlVpcNIiIZRolfRCTDKPGLCBCaTUr6S8XfSYlfRKhXrx4bN25U8k9z7s7GjRsrfEOX\nRuASEdq0aUNubi7qLiX91atXjzZt2lRoG0r8IkLt2rVp37593GFIFVFVj4hIhlHiFxHJMEr8IiIZ\nRolfRCTDKPGLiGQYJX4RkQyjxC8ikmGU+EVEMowSv4hIhlHiFxHJMMkMvTjFzD43s38VM/80M9ti\nZgujx88T5g0ws2VmtsLMxqUycBERKZ9kSvyPAgNKWebv7t4tekwAMLOawAPAWUAWMMzMsioSrIiI\nVFypid/d5wBflGPbPYEV7v6xu38NTAcGlWM7IiKSQqmq4z/FzN43s1fMrFM0rTXwacIyudG0IpnZ\naDPLMbMcdQ0rIlJ5UpH4FwBt3b0rcD/wfHk24u6T3T3b3bNbtmyZgrBERKQoFU787v6lu2+LXr8M\n1DazFsAa4KiERdtE00REJEYVTvxmdoSZWfS6Z7TNjcA84Hgza29mdYChwMyK7k9ERCqm1BG4zOwp\n4DSghZnlArcAtQHc/Q/AEGCMmeUBO4GhHgbuzDOzscAsoCYwxd0/qJRPISIiSbN0HFw5Ozvbc3Jy\n4g5DROSgYWbz3T07mWV1566ISIZR4hcRyTBK/CIiGUaJX0Qkwyjxi4hkGCV+EZEMo8QvIpJhlPhF\nRDKMEr+ISIZR4hcRyTBK/CIiGUaJX0Qkwyjxi4hkGCV+EZEMo8QvIpJhlPhFRDKMEr+ISIYpNfGb\n2RQz+9zM/lXM/OFmtsjMFpvZ22bWNWHeymj6QjPTkFoiImkgmRL/o8CAEuZ/AvR19xOB24DJ+83v\n5+7dkh0STEREKlepid/d5wBflDD/bXffFL19F2iTothEUub66+G88yA/P+5IROKX6jr+S4FXEt47\n8JqZzTez0SWtaGajzSzHzHLWr1+f4rAkk739Ntx9N8yYAY88Enc0IvEzdy99IbN2wIvu3rmEZfoB\nvwdOdfeN0bTW7r7GzA4DXgeujs4gSpSdne05ObokIBWXnw+9esG6ddC2LXz4ISxbBi1axB2ZSGqZ\n2fxkq9RTUuI3sy7Aw8CggqQP4O5roufPgRlAz1TsTyRZjz0GOTlwxx3w0EOwZQvceGPcUYnEq8KJ\n38yOBp4DRrr7vxOmNzCzRgWvgf5AkS2DRCrDl1+GJP/Nb8KwYdC5M1x3HTz8MLzzTtzRicSnVmkL\nmNlTwGlACzPLBW4BagO4+x+AnwPNgd+bGUBedLpxODAjmlYLeNLdX62EzyBSpIkT4bPP4MUXIfwb\nwi23wPTpMGZMOBOoVeovQKT6SaqOv6qpjl8qavly6NQJRoyAKVMKz3vuOTj//HDB99pr44lPJNWq\nvI5fJN3ccAPUrQu/+tWB8wYPhrPOgp/9DNasqfrYROKmxC/VzmuvwcyZIbEfccSB883g/vshLy+0\n7xfJNEr8Uq3s3h2qb447Dn70o+KXO/ZYGD8enn46HChEMokSv1QrDz4IS5fCXXeFqp6S/OQncPzx\ncNVV8NVXVROfSDpQ4pdqY8OG0Gqnf38455zSl69XD373O1ixAn7zm8qPTyRdKPFLtfHzn8PWraG1\nTkHzzdL07w8XXBCafn70UeXGJ5IulPilWli0KNyZe9VVkJVVtnXvvhvq1IGxYyENWzeLpJwSvxz0\n3MMF3aZN4dZby75+q1YwYQK8+mroyE2kulPil4Pec8/B7Nlw220h+ZfH2LHQtWtoCbRtW2rjE0k3\nSvxyUNu5M9ys1aULjC6x4++S1aoVWgTl5sIvfpG6+ETSkRK/HNTuugtWroR77oGaNSu2rVNOgcsu\nC3X+/1J3glKNKfHLQWvNmtAlw/nnQ79+qdnmpEnQpEnoxE0XeqW6UuKXg9a4cbBnT2rb4DdvDrff\nDm+9FfryF6mOlPjloPTOOzB1aqjfb98+tdu+5JLQh/9PfgJfFDvatMjBS4lfDjr5+aH1TatWodSf\najVqhAu9mzaF/nxEqhslfjnoPP44zJsXqmQaNqycfXTpAtdcA5Mnw9y5lbMPkbhoIBY5qHz5JfzX\nf8Exx8A//pF81wzlsXUrdOgAhx8O772n0bokvWkgFqm2fvWrMJzivfdWbtIHaNQoNBP95z9D1Y9I\ndZFU4jezKWb2uZkV2brZgvvMbIWZLTKzHgnzRpnZ8ugxKlWBS+ZZsSK0sb/4YvjGN6pmn0OGhI7c\nbr4Z1q2rmn2KVLZkS/yPAgNKmH8WcHz0GA08CGBmzQiDs/cCegK3mFk5b6qXTHfDDaEztaKGU6ws\nZqHr5l274Mc/rrr9ilSmpBK/u88BSmrYNgh43IN3gSZmdiTwHeB1d//C3TcBr1PyAUSkSK+/Dn/5\nSyh5H3lk1e77+OND66GnnoI33qjafYtUhlTV8bcGPk14nxtNK276AcxstJnlmFnO+vXrUxSWVAcF\nwykee2x4jsNPfxr2f9VVofQvcjBLm4u77j7Z3bPdPbtly5ZxhyNp5A9/gCVL4Le/LX04xcpyyCFh\ngPZly+DOO+OJQSRVUpX41wBHJbxvE00rbrpIUjZsCCNrffvbMHBgvLGcdVboF+iXv4RPPok3FpGK\nSFXinwlcFLXuORnY4u7rgFlAfzNrGl3U7R9NE0nKLbeUfTjFylTQC+jVV6sTNzl4Jduc8yngHeAE\nM8s1s0vN7AozuyJa5GXgY2AF8EfgSgB3/wK4DZgXPSZE00RKtXhxqOa58kro1CnuaII2bUJ//S+9\nBDNnxh2NSPnozl1JS+5wxhnw/vuwfDk0axZ3RPvs3g09eoS7iJcsgQYN4o5Iqov8/NBXVHnozl05\n6M2YsW84xXRK+gC1a4c7eVevDvGJlMfOnaErkAcfhB/+EE46CU48sWr2rd5HJO189VW4Wapz54oN\np1iZTj01dN/829/CRRdBVlbcEUk627YtnL0uWADz54fnJUvCeBIQCjc9eoTk717517OU+CXtFAyn\n+MYb6d0x2u23w/PPh2sQs2enx8Vnid+WLaF/p8Qkv2zZvsYAhx0WEvzAgeG5Rw84+uiq/f9J45+V\nZKKC4RTPOw9OPz3uaErWsmUYqvHyy8OgMCNHxh2RVLUNGw5M8h99tG9+mzYhsQ8dui/JH3lk/IUE\nXdyVtHLRRfD00+E0+Jhj4o6mdPn5YbSuTz6BDz+EpuqJqtr6z39CYk9M8qtX75vfvn1I7AVVNt27\nh9J9VSnLxV2V+CVtvPsuPPFEGPXqYEj6sG+0ruzs0I/QAw/EHZFUlDvk5h6Y5BN7Z/2v/woH/LFj\n9yX5g+mgrxK/pIX8fDj55PCD+/e/K29krcryox+FLh3mzq26LqMlNb78El57bV+iX7AACroLq1ED\nOnYsXJLv2hUOPTTemIuiEr8cdJ54Igyn+PjjB1/SB5gwIVRRjRkTkn/NmnFHJKWZPx8eegiefBK2\nbw8NCTp3hnPP3Vcf36UL1K8fd6Spp8Qvsdu6NXR73KsXDB8edzTl07hx6FZi2LCQTK68Mu6IpCjb\nt4futR96CHJyQud7Q4fCD34QztTi6gSwqukGLondr34VLpzde2/571pMB9//frjbePz48HkkfSxa\nFLrUbtUq3Cy1c2eomlu7FqZMCfdlZErSByV+idlHH4V2+6NGhRL/wcwsXNzduRN+8pO4o5GdO+Gx\nx8JF2K5d4U9/Cm3n33or9AM1diw0aRJ3lPFQ4q8EW7eq58ZkFQyn+Otfxx1JapxwAvzP/4R2/X/7\nW9zRZKalS8OAPa1ahfGZN24MhYs1a8K1pN69429HHzcl/hTZvBkmT4Y+fcIV/65dw1itmzfHHVn6\n+utfw52vN91U9cMpVqbx40Ob7iuvhK+/jjuazLBrV7hI27dv6D7j97+HAQPCHdUffgjXXQfNm8cd\nZfpQ4q+A3bvhhRfge9+DI44Id3Bu3BhKfHXqhD7bC0od77yjs4BEeXmhCeQxx8Q3nGJlOeQQuO++\nUPK86664o6neli8P1WqtW4eGAWvWhK40cnPDRdzTTlPpvkjunnaPk046ydNVfr77e++5jx3r3qKF\nO7i3bOl+zTXu8+aF+QVyctwvv9y9YcOwXOfO7vfd5/7FF/HFny7uvz98JzNmxB1J5fnv/3Y/5BD3\nlSvjjqR62bXL/c9/dj/99PA/VLOm+/nnu7/2mvuePXFHFx8gx5PMsbEn+aIe6Zj4V650nzjR/YQT\nwrdWt677BRe4v/CC+9dfl7zu1q3ukye7Z2eHdevVcx81yv0f/yh8oMgUGza4N23qfsYZ1fvzr1rl\nXr+++6BBcUdSPXz0kfu4ce6HHRZ+R23buv/yl+5r18YdWXpQ4k+RLVvc//Qn9759wzcF7t/6lvsf\n/+i+aVP5tjl/vvsVV7g3ahS216mT+733ZtZZwFVXhVLa4sVxR1L5br89/J1nzow7koPT7t3uzz3n\n/p3vuJu516jhPnCg+0svueflxR1dekl54gcGAMsIQyuOK2L+3cDC6PFvYHPCvD0J82Yms784E//u\n3eGfaujQUDIH9+OPd7/tNvePP07dfrZuDQeQb3xj31nARRe5v/VW9S4FL1oUfrxjx8YdSdXYtcs9\nKyuUTrdvjzuag8eqVe4/+5l7q1bh99G6tfstt7ivXh13ZOkrpYkfqAl8BBwD1AHeB7JKWP5qYErC\n+23JBlMVe9NmAAANw0lEQVTwqOrEn58fSuLXXut++OHhW2nWzP3KK93ffbfyE/GCBe5jxlT/s4D8\n/FAv26yZ+8aNcUdTdf72t/B3HT8+7kjSW15eqDo955xQODBzP+ss9+efDwUyKVmqE/8pwKyE9zcC\nN5aw/NvAtxPep23i//RT90mTQqIF9zp13M87L/yj7dpVvm1OnRpKd2bheerU5NfdutX94Yfde/bc\ndxYwcqT73/9ePc4CnnsufK7f/S7uSKreRRe5167tvnRp3JGknzVr3CdMcD/qqPD/cfjh4SD5ySdx\nR3ZwKUviL7V3TjMbAgxw98ui9yOBXu4+tohl2wLvAm3cfU80LS+q5skDJrn788XsZzQwGuDoo48+\nadWqVSXGVV5bt8Jzz4UbOd58M9Tcf/ObYRCNCy6o2Piu06aFoQJ37Ng3rX790L6/rH3QLFwIf/xj\nuBHoyy9D2+TRo0Oc6TYGbTK++ip8hvr1w2dL55G1KsNnn0GHDqH73jfeqP5NDN3DcIObNoV7WQqe\nE19v2hTu3H711TAE4ZlnwhVXhLtra9eO+xMcfMrSO2eqE/9PCUn/6oRprd19jZkdA7wJnOHuH+2/\nbqJUd8u8Z0/4sT3+eBjEe8eO0H585EgYMQKOOy41+2nXDoo6XrVtG4YSLI/t2+HPfw4Hj7lzQ38i\n3/teOAiceurBk0B+/etwY9Nf/xr6s8lEDz4YbuoaPjyMzHTIIVCvXvme69at/L/9118fmLiLS+D7\nz9u8ed94ssU59FBo0QLOPz/8P6fqd5ipUp34TwFudffvRO9vBHD3A26yN7N/Ale5+9vFbOtR4EV3\nf6akfaYq8S9aFEr206aFQRSaNAkdaY0cGUr5qf7h1KhR9E1aZqG/+Yp6//1wFvDEE+EsoGPHfWcB\n6XxX4tq1YeCKb387HHgz1Z494aD9xhuhH5nduyu2vfIeNOrVC4/t20tO4Dt3lrz/unXD4CNNmoRH\nweuSphU8H3po5p31VbZUJ/5ahJY6ZwBrgHnAhe7+wX7LdQBeBdpH9U2YWVNgh7vvMrMWwDvAIHdf\nUtI+K5L4160Lt24//nhI/LVqwXe/G4b0O/vs8A9fWSqjxF+U7dtD3++TJ4dRq+rWhSFDwkGgT5/0\nOwsYNQqmTw/DKR57bNzRpI89e0IV2FdfhSSbzHNZli3uedeusH+z0hN0SdMq87ckZZfSgVjcPc/M\nxgKzCC18prj7B2Y2gXAxYWa06FBguhc+knQEHjKzfEL3EJNKS/rltW1bSH6vvx5K1z17hr5yvv/9\ncDpZFSZOLLqOf+LE1O6nQQO45JLwWLRo31nAtGmhHnn06HCgK+9ZgHtIDql4bNkSDsI33qikv7+a\nNcPfskGDqt1vfn7429Ste3B3gy3lV62GXjz33NA52siRoZfEOEybFjodW70ajj46JP2qGFxkxw74\n3/8NA0y8807oK+iss8KBp6zJuqJVEIlq1Qp/k9mzoVGj1G1XRApLaVVPHDTmbsUsXhzOAl56KZzO\n161bsUe9euVbr04dDUEoUlWU+EVEMkxZEr9q+EREMowSv4hIhlHiFxHJMEr8IiIZRolfRCTDKPGL\niGQYJX4RkQyjxC8ikmGU+EVEMowSv4hIhlHiFxHJMEr8IiIZRolfRCTDKPGLiGQYJX4RkQyTVOI3\nswFmtszMVpjZuCLmX2xm681sYfS4LGHeKDNbHj1GpTJ4EREpu1LH3DWzmsADwLeBXGCemc0sYuzc\nP7v72P3WbQbcAmQDDsyP1t2UkuhFRKTMkinx9wRWuPvH7v41MB0YlOT2vwO87u5fRMn+dWBA+UKV\nZE2bBu3ahYG027UL70VECiST+FsDnya8z42m7e98M1tkZs+Y2VFlXBczG21mOWaWs379+iTCkqJM\nmwajR8OqVeAenkePVvIXkX1SdXH3BaCdu3chlOofK+sG3H2yu2e7e3bLli1TFFbmuekm2LGj8LQd\nO8J0ERFILvGvAY5KeN8mmraXu290913R24eBk5JdV1Jr9eqyTReRzJNM4p8HHG9m7c2sDjAUmJm4\ngJkdmfB2ILA0ej0L6G9mTc2sKdA/miaV5OijyzZdRDJPqYnf3fOAsYSEvRR42t0/MLMJZjYwWuwa\nM/vAzN4HrgEujtb9AriNcPCYB0yIpkklmTgR6tcvPK1+/TBdRATA3D3uGA6QnZ3tOTk5cYdx0Jo2\nLdTpr14dSvoTJ8Lw4XFHJSKVyczmu3t2MsuW2o5fDj7DhyvRi0jx1GWDiEiGUeIXEckwSvwiIhlG\niV9EJMMo8YuIZBglfqk06ixOJD2pOadUioLO4gr6DSroLA7U1FQkbirxS6VQZ3Ei6UuJXyqFOosT\nSV9K/FIp1FmcSPpS4pdKoc7iRNKXEr9UiuHDYfJkaNsWzMLz5Mm6sCuSDtSqRyqNOosTSU8q8YuI\nZBglfhGRDKPEL9We7iAWKSypxG9mA8xsmZmtMLNxRcy/3syWmNkiM3vDzNomzNtjZgujx8z91xWp\nTAV3EK9aBe777iBW8pdMVmriN7OawAPAWUAWMMzMsvZb7J9Atrt3AZ4B7kiYt9Pdu0WPgYhUId1B\nfCCdAUkyJf6ewAp3/9jdvwamA4MSF3D32e5e8PN6F2iT2jBFykd3EBemMyCB5BJ/a+DThPe50bTi\nXAq8kvC+npnlmNm7Zvbf5YhRpNx0B3FhOgMSSPHFXTMbAWQDv0mY3DYa+f1C4B4zO7aYdUdHB4ic\n9evXpzIsyWC6g7gwnQEJJJf41wBHJbxvE00rxMzOBG4CBrr7roLp7r4mev4Y+BvQvaiduPtkd892\n9+yWLVsm/QFESqI7iAvTGZBAcol/HnC8mbU3szrAUKBQ6xwz6w48REj6nydMb2pmdaPXLYDewJJU\nBS+SjOHDYeVKyM8Pz5ma9EFnQBKUmvjdPQ8YC8wClgJPu/sHZjbBzApa6fwGaAj8737NNjsCOWb2\nPjAbmOTuSvySkdKhNY3OgATA3D3uGA6QnZ3tOTk5cYchkjL7j0gGoaStpCupYmbzo+uppdKduyJV\nQK1pJJ0o8YtUAbWmkXSixC9SBdSaRtKJEr9IFVBrGkknSvwiVUCtaSSdaAQukSqiEckkXajELyKS\nYZT4RUQyjBK/iEiGUeIXEckwSvwiIhlGiV9EJMMo8YtILNKht9JMpXb8IlLl9u+ttGDsX9C9DlVB\nJX4RqXLp0ltppp51KPGLSJVLh95KC846Vq0C931nHXEk/6o+ACnxi0iVS4feStPprKOqD0BK/CJS\n5dKht9J0OOuAeA5ASSV+MxtgZsvMbIWZjStifl0z+3M0f66ZtUuYd2M0fZmZfSd1oYvIwSodeitN\nh7MOiOcAVGriN7OawAPAWUAWMMzMsvZb7FJgk7sfB9wN3B6tmwUMBToBA4DfR9sTkQw3fDisXAn5\n+eG5qlvzpMNZB8RzAEqmxN8TWOHuH7v718B0YNB+ywwCHotePwOcYWYWTZ/u7rvc/RNgRbQ9EZFY\npcNZB8RzAEom8bcGPk14nxtNK3IZd88DtgDNk1wXADMbbWY5Zpazfv365KIXEamAuM86CmKo6gNQ\n2tzA5e6TgckA2dnZHnM4IiJVpqoH6UmmxL8GOCrhfZtoWpHLmFktoDGwMcl1RUSkCiWT+OcBx5tZ\nezOrQ7hYO3O/ZWYCo6LXQ4A33d2j6UOjVj/tgeOB91ITuoiIlEepVT3unmdmY4FZQE1girt/YGYT\ngBx3nwn8CXjCzFYAXxAODkTLPQ0sAfKAq9x9TyV9FhERSYKFgnl6yc7O9pycnLjDEBE5aJjZfHfP\nTmZZ3bkrIpJh0rLEb2brgVXlXL0FsCGF4RzM9F0Upu+jMH0f+1SH76Ktu7dMZsG0TPwVYWY5yZ7u\nVHf6LgrT91GYvo99Mu27UFWPiEiGUeIXEckw1THxT447gDSi76IwfR+F6fvYJ6O+i2pXxy8iIiWr\njiV+EREpgRK/iEiGqTaJv7RRwjKJmR1lZrPNbImZfWBmP4o7priZWU0z+6eZvRh3LHEzsyZm9oyZ\nfWhmS83slLhjipOZXRf9Tv5lZk+ZWb24Y6ps1SLxJzlKWCbJA37s7lnAycBVGf59APwIWBp3EGni\nXuBVd+8AdCWDvxczaw1cA2S7e2dCf2RD442q8lWLxE9yo4RlDHdf5+4LotdbCT/sIgfAyQRm1gY4\nG3g47ljiZmaNgW8ROlbE3b92983xRhW7WsAhUZfy9YG1McdT6apL4k96pK9MEw183x2YG28ksboH\n+B8gP+5A0kB7YD3wSFT19bCZNYg7qLi4+xrgTmA1sA7Y4u6vxRtV5asuiV+KYGYNgWeBa939y7jj\niYOZnQN87u7z444lTdQCegAPunt3YDuQsdfEzKwpoXagPdAKaGBmI+KNqvJVl8Svkb72Y2a1CUl/\nmrs/F3c8MeoNDDSzlYQqwNPNbGq8IcUqF8h194IzwGcIB4JMdSbwibuvd/fdwHPAN2OOqdJVl8Sf\nzChhGcPMjFCHu9Td74o7nji5+43u3sbd2xH+L95092pfoiuOu/8H+NTMTogmnUEYKClTrQZONrP6\n0e/mDDLgYnfaDLZeEcWNEhZzWHHqDYwEFpvZwmjaeHd/OcaYJH1cDUyLCkkfA5fEHE9s3H2umT0D\nLCC0hvsnGdB9g7psEBHJMNWlqkdERJKkxC8ikmGU+EVEMowSv4hIhlHiFxHJMEr8IiIZRolfRCTD\n/D/w0XtHon6dFgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7ff68fa53320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The model quickly starts overfitting, unsurprisingly given the small number of training samples. Validation accuracy has high variance for \n",
    "the same reason, but seems to reach high 50s.\n",
    "\n",
    "Note that your mileage may vary: since we have so few training samples, performance is heavily dependent on which exact 200 samples we \n",
    "picked, and we picked them at random. If it worked really poorly for you, try picking a different random set of 200 samples, just for the \n",
    "sake of the exercise (in real life you don't get to pick your training data).\n",
    "\n",
    "We can also try to train the same model without loading the pre-trained word embeddings and without freezing the embedding layer. In that \n",
    "case, we would be learning a task-specific embedding of our input tokens, which is generally more powerful than pre-trained word embeddings \n",
    "when lots of data is available. However, in our case, we have only 200 training samples. Let's try it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_5 (Embedding)      (None, 100, 100)          1000000   \n",
      "_________________________________________________________________\n",
      "flatten_4 (Flatten)          (None, 10000)             0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 32)                320032    \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 1,320,065\n",
      "Trainable params: 1,320,065\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 200 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "200/200 [==============================] - 1s - loss: 0.6941 - acc: 0.4750 - val_loss: 0.6920 - val_acc: 0.5213\n",
      "Epoch 2/10\n",
      "200/200 [==============================] - 0s - loss: 0.5050 - acc: 0.9900 - val_loss: 0.6949 - val_acc: 0.5138\n",
      "Epoch 3/10\n",
      "200/200 [==============================] - 0s - loss: 0.2807 - acc: 1.0000 - val_loss: 0.7131 - val_acc: 0.5125\n",
      "Epoch 4/10\n",
      "200/200 [==============================] - 0s - loss: 0.1223 - acc: 1.0000 - val_loss: 0.6997 - val_acc: 0.5214\n",
      "Epoch 5/10\n",
      "200/200 [==============================] - 0s - loss: 0.0584 - acc: 1.0000 - val_loss: 0.7043 - val_acc: 0.5183\n",
      "Epoch 6/10\n",
      "200/200 [==============================] - 0s - loss: 0.0307 - acc: 1.0000 - val_loss: 0.7051 - val_acc: 0.5248\n",
      "Epoch 7/10\n",
      "200/200 [==============================] - 0s - loss: 0.0166 - acc: 1.0000 - val_loss: 0.7345 - val_acc: 0.5282\n",
      "Epoch 8/10\n",
      "200/200 [==============================] - 0s - loss: 0.0098 - acc: 1.0000 - val_loss: 0.7173 - val_acc: 0.5199\n",
      "Epoch 9/10\n",
      "200/200 [==============================] - 0s - loss: 0.0058 - acc: 1.0000 - val_loss: 0.7201 - val_acc: 0.5253\n",
      "Epoch 10/10\n",
      "200/200 [==============================] - 0s - loss: 0.0035 - acc: 1.0000 - val_loss: 0.7244 - val_acc: 0.5264\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_words, embedding_dim, input_length=maxlen))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.summary()\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['acc'])\n",
    "history = model.fit(x_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=32,\n",
    "                    validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt4VPW97/H3lxAM4X4JgkQItVbugRBBD94oaNEqPCq1\nIO4W3Yp6xFqqZ28qtvJose5q3dqW7ZZ6tLUilKPVYuul1WLRWpWgggIqbAUNIIarCCgEv+ePtSZM\nhkkyCRMms/J5Pc88sy6/Wes7a5LPWvNbM2vM3RERkWhpkekCREQk/RTuIiIRpHAXEYkghbuISAQp\n3EVEIkjhLiISQQr3CDOzHDP7zMx6pbNtJpnZV80s7Z/fNbMxZrYubvxdMzs1lbYNWNf9ZnZjQx8v\nkoqWmS5ADjKzz+JG84EvgAPh+JXuPq8+y3P3A0DbdLdtDtz9hHQsx8wuBy5x9zPiln15OpYtUhuF\nexPi7lXhGh4ZXu7uz9XU3sxaunvlkahNpC76e2xa1C2TRczsJ2b2ezObb2a7gEvM7GQze8XMdpjZ\nJjP7hZnlhu1bmpmbWVE4/nA4/2kz22Vm/zSzPvVtG84/28zeM7OdZvZLM/uHmU2poe5UarzSzNaa\n2XYz+0XcY3PM7D/NbKuZvQ+MrWX7zDSzBQnT5pjZXeHw5Wa2Onw+/xMeVde0rHIzOyMczjez34W1\nrQSGJbS9yczeD5e70szGhdMHAb8CTg27vLbEbdtZcY+/KnzuW83sCTPrkcq2qc92jtVjZs+Z2TYz\n+9jM/i1uPT8Kt8mnZlZmZsck6wIzs5dir3O4PZeE69kG3GRmx5vZ4nAdW8Lt1iHu8b3D51gRzr/H\nzPLCmvvFtethZnvMrEtNz1fq4O66NcEbsA4YkzDtJ8A+4DyCHXNr4ERgBMG7sK8A7wHTwvYtAQeK\nwvGHgS1AKZAL/B54uAFtuwG7gPHhvB8A+4EpNTyXVGr8I9ABKAK2xZ47MA1YCRQCXYAlwZ9t0vV8\nBfgMaBO37E+A0nD8vLCNAV8H9gKDw3ljgHVxyyoHzgiH7wReADoBvYFVCW0vAnqEr8nFYQ1Hh/Mu\nB15IqPNhYFY4fFZY4xAgD/gv4G+pbJt6bucOwGbgOuAooD0wPJz3Q2A5cHz4HIYAnYGvJm5r4KXY\n6xw+t0rgaiCH4O/xa8BooFX4d/IP4M645/N2uD3bhO1HhvPmArPj1nM98Him/w+z+ZbxAnSr4YWp\nOdz/VsfjbgD+XzicLLD/O67tOODtBrS9DHgxbp4Bm6gh3FOs8aS4+X8AbgiHlxB0T8XmnZMYOAnL\nfgW4OBw+G3i3lrZ/Aq4Jh2sL9w/jXwvgf8e3TbLct4FvhsN1hftvgdvi5rUnOM9SWNe2qed2/hdg\naQ3t/idWb8L0VML9/TpqmBBbL3Aq8DGQk6TdSOADwMLxN4EL0v1/1Zxu6pbJPh/Fj5hZXzP7c/g2\n+1PgFqBrLY//OG54D7WfRK2p7THxdXjw31he00JSrDGldQHra6kX4BFgUjh8cTgeq+NcM3s17DLY\nQXDUXNu2iulRWw1mNsXMloddCzuAvikuF4LnV7U8d/8U2A70jGuT0mtWx3Y+liDEk6ltXl0S/x67\nm9lCM9sQ1vCbhBrWeXDyvhp3/wfBu4BTzGwg0Av4cwNrEtTnno0SPwZ4H8GR4lfdvT3wY4Ij6ca0\nieDIEgAzM6qHUaLDqXETQSjE1PVRzYXAGDPrSdBt9EhYY2vgUeCnBF0mHYG/pFjHxzXVYGZfAe4l\n6JroEi73nbjl1vWxzY0EXT2x5bUj6P7ZkEJdiWrbzh8Bx9XwuJrm7Q5ryo+b1j2hTeLz+w+CT3kN\nCmuYklBDbzPLqaGOh4BLCN5lLHT3L2poJylQuGe/dsBOYHd4QurKI7DOPwElZnaembUk6MctaKQa\nFwLfN7Oe4cm1f6+tsbt/TNB18BuCLpk14ayjCPqBK4ADZnYuQd9wqjXcaGYdLfgewLS4eW0JAq6C\nYD93BcGRe8xmoDD+xGaC+cC/mtlgMzuKYOfzorvX+E6oFrVt50VALzObZmZHmVl7Mxsezrsf+ImZ\nHWeBIWbWmWCn9jHBifscM5tK3I6olhp2AzvN7FiCrqGYfwJbgdssOEnd2sxGxs3/HUE3zsUEQS+H\nQeGe/a4HvktwgvM+ghOfjcrdNwPfBu4i+Gc9DniD4Igt3TXeCzwPvAUsJTj6rssjBH3oVV0y7r4D\nmA48TnBScgLBTioVNxO8g1gHPE1c8Lj7CuCXwGthmxOAV+Me+1dgDbDZzOK7V2KPf4ag++Tx8PG9\ngMkp1pWoxu3s7juBM4ELCXY47wGnh7PvAJ4g2M6fEpzczAu7264AbiQ4uf7VhOeWzM3AcIKdzCLg\nsbgaKoFzgX4ER/EfErwOsfnrCF7nL9z95Xo+d0kQO3kh0mDh2+yNwAR3fzHT9Uj2MrOHCE7Szsp0\nLdlOX2KSBjGzsQSfTNlL8FG6/QRHryINEp6/GA8MynQtUaBuGWmoU4D3CfqavwGcrxNg0lBm9lOC\nz9rf5u4fZrqeKFC3jIhIBOnIXUQkgjLW5961a1cvKirK1OpFRLLSsmXLtrh7bR89BjIY7kVFRZSV\nlWVq9SIiWcnM6vqWNqBuGRGRSFK4i4hEkMJdRCSCFO4iIhGkcBcRiaA6w93MHjCzT8zs7RrmW/gz\nW2vNbIWZlaS/TElm3jwoKoIWLYL7efX6+WzVEdUaVIfqAOr+JSbgNKCE8Fd4ksw/h+BKeQacBLya\nyq+EDBs2zKXhHn7YPT/fHQ7e8vOD6aojM3U0hRpUR/TrAMo8hYxN6eeaCH67saZwvw+YFDf+LtCj\nrmUq3A9P797V/0hit969VUem6mgKNaiO6NeRarindG0ZMysC/uTuA5PM+xNwu7u/FI4/D/y7ux/y\nDaXwYv9TAXr16jVs/fqUPosvSbRoEfxpJDKDL79UHZmooynUoDqiX4eZLXP30jrXV5/iDpe7z3X3\nUncvLSio89uzTVZT6L/rVcOPzdU0XXU0jxpUh+qISUe4b6D670sW0rDff8wK8+bB1Kmwfn2wF16/\nPhg/0gE/ezbk51eflp8fTFcdmamjKdSgOlRHlVT6bqi9z/2bVD+h+loqy8zWPvem0n/nHpyI6d3b\n3Sy4P9IniFRH06xBdUS7DtLV525m84EzgK4Ev714M5Ab7hj+28wM+BUwFtgDXOpJ+tsTlZaWejZe\nOKyp9N+JSPOUap97nVeFdPdJdcx34Jp61JbVevUKumKSTRcRaSr0DdV6air9dyIitVG419PkyTB3\nLvTuHXTF9O4djE+enOnKREQOytiPdWSzyZMV5iLStOnIXUQkghTuIiIRpHAXEYkghbuISAQp3EVE\nIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4\ni4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRJDCXUQkghTuIiIR\nlFK4m9lYM3vXzNaa2Ywk83ub2fNmtsLMXjCzwvSXKiIiqaoz3M0sB5gDnA30ByaZWf+EZncCD7n7\nYOAW4KfpLlRERFKXypH7cGCtu7/v7vuABcD4hDb9gb+Fw4uTzBcRkSMolXDvCXwUN14eTou3HLgg\nHD4faGdmXRIXZGZTzazMzMoqKioaUq+IiKQgXSdUbwBON7M3gNOBDcCBxEbuPtfdS929tKCgIE2r\nFhGRRC1TaLMBODZuvDCcVsXdNxIeuZtZW+BCd9+RriJFRKR+UjlyXwocb2Z9zKwVMBFYFN/AzLqa\nWWxZPwQeSG+ZIiJSH3WGu7tXAtOAZ4HVwEJ3X2lmt5jZuLDZGcC7ZvYecDQwu5HqFRGRFJi7Z2TF\npaWlXlZWlpF1i4hkKzNb5u6ldbXTN1RFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcR\niSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuISAQp3EVEIkjh\nLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hE\nUErhbmZjzexdM1trZjOSzO9lZovN7A0zW2Fm56S/VBERSVWd4W5mOcAc4GygPzDJzPonNLsJWOju\nQ4GJwH+lu1AREUldKkfuw4G17v6+u+8DFgDjE9o40D4c7gBsTF+JIiJSXy1TaNMT+ChuvBwYkdBm\nFvAXM7sWaAOMSUt1IiLSIOk6oToJ+I27FwLnAL8zs0OWbWZTzazMzMoqKirStGoREUmUSrhvAI6N\nGy8Mp8X7V2AhgLv/E8gDuiYuyN3nunupu5cWFBQ0rGIREalTKuG+FDjezPqYWSuCE6aLEtp8CIwG\nMLN+BOGuQ3MRkQypM9zdvRKYBjwLrCb4VMxKM7vFzMaFza4HrjCz5cB8YIq7e2MVLSIitUvlhCru\n/hTwVMK0H8cNrwJGprc0ERFpKH1DVUQkghTuIiIRlFK3jIhEx/79+ykvL+fzzz/PdClSi7y8PAoL\nC8nNzW3Q4xXuIs1MeXk57dq1o6ioCDPLdDmShLuzdetWysvL6dOnT4OWoW4ZkWbm888/p0uXLgr2\nJszM6NKly2G9u1K4izRDCvam73BfI4W7iBxRW7duZciQIQwZMoTu3bvTs2fPqvF9+/altIxLL72U\nd999t9Y2c+bMYd68eekoOSupz11EajVvHsycCR9+CL16wezZMHlyw5fXpUsX3nzzTQBmzZpF27Zt\nueGGG6q1cXfcnRYtkh9/Pvjgg3Wu55prrml4kRGgI3cRqdG8eTB1KqxfD+7B/dSpwfR0W7t2Lf37\n92fy5MkMGDCATZs2MXXqVEpLSxkwYAC33HJLVdtTTjmFN998k8rKSjp27MiMGTMoLi7m5JNP5pNP\nPgHgpptu4u67765qP2PGDIYPH84JJ5zAyy+/DMDu3bu58MIL6d+/PxMmTKC0tLRqxxPv5ptv5sQT\nT2TgwIFcddVVxL6A/9577/H1r3+d4uJiSkpKWLduHQC33XYbgwYNori4mJkzZ6Z/Y6VA4S4iNZo5\nE/bsqT5tz55gemN45513mD59OqtWraJnz57cfvvtlJWVsXz5cv7617+yatWqQx6zc+dOTj/9dJYv\nX87JJ5/MAw88kHTZ7s5rr73GHXfcUbWj+OUvf0n37t1ZtWoVP/rRj3jjjTeSPva6665j6dKlvPXW\nW+zcuZNnnnkGgEmTJjF9+nSWL1/Oyy+/TLdu3XjyySd5+umnee2111i+fDnXX399mrZO/SjcRaRG\nH35Yv+mH67jjjqO0tLRqfP78+ZSUlFBSUsLq1auThnvr1q05++yzARg2bFjV0XOiCy644JA2L730\nEhMnTgSguLiYAQMGJH3s888/z/DhwykuLubvf/87K1euZPv27WzZsoXzzjsPCD6Xnp+fz3PPPcdl\nl11G69atAejcuXP9N0QaqM9dRGrUq1fQFZNsemNo06ZN1fCaNWu45557eO211+jYsSOXXHJJ0o8G\ntmrVqmo4JyeHysrKpMs+6qij6myTzJ49e5g2bRqvv/46PXv25KabbsqKL4DpyF1EajR7NuTnV5+W\nnx9Mb2yffvop7dq1o3379mzatIlnn3027esYOXIkCxcuBOCtt95K+s5g7969tGjRgq5du7Jr1y4e\ne+wxADp16kRBQQFPPvkkEHx/YM+ePZx55pk88MAD7N27F4Bt27alve5U6MhdRGoU+1RMOj8tk6qS\nkhL69+9P37596d27NyNHpv/Cs9deey3f+c536N+/f9WtQ4cO1dp06dKF7373u/Tv358ePXowYsTB\nXxmdN28eV155JTNnzqRVq1Y89thjnHvuuSxfvpzS0lJyc3M577zzuPXWW9Nee10sU5ddLy0t9bKy\nsoysW6Q5W716Nf369ct0GU1CZWUllZWV5OXlsWbNGs466yzWrFlDy5ZN47g32WtlZsvcvbSGh1Rp\nGs9ARCQDPvvsM0aPHk1lZSXuzn333ddkgv1wReNZiIg0QMeOHVm2bFmmy2gUOqEqIhJBCncRkQhS\nuIuIRJDCXUQkghTuInJEjRo16pAvJN19991cffXVtT6ubdu2AGzcuJEJEyYkbXPGGWdQ10es7777\nbvbEXTDnnHPOYceOHamUnlUU7iJyRE2aNIkFCxZUm7ZgwQImTZqU0uOPOeYYHn300QavPzHcn3rq\nKTp27Njg5TVVCncROaImTJjAn//856of5li3bh0bN27k1FNPrfrceUlJCYMGDeKPf/zjIY9ft24d\nAwcOBIJLA0ycOJF+/fpx/vnnV33lH+Dqq6+uulzwzTffDMAvfvELNm7cyKhRoxg1ahQARUVFbNmy\nBYC77rqLgQMHMnDgwKrLBa9bt45+/fpxxRVXMGDAAM4666xq64l58sknGTFiBEOHDmXMmDFs3rwZ\nCD5Lf+mllzJo0CAGDx5cdfmCZ555hpKSEoqLixk9enRatm08fc5dpBn7/vchyeXLD8uQIRDmYlKd\nO3dm+PDhPP3004wfP54FCxZw0UUXYWbk5eXx+OOP0759e7Zs2cJJJ53EuHHjavzJuXvvvZf8/HxW\nr17NihUrKCkpqZo3e/ZsOnfuzIEDBxg9ejQrVqzge9/7HnfddReLFy+ma9eu1Za1bNkyHnzwQV59\n9VXcnREjRnD66afTqVMn1qxZw/z58/n1r3/NRRddxGOPPcYll1xS7fGnnHIKr7zyCmbG/fffz89+\n9jN+/vOfc+utt9KhQwfeeustALZv305FRQVXXHEFS5YsoU+fPo1y/RkduYvIERffNRPfJePu3Hjj\njQwePJgxY8awYcOGqiPgZJYsWVIVsoMHD2bw4MFV8xYuXEhJSQlDhw5l5cqVSS8KFu+ll17i/PPP\np02bNrRt25YLLriAF198EYA+ffowZMgQoObLCpeXl/ONb3yDQYMGcccdd7By5UoAnnvuuWq/CtWp\nUydeeeUVTjvtNPr06QM0zmWBdeQu0ozVdoTdmMaPH8/06dN5/fXX2bNnD8OGDQOCC3FVVFSwbNky\ncnNzKSoqatDldT/44APuvPNOli5dSqdOnZgyZcphXaY3drlgCC4ZnKxb5tprr+UHP/gB48aN44UX\nXmDWrFkNXl866MhdRI64tm3bMmrUKC677LJqJ1J37txJt27dyM3NZfHixaxPdjH5OKeddhqPPPII\nAG+//TYrVqwAgssFt2nThg4dOrB582aefvrpqse0a9eOXbt2HbKsU089lSeeeII9e/awe/duHn/8\ncU499dSUn9POnTvp2bMnAL/97W+rpp955pnMmTOnanz79u2cdNJJLFmyhA8++ABonMsCK9xFJCMm\nTZrE8uXLq4X75MmTKSsrY9CgQTz00EP07du31mVcffXVfPbZZ/Tr148f//jHVe8AiouLGTp0KH37\n9uXiiy+udrngqVOnMnbs2KoTqjElJSVMmTKF4cOHM2LECC6//HKGDh2a8vOZNWsW3/rWtxg2bFi1\n/vybbrqJ7du3M3DgQIqLi1m8eDEFBQXMnTuXCy64gOLiYr797W+nvJ5UpXTJXzMbC9wD5AD3u/vt\nCfP/E4htqXygm7vX+tkiXfJXJDN0yd/s0aiX/DWzHGAOcCZQDiw1s0XuXnV2wt2nx7W/Fkh9dyci\nImmXSrfMcGCtu7/v7vuABcD4WtpPAuanozgREWmYVMK9J/BR3Hh5OO0QZtYb6AP8rYb5U82szMzK\nKioq6luriIikKN0nVCcCj7r7gWQz3X2uu5e6e2lBQUGaVy0iqcrUz2tK6g73NUol3DcAx8aNF4bT\nkpmIumREmrS8vDy2bt2qgG/C3J2tW7eSl5fX4GWk8iWmpcDxZtaHINQnAhcnNjKzvkAn4J8NrkZE\nGl1hYSHl5eWoa7Rpy8vLo7CwsMGPrzPc3b3SzKYBzxJ8FPIBd19pZrcAZe6+KGw6EVjgOhwQadJy\nc3OrvvYu0ZXS5Qfc/SngqYRpP04Yn5W+skRE5HDoG6oiIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJB\nCncRkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1E\nJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCF\nu4hIBCncRUQiSOEuIhJBKYW7mY01s3fNbK2ZzaihzUVmtsrMVprZI+ktU0RE6qNlXQ3MLAeYA5wJ\nlANLzWyRu6+Ka3M88ENgpLtvN7NujVWwiIjULZUj9+HAWnd/3933AQuA8QltrgDmuPt2AHf/JL1l\niohIfaQS7j2Bj+LGy8Np8b4GfM3M/mFmr5jZ2GQLMrOpZlZmZmUVFRUNq1hEROqUrhOqLYHjgTOA\nScCvzaxjYiN3n+vupe5eWlBQkKZVi4hIolTCfQNwbNx4YTgtXjmwyN33u/sHwHsEYS8iIhmQSrgv\nBY43sz5m1gqYCCxKaPMEwVE7ZtaVoJvm/TTWKSIi9VBnuLt7JTANeBZYDSx095VmdouZjQubPQts\nNbNVwGLg/7j71sYqWkREamfunpEVl5aWellZWUbWLSKSrcxsmbuX1tVO31AVEYkghbuISAQp3EVE\nIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4\ni4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBLTNdgEhzd+AA7N0b3PbsCW6pDKfS\nzgy6dYPu3YPb0UcfOlxQALm5md4K2cU9eN0qK4P7mm41zS8shK5dG7dGhbtIA+zfD9u2wdatye+3\nb089gL/4omE15OVBfn5wa926+nCnTsHwl1/C5s3wxhvw8cfw6aeHLscMunSpOfzjh7t0gZycw9t2\n6eAOn38OO3bAzp3BLTZc27QvvkgtfOua73549d97L1x1VXq2RU2yLtw/+CC4desWHHF06QIts+5Z\nRMOXX0JFBWzaFAQHBIHTuvWht9j0Fk2sI/DAgSCIawvqZPe7dtW8zJYtg3Bt06Z66HboAD161BzI\n9RnOy2vYtty7Nwj7jz8ObsmGX345uN+799DH5+QE/3ep7Ag6dQp2HMlUVtYvlJMN799f+3Nt0SLY\n5h07BvcdOgQ15eQcvLVsWX088dZY84cMqf9rV19ZF4sLF8KMGQfHzaBz5+APLhb48cOJ05rKkUdT\nduAAfPJJENobNwb38cOx+82bg3/S+mjVqubgr22nkOq8Vq2Co9Pagjl+eMeOmmtt0SIIgy5dgr+x\nHj1gwIBgPDYt2X3btjWHWqa1bg1FRcGtNu7w2Wc17wBiw6tWBcPJgjY392DQ5+ZWD+jdu+uutW3b\n6uHcrRt87WuHBnay4Y4dg51rU30djgTzw31/0UClpaVeVlZW78dt2gTvvBMcMVZUBCEUfx8b3rYt\n+Vun2FvQunYCsfvOnaOzM6isDP4hawvsWGh/+eWhjy8oCAKuRw845pjq9927B2EY6zuOv33+efLp\n9ZlX351Ioo4daw/kZPcdOjS9dxpNkXsQ2jXtADZtCg4YagvixGnt2+sdeU3MbJm7l9bVLus2Xyxc\n6lJZGRyZJe4EEncIb78dDG/dmnw5sZ1Bsp1Ahw7BH2DsFnsblmw81Xmptos/Itm3r3po1xTeFRWH\n7vBiJ9xiQT10aPLwPvro4Kg4UyorU9sp7NsXvC7xIR17Ky6NwyzYxp06Qb9+ma5GYrIu3FPVsmUQ\nSEcfnVr72M6gpp1AbNqKFcH9tm2NW39dzA4GfbK+0RYtgufeo0dwZv7EEw8GdWJoZ8MRUsuW0K5d\ncBORumXBv/WRUd+dwf79QZ9k/Bn1ysrah1NtV5/H7N8f9E0ec0z18O7WTUerIs1ZSuFuZmOBe4Ac\n4H53vz1h/hTgDmBDOOlX7n5/GutscnJzg7ehIiJNUZ3hbmY5wBzgTKAcWGpmi9x9VULT37v7tEao\nUURE6imVzwIMB9a6+/vuvg9YAIxv3LJERORwpBLuPYGP4sbLw2mJLjSzFWb2qJkdm5bqRESkQdL1\nKd4ngSJ3Hwz8FfhtskZmNtXMysysrKKiIk2rFhGRRKmE+wYg/ki8kIMnTgFw963uHrtCxv3AsGQL\ncve57l7q7qUFBQUNqVdERFKQSrgvBY43sz5m1gqYCCyKb2Bm8V8rGgesTl+JIiJSX3V+WsbdK81s\nGvAswUchH3D3lWZ2C1Dm7ouA75nZOKAS2AZMacSaRUSkDll3bRkRkeYs1WvL6LJIIiIRlFXhPm9e\ncKnSFi2C+3nzMl2RiEjTlDXXlpk3D6ZODX65BmD9+mAcYPLkzNUlItIUZc2R+8yZB4M9Zs+eYLqI\niFSXNeH+4Yf1my4i0pxlTbj36lW/6SIizVnWhPvs2cEPA8fLzw+mi4hIdVkT7pMnw9y50Lt38CtE\nvXsH4zqZKiJyqKz5tAwEQa4wFxGpW9YcuYuISOoU7iIiEaRwFxGJIIW7iEgEKdxFRCIoY5f8NbMK\nYH0DH94V2JLGcrKdtkd12h4HaVtUF4Xt0dvd6/wpu4yF++Ews7JUrmfcXGh7VKftcZC2RXXNaXuo\nW0ZEJIIU7iIiEZSt4T430wU0Mdoe1Wl7HKRtUV2z2R5Z2ecuIiK1y9YjdxERqYXCXUQkgrIu3M1s\nrJm9a2ZrzWxGpuvJFDM71swWm9kqM1tpZtdluqamwMxyzOwNM/tTpmvJNDPraGaPmtk7ZrbazE7O\ndE2ZYmbTw/+Tt81svpnlZbqmxpZV4W5mOcAc4GygPzDJzPpntqqMqQSud/f+wEnANc14W8S7Dlid\n6SKaiHuAZ9y9L1BMM90uZtYT+B5Q6u4DgRxgYmaranxZFe7AcGCtu7/v7vuABcD4DNeUEe6+yd1f\nD4d3Efzj9sxsVZllZoXAN4H7M11LpplZB+A04P8CuPs+d9+R2aoyqiXQ2sxaAvnAxgzX0+iyLdx7\nAh/FjZeiU3TYAAABjElEQVTTzAMNwMyKgKHAq5mtJOPuBv4N+DLThTQBfYAK4MGwm+p+M2uT6aIy\nwd03AHcCHwKbgJ3u/pfMVtX4si3cJYGZtQUeA77v7p9mup5MMbNzgU/cfVmma2kiWgIlwL3uPhTY\nDTTLc1Rm1ongHX4f4BigjZldktmqGl+2hfsG4Ni48cJwWrNkZrkEwT7P3f+Q6XoybCQwzszWEXTX\nfd3MHs5sSRlVDpS7e+zd3KMEYd8cjQE+cPcKd98P/AH4XxmuqdFlW7gvBY43sz5m1orgpMiiDNeU\nEWZmBP2pq939rkzXk2nu/kN3L3T3IoK/i7+5e+SPzmri7h8DH5nZCeGk0cCqDJaUSR8CJ5lZfvh/\nM5pmcHI5q34g290rzWwa8CzBGe8H3H1lhsvKlJHAvwBvmdmb4bQb3f2pDNYkTcu1wLzwQOh94NIM\n15MR7v6qmT0KvE7wKbM3aAaXIdDlB0REIijbumVERCQFCncRkQhSuIuIRJDCXUQkghTuIiIRpHAX\nEYkghbuISAT9f9OWSH2mH2LTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7ff68c18a048>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X98FfWd7/HXJ/wwIJTwI1YlQBCpEAoiHrFeRMRqF6uC\nWuqCwVa3bqoPLLraXanYrlJ5VF2vWnq5ttRifxibsri2bKtle5UtWCsSEKOACGKQAGqkgkhgIfC5\nf8wknMST5ISc5JxM3s/H4zzOmZnvmfmcCbzPzHfmzJi7IyIi0ZKV7gJERCT1FO4iIhGkcBcRiSCF\nu4hIBCncRUQiSOEuIhJBCndJyMw6mdknZjYwlW3TycxON7OUn/trZhebWXnc8CYzG59M2+NY1uNm\ndtfxvr+R+d5nZj9P9XwlfTqnuwBJDTP7JG6wO/A/wJFw+JvuXtyc+bn7EaBHqtt2BO5+RirmY2Y3\nAjPc/cK4ed+YinlL9CncI8Lda8M13DK80d3/X0Ptzayzu1e3RW0i0vbULdNBhLvdvzGzX5vZPmCG\nmZ1nZi+b2R4z22Vm882sS9i+s5m5meWHw0+G058zs31m9lczG9zctuH0S83sLTPba2Y/MrO/mNn1\nDdSdTI3fNLMtZvaRmc2Pe28nM3vEzHab2VZgUiPrZ46ZldQbt8DMHg5f32hmG8PP83a4Vd3QvCrM\n7MLwdXcz+1VY23rg7Hpt7zazreF815vZ5HD8SOD/AOPDLq8P49btPXHvvyn87LvN7Ldmdkoy66Yp\nZnZVWM8eM3vBzM6Im3aXme00s4/N7M24z/oFM1sbjn/fzP4t2eVJK3B3PSL2AMqBi+uNuw84BFxB\n8KXeDTgHOJdgD+404C3glrB9Z8CB/HD4SeBDIAZ0AX4DPHkcbU8C9gFTwmm3A4eB6xv4LMnU+Dug\nF5AP/K3mswO3AOuBPKAvsCL4J59wOacBnwAnxs37AyAWDl8RtjHgIuAAMCqcdjFQHjevCuDC8PVD\nwH8DvYFBwIZ6ba8BTgn/JteGNXw2nHYj8N/16nwSuCd8/aWwxtFANvB/gReSWTcJPv99wM/D18PD\nOi4K/0Z3AZvC1yOAbcDJYdvBwGnh69XA9PB1T+DcdP9f6MgPbbl3LC+6+3+6+1F3P+Duq919lbtX\nu/tWYCEwoZH3L3H3Unc/DBQThEpz214OrHP334XTHiH4IkgoyRp/4O573b2cIEhrlnUN8Ii7V7j7\nbuD+RpazFXiD4EsH4BLgI3cvDaf/p7tv9cALwPNAwoOm9VwD3OfuH7n7NoKt8fjlLnb3XeHf5CmC\nL+ZYEvMFKAQed/d17n4QmA1MMLO8uDYNrZvGTAOWuvsL4d/ofoIviHOBaoIvkhFh19474bqD4Et6\nqJn1dfd97r4qyc8hrUDh3rFsjx8ws2Fm9gcze8/MPgbmAv0aef97ca+raPwgakNtT42vw92dYEs3\noSRrTGpZBFucjXkKmB6+vjYcrqnjcjNbZWZ/M7M9BFvNja2rGqc0VoOZXW9mr4XdH3uAYUnOF4LP\nVzs/d/8Y+AjoH9emOX+zhuZ7lOBv1N/dNwF3EPwdPgi7+U4Om94AFACbzOwVM/tykp9DWoHCvWOp\nfxrgTwi2Vk93988A3yPodmhNuwi6SQAwM6NuGNXXkhp3AQPihps6VXMxcLGZ9SfYgn8qrLEbsAT4\nAUGXSQ7wX0nW8V5DNZjZacBjwM1A33C+b8bNt6nTNncSdPXUzK8nQffPjiTqas58swj+ZjsA3P1J\ndx9H0CXTiWC94O6b3H0aQdfb/waeNrPsFtYix0nh3rH1BPYC+81sOPDNNljm74ExZnaFmXUGbgVy\nW6nGxcBtZtbfzPoCdzbW2N3fA14Efg5scvfN4aQTgK5AJXDEzC4HvtiMGu4ysxwLfgdwS9y0HgQB\nXknwPfePBFvuNd4H8moOICfwa+AbZjbKzE4gCNmV7t7gnlAzap5sZheGy/5nguMkq8xsuJlNDJd3\nIHwcJfgA15lZv3BLf2/42Y62sBY5Tgr3ju0O4OsE/3F/QnDgs1W5+/vA3wMPA7uBIcCrBOflp7rG\nxwj6xl8nONi3JIn3PEVwgLS2S8bd9wD/BDxDcFByKsGXVDL+lWAPohx4Dvhl3HzLgB8Br4RtzgDi\n+6n/BGwG3jez+O6Vmvf/kaB75Jnw/QMJ+uFbxN3XE6zzxwi+eCYBk8P+9xOABwmOk7xHsKcwJ3zr\nl4GNFpyN9RDw9+5+qKX1yPGxoMtTJD3MrBNBN8BUd1+Z7npEokJb7tLmzGxS2E1xAvBdgrMsXklz\nWSKRonCXdDgf2Eqwy/93wFXu3lC3jIgcB3XLiIhEkLbcRUQiKG0XDuvXr5/n5+ena/EiIu3SmjVr\nPnT3xk4fBtIY7vn5+ZSWlqZr8SIi7ZKZNfVLa0DdMiIikaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEK\ndxGRCFK4i4hEUNrOcxeR1uEO77wDL74IO3fC4MEwZEjw6N073dVJW1G4i7Rz1dVQVhaEec1j167E\nbfv0ORb09R+nnAJZ2pePDIW7SDuzfz+88sqxIP/rX2HfvmDawIEwcSKcf37wyM+H8nJ4+23YsiV4\nfvvt4P3//u9w5Mix+WZnNxz8+fnQpaH7QUlGUriLZLgPPoC//OVYmK9dG2ytm8HIkXDddUGQjxsX\nhHt9I0cGj/oOH4Z3360b+jWPP/0JDhw41jYrCwYNajj8eyRz2+12xD1YPwcOJH4cPNjwtGTazp4N\nV1/dup9B4S6SQdyDsI3vYnnrrWDaCSfAuefCv/xLEObnnQc5Oce/rC5djoVzojreey9x8C9ZArt3\n121/0klw+umJgz83N/giSuTIETh0KAjSQ4da/3VjoVx/2tEW3P21W7eGH337Bn/L1pa267nHYjHX\nhcOkozt8GNatqxvmH3wQTOvT51j3yvnnw5gxbRMKydiz59OhX9P1s2NH8OVQo2fP4EBuosBtSYA2\npXPn4Ausa9fg0aVL0PVUE7Lxr5t6NKdt164Nf5mlgpmtcfdYk5+/9UqQVHMPDpRt3hw83nor+M8E\n0L8/5OUde6553a1bemuWuvbtg5dfPhbkL78MVVXBtNNOg0mTjoX5GWdk7gHOnBw4++zgUd/Bg8HZ\nOvGh//HHwRdT/bCNf27J6/rjunTJ3HXXVpIKdzObBPwQ6AQ87u7315v+CDAxHOwOnOTuLdhh7Ljc\n4cMPjwV4TYjXvN6//1jbrl2DQMjKguefD/4D1denz6eDv/5zTk7rbml0ZLt21d0qX7cu2FrNyoLR\no+HGG4/1l596arqrTY3sbBg+PHhI+jQZ7uHd6RcAlwAVwGozW+ruG2rauPs/xbX/FnBWK9QaKXv2\nNBzge/Yca9epU3Ce8tChcMEFwfPnPhc8DxwYTK+xb1+wS7xjB1RUfPp57Vp4//1P19K9e9NfACed\nVHdZbc092JWvqgr6QxM9V1UFW41HjgQB6h48J/M61dM/+ig4i2Xr1qD+bt3gC1+AOXNg/Pjgdc+e\n6VufEn3JbLmPBba4+1YAMysBpgAbGmg/HfjX1JRXV3Fx8J/j3XeDYJs3DwoLW2NJqbF/f8MBXll5\nrJ0ZDBgQhPb06XUDfPDg5E9B69kThg0LHg05dCjYmkwU/jt2wMqVwQ9fDh+u+75OnYIty4a+APr3\nDz5HosBN1bjW7J9NRlZW8Bnjnxt63a0bjB0LM2cGW+ZnnaVTCaVtJRPu/YHtccMVwLmJGprZIGAw\n8ELLS6uruBiKio71T27bFgxDegP+4MFg6yw+uGte79xZt+0ppwShPWVK3QAfMiTYlW0LXbsGp7QN\nGtRwm6NHgy+f+uFf8/r11+G55+p2ETVXzcGn7t2PPde87t274WlNjcvODg6kNRW+zZ2ubitpb1J9\nQHUasMTdjySaaGZFQBHAwEQn5DZizpxjwV6jqgruvDPYxa05Ap/oNKjmjGvO+yorg72I+DMD+vUL\nQvuSS4Lgrgnx009vP+cCZ2XBZz8bPMaMSdzGPejjrwn+mi+ypsK35syDjn6wS6S1NXkqpJmdB9zj\n7n8XDn8HwN1/kKDtq8BMd3+pqQU391TIrKy6IZpqnTsfO9pe/+h7Q+NycuoG+NChLTvvWESkKak8\nFXI1MNTMBgM7CLbOr02wwGFAb+Cvzaw1KQMHBl0x9fXpA48+2nAAJxPSXbpot1tEoqXJcHf3ajO7\nBVhGcCrkIndfb2ZzgVJ3Xxo2nQaUeCv9KmrevLp97hDs5s+fn9kHVUVE0iGpPnd3fxZ4tt6479Ub\nvid1ZX1aTYC3p7NlRETSpV39QrWwUGEuIpIMnbMgIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRpHAX\nEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI\n4S4iEkFJhbuZTTKzTWa2xcxmN9DmGjPbYGbrzeyp1JYpIiLN0eRt9sysE7AAuASoAFab2VJ33xDX\nZijwHWCcu39kZie1VsEiItK0ZLbcxwJb3H2rux8CSoAp9dr8I7DA3T8CcPcPUlumiIg0RzLh3h/Y\nHjdcEY6L9zngc2b2FzN72cwmpapAERFpvia7ZZoxn6HAhUAesMLMRrr7nvhGZlYEFAEMHDgwRYsW\nEZH6ktly3wEMiBvOC8fFqwCWuvthd38HeIsg7Otw94XuHnP3WG5u7vHWLCIiTUgm3FcDQ81ssJl1\nBaYBS+u1+S3BVjtm1o+gm2ZrCusUEZFmaDLc3b0auAVYBmwEFrv7ejOba2aTw2bLgN1mtgFYDvyz\nu+9uraJFRKRx5u5pWXAsFvPS0tK0LFtEpL0yszXuHmuqnX6hKiISQQp3EZEIUriLiESQwl1EJIIU\n7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hI\nBCncRUQiSOEuIhJBCvfjUFwM+fmQlRU8FxenuyIRkbqSCnczm2Rmm8xsi5nNTjD9ejOrNLN14ePG\n1JeaGYqLoagItm0D9+C5qEgBLyKZpclwN7NOwALgUqAAmG5mBQma/sbdR4ePx1NcZ8aYMweqquqO\nq6oKxouIZIpkttzHAlvcfau7HwJKgCmtW1bmevfd5o0XEUmHZMK9P7A9brgiHFffV8yszMyWmNmA\nlFSXgQYObN54EZF0SNUB1f8E8t19FPAn4BeJGplZkZmVmllpZWVlihbdtubNg+7d647r3j0YLyKS\nKZIJ9x1A/JZ4Xjiulrvvdvf/CQcfB85ONCN3X+juMXeP5ebmHk+9aVdYCAsXwqBBYBY8L1wYjBcR\nyRSdk2izGhhqZoMJQn0acG18AzM7xd13hYOTgY0prTLDFBYqzEUkszUZ7u5ebWa3AMuATsAid19v\nZnOBUndfCswys8lANfA34PpWrFlERJpg7p6WBcdiMS8tLU3LskVE2iszW+Pusaba6ReqIiIRpHAX\nEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI\n4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCUV7mY2ycw2mdkWM5vdSLuvmJmb\nWZP39xMRkdbTZLibWSdgAXApUABMN7OCBO16ArcCq1JdpIiINE8yW+5jgS3uvtXdDwElwJQE7b4P\nPAAcTGF9IiJyHJIJ9/7A9rjhinBcLTMbAwxw9z80NiMzKzKzUjMrraysbHaxIiKSnBYfUDWzLOBh\n4I6m2rr7QnePuXssNze3pYsWEZEGJBPuO4ABccN54bgaPYHPA/9tZuXAF4ClOqgqIpI+yYT7amCo\nmQ02s67ANGBpzUR33+vu/dw9393zgZeBye5e2ioVi4hIk5oMd3evBm4BlgEbgcXuvt7M5prZ5NYu\nUEREmq9zMo3c/Vng2XrjvtdA2wtbXpaIiLSEfqEqIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRpHAX\nEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI\n4S4iEkEKdxGRCEoq3M1skpltMrMtZjY7wfSbzOx1M1tnZi+aWUHqSxURkWQ1Ge5m1glYAFwKFADT\nE4T3U+4+0t1HAw8CD6e8UhERSVoyW+5jgS3uvtXdDwElwJT4Bu7+cdzgiYCnrkQREWmuzkm06Q9s\njxuuAM6t38jMZgK3A12BixLNyMyKgCKAgQMHNrdWERFJUsoOqLr7AncfAtwJ3N1Am4XuHnP3WG5u\nbqoWLSIi9SQT7juAAXHDeeG4hpQAV7akKElOcTHk50NWVvBcXJzuikQkUyQT7quBoWY22My6AtOA\npfENzGxo3OBlwObUlSiJFBdDURFs2wbuwXNRkQJeRAJNhru7VwO3AMuAjcBid19vZnPNbHLY7BYz\nW29m6wj63b/eahULAHPmQFVV3XFVVcF4ERFzT8+JLbFYzEtLS9Oy7CjIygq22Oszg6NH274eEWkb\nZrbG3WNNtdMvVNuphk420klIIgIK93Zr3jzo3r3uuO7dg/EiIgr3dqqwEBYuhEGDgq6YQYOC4cLC\ndFcmIpkgmR8xSYYqLFSYi0hi2nIXEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU\n7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEUFLhbmaTzGyTmW0xs9kJpt9uZhvM\nrMzMnjezQakvVUREktVkuJtZJ2ABcClQAEw3s4J6zV4FYu4+ClgCPJjqQkVEJHnJbLmPBba4+1Z3\nPwSUAFPiG7j7cnevCgdfBvJSW6aIiDRHMuHeH9geN1wRjmvIN4DnEk0wsyIzKzWz0srKyuSrFBGR\nZknpAVUzmwHEgH9LNN3dF7p7zN1jubm5qVy0iIjESeYeqjuAAXHDeeG4OszsYmAOMMHd/yc15YmI\nyPFIZst9NTDUzAabWVdgGrA0voGZnQX8BJjs7h+kvkwREWmOJsPd3auBW4BlwEZgsbuvN7O5ZjY5\nbPZvQA/g381snZktbWB2IiLSBpLplsHdnwWerTfue3GvL05xXSIi0gL6haqISAQp3EVEIkjhLiIS\nQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4S4sVF0N+\nPmRlBc/FxemuSESSuiqkSEOKi6GoCKrCO+hu2xYMAxQWpq8ukY5OW+7SInPmHAv2GlVVwXgRSR+F\nu7TIu+82b7yItA2Fu7TIwIHNGy8ibUPhLi0ybx507153XPfuwXgRSZ+kwt3MJpnZJjPbYmazE0y/\nwMzWmlm1mU1NfZmSqQoLYeFCGDQIzILnhQt1MFUk3Zo8W8bMOgELgEuACmC1mS119w1xzd4Frge+\n3ZJiDh8+TEVFBQcPHmzJbKSNZGdnk5eXR2FhF4W5SIZJ5lTIscAWd98KYGYlwBSgNtzdvTycdrQl\nxVRUVNCzZ0/y8/Mxs5bMSlqZu7N7924qKioYPHhwussRkXqS6ZbpD2yPG64IxzWbmRWZWamZlVZW\nVn5q+sGDB+nbt6+CvR0wM/r27au9LJEM1aYHVN19obvH3D2Wm5ubsI2Cvf3Q30okcyUT7juAAXHD\neeE4ERHJUMmE+2pgqJkNNrOuwDRgaeuWlZxUX9Nk9+7djB49mtGjR3PyySfTv3//2uFDhw4lNY8b\nbriBTZs2NdpmwYIFFKfoAiznn38+69atS8m8RCQ6mjyg6u7VZnYLsAzoBCxy9/VmNhcodfelZnYO\n8AzQG7jCzO519xGtWXhrXNOkb9++tUF5zz330KNHD7797bonALk77k5WVuLvxSeeeKLJ5cycOfP4\nChQRSVJSfe7u/qy7f87dh7j7vHDc99x9afh6tbvnufuJ7t63tYMd2vaaJlu2bKGgoIDCwkJGjBjB\nrl27KCoqIhaLMWLECObOnVvbtmZLurq6mpycHGbPns2ZZ57JeeedxwcffADA3XffzaOPPlrbfvbs\n2YwdO5YzzjiDl156CYD9+/fzla98hYKCAqZOnUosFmtyC/3JJ59k5MiRfP7zn+euu+4CoLq6muuu\nu652/Pz58wF45JFHKCgoYNSoUcyYMSPl60xE0qvdXhWyra9p8uabb/LLX/6SWCwGwP3330+fPn2o\nrq5m4sSJTJ06lYKCgjrv2bt3LxMmTOD+++/n9ttvZ9GiRcye/anfgOHuvPLKKyxdupS5c+fyxz/+\nkR/96EecfPLJPP3007z22muMGTOm0foqKiq4++67KS0tpVevXlx88cX8/ve/Jzc3lw8//JDXX38d\ngD179gDw4IMPsm3bNrp27Vo7TkSio91efqCtr2kyZMiQ2mAH+PWvf82YMWMYM2YMGzduZMOGDZ96\nT7du3bj00ksBOPvssykvL08476uvvvpTbV588UWmTZsGwJlnnsmIEY3vDK1atYqLLrqIfv360aVL\nF6699lpWrFjB6aefzqZNm5g1axbLli2jV69eAIwYMYIZM2ZQXFxMly5dmrUuRCTztdtwb+trmpx4\n4om1rzdv3swPf/hDXnjhBcrKypg0aVLC8727du1a+7pTp05UV1cnnPcJJ5zQZJvj1bdvX8rKyhg/\nfjwLFizgm9/8JgDLli3jpptuYvXq1YwdO5YjR46kdLnpoJuGiBzTbsM9ndc0+fjjj+nZsyef+cxn\n2LVrF8uWLUv5MsaNG8fixYsBeP311xPuGcQ799xzWb58Obt376a6upqSkhImTJhAZWUl7s5Xv/pV\n5s6dy9q1azly5AgVFRVcdNFFPPjgg3z44YdU1T+A0c7UHGDftg3cjx1gV8BLR9Vu+9whCPJ0XNNk\nzJgxFBQUMGzYMAYNGsS4ceNSvoxvfetbfO1rX6OgoKD2UdOlkkheXh7f//73ufDCC3F3rrjiCi67\n7DLWrl3LN77xDdwdM+OBBx6gurqaa6+9ln379nH06FG+/e1v07Nnz5R/hrbU2AF2XfdGOiJz97Qs\nOBaLeWlpaZ1xGzduZPjw4WmpJ9NUV1dTXV1NdnY2mzdv5ktf+hKbN2+mc+fM+j7OlL9ZVlawxV6f\nGRxt0RWPRDKLma1x91hT7TIrKaTWJ598whe/+EWqq6txd37yk59kXLBnkoEDg66YRONFOiKlRYbK\nyclhzZo16S6j3Zg3r+6P2kA3DZGOrd0eUBWJp5uGiNSlLXeJjHQdYBfJRNpyF0khnWsvmUJb7iIp\n0hoXsxM5XtpyjzNx4sRP/SDp0Ucf5eabb270fT169ABg586dTJ2a+P7gF154IfVP/azv0UcfrfNj\noi9/+cspue7LPffcw0MPPdTi+Ujj2vJidiJNUbjHmT59OiUlJXXGlZSUMH369KTef+qpp7JkyZLj\nXn79cH/22WfJyck57vlJ22rri9k1Rt1DkrHdMrfdBqm+B8Xo0RBeaTehqVOncvfdd3Po0CG6du1K\neXk5O3fuZPz48XzyySdMmTKFjz76iMOHD3PfffcxZcqUOu8vLy/n8ssv54033uDAgQPccMMNvPba\nawwbNowDBw7Utrv55ptZvXo1Bw4cYOrUqdx7773Mnz+fnTt3MnHiRPr168fy5cvJz8+ntLSUfv36\n8fDDD7No0SIAbrzxRm677TbKy8u59NJLOf/883nppZfo378/v/vd7+jWrVuDn3HdunXcdNNNVFVV\nMWTIEBYtWkTv3r2ZP38+P/7xj+ncuTMFBQWUlJTw5z//mVtvvRUIbqm3YsWKdv9L1taUKefaq3tI\nQFvudfTp04exY8fy3HPPAcFW+zXXXIOZkZ2dzTPPPMPatWtZvnw5d9xxB439uvexxx6je/fubNy4\nkXvvvbfOOevz5s2jtLSUsrIy/vznP1NWVsasWbM49dRTWb58OcuXL68zrzVr1vDEE0+watUqXn75\nZX7605/y6quvAsFFzGbOnMn69evJycnh6aefbvQzfu1rX+OBBx6grKyMkSNHcu+99wLBJYxfffVV\nysrK+PGPfwzAQw89xIIFC1i3bh0rV65s9EtD2v5idg3JpO4h7UGkT8ZuuTe2hd2aarpmpkyZQklJ\nCT/72c+A4Jrrd911FytWrCArK4sdO3bw/vvvc/LJJyecz4oVK5g1axYAo0aNYtSoUbXTFi9ezMKF\nC6murmbXrl1s2LChzvT6XnzxRa666qraK1NeffXVrFy5ksmTJzN48GBGjx4NNH5ZYQiuL79nzx4m\nTJgAwNe//nW++tWv1tZYWFjIlVdeyZVXXgkEFy+7/fbbKSws5OqrryYvLy+ZVdhh1WwVz5kTdMUM\nHBgEe1tvLWdK91Am7UEUF6f/79LWtOVez5QpU3j++edZu3YtVVVVnH322QAUFxdTWVnJmjVrWLdu\nHZ/97GcTXua3Ke+88w4PPfQQzz//PGVlZVx22WXHNZ8aNZcLhpZdMvgPf/gDM2fOZO3atZxzzjlU\nV1cze/ZsHn/8cQ4cOMC4ceN48803j7vOjqKwEMrLg+vZlJenJ0Da+l4HDcmUPYhMumJoW+7JJBXu\nZjbJzDaZ2RYz+9SthMzsBDP7TTh9lZnlp7rQttKjRw8mTpzIP/zDP9Q5kLp3715OOukkunTpwvLl\ny9mWqHM1zgUXXMBTTz0FwBtvvEFZWRkQXC74xBNPpFevXrz//vu1XUAAPXv2ZN++fZ+a1/jx4/nt\nb39LVVUV+/fv55lnnmH8+PHN/my9evWid+/erFy5EoBf/epXTJgwgaNHj7J9+3YmTpzIAw88wN69\ne/nkk094++23GTlyJHfeeSfnnHOOwr2dyJTuoUzZg+ioXzJNdsuYWSdgAXAJUAGsNrOl7h5/gfFv\nAB+5++lmNg14APj71ii4LUyfPp2rrrqqzpkzhYWFXHHFFYwcOZJYLMawYcMancfNN9/MDTfcwPDh\nwxk+fHjtHsCZZ57JWWedxbBhwxgwYECdywUXFRUxadKk2r73GmPGjOH6669n7NixQHBA9ayzzmq0\nC6Yhv/iKUo6yAAAEUUlEQVTFL2oPqJ522mk88cQTHDlyhBkzZrB3717cnVmzZpGTk8N3v/tdli9f\nTlZWFiNGjKi9q5RktkzpHsqUA8zt4UumNf42TV7y18zOA+5x978Lh78D4O4/iGuzLGzzVzPrDLwH\n5HojM9clf6NBfzNpSP0+dwj2INr6mj/5+Ym/ZAYNCrrO2kqqLkud7CV/k+mW6Q9sjxuuCMclbOPu\n1cBeoG+CoorMrNTMSisrK5NYtIi0V5lyMbdM6aZq62MhbXpA1d0XunvM3WO5ubltuWgRSYNMOMDc\nUb9kkjkVcgcwIG44LxyXqE1F2C3TC9h9PAXV3A5OMl+67uIl0lyZcMXQtj4WksyW+2pgqJkNNrOu\nwDRgab02S4Gvh6+nAi801t/ekOzsbHbv3q3QaAfcnd27d5OdnZ3uUkTajbbck2lyy93dq83sFmAZ\n0AlY5O7rzWwuUOruS4GfAb8ysy3A3wi+AJotLy+PiooK1B/fPmRnZ+uHTSIZKqNukC0iIo1L5dky\nIiLSzijcRUQiSOEuIhJBaetzN7NKoPELtDSsH/BhCstp77Q+6tL6OEbroq4orI9B7t7kD4XSFu4t\nYWalyRxQ6Ci0PurS+jhG66KujrQ+1C0jIhJBCncRkQhqr+G+MN0FZBitj7q0Po7Ruqirw6yPdtnn\nLiIijWuvW+4iItIIhbuISAS1u3Bv6n6uHYWZDTCz5Wa2wczWm9mt6a4pE5hZJzN71cx+n+5a0s3M\ncsxsiZm9aWYbw7uqdUhm9k/h/5M3zOzXZhb5y5m2q3CPu5/rpUABMN3MCtJbVdpUA3e4ewHwBWBm\nB14X8W4FNqa7iAzxQ+CP7j4MOJMOul7MrD8wC4i5++cJrm57XFeubU/aVbgDY4Et7r7V3Q8BJcCU\nNNeUFu6+y93Xhq/3EfzHrX/7ww7FzPKAy4DH011LuplZL+ACgstx4+6H3H1PeqtKq85At/BmQt2B\nnWmup9W1t3BP5n6uHY6Z5QNnAavSW0naPQr8C9CM2w1H1mCgEngi7KZ63MxOTHdR6eDuO4CHgHeB\nXcBed/+v9FbV+tpbuEs9ZtYDeBq4zd0/Tnc96WJmlwMfuPuadNeSIToDY4DH3P0sYD/QIY9RmVlv\ngj38wcCpwIlmNiO9VbW+9hbuydzPtcMwsy4EwV7s7v+R7nrSbBww2czKCbrrLjKzJ9NbUlpVABXu\nXrM3t4Qg7Duii4F33L3S3Q8D/wH8rzTX1OraW7gncz/XDsGCu4j/DNjo7g+nu550c/fvuHueu+cT\n/Lt4wd0jv3XWEHd/D9huZmeEo74IbEhjSen0LvAFM+se/r/5Ih3g4HKT91DNJA3dzzXNZaXLOOA6\n4HUzWxeOu8vdn01jTZJZvgUUhxtCW4Eb0lxPWrj7KjNbAqwlOMvsVTrAZQh0+QERkQhqb90yIiKS\nBIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSC/j8V/tiI4+5lRgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7ff687df5048>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Validation accuracy stalls in the low 50s. So in our case, pre-trained word embeddings does outperform jointly learned embeddings. If you \n",
    "increase the number of training samples, this will quickly stop being the case -- try it as an exercise.\n",
    "\n",
    "Finally, let's evaluate the model on the test data. First, we will need to tokenize the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_dir = os.path.join(imdb_dir, 'test')\n",
    "\n",
    "labels = []\n",
    "texts = []\n",
    "\n",
    "for label_type in ['neg', 'pos']:\n",
    "    dir_name = os.path.join(test_dir, label_type)\n",
    "    for fname in sorted(os.listdir(dir_name)):\n",
    "        if fname[-4:] == '.txt':\n",
    "            f = open(os.path.join(dir_name, fname))\n",
    "            texts.append(f.read())\n",
    "            f.close()\n",
    "            if label_type == 'neg':\n",
    "                labels.append(0)\n",
    "            else:\n",
    "                labels.append(1)\n",
    "\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "x_test = pad_sequences(sequences, maxlen=maxlen)\n",
    "y_test = np.asarray(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's load and evaluate the first model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24736/25000 [============================>.] - ETA: 0s"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.93747248332977295, 0.53659999999999997]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.load_weights('pre_trained_glove_model.h5')\n",
    "model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get an appalling test accuracy of 54%. Working with just a handful of training samples is hard!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}