09.06-14_[End-to-End_Example]_Recommender_Take_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Helpful Links: Where the Data Lives**\n",
    "\n",
    "Open Academic Society: [Project Page](https://www.openacademic.ai/oag/)\n",
    "\n",
    "Microsoft Research: [MS Academic Graph](https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sys import getsizeof"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_df = pd.read_json('data/mag_papers_0/mag_subset20K.txt', lines=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10399, 6)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# filter out non-English articles\n",
    "# keep abstract, authors, fos, keywords, year, title\n",
    "\n",
    "model_df = model_df[model_df.lang == 'en'].drop(['doc_type', 'doi', 'id', 'issue', 'lang', 'n_citation', 'page_end', \n",
    "                            'page_start', 'publisher', 'references', 'url', 'venue', 'volume'], axis=1)\n",
    "\n",
    "model_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start with a simple example of building a recommender with just a few fields, building sparse arrays of available features to calculate for the Jaccard similary between papers. We will see if reasonably similar papers can be found in a timely manner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (2) engineering our current features, pipe, outcome"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numerical features broadly distributed across a dataset can unnecessarily increase the size of our feature space. Let's wrangle this in first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19994    1951\n",
       "19995    2017\n",
       "19997    1971\n",
       "19998    1986\n",
       "19999    2015\n",
       "Name: year, dtype: int64"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_df['year'].tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Year spread:  1831  -  2017\n",
      "Quantile spread:\n",
      " 0.25    1990.0\n",
      "0.50    2005.0\n",
      "0.75    2012.0\n",
      "Name: year, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print(\"Year spread: \", model_df['year'].min(),\" - \", model_df['year'].max())\n",
    "print(\"Quantile spread:\\n\", model_df['year'].quantile([0.25, 0.5, 0.75]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Occurrence')"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEQCAYAAACN2GLgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3X+YXVV97/H3R4KEZhLk5yBYiPzQ\n2ECDZiy3eoFEREEvlRrqA0QhcEsQCmrhuZbbCyQiFAHLU3qL0PDDCCqhWLAKV6zQBBDUSuoDcSDS\n8iNIJBAghkyAgPC9f6x1YM/J/DqTvfecmXxez3OezFlrr72/Z2fO+c7aa5+1FBGYmZmV7S0jHYCZ\nmY1NTjBmZlYJJxgzM6uEE4yZmVXCCcbMzCrhBGNmZpVwgjEzs0o4wZiZWSWcYMzMrBLj6jiIpJ6m\noq2Br0XEabn+YOAyYDfgZ8CciFiR67YCLgeOBF4ELoqISwY75g477BCTJ0/uVbZ+/XomTJiwaS+m\nYo6xHI6xHI6xHO0eYzG+pUuXPhsRO5ay44io9QFMAHqAA/PzHYC1wJ8B44GLgZ8Wtr8AuBvYFngP\nsAo4dLDjTJ8+PZotXrx4o7J24xjL4RjL4RjL0e4xFuMD7ouSPu9H4hLZkcAzOWkAfBLojogbI+Jl\nYD4wTdKUXH8s8OWIWBMRDwFXAnPqDdnMzFo1EgnmOODanCkBpgL3NyojYj3wCDBV0rbALsX6/PPU\nmmI1M7Nh0puf8zUcTNoNeAzYKyIey2VXA6sj4szCdveQeip3AE8AW+feDZIOAa6MiMl97H8uMBeg\ns7Nz+qJFi3rV9/T00NHRUcErK49jLIdjLIdjLEe7x1iMb+bMmUsjoquUHZd1rW0oD+As4M6msktJ\nA/7FsmXALNK4SwA7FepmAcsGO5bHYKrjGMvhGMvhGDfdWBmDORb4RlNZNzCt8UTSBGBP0rjMGuCp\nYn3+ubviOM3MbBPVlmAkfQDYFbixqepmYB9JsySNB84BHoiI5bn+WuAsSdvmgf8TgYU1hW1mZsNU\nZw/mOOCmiFhXLIyI1aTLXucDa4D9gaMKm8wjDfqvAO4ELo6I22qJ2MzMhq2WL1oCRMRJA9TdDkzp\np24DcEJ+mJnZKOGpYszMrBJOMGZmY8TkM28d6RB6cYIxM7NKOMGYmVklnGDMzKwSTjBmZlYJJxgz\nM6uEE4yZmVXCCcbMzCrhBGNmZpVwgjEzs0o4wZiZWSWcYMzMrBJOMGZmVgknGDMzq4QTjJmZVcIJ\nxszMKuEEY2ZmlXCCMTOzSjjBmJlZJZxgzMysErUmGElHSXpI0npJj0g6IJcfLGm5pBclLZa0e6HN\nVpKukfSCpFWSTq8zZjMzG57aEoykQ4ALgeOBicCBwKOSdgBuAs4GtgPuA24oNJ0P7A3sDswEvijp\n0LriNjOz4amzB/Ml4NyI+GlEvB4RKyNiJfBJoDsiboyIl0kJZZqkKbndscCXI2JNRDwEXAnMqTFu\nMzMbhloSjKQtgC5gR0n/JelJSf8gaWtgKnB/Y9uIWA88AkyVtC2wS7E+/zy1jrjNzGz4FBHVH0Ta\nBVgJLAUOB14F/gVYAuwMrI6IMwvb30PqqdwBPAFsnXs3jUttV0bE5D6OMxeYC9DZ2Tl90aJFvep7\nenro6Ogo+dWVyzGWwzGWwzGWo64Yl61cy767btNyu2J8M2fOXBoRXaUEFBGVP4BtgQCOK5TNAn4B\nXAp8rWn7Zbm+0W6npnbLBjvm9OnTo9nixYs3Kms3jrEcjrEcjrEcdcW4+1/dMqx2xfiA+6Kkz/5a\nLpFFxBrgyZwsmnUD0xpPJE0A9iSNy6wBnirW55+7q4vWzMzKUOcg/9eB0yTtlMdWvgDcAtwM7CNp\nlqTxwDnAAxGxPLe7FjhL0rZ54P9EYGGNcZuZ2TDUmWC+DPwceBh4iHR57PyIWE267HU+sAbYHziq\n0G4eadB/BXAncHFE3FZj3GZmNgzj6jpQRLwKnJIfzXW3A1M2apTqNgAn5IeZmY0SnirGzMwq4QRj\nZmaVcIIxM7NKOMGYmVklnGDMzKwSTjBmZlYJJxgzM6uEE4yZmVXCCcbMzCrhBGNmZpVwgjEzs0o4\nwZiZWSWcYMzMrBJOMGZmVgknGDMzq4QTjJmZVcIJxszMKuEEY2ZmlXCCMTOzSowb6QDMzGzTTD7z\n1pEOoU+19WAkLZH0sqSe/PhVoe4YSSskrZf0XUnbFeq2k3Rzrlsh6Zi6YjYzs+GruwdzakRcVSyQ\nNBX4R+DjwH8AC4CvAUflTS4DXgE6gf2AWyXdHxHdtUVtZtaG2rXn0tAOl8hmA9+PiLsAJJ0NPCRp\nIvA6MAvYJyJ6gB9L+h7wGeDMkQrYzMwGV/cg/wWSnpV0j6QZuWwqcH9jg4h4hNRjeVd+vBYRDxf2\ncX9uY2ZmbUwRUc+BpP2BB0nJ4yjgH0iXvBYAN0bEFYVtV5J6Nq/lup0LdScCsyNiRh/HmAvMBejs\n7Jy+aNGiXvU9PT10dHSU+8JK5hjL4RjL4RjLUUWMy1au7bN83123aXlfxfhmzpy5NCK6Nim4rLZL\nZBHxs8LTb0g6GvgY0ANMatp8ErCOdImsv7q+jrGAlLDo6uqKGTNm9KpfsmQJzWXtxjGWwzGWwzGW\no4oY5/Qz/vL47NaPU9U5HMnvwQQgoBuY1iiUtAewFfBwfoyTtHeh3bTcxszM2lgtCUbS2yR9VNJ4\nSeMkzQYOBH4IfAs4XNIBkiYA5wI3RcS6iFgP3AScK2mCpA8CnwCuqyNuMzMbvroukW0JnAdMIY2r\nLAeOiIhfAUj6LCnRbA/cDhxfaHsKcA3wDPAccLJvUTYza3+1JJiIWA28f4D6bwPf7qfueeCIikIz\nM7OKeC4yMzOrhBOMmZlVwgnGzMwq4QRjZmaVcIIxM7NKOMGYmVklWk4wkn5f0n+rIhgzMxs7hpxg\nJO0m6R7SlyRvz2VHSrpq4JZmZrY5aqUH84/ArcBE4NVc9iPgkLKDMjOzN00+89a2X1ysL618k/+P\ngI9HxOuSAiAi1kpqfW5oMzMb81pJME8De5FmOAZA0h8AT5QdlJmZbWy09WJauUT2VeAWSceTptA/\nGrgBuLCSyMzMbFQbcg8mIq6R9DxpxchfA8cCZ0fEd6sKzszMRq+WZlPOycQJxczMBtXKbcp/L+kD\nTWUfkPR35YdlZmajXStjMEcD9zWVLQWOKS8cMzMbK1pJMNHH9lu0uA8zM9tMtJIc7gbOk/QWgPzv\n/FxuZmbWSyuD/J8HbgGekrQC2A14Cji8isDMzGx0a+U25SclvQ/YH3gH6Vblf4+I16sKzszMRq9W\nb1N+HfhJ4zIZpEtlTjJmZtaslduU3yfpJ5LWkya7fBX4HW9OfDnU/ewt6WVJ3yyUHSNphaT1kr4r\nabtC3XaSbs51KyT5rjUzs1GglUH+bwCLgS5gj/x4Z/63FZcBP288kTSVNFPzZ4BO4EXga03bv5Lr\nZgOX5zZmZtbGWrlEtjvwfyIihnswSUcBvwXuJU2cCSlpfD8i7srbnA08JGki8DowC9gnInqAH0v6\nHikZnTncOMzMrHqt9GBuBj4y3ANJmgScC5zRVDUVuL/xJCIeIfVY3pUfr0XEw4Xt789tzMysjWmo\nHRJJN5BuSf4xsKpYFxHHDqH9pcBvIuJCSfOBvSLi05LuAG6MiCsK264k9Wxey3U7F+pOBGZHxIw+\njjGXNBknnZ2d0xctWtSrvqenh46OjiG93pHiGMvhGMvhGMuxqTEuW7l2yNvuu2vrS3QV45s5c+bS\niOhqeSd9aOUS2YP50TJJ+wEfBt7bR3UPMKmpbBKwjnSJrL+6jUTEAmABQFdXV8yYMaNX/ZIlS2gu\nazeOsRyOsRyOsRybGuOcFtaBeXx268ep6hy28j2YL23CcWYAk4EnJAF0AFvkBctuA6Y1NpS0B7AV\naWGz10lrz+wdEf+ZN5kGdG9CLGZmVoOW5hGTdIikqyV9Pz/vkvShITRdAOwJ7JcfVwC3Ah8FvgUc\nLukASRNI4zQ3RcS6iFgP3AScK2mCpA8CnwCuayVuM7PRZrStXtmXIfdgJJ1Gmi7mKuDIXPwS8PfA\nB/prBxARL5JuP27sqwd4OSJWA6slfZaUaLYHbgeOLzQ/BbgGeAZ4Djg5ItyDMbMxb7QnmVbGYL4A\nHBwRj0v6q1y2HHh3qweNiPlNz78NfLufbZ8Hjmj1GGZmNrJauUQ2kTT/GKSp+wG2JN1SbGZm1ksr\nCeYuNv5y4+dI3+43MzPrpZVLZKcB38/fQ5ko6VfAC3i6fjMz60MrCeZp4P35sTuert/MzAYwpAQj\naQvSFyLfFhH/Dvx7pVGZmdmoN6QxmIh4jfTFx+2rDcfMzMaKVi6RfQu4Jc8p9iRv3klGRPxb2YGZ\nmdno1kqCOTn/O7+pPGh9TRgzMxvjWkkwe+VLZWZmZoMa0hhMY5Bf0lYVx2NmZmOEB/nNzKwSHuQ3\nM7NKeJDfzKyNjPYZlItaWXDsnVUGYmZmY0tLC46ZmZkNVSsLjv2awrhLUUTsVlpEZmY2JrQyBvPp\npudvJ61wuai8cMzMbKxoZQzmzuYySUuA24BLS4zJzMzGgE0dg9kAePDfzMw20soYzLlNRb8HfAz4\nQakRmZnZmNDKGMzvNz1fD1wCXFdeOGZmNla0MgZz/KYcSNI3gYOBCcAq4KKIuCrXHQxcBuwG/AyY\nExErct1WwOXAkcCLud0lmxKLmZlVb8hjMJLOlPT+prI/kvTFIe7iAmByREwC/gQ4T9J0STsANwFn\nA9sB9wE3FNrNB/YmLdM8E/iipEOHGreZmY2MVgb5Pw882FT2IPCFoTSOiO6I2NB4mh97Ap8EuiPi\nxoh4mZRQpkmakrc9FvhyRKyJiIeAK4E5LcRtZmYjQBF9fndy4w2l54C3R8QrhbK3AqsiYrsh7uNr\npOSwNfAL4EDgfOCtEXFyYbtfAvOAfwOeB3aOiKdz3ZHAvIjYt4/9zwXmAnR2dk5ftKj3V3R6enro\n6OgY0usdKY6xHI6xHI6xHK3EuGzl2k061r67btNym2J8M2fOXBoRXZsURNbKIP9S4BTg7wplnwX+\nY6g7iIhTJJ0G/DEwg3SbcwewumnTtcDEXNd43lzX1/4XAAsAurq6YsaMGb3qlyxZQnNZu3GM5XCM\n5XCM5WglxjmbONnl47OHdpyiqs5hKwnmL4EfSfoM8AiwF9AJHNLKAfPaMj+W9GnSDM09wKSmzSYB\n63Jd4/nLTXVmZtbGhjwGExHdwLuAi4GfAxcB746I5nGZoRpHGoPpBqY1CiVNaJRHxBrgqWJ9/rl7\nmMc0M7OatHIX2a7AlhGxKCIujohFwJaSdhlC250kHSWpQ9IWkj4KHE0aY7kZ2EfSLEnjgXOAByJi\neW5+LXCWpG3zwP+JwMKWXqWZmdWulbvIvgu8o6nsHaQEMZggXQ57ElgDfBX4QkT8S0SsBmaRBvvX\nAPsDRxXaziNdklsB3AlcHBG3tRC3mVlbG0uLjBW1MgbzrohYViyIiGWF24n7lZPIQQPU3w70uZ98\na/MJ+WFmNiaNxSTTSg9mtaS9igX5+XPlhmRmZmNBKwnmGuCfJR0u6Q8kHQ58B7iqmtDMzGw0a+US\n2VeAV0h3kb0D+DVwNWnCSzMzs16GlGAkjSOtaPle4AnSlytvB66LiNerC8/MzEarQS+RSdoGuBe4\nEHiV9I3+V0iTV96b683MzHoZSg/mAtJULjMjYn2jMH8h8p9y/SnVhGdmZqPVUAb5jwBOLiYXgPz8\nL4A/rSIwMzMb3YaSYLYBVvZT9yQbzyNmZmY2pEtkjwAfAn7UR93BwKOlRmRmtpkYi1+uLBpKD+YS\n4No8V9hbACS9Ja/LshDfpmxmZn0YtAcTEQslbU9KJtdLehbYgbSWy7kR8fVqQzQzs9FoSN+DiYi/\nlbQA+AApuTwL/CQiXqgyODMzG72G/E3+iFgH/LDCWMzMbAxpZS4yMzOzIXOCMTOzSjjBmJnVaKzf\nmlzkBGNmZpVoZbp+MzMrwebSi3EPxszMKuEEY2ZmlaglwUjaStLVklZIWifpF5IOK9QfLGm5pBcl\nLZa0e1PbayS9IGmVpNPriNnMzDZNXT2YcaQllg8izc58NvBPkiZL2gG4KZdtB9wH3FBoOx/YG9gd\nmAl8UdKhNcVtZmbDVMsgf147Zn6h6BZJjwHTge2B7oi4EUDSfOBZSVMiYjlwLHB8RKwB1ki6EpgD\n3FZH7GZmNjwjMgYjqRN4F9ANTAXub9TlZPQIMFXStsAuxfr889T6ojUzs+FQRNR7QGlL4AfAIxFx\nkqSrgdURcWZhm3uAK4E7gCeArSPi5Vx3CHBlREzuY99zgbkAnZ2d0xctWtSrvqenh46OjkpeV1kc\nYzkcYzkcYzmKMS5bubbSY+276zYttynGN3PmzKUR0VVGLLV+DyavJ3Md8Apwai7uYeNVMScB63Jd\n4/nLTXUbiYgFwAKArq6umDFjRq/6JUuW0FzWbhxjORxjORzjpml832XhoR3Mua2x6ny1H7uPz57R\ncpuqzmFtl8gkCbga6ARmRcSruaobmFbYbgKwJ2lcZg3wVLE+/9xdS9BmZjZsdY7BXA68Bzg8Il4q\nlN8M7JNXzBwPnAM8kAf4Aa4FzpK0raQpwImkxc/MzKyN1fU9mN2Bk4D9gFWSevJjdkSsBmYB5wNr\ngP2BowrN55EG/VcAdwIXR4TvIDMza3N13aa8AtAA9bcDU/qp2wCckB9mZqNOGtjf/KZ+9FQxZmZW\nCScYMzOrhBOMmZlVYvO7KGhmVrHNZb2XwbgHY2ZWIieXNznBmJlZJXyJzMysBO65bMw9GDMzq4R7\nMGZmm8A9l/65B2NmNkxOLgNzgjEzs0o4wZiZWSWcYMzMBjD5zFvfuBRW/NkG5wRjZmaV8F1kZmYt\nci9maNyDMTOzSjjBmJlZJZxgzMysEk4wZmZWCScYM7N+eDB/0zjBmJlZJWpLMJJOlXSfpA2SFjbV\nHSxpuaQXJS2WtHuhbitJ10h6QdIqSafXFbOZWYN7M62rswfzG+A84JpioaQdgJuAs4HtgPuAGwqb\nzAf2BnYHZgJflHRoDfGa2WakmED8jf1y1JZgIuKmiPgu8FxT1SeB7oi4MSJeJiWUaZKm5PpjgS9H\nxJqIeAi4EphTU9hmZjZMioh6DyidB7wjIubk55cCb42Ikwvb/BKYB/wb8Dywc0Q8neuOBOZFxL59\n7HsuMBegs7Nz+qJFi3rV9/T00NHRUcXLKo1jLIdjLMfmEOOylWvf+HnfXbfZqKwMnVvD0y+Vust+\nNV5DK4rncObMmUsjoquMWNphqpgOYHVT2VpgYq5rPG+u20hELAAWAHR1dcWMGTN61S9ZsoTmsnbj\nGMvhGMuxOcQ4p3Ap7PHZMzYqK8MZ+/6Ov11Wz8dt4zW0oqr/53a4i6wHmNRUNglYl+toqm/UmZlZ\nG2uHHkw3cFzjiaQJwJ6kcZk1kp4CpgE/yptMy23MzIbNg/jVq/M25XGSxgNbAFtIGi9pHHAzsI+k\nWbn+HOCBiFiem14LnCVp2zzwfyKwsK64zcxseOrswZxFGrhv+DTwpYiYL2kW8A/AN4GfAUcVtpsH\nXA6sAF4CLoyI2+oJ2cxGk0av5PGvfHyjsuZyq15tCSYi5pNuQe6r7nZgSj91G4AT8sPMbNj6SkB9\n1Vs52mGQ38zMxiAnGDMb1cr+zoqVxwnGzEaVoUzj4ktd7cEJxsw2O05A9WiH78GYmW2SvhKGk8jI\ncw/GzNqWZzUe3dyDMbMRM9Btw83T5/dXf8ZG095au3APxszMKuEejJm1DV8OG1vcgzEzs0o4wZiZ\nWSV8iczMRpwvjY1N7sGYWS2abzl2Uhn7nGDMzKwSTjBmVqqhzBPm3svmwQnGzErnJGLgBGNmLWoe\nR3Eysf74LjIz28ib07D8jhlNZYO1MWtwgjGzlrWaTJx8Nk9OMGajyGBrype1/8HKzIZiVIzBSNpO\n0s2S1ktaIemYkY7JrEx1jGOMRKLw+MzmbbT0YC4DXgE6gf2AWyXdHxHdIxuWWbkmn3nrG72T4gfz\ncHosffV2BiozK1vbJxhJE4BZwD4R0QP8WNL3gM8AZ45ocLZZKX44D+VSVTFZ9LWfM/b9HXOGeEmq\nuay/RDRQ3GZ1a/sEA7wLeC0iHi6U3Q8cVNUBB3pDFj9cGs/7azvQh0vzh1Tzh05/f3VuyjX4vvbT\n/NoGa9sc40D7GeoHW3/76SuuwV5Dc4xVaGW6k+b6MsdONiVxOOlYHRQRIx3DgCQdANwYETsXyk4E\nZkfEjKZt5wJz89N3A79q2t0OwLPVRVsKx1gOx1gOx1iOdo+xGN/uEbFjGTsdDT2YHmBSU9kkYF3z\nhhGxAFjQ344k3RcRXeWGVy7HWA7HWA7HWI52j7Gq+EbDXWQPA+Mk7V0omwZ4gN/MrI21fYKJiPXA\nTcC5kiZI+iDwCeC6kY3MzMwG0vYJJjsF2Bp4BrgeOHmYtyj3e/msjTjGcjjGcjjGcrR7jJXE1/aD\n/GZmNjqNlh6MmZmNMk4wZmZWiVGZYCSdKuk+SRskLWyq+5SkhyStk/SgpCMKdcdJWirpBUlPSrpI\n0rhC/RJJL0vqyY/m79HUEeMcSa8VYuiRNKNQP1nSYkkvSlou6cMjEOMVTfFtkLSuUF/XefxzSf+V\nj3GbpF0KdZJ0oaTn8uMiSSrU75d/F17M/+43AjH+L0m/zOf4MUn/q6nt45JeKpzHfx2BGOdLerXp\n/3uPQn07nMcfNMX3iqRlhfpSzqOkrSRdrTQf4jpJv5B0WKH+4PyefDG/R3dvanuN0mfPKkmnN+27\n37Y1xvhVSf+Z2y2XdGzTvkNpPsjGebxq0IAiYtQ9gE8CRwCXAwsL5buS5iw7DBDwceBFYKdcfzJw\nAPDWvO1S4MxC+yXAn49wjHOAHw+w358Al5BuepgF/BbYsc4Y+9jPQuCams/jQaSbPqbm/8/LgTsL\n9SeRvmj7jvx6HgQ+m+veCqwA/hLYCvhcfv7WmmP8IvA+0vfR3p1jOKpQ/zjw4RE+j/OBb/azz7Y4\nj33sZwlwTtnnEZiQz8dk0h/n/4P0fbzJpC8qrgX+DBgPXAz8tND2AuBuYFvgPcAq4NBcN2DbGmP8\nEjAlt9sfWAN8oFAfwF4txVPGL+9IPYDzmn4R9weeadpmNfDH/bQ/Hfh+0y9mKR+Mw42RARIMadqc\nDcDEQtnd5A/OkTiP+Rd6HXBQzefxq8Blhee75DfAnvn5vcDcQv3/bLyZgI8AK8k3ueSyJxpv+Lpi\n7KP93wP/t/D8cUpKMJtwHufTf4Jpu/NI+iB9DXhnleexsO8HSH/ozQXuLZRPAF4CpuTnK4GPFOq/\nDCzKPw/Ytq4Y+2j3PeCMwvOWE8yovEQ2gPuAhyT9iaQtlC7rbCCd4L4cyMZf2LxA0rOS7lHh0lTN\nMb43x/CwpLP15mW8qcCjEVGcxeD+XF53jA2zSMnnrqbyqs+j8qP4HGCf/O9U0rlpKJ6nqcADkd81\n2QOUfx4Hi/HNCkmk3nXz7+O3JK2W9K+SppUc31BjPFzS85K6JZ1cKG+78wgcC9wdEY81lZd+HiV1\nkv7o66bp9y3S9/ceAaZK2paUFAf6feyzbV0x9tFua+D9bPz7eFe+xHeTpMmDHX9MJZiIeA24Fvg2\n6QPx28BJ+UT2Iul4oIv011HDXwF7kC6pLAC+L2nPmmO8i/TG2Yn04X000Lg230Hq4hatBSbWHGPR\nccC1TR8ylZ9H4P8Bn5L0h/nNcA7pL6zfy/XN52ot0JE/yGs5j0OIsWg+6f349ULZbNJf5LsDi4Ef\nSnpbzTH+E+mSzo7AicA5ko7Ode14Ho8lXbItKv08StoS+BbwjYhYzsDnoqPwvLmOQdrWFWOzK0jJ\n6IeFsoNI53EK8BvglsIfv30aUwlGacD7ImAG6VrtQcBVzQOP+S/yrwCHRcQbE9BFxM8iYl1EbIiI\nbwD3AB+rM8aIeDQiHouI1yNiGXAucGRuPuR52aqMsbDd7+e6a4vldZzHiLgDmAf8M+m6/+Ok8/Bk\n3qT5XE0CenIirOU8DiFGIA1ukz4YPx4RGwrt74mIlyLixYi4gDTedkCdMUbEgxHxm4h4LSLuBS6l\n5t/HFs7jfwd2Br7T1L7U8yjpLaSZRF4BTs3FA52LnsLz5rrB2tYVY7HtxaQ/cj9V/MMxIu6KiFci\n4rfA54F3kv746NeYSjCkxcjuioj78gf0z4GfAW/caSXpUOBK4PD8AT6QoHfXvJYYB4ihG9hDUvEv\njirmZRtqjMeSruk+Osj+qjiPRMRlEbF3ROxE+vAZB/wyV3eTzk1D8Tx1A3+YezMNf0gF89sNEiOS\nTiCta3RwRDzZz27e2B31n8eBYmib85gdB9wUad2oAXfHMM9jfq1XkxY/nBURr+aqXr9vSutY7Ql0\nR8Qa4CkG/n3ss21dMRbKvkS6uecjEfHCIIca/DyWMYhU94P0yzWedGfGdfnncaS/pp8F9svbvRd4\nLp8sgA/l5wf2sc+3AR8t7Gs2sB54d80xHgZ05p+nkN5E8wr7/Snpst544E/ZtLvIhhVjof2vgBNG\n6DyOJ/2VJWA30o0Ff1No91ngIdJlul1Ib6Lmu8g+T7r76VQ27e6n4cY4m3Q30Xv62OduwAdzrONJ\nl0lXA9vXHOMnSHc+Cfgj0mD1ce10HnPbrfN74UMVn8crSO/BjqbyHUmXm2bl41xI7zu0vgLcmc/l\nFFLCOXQobWuM8X8D/wm8vY99TiX94bkF6VLb35He/1sOGMtwX8RIPkjXq6PpMT/XnQr8F6nb9yi9\n74JYDPyO1FVsPH5QOPk/z+1+m/+DDhmBGL8KPE36UH6UdIlsy0L95PwGeyn/Bw/77pjhxpjr/zjH\nOLGpvJbzSEpkD+QYVpE+lLYotBPpMt/z+XERve92ei/pNvWXgP8A3jsCMT4GvNr0+3hFrptaaPsc\ncAfQNQIxXp+P3wMsBz7XtN8RP4+57dGk5Kam8tLOI2kMJ4CXm/7PZuf6D+dz9BLpPTq50HYr4Brg\nBdL7+/SmfffbtsYYgzTmWmz317nuQ6TPm/WkW8a/C+w9WDyei8zMzCox1sZgzMysTTjBmJlZJZxg\nzMysEk4wZmZWCScYMzOrhBN7Zbi6AAADKklEQVSMmZlVwgnGzMwq4QRjmxVJ35J0TVPZQUqLkr29\nphgOk3R3XtjpGaUF2j5ew3GfrGhma7M+OcHY5uZzwMckHQIgaTxpbrozIuKpMg8kaYs+yo4CbiB9\nq3tX4O2khZ7+pMxjm7UDJxjbrETEc8BpwII82d884JGIWAhpFlpJfy3pkbyezaK8nkej7jt5PYzf\n5p7HG7PJSvqmpMuUlvRdT9OMvXmG278lzS339Yh4IdIsxYsj4qTCMc5RWvL2GUkLJU3KdR+W9HjT\nPt/olUg6T9L1OY51Sssxvy/XXU+ak62xvHCvJXvNquAEY5udiLiRNH/W9aRV/k4qVJ9OWiL6QNJy\ny+tJK0023ALsTZoW/pekSRmLjiH1SCaSlrcu+gPSh/x36N+fA58mLZWwJ2lyxEuH9MKSI3JMbwN+\n0Ig9Io4mreFxWER0RMQlLezTbFicYGxz9RekCfzOjYgnCuUnkSb4WxkRL5MmW/yUpLdEWrpgYaS1\nbhp103NPqOHmiPhJ3nYDvW2f/x3oUtxs4KuR1gRaB/w1cEzu/QzFnRHxw0iLxl1HmgHXbEQMuBqZ\n2VgVEU9LepaN19zYjbQC5+vFzYGdJK0mzeR7JLAD0NhmB1JPB+DXAxz2ufzv2wfYbhfSrMANK0hT\nze84wH6LVhV+fpG07rrZiHAPxqy3J0nLC7yt8BgfEatIC6x9jNTz2QbYK7cpLro00PTkD5IuU80a\nYJvfkKZcb9iNtCrhalISe2OZ4Lxc7fYMnadOt1o5wZj1dgXwN5J2A5C0k6TGHV4TSetlPEf6oD+/\nlR1HxOvAGcB8ScdJmpQH9Q+QdEXe7HrgdEmT88ql5wPX57bLgYmSPprXW58HbNlCCE8De7QSs9mm\ncIIx6+0S4DbgDknrgHuB9+e6r5N6GL8hXVq7t9WdR8Qi0o0AJ+b9rCItKvcveZMrSbcx301a6G0d\nacVIIi29exrwDdLKks/T+5LYYP4G+FK+A+4LrcZu1iovOGZmZpVwD8bMzCrhBGNmZpVwgjEzs0o4\nwZiZWSWcYMzMrBJOMGZmVgknGDMzq4QTjJmZVcIJxszMKvH/AaaZmn2vwSS6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11f3cc470>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot years to see the distribution\n",
    "fig, ax = plt.subplots()\n",
    "model_df['year'].hist(ax=ax, bins= model_df['year'].max() - model_df['year'].min())\n",
    "ax.tick_params(labelsize=12)\n",
    "ax.set_xlabel('Year Count', fontsize=12)\n",
    "ax.set_ylabel('Occurrence', fontsize=12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from the uneven distribution that this is an excellent candidate for binning and dummy coding. Lucky for us, pandas can do all these things using built-in functions. Our results will be easy to interpret."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## years: binning + dummy coding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "186"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we'll base our bins on the range of the variable, rather than the unique number of features\n",
    "model_df['year'].max() - model_df['year'].min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# binning here (by 10 years)\n",
    "bins = int(round((model_df['year'].max() - model_df['year'].min()) / 10))\n",
    "\n",
    "temp_df = pd.DataFrame(index = model_df.index)\n",
    "temp_df['yearBinned'] = pd.cut(model_df['year'].tolist(), bins, precision = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have reduced from 156 to 19 features representing the year.\n"
     ]
    }
   ],
   "source": [
    "# now we only have as many bins as we created(grouping together by 10 years)\n",
    "print('We have reduced from', len(model_df['year'].unique()),\n",
    "      'to', len(temp_df['yearBinned'].values.unique()), 'features representing the year.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>(1831.0, 1841.0]</th>\n",
       "      <th>(1841.0, 1851.0]</th>\n",
       "      <th>(1851.0, 1860.0]</th>\n",
       "      <th>(1860.0, 1870.0]</th>\n",
       "      <th>(1870.0, 1880.0]</th>\n",
       "      <th>(1880.0, 1890.0]</th>\n",
       "      <th>(1890.0, 1900.0]</th>\n",
       "      <th>(1900.0, 1909.0]</th>\n",
       "      <th>(1909.0, 1919.0]</th>\n",
       "      <th>(1919.0, 1929.0]</th>\n",
       "      <th>(1929.0, 1939.0]</th>\n",
       "      <th>(1939.0, 1948.0]</th>\n",
       "      <th>(1948.0, 1958.0]</th>\n",
       "      <th>(1958.0, 1968.0]</th>\n",
       "      <th>(1968.0, 1978.0]</th>\n",
       "      <th>(1978.0, 1988.0]</th>\n",
       "      <th>(1988.0, 1997.0]</th>\n",
       "      <th>(1997.0, 2007.0]</th>\n",
       "      <th>(2007.0, 2017.0]</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   (1831.0, 1841.0]  (1841.0, 1851.0]  (1851.0, 1860.0]  (1860.0, 1870.0]  \\\n",
       "0                 0                 0                 0                 0   \n",
       "1                 0                 0                 0                 0   \n",
       "2                 0                 0                 0                 0   \n",
       "5                 0                 0                 0                 0   \n",
       "7                 0                 0                 0                 0   \n",
       "\n",
       "   (1870.0, 1880.0]  (1880.0, 1890.0]  (1890.0, 1900.0]  (1900.0, 1909.0]  \\\n",
       "0                 0                 0                 0                 0   \n",
       "1                 0                 0                 0                 0   \n",
       "2                 0                 0                 0                 0   \n",
       "5                 0                 0                 0                 0   \n",
       "7                 0                 0                 0                 0   \n",
       "\n",
       "   (1909.0, 1919.0]  (1919.0, 1929.0]  (1929.0, 1939.0]  (1939.0, 1948.0]  \\\n",
       "0                 0                 0                 0                 0   \n",
       "1                 0                 0                 0                 0   \n",
       "2                 0                 0                 0                 0   \n",
       "5                 0                 0                 0                 0   \n",
       "7                 0                 0                 0                 0   \n",
       "\n",
       "   (1948.0, 1958.0]  (1958.0, 1968.0]  (1968.0, 1978.0]  (1978.0, 1988.0]  \\\n",
       "0                 0                 0                 0                 0   \n",
       "1                 0                 0                 0                 0   \n",
       "2                 0                 0                 0                 0   \n",
       "5                 0                 1                 0                 0   \n",
       "7                 0                 0                 1                 0   \n",
       "\n",
       "   (1988.0, 1997.0]  (1997.0, 2007.0]  (2007.0, 2017.0]  \n",
       "0                 0                 0                 1  \n",
       "1                 0                 0                 1  \n",
       "2                 0                 0                 1  \n",
       "5                 0                 0                 0  \n",
       "7                 0                 0                 0  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_yrs = pd.get_dummies(temp_df['yearBinned'])\n",
    "X_yrs.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntervalIndex([(1831.0, 1841.0], (1841.0, 1851.0], (1851.0, 1860.0], (1860.0, 1870.0], (1870.0, 1880.0] ... (1968.0, 1978.0], (1978.0, 1988.0], (1988.0, 1997.0], (1997.0, 2007.0], (2007.0, 2017.0]]\n",
       "              closed='right',\n",
       "              dtype='interval[float64]')"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_yrs.columns.categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Counts')"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAFGCAYAAAB5SqiYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmcZEWZ7vHfQ7PI1jRLo8KIiCuL\noAiKIiKKG+53REfHDXUYZrzXXcEVRR0WHRkEFRkcRBkUGREEFLChgWYTGlRcAZdWQRFoNmUb6X7v\nHxFlZydV1RGVUVmns5/v55OfyjpZ9dSbWVn1ZkacE0cRgZmZ2YqsNtMFmJnZysENw8zMirhhmJlZ\nETcMMzMr4oZhZmZF3DDMzKyIG4aZmRVxwzAzsyJuGGZmVsQNw8zMiqw+0wW0tMkmm8SWW24502WY\nma1UrrzyylsiYu6Kvm6kGsaWW27JwoULZ7oMM7OViqTflnydh6TMzKyIG4aZmRVxwzAzsyJuGGZm\nVsQNw8zMirhhmJlZETcMMzMr4oZhZmZFRurAPTOzVd2WB5y5wq9ZdMgLp5TtdxhmZlbEDcPMzIq4\nYZiZWRE3DDMzK+KGYWZmRdwwzMysiBuGmZkVccMwM7MibhhmZlbEDcPMzIq4YZiZWZGhNgxJ75J0\nUb5+uKQFko7oub1om5mZDd/QGoaktYAd8vUdgXUjYjdgTUk7l24bVr1mZra8Yb7DeAtwfL7+VGBe\nvj4P2KVim5mZzYChNAxJawC7R8R5edMc4M58/Q5gw4ptZmY2A4Z1PozXASf2fH47MDtfn50/X1K4\nbTmS9gX2Bdhiiy1a121mZtmwhqQeC/yLpLOAbYFNgGfn2/YELgMuLdy2nIg4JiJ2ioid5s6dO333\nwMxsFTeUhhER+0fE8yLi+cBPI+JjwL2SFgBLI+LyiLiqZNsw6jUzswca+ilaI+Lp+ePbx7mtaJuZ\nmQ2fD9wzM7MibhhmZlbEDcPMzIq4YZiZWRE3DDMzK+KGYWZmRdwwzMysiBuGmZkVccMwM7Mibhhm\nZlbEDcPMzIq4YZiZWRE3DDMzK+KGYWZmRdwwzMysiBuGmZkVccMwM7MibhhmZlbEDcPMzIq4YZiZ\nWRE3DDMzK+KGYWZmRdwwzMysiBuGmZkVccMwM7MibhhmZlbEDcPMzIq4YZiZWRE3DDMzK+KGYWZm\nRdwwzMysiBuGmZkVccMwM7MibhhmZlbEDcPMzIq4YZiZWRE3DDMzK+KGYWZmRdwwzMysiBuGmZkV\nGUrDkLSdpEskLZB0nJLD8+dH9Hxd0TYzMxu+Yb3DuCYinhYRu+XPnwysmz9fU9LOknYs2Takes3M\nrM/qw/ghEfHXnk/vA/YE5uXP5wG7AEsLt10x3fWamdkDDW0OQ9JLJP0E2JTUqO7MN90BbAjMKdxm\nZmYzYGgNIyK+HRHbATcA9wOz802zgdvzpWTbciTtK2mhpIU333zzNN4DM7NV27Amvdfq+fROIIBn\n58/3BC4DLi3ctpyIOCYidoqInebOnTsN1ZuZGQzvHcbzJV0g6QLgwcAhwL2SFgBLI+LyiLiqZNuQ\n6jUzsz7DmvQ+DTitb/Pbx/m6om1mZjZ8PnDPzMyKuGGYmVkRNwwzMyvihmFmZkXcMMzMrIgbhpmZ\nFXHDMDOzIm4YZmZWxA3DzMyKuGGYmVkRNwwzMysy5YYhaQ9Jz2hZjJmZdVdxw8irze6ar+8PfB34\nmqQPTFdxZmbWHTXvMLZj2fko/gl4JumUqfs1rsnMzDqoZnnz1YCQ9EhAEfFzAEk+baqZ2SqgpmFc\nBBwFPBT4FkBuHrdMQ11mZtYxNUNSbySdU/tq4MC87XHAEY1rMjOzDqp5h/GsiFhugjsizpT0isY1\nmZlZB9W8w/jSBNuPaVGImZl12wrfYUjaKl9dTdIjAPXcvBVw73QUZmZm3VIyJPVLIEiN4ld9t90I\nfLRxTWZm1kErbBgRsRqkA/ciYvfpL8nMzLqoeA7DzcLMbNVWvJdUnr/4JPAEYL3e2yJii8Z1mZlZ\nx9TsVnsiaQ7j3cDd01OOmZl1VU3D2BbYNSKWTlcxZmbWXTXHYVwIPHG6CjEzs26reYexCDhb0imk\n3Wn/JiI+0rIoM7NVzZYHnLnCr1l0yAuHUMnEahrGusDpwBrAw6anHDMz66rihhER+0xnIWZm1m01\nu9VuNdFtEfHrNuWYmVlX1QxJ9S4RMibyx1nNKjIzs06qGZJabo8qSQ8hnRdjQeuizMyse2p2q11O\nRNwIvAM4uF05ZmbWVVNuGNljgXVaFGJmZt1WM+m9gGVzFpAaxbbAQa2LMjOz7qmZ9D627/O7gB9F\nxHUN6zEzs46qmfQ+fjoLMTOzbiuew5C0hqSPSfq1pHvzx49JWnM6CzQzs26oGZI6DHgysB/wW+Dh\nwIeB2cA725dmZmZdUrOX1N7ASyLinIi4JiLOAV4OvHJF3yjpKZIukbRA0uF523slXSTpvyWtUbPN\nzMyGr6ZhqHJ7r98Cz4qI3YBNJe0G7BERTweuBl4maW7Jtop6zcysoZqGcTJwuqTnSdpa0vOBU/P2\nSUXEjRFxb/70fmB74Pz8+TxgF9JwV8k2MzObATVzGO8DPgR8DtgMuAH4GvCJ0gBJ2wObALcDS/Lm\nO4ANgTnAnQXb+jP3BfYF2GILn1rczGy6rPAdhqRdJR0aEf8bER+JiEdFxDoR8WhgLWDHkh8kaSPg\nKODNpIYxO980O39eum05EXFMROwUETvNnTu3pBQzM5uCkiGpD5BOzzqe+cAHVxQgaXXgBOC9eQ2q\nK4Dd8817ApdVbDMzsxlQ0jCeAJw1wW3zgCcVZOwN7AwcKul84JHAhZIuyvmnRsRNJdsKfpaZmU2D\nkjmM2cCawD3j3LYGsP6KAiLia6T5jl6XAof2fd2hJdvMzGz4St5h/AJ47gS3PTffbmZmI67kHcbh\nwBclzSINEy2VtBrpmIjPAe+azgLNzKwbVtgwIuLEfHa944G1JN1C2jX2XuDAPNxkZmYjrug4jIj4\njKRjgacCGwOLgUsj4s7Jv9PMzEZFzfLmdwJnT2MtZmbWYYOeotXMzFYRbhhmZlbEDcPMzIq4YZiZ\nWRE3DDMzK+KGYWZmRdwwzMysiBuGmZkVccMwM7MibhhmZlbEDcPMzIq4YZiZWRE3DDMzK+KGYWZm\nRdwwzMysiBuGmZkVccMwM7MibhhmZlbEDcPMzIq4YZiZWZHVZ7oAM7OV3ZYHnDnp7YsOeeGQKple\nfodhZmZF3DDMzKyIG4aZmRVxwzAzsyJuGGZmVsQNw8zMirhhmJlZETcMMzMr4oZhZmZF3DDMzKyI\nG4aZmRVxwzAzsyJuGGZmVsQNw8zMigylYUjaTNJVku6VtHredrikBZKO6Pm6om1mZjZ8w3qHcSvw\nbOAyAEk7AutGxG7AmpJ2Lt02pHrNzKzPUE6gFBH3AvdKGtv0VGBevj4P2AVYWrjtiiGUbGZmfWZq\nDmMOcGe+fgewYcW25UjaV9JCSQtvvvnmaS3azGxVNlMN43Zgdr4+O39eum05EXFMROwUETvNnTt3\nWos2M1uVzVTDuJQ0pwGwJ2luo3SbmZnNgGHtJbWGpHnADsDZwBqkOY0FwNKIuDwirirZNox6zczs\ngYY16f1X0juEXt8f5+veXrLNzMyGzwfumZlZETcMMzMr4oZhZmZFhjKHYWbWRVsecOYKv2bRIS8c\nQiUrB7/DMDOzIm4YZmZWxA3DzMyKuGGYmVkRNwwzMyvihmFmZkXcMMzMrIgbhpmZFXHDMDOzIm4Y\nZmZWxA3DzMyKeC0pM1vpeA2omeF3GGZmVsQNw8zMirhhmJlZETcMMzMr4oZhZmZF3DDMzKyIG4aZ\nmRVxwzAzsyJuGGZmVsRHepvZUPko7ZWX32GYmVkRNwwzMyvihmFmZkXcMMzMrIgbhpmZFXHDMDOz\nIt6t1syKeHdY8zsMMzMr4oZhZmZF3DDMzKyIG4aZmRVxwzAzsyJuGGZmVsS71Zp1WKtdWVeU491h\nrcRK8Q5D0uGSFkg6YqZrMTNbVXX+HYakHYF1I2I3SV+QtHNEXDHTdZlNxge52SjqfMMAngrMy9fn\nAbsAbhgjZFjDLsPOMRs1ioiZrmFSkj4IXBkRZ0naE3haRBzUc/u+wL7508cC16wgchPglgaltcjp\nUi2tclzL9OZ0qZZWOa5lenNKMh4eEXNXFLQyvMO4HZidr8/On/9NRBwDHFMaJmlhROw0aFEtcrpU\nS6sc1zK9OV2qpVWOa5nenFa1wMox6X0p8Ox8fU/gshmsxcxsldX5hhERVwH3SloALI2Iy2e6JjOz\nVdHKMCRFRLy9YVzx8NUQcrpUS6sc1zK9OV2qpVWOa5nenFa1dH/S28zMuqHzQ1JmZtYNbhhmZlZk\npZjDmCpJb5rk5j9HxMkFGQdNcvNtEXF4YS0D53SplpY5o0bSZC/EIgrHgVvlWPetLL/rkZ7DkPRz\n4GBA49z81oh4ckHGD4C3T5BxSEQ8tbCWgXO6VEurHElfmeTmmyPi3YW1XDjRTcANEfEPhTnXAjdO\nkLM0InYvyLgN+EH+nt4/MAEPjohtCmsZOKfh49KZnC7VknO69JwZuJZJRcTIXoBdp3Jb39c9ciq3\nTUdOl2ppeJ8uBR4+zmVLYEFFLadP5bZxvvY/p3Jb39d9fCq3TUdOw8elMzldqqWDz5mBa5nsMtLv\nMAAkbQrsDGxIOkr8ioj4U2XGk0hrWI1lXBYRC6dQy8A5XaqlRY6kdSPirtrbzGz4RrphSHofsBsw\nH7iDtLTIHsAlEXFIYcbhwFqkhQ/HMvYElkTE2ypqGTinS7U0vE+zgJfR13SAUyPi/opa1gP2yzlz\nenK+GBF/rsjZDPgAsA1pp5ClwM9IQ2zXF2ZsB3yC9HisRhpiuB04MCKurqhl4JyGj0tncrpUS87p\n0nNm4FomNehblC5fgAtrtk9XxijW0vA+fRV4L7Aj8EjgifnzEypr+TawN7ARMIvUfPamYmgh55wL\n7Ny37cnAuRUZC4CH9m3bjIohtlY5DR+XzuR0qZYOPmcGrmWyy0jvJQUsknQA6RXwnaTu/WzgdxUZ\nCyUdPU7GVZW1tMjpUi2tcraMiNf1bftBXgqmxsbANyNiaf78NknfBN5RmbM28NO+bT/N22v07wgw\n3o4Bw8hp9bh0KadLtUC3njOtahnXqA9JzQJeQjqnxthbzktJryBqhjue2J8RET+YQj0D53SplhY5\nkt4DPBM4n2VNZ3fSK6vDKnJeQxpeuDrnbABsCxwbESdU5OwBfBi4uyfnQcC/RcS5hRnbAh8nvWId\nG15YDHw0In5cUcvAOQ0fl87kdKmWnNOl58zAtUyaP8oNYyKSFAPecUkPjYg/Nqhl4Jwu1TKVHEmb\nkN42jzWdKyLi5in83NWBx/TkXFvzwqAva23SH+9tEXHPVDK6otXj0qWcLtXSk9WZ58y01dJiXGtl\nuwCHN8jYv1EtA+d0qZaG9+mJjWp5QaOcfRpkvL9RLQPnNHxcOpPTpVpyTpeeMwPXEhGr5tIgEfHO\nBhmHNqpl4Jwu1dIwZ/sGGZD2Emmhdn5nPP/VIKNVTqvHpUs5XaoFuvWcaVHLKjsk9aGI+MSAGUdG\nxP+r+PoWu0h2ZnfNljk5a13ybrUR8Zea7+3LWYs0vHBLRCyZYsZqwKNyPX+KiEVTyNgI2Gksg3R8\nyr0zkSNpNnBfRNwnaQtg/YjonxhdqXI6VsuciLh9xV85acb2tX8z01XLpPmj3DAmWEtKwDsjYrvC\njPHWSxLwmoh4ZEUtC4BXRs/Yft5n+qSI2G1YGV3LkfQs0iTdnSyb9F4f+GRUTNJJ2hX4ELAuaeLy\natLecB+JiN9W5OwNvIX0z3kX4OfAfcDBUTiZL+ltpIn7nwLPAG4gNdTjIuKciloGzpH0MdJOCUHa\n4WN70jEz90XEfhW1dCanS7XknL8Al5N20/1WzfOtJ+NWYBFwIfAt0q7p1f+cW9QyqRbjWl29ADcB\nrwfe0Hf5cUXGDaQ/2v7LwspaFgCb9W3bnLp96gfO6FoOcBGwTt+2dYGLK2u5AFg7X58LnEzabbJ2\nn/rzWfZCah3gdNJeJudUZJzXc3014Kz8cV5lLQPnjP0uSMcZXNv7eNX+rruS06Va8tfPz8+1fUj/\nqC8mvXjZpjJjFunA4iNJ60odC+w17Fomu4z6cRgnA2dH31IgkjavyPgc8LPo22tH0ucra9kPOCoP\nMYwdgbkY+NcpZPTveleT0aqWVjn3AY8Hvt+z7fFA7fDNLNJQ1D3ApqShhcV5qKvGEmBnST8mvTD4\na0Tcm/emKXWXpFeR3uXsDtwUEUs1+Yqk05Wj/O5rQ2CppB2B26hfqbpLOV2qBYCIWAwcBxyXn3N7\nAR8E/rEiYwnpH/78XNzOwMuB7wy7lomM9JCUdZ+khwIHkIYDViP9w74a+FRE3FCRswNwEOmP/ybS\n3iXXSXpTRBRPHEp6GLA/sBXwC9KSCjdJ2isiiv5w87j4vjnjGtKib3dL2jHSOepLaxk4R9LWpMb+\nE9I/orG5u8Mqa+lMTpdqyTn7x4A7ekh6Q0QcP0hGq1omzR/lhiFpq/5NwBeAf46I3xRmPDMizpc0\nhzTWvi3wK+ATUXeswfsj4mBJTwE+TfrHuHrOOasw4xTgDOC0/CpiSiS9GHg3abz2SOBAYA3gmMp/\nrpuS/tk/hbS8wrWkYaojI+K+yprWY9mkd/E6PpPk3BYDTJ6Pk7tmRPzvgBlrTeFx2Ri4MyL+mpvi\nPRFx7YB17BYRtUfSNydpLvBoYFFE/GGGauid9H44sF5MYfI8Zw20w4UaLJQ6SfbAz18Y/YZxD2ky\na9HYJuA5pPHoyU6u1JtxXkQ8S9KJpDHBU4GnA++NiOdV1DIvIvaUdC7w6vyqdV3SeHTp+SeuAr5C\nOnp9KXAaaZG+35fWkXMuJY2Vrk/a3W5r0lDORaW15JzvkN7qXg08C3hBrun1EfHmwozeSe87SEem\nrk86MnVeRS0T5VRNnk+Sf05EPLfwa19Nasj3k54vh0ZEjD2XKn7mx4Gnkc5vsBh4GHAX8KuIOLAw\nY7ydNl4NfC0iPlJRyzsi4j9y0zqSNBy6OnBATfORdFJEvErSW0hDJGOTzZdF4Z6LkhYDZwKnAGfF\nFPY+yzmtJr0H3uFCDRZKXUF+8fN3Ui0mQrp6IS3edSjweWDrvO27lRnzSUMl8/u2106MnUQ6ovR4\n4DF520P6c1eQ0TsJOpe0N88Z1E9Wf5+0yuwmpCf2uvk+XlaZc0nP9XXG7gt1i661mvRulXPhOJcF\nwK01jwuwer6+H6mJzqn5XefvvTh/nAVcx7IXeDWLO55Gmix/CWlPq93z7/8ZlbWclz+eAzwqX99k\nCo/v9/LH+cBqPdtrdpSYDzwB+Fi+L98EXgtsUFlLq0nvgXe4mOh3WvO7bvX8newy0pPekd7m7q+0\n9MQ78vj07ClEnUuaFJsTEbdLWp/0D7LGfqShn82BH0q6hjSEUzNB/LfFyCJNwh8LHJvrqfFp4DzS\nGP2/kv7Z3ksarqvxBUmXANeTTng09mq25i19q0nvVjlzge0j4q+9GyV9ryJDkZeXiIijlc5MeDpp\nMr7G0pyxRNJhkf8jkIYzi0TESyVtD7yNtKvwEcDiiJjobHMT2Si/i9soIn6Zs2+RVDtE8UNJ+wBX\nAm+SdAGwA2neqVRExA+BHwIHSnoUaXL4NNK6ZKVaTXr37nAxl6ntcLFIgy+UCm2evxMa6SGpfpI2\nIL26v2KmaxkVSgs8bkwat60+QrZv0nsW6Z/hj6if9G6V83zSAop39G1/UkRcWZjxFtKw5+96tm0O\nfDjqhjqeQ3plv6Rn25qkZR6+WJrT872PBd4FPDwinl/5vb1DYEf0vHD6VOV9EvAK4EXAg0nDLxcD\n/xWFc06SDo8GqzU0nPQeeIcLLb9Q6gYsOzdH7UKpAz9/J80f5YaRJ5j3B24BPksanlqH9CQv3eNl\noozDIuK7FbUMPNHc4v6soJYvRsRxFTlNHpsuGnQCs2HGwEedt6qlpS7V05Va8qT3k8k7bTDFSW+l\nXcAfy7LJ82v633FMWYtxra5eSK9cNiNNRP0+X1+firH6Fhk551LSAWBzc856pFfCl66MtTR8fB8D\nfJ10dOqfSePthwMbVtbSKmdX4Luksd/FpFeex5NelQ8tI+e8AjibtKPDtaQhl29QsTBjw1rGHt8r\nGj2+Fwzw+LaupdVjM+XnHvA+0k417wTenD9+m7RTQU0tr82P7RGk5dKPIB2M+rqanAnzW4R09TL2\njyv/c1wEzMqfXzLMjPz1A080d6mWho/vucBD8vXHkRZbexTwjcpaWuW0mMDs0lHnrWrp0uPbmVpa\n1UO7Se8FY8+Znm2zSHtAFudMdBnpSW/gxLwr6nXAwcAlku4i/eENMwPgUww+0dylWlrVsw7pbTPA\nraRXd7/MOyrUaJXTYgKzRQa0Oeq8VS1deny7VEurelpNet8GvEpSb86eefvgWnSdleVCGi5Zd6Yz\nunR/Zrqe/GQe2/3vAuApefsHZihnB9LQz4XA/7BsF9I3DTMjf/3DgKNIS0N8Btg0by9eX6hhLV16\nfDtTS6t6SM3r5cBhwDH548vJu2dX5KxHGs76BmkX6JNIp5xdvyZnostIT3pPRG2WNx84I+dULZPe\n9Vpa1mPja3XUrnXfVFYHmE4j3TDUZnnzgTNyzsDLpHeplpb1TJDdqnm1yjk5Ivae6YycM/BRuw1r\n6dLj25laautpuDrAHsBHScfwfDEivp63fysiXl57H/qN+hzGIcB76DngLavpki0yIO358Jpxtr9g\nJa2lST2TNK+9gJoTVLXK+coEOU8bZkbOGe/gOpH2Sht2LV16fDtTS8N6/h+wS0TcL2k/4FRJb+CB\nf1sr8on8c+8DPpoPtnwraa5mYKPeMFosb94iA9osk96lWlrV06p5tcp5OmlMund/fJGOZB9mBrQ5\nardVLV16fLtUS6t6Wq0OoFh20N4Bkl5GmgPbqDJnfC0mQnzxZaoX4APA3HG21048tsrZh7T8Rf/2\nFw4zI3/98xlnfSTgSTNQS5ce387U0qoe0rpwW/Rt2xw4urKWj9B3HAlp9YOzanImuoz0HIaZmbVT\newYwMzNbRa1yDUPS6nmhuhnNyDkPklR8/omu19KyHjPrnlWuYZAW5DpM0lEznAGwNrCXpCNGpJYm\n9TRsXq1y1pf0atUvI980I+dsKOn9euDZJGeili49vp2ppVU9LX7XOafJfYIRPw6ji/KyA2OnIm12\nCtGVvZZekjYkLcM9OyLe3oGc9YBXAVtFxAdnKiPnPIi0aN52ETGl5t6wli49vp2ppVU9LX7XOafJ\nfYIRbxiStiPtlzyb9G4qSGu+fCQifjysjJzTewrRsTVeqk4h2qVaWtaTs5o0rxY5arA8dIuMnDPw\neZ5b1ZKzOvH4dq2WFvW0+F3nnGlb3nzUG8YC4JUR8ceebZsBJ0XEbsPKyN9zEfDciLi7Z9u6pJVH\nd13ZamlVT8Pm1SrndaRdHH/Yk7MD8KWI+OqwMnLOROd5vjgiDh1yLV16fDtTS6t6WvyuW96nCbXY\nN7erF9JSv5v1bducuvMHD5yRv+dc8qJkPdt2oe78152ppeHj27Vzeg+8PHSLjPw9Ay953bCWLj2+\nnamlVT0tftct79NEl1E/0ns/4Kg8njg2ZLKYuvNot8iAdGKTAyQdknOWAFcDr19Ja2lVT9fO6d1i\neehWS0y3WPK6VS1deny7VEurelaK5c1HekjKuk/Ln4u7t3kNck7vQXLWA/6JdG7lOaQ/tEtJb+n/\nPKyMnNN7nuc5pPHoS6k4z3PDWrr0+Hamllb1tPhdt7xPE+avig1D0vsj4uCZzsg5n42It3WkliNi\ngD1MWtdj45OkWBX/cFdBXftdr4rHYUA6hWIXMhi0WWStahm4WWQD16PBjwfpXE6rWoB/HzSgS49L\nzvnsKNWSc1rUM/DvGho+Nh1qXkMjaeeIuGKmM3LOiyOi+JSmkp5EmqAe22XusohYWPkzZwEv688B\nTq15+9uqHjNbOYx0w5A03jsokVZufM6wMnLOeEdrCvhylO+CejiwFmlibGzXuz2BJTXvVCR9lTTG\nem5fzg4R8dqKnIHradW8GuYIeBFpDHhD4E/AmTUvDlpk5Jy1WX48eizn5BmoZSPgH0k7NZwCvJf0\n+/58RPymIueRpPMz3AJ8Cfi3nHNIRFy5stXSqh5JnwHOAM6PiKWlP3ucnLcBZ0TEr6eaMWn+iDeM\nu0n/NMSyk/qIdI6BjYeVkXPuJJ03uP+EKM+JiL8rzLgwIp5Run2SnAXjNamJtk9nPQ2bV6ucY4Gf\nAj8CnkXan/5W4L6IOGRYGTnnZODEnpxHAT8nPffePeRazgG+TGpc+5HO6rYY+FhEPLMi5wLg/aTm\ndSTwilzPf0f58UidqaVVPZJ+RjoH9zNIv6tTSMdFVZ2eVdKv8/c/BDgLOCUqD6KdVIt9c7t6Aa4E\n5oyz/XuVGeOdk6A4I3/9JYy/Zv5JFRmfAY4mPbGfmz9+AfiPylreQ3o18x5g3/zxdOB9lTnj1XN0\nTT1McMzGRNuHkDO/7/Nzp/CcGTijv3bSvvTn5evzZqCW83uu/2yi/IKci/LH1YDrerbXHFvSmVpa\n1TP2u83Xn0RaQeFy4BuVtczPH9cB/h44AVgIHFaTM9Fl1I/DeBFw9zjba86E9SLgngEzAJ4R4wyN\nRMSrSgMi4l2SnkgaXngMaZe5YyLiBzWFRMSnJX0ZeDLpVdH1wPHRdwa+KdZzdET8sCLmNElnAOeT\n9hvfgPQqq3hep3HOjyV9gfRu5ZmkI2+h7uyUvRm7TzED4AxJpwM/I/2uTsjbb52BWn6f360EcKWk\nI3Mdt1TmXJyPEVgMnCzpNNLwYc2r4C7V0qqev408RBoOuxL4kKTHVNYylnE38E3gm0pLhRSfF3zS\nInM3Gll5Uvb3pCfFi4B7IuKcATPfGhGfG+D7twO2A34VdWPjD42IP+Z5lZcCWwO/Bv5nvGY0Sc5L\nSK9Sx2umxSStQTor3OKIuCQvSzCb9Jb+9oqcTVjWvMbW0KlqXn05G+SchVPM2Ql4BGkNnqvztqrd\nG3PGVjnjR1PJyN/zEGAL4NeSug/RAAAKqElEQVQRUfsPsVkteS7kCcANpH+Ez803fS8ilkz4jeNn\nbQj8JSL+KulxpP9DP18Za2lVj6QHxxTWjRonZ4ex3/F0GOmGIelLpM59H+n8yH8gvfrcNCL2LcxY\nwPJzFwDbAj+JunmDsyLi+ZLeQTqC80zSSpQ3RMQBhRnnRcSz8i5yd5NeLT4B2CkiXllRyx+A35Im\nQL8FfDsiqo8ElfQt4ArSP/onke7TYuA1EfG8ipzHA09j2cTuORHxh8paxia9xyZ3b2Pqk+cvJ02e\njzWwKe1BNk527R5xAvYiHQh2TuTJUEkvjYjTCjPWIr1Qug74DfAm0jvmr0RE7ZHR/dkHRcRHKr9n\nztiLCUkvIr9wIr3oKfpnJOmJEfGDvFPAPwPbkF44HV35QqXJBHGjSe9WOyc02wNy3PwRbxgXRMTu\n+fqPI+Lx+fr8iNijMONdpCM4vxwR5+dt342IqiGpnn/2FwB79PzxXxQRTy/MmBcRe4597NlefH96\nv17SI4D/A7yY1FRPi4jP1+bk6z+JiO1q61FanmRt0kTdHqTlFJYAl0TEVypq+SppKKF/j62hT56r\nwR5xOecEYBFwP+lFxlsi4pqx51Jhxqmk4Y3VSY/vqaQXTc+LiL0ravkdaZmKpQz2wmns7+BgUkM+\njfTC6e8iYp/KjONJRzGfR3rh9MaI2KuiliYTxI0mvVvtnNBk548JtZgI6eqFnsW/gBf3XD+/MmdN\n0vpIXycdvv/dKdRyI/AV0nzB2j3bF1ZkvA44FjiONJ79T8BRpCUIqifG+rY9GNi3MucM4EOkCbrv\nAe8mvYItPuE8fQsekidjqZjYzV/fmclz0j/k/8q/p97L9ZW1nN9zfTPSXjQvpWeCtOZ3TRrqG/dx\nL8j5e9IeW28EVs/bpvJ3MDZxf0Hf9gsqMs4lNa2zYdlCezUZvY8NA04Q02bSe37f5wPvKFGyvfYy\n6pPe+0qaFRFLIg8F5HH3z9SERMT/Ap+XdAzpn3bNpO6Yp+SPHya9gkZpSfEPV9TxVUnnAs8j/YOf\nRZr0vrqylge8Yok0fnpMZc7epDmMXwGfBN5Aaq7FE/nATZL2Z9mk7M/y9lmVtXRp8vwnwP7RN38i\n6aTKWlaTNDsi7oyIP+QhnGNIw3+l1uy53rsoZNXjGxFjE6h7ASdIugRYoyYj2zEP8249NjyV5+TW\nq8g4GPgGabjlfKXl+rcmDa9Wi8EniFtMerfaOaHV38G4Rn1Iqn9S9rWkB7B4UrbhxG6Xa5mxnJ45\ng62Aa0jzKSFps6ifx2g9eT6lHEmrR4vxYmlL4LaIuKNv+y4RcVlhxsbArdHzh55/b0+IAVYqkLQH\n6UxwR041oydrHeDRUTFZq3Q2uqeSXjjdQfodVe0U0GqCeIJJb5HmnYon4RvuKNFk549xs0e8YfRP\nyn6H9AstnpRtOLE7UrW0qqdFI+3JGnjyvFVOzuidwBykloFyulRLq5wu1dIqp0s7f0yYP+INY34M\nPik7cMYo1tLwPrVqXq0mzwfOmeZalpLm5rpQy1Rz1iEN63bx8V3ZnzNNdv6YUIuJkK5eWDYp+3GW\nTcruQ92k7MAZo1hLw/s0v+f6T8bbXpjTavJ84BzX4vs0g/fJk94D6J2U/TfSpOyDqJuUbZExirW0\nyrlL0odIixj+UdK7ybsTVtbSavK8RY5rmd6cLtXSKme6dv6YnfM86T1V6tby5iNVS22O0sFXY03n\nOlLTEXBi9E32riCnf/L89IhYWjt53iLHtfg+zdR9yllNdv4YV4u3KV29kBYVG+9Ss5DcwBmjWEvL\nnAmyd270HOhMjmvxfRpGBumAysf1bXtKi/s00u8w1K3lzUeqllY5Gv98IwBnR935Rlqdt6RL51AZ\nqVpa5XSpllY5DWv5d9KuxvcDGwNvioibVbE6wGRGfQ7j58DL44H7sX9vyBmjWEurnL8wQdOprKU3\nh5w1UzmuZXpzulRLq5xWtewUy5ZD2p60Cu97KzMm1uJtSlcvwEOBNcfZvvowM0axlob3qdX5RjqT\n41p8n2bwPl3c+zdJOhbjTOBPNTkT5rcI6eqFtO9x9W2tM0axlob3qTPNq1WOa/F9msH79GTSaty9\n22YB/1CTM9Fl1Iek9pR0IOl8GIvytkcAfwdcRDpIZhgZo1hLq5xNI+KP/Rsj4n7VLd3QpRzXMr05\nXaqlVU6rWu6LiJv6MpYAX6/MGddIT3oDY+u87AA8mjQmeC3wo6i44y0yRrGWFjn5uItdmaDpRETR\nQpFdynEt05vTpVpa5TSu5WmkVbGnnDNh/qg3DOu+rjSvljmuZXpzulRLq5wu1TJhthuGmZmVmGgf\neDMzs+W4YZiZWRE3DFulSDpaUvFZDhv+3DcqnRnObKXlhmEjRdIiSfdI+ouk2ySdKelhY7dHxH4R\n8fGZrLGXpLUlXSfp9X3bD5R08SRLp5gNnZ+MNopeHBHrkQ6G+hMw8GlEp0tE3AO8GfiMpAcDSNoa\neBfw5ohY2upnKZ2v2mzK3DBsZEXEvcD/ANuMbZP0ZUmfyNefKel6Se+WdJOkP0rap+9rP5ffpfxZ\n0vclPbLn9sdJ+p6kWyVdI+mVPbdtLOnbku6UdDnwt+8bp84LgZOAo/Iukf8JHBwRv8hZcyQdl+u7\nXtJBY+88JD1a0nxJiyXdIumrkjboqeN6Se+V9GPg7rztA5L+kGv7haRnDvRA2yrDDcNGlqR1SCdz\numySL3sI6Rzim5Ne6X9O0oY9t78a+BhpTZ5fAp/M2euSzjJ4IrBp/rrPS9o2f9/nSKfZfCjwpnyZ\nzP7AzsA3SSeh+lTPbScA95Cazk7AC0lnNoS0n/0n8s/ZhnQ+hf45mn8AXgBskOv7Z2DHiJidt/9u\nBbWZAaO/Wq2tmk6VdD+wHnATMNm5wf8KHBQR9wPfkfQX4LEsazKnRMTlAJL+Gxg7UvZFwKKIOC5/\nfpWkbwKvkPQL4O+Bx0fEXcBPJB0PPGOiIiLiL5LeSjrt7RPycg5I2hx4NjAnIu4D7pb0H8DrgS9F\nxLWkA7MgnbXtcFLz6XVERFyf8+4nNaRtJd0SEb+Z5LExW44bho2il0XEPKWzmL0UuEDSNhFx4zhf\nuzg3izF3kxrNmBsnuO3hwFMk3d5z++rAV4G5+frve277bUHdP+37OPZz1gL+lEargDQysAhA0kOA\nz5KWlVg/39Z/drW/1RER1+TlIw4CtpZ0NvDOCR4bs+V4SMpGVkQsiYhTgCXA0xvH/x64ICLm9FzW\ni4h/If3Dvh94WM/XbzHAz7kb2Kjn58yOiLHzJBxKOv/54/MQ0xtZdk6FMcst5xARJ0TErqQ1hmYB\nB0+xNlvFuGHYyFLyUtL8w88bx58BPEbS6yStkS87S9o6DyedAnxU0jqStiGdq7xaRPweuAD4tKTZ\nklaT9ChJY8Nb6wN3AXfk3YffM1mepK0l7SFpLdK8yD2khmq2Qm4YNopOz3MRd5Imqd8QET9dwfdU\niYg/A88lTSj/gTR0dShp+Ajg/5KGr24Evgwc98CUYq8F1gV+BtwGnEyarAc4kHQOhDuAb5MmzSez\nFnAYcEuubUPgQwPUZqsQLz5oZmZF/A7DzMyKuGGYmVkRNwwzMyvihmFmZkXcMMzMrIgbhpmZFXHD\nMDOzIm4YZmZWxA3DzMyK/H+q3y4YYb68mAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11f2b1550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's look at the new distribution\n",
    "fig, ax = plt.subplots()\n",
    "X_yrs.sum().plot.bar(ax = ax)\n",
    "ax.tick_params(labelsize=8)\n",
    "ax.set_xlabel('Binned Years', fontsize=12)\n",
    "ax.set_ylabel('Counts', fontsize=12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have preserved the underlying distribution of the original variable through binning by decades. If we desired to use a method that would benefit from a different distribution, we could alter our binning choices to change how this variable presents itself to the model. Since we are using a cosine similarity, this is fine.\n",
    "\n",
    "Let's move on to the next feature we originally included in our model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## fields of study: convert to sparse array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This feature contributed significantly to the original model's size and processing time. We we will aim to reduce these. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's leverage work we have already done. We have a sparse array of parsed field of study fields. We'll convert from a pandas DataFrame to a numpy sparse array to see if that helps to save on computation time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "unique_fos = sorted(list({ feature\n",
    "                          for paper_row in model_df.fos.fillna('0')\n",
    "                          for feature in paper_row }))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def feature_array(x, unique_array):\n",
    "    row_dict = {}\n",
    "    for i in x.index:\n",
    "        var_dict = {}\n",
    "        \n",
    "        for j in range(len(unique_array)):\n",
    "            if type(x[i]) is list:\n",
    "                if unique_array[j] in x[i]:\n",
    "                    var_dict.update({unique_array[j]: 1})\n",
    "                else:\n",
    "                    var_dict.update({unique_array[j]: 0})\n",
    "            else:    \n",
    "                if unique_array[j] == str(x[i]):\n",
    "                    var_dict.update({unique_array[j]: 1})\n",
    "                else:\n",
    "                    var_dict.update({unique_array[j]: 0})\n",
    "        \n",
    "        row_dict.update({i : var_dict})\n",
    "    \n",
    "    feature_df = pd.DataFrame.from_dict(row_dict, dtype='str').T\n",
    "    \n",
    "    return feature_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 28min 55s, sys: 13.3 s, total: 29min 8s\n",
      "Wall time: 29min 38s\n"
     ]
    }
   ],
   "source": [
    "%time fos_features = feature_array(model_df['fos'], unique_fos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>0-10 V lighting control</th>\n",
       "      <th>1/N expansion</th>\n",
       "      <th>10G-PON</th>\n",
       "      <th>14-3-3 protein</th>\n",
       "      <th>2-choice hashing</th>\n",
       "      <th>20th-century philosophy</th>\n",
       "      <th>2D computer graphics</th>\n",
       "      <th>2DEG</th>\n",
       "      <th>3-D Secure</th>\n",
       "      <th>...</th>\n",
       "      <th>k-nearest neighbors algorithm</th>\n",
       "      <th>m-derived filter</th>\n",
       "      <th>microRNA</th>\n",
       "      <th>pH</th>\n",
       "      <th>photoperiodism</th>\n",
       "      <th>route</th>\n",
       "      <th>strictfp</th>\n",
       "      <th>string</th>\n",
       "      <th>Ćuk converter</th>\n",
       "      <th>μ operator</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 7604 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0 0-10 V lighting control 1/N expansion 10G-PON 14-3-3 protein  \\\n",
       "0  0                       0             0       0              0   \n",
       "1  0                       0             0       0              0   \n",
       "\n",
       "  2-choice hashing 20th-century philosophy 2D computer graphics 2DEG  \\\n",
       "0                0                       0                    0    0   \n",
       "1                0                       0                    0    0   \n",
       "\n",
       "  3-D Secure    ...     k-nearest neighbors algorithm m-derived filter  \\\n",
       "0          0    ...                                 0                0   \n",
       "1          0    ...                                 0                0   \n",
       "\n",
       "  microRNA pH photoperiodism route strictfp string Ćuk converter μ operator  \n",
       "0        0  0              0     0        0      0             0          0  \n",
       "1        0  0              0     0        0      0             0          0  \n",
       "\n",
       "[2 rows x 7604 columns]"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fos_features.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_fos = fos_features.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Our pandas Series, in bytes:  2530632380\n",
      "Our hashed numpy array, in bytes:  112\n"
     ]
    }
   ],
   "source": [
    "# We can see how this will make a difference in the future by looking at the size of each\n",
    "print('Our pandas Series, in bytes: ', getsizeof(fos_features))\n",
    "print('Our hashed numpy array, in bytes: ', getsizeof(X_fos))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Putting it back together, we'll pipe our features together and re-run our recommender to see if we have improved results. Since we are starting to use sklearn, we'll take advantage of their cosine similarity function, reducing the computational time by only focusing on one item at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7623"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_yrs.shape[1] + X_fos.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 700 ms, sys: 1.02 s, total: 1.72 s\n",
      "Wall time: 2.14 s\n",
      "Size of second feature array, in bytes:  634172728\n"
     ]
    }
   ],
   "source": [
    "# now looking at 10399 x  7623 array for our feature space\n",
    "\n",
    "%time second_features = np.append(X_fos, X_yrs, axis = 1)\n",
    "\n",
    "second_size = getsizeof(second_features)\n",
    "\n",
    "print('Size of second feature array, in bytes: ', second_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The power of feature engineering saves us, in bytes:  168066769\n"
     ]
    }
   ],
   "source": [
    "print(\"The power of feature engineering saves us, in bytes: \", 802239497 - second_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Converting between pandas DataFrames and numpy matrices can make indices tricky. We have the same size index, but the index assigments are not the same.\n",
    "\n",
    "pandas assists with this differentiating between .iloc, .loc and .get_loc. \n",
    "\n",
    ".loc will return the index based on the original pandas DataFrame, allowing us to reference specific papers. We use .iloc to specify the integer location, which is the same index as our numpy array. .get_loc assists us to find the integer location when we know the DataFrame index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "abstract    A microprocessor includes hardware registers t...\n",
       "authors                      [{'name': 'Mark John Ebersole'}]\n",
       "fos         [Embedded system, Parallel computing, Computer...\n",
       "keywords                                                  NaN\n",
       "title       Microprocessor that enables ARM ISA program to...\n",
       "year                                                     2013\n",
       "Name: 21, dtype: object"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_df.loc[21]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "abstract                                                  NaN\n",
       "authors     [{'name': 'Nicola M. Heller'}, {'name': 'Steph...\n",
       "fos         [Biology, Medicine, Post-transcriptional regul...\n",
       "keywords    [glucocorticoids, post transcriptional regulat...\n",
       "title       Post-transcriptional regulation of eotaxin by ...\n",
       "year                                                     2002\n",
       "Name: 30, dtype: object"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_df.iloc[21]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_df.index.get_loc(30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "def piped_collab_filter(features_matrix, index, top_n):\n",
    "                \n",
    "    item_similarities = 1 - cosine_similarity(features_matrix[index:index+1], features_matrix).flatten() \n",
    "    related_indices = [i for i in item_similarities.argsort()[::-1] if i != index]\n",
    "\n",
    "    return [(index, item_similarities[index]) for index in related_indices][0:top_n]\n",
    "\n",
    "def paper_recommender(items_df, paper_index, top_n):\n",
    "    if paper_index in model_df.index:\n",
    "        \n",
    "        print('Based on the paper:')\n",
    "        print('Paper index = ', model_df.loc[paper_index].name)\n",
    "        print('Title :', model_df.loc[paper_index]['title'])\n",
    "        print('FOS :', model_df.loc[paper_index]['fos'])\n",
    "        print('Year :', model_df.loc[paper_index]['year'])\n",
    "        print('Abstract :', model_df.loc[paper_index]['abstract'])\n",
    "        print('Authors :', model_df.loc[paper_index]['authors'], '\\n')\n",
    "        \n",
    "        # define the location index for the DataFrame index requested\n",
    "        array_ix = model_df.index.get_loc(paper_index)\n",
    "\n",
    "        top_results = piped_collab_filter(items_df, array_ix, top_n)\n",
    "        \n",
    "        print('\\nTop',top_n,'results: ') \n",
    "        \n",
    "        order = 1\n",
    "        for i in range(len(top_results)):\n",
    "            print(order,'. Paper index = ', model_df.iloc[top_results[i][0]].name)\n",
    "            print('Similarity score: ', top_results[i][1])\n",
    "            print('Title :', model_df.iloc[top_results[i][0]]['title'])\n",
    "            print('FOS :', model_df.iloc[top_results[i][0]]['fos'])\n",
    "            print('Year :', model_df.iloc[top_results[i][0]]['year'])\n",
    "            print('Abstract :', model_df.iloc[top_results[i][0]]['abstract'])\n",
    "            print('Authors :', model_df.iloc[top_results[i][0]]['authors'], '\\n')\n",
    "            if order < top_n: order += 1\n",
    "    \n",
    "    else:\n",
    "        print('Whoops! Choose another paper. Try something from here: \\n', model_df.index[100:200])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Based on the paper:\n",
      "Paper index =  2\n",
      "Title : Should endometriosis be an indication for intracytoplasmic sperm injection (ICSI) in fresh IVF cycles\n",
      "FOS : nan\n",
      "Year : 2015\n",
      "Abstract : nan\n",
      "Authors : [{'name': 'Jovana P. Lekovich', 'org': 'Weill Cornell Medical College, New York, NY'}, {'name': 'G.D. Palermo', 'org': 'Weill Medical College of Cornell University, New York, NY'}, {'name': 'Nigel Pereira', 'org': 'The Ronald O. Perelman and Claudia Cohen Center, New York, NY'}, {'name': 'Zev Rosenwaks', 'org': 'Weill Cornell Medical College, New York, NY'}] \n",
      "\n",
      "\n",
      "Top 3 results: \n",
      "1 . Paper index =  10055\n",
      "Similarity score:  1.0\n",
      "Title : [Diagnosis of cerebral tumors; comparative studies on arteriography, ventriculography and electroencephalography].\n",
      "FOS : ['Radiology', 'Pathology', 'Surgery']\n",
      "Year : 1953\n",
      "Abstract : nan\n",
      "Authors : [{'name': 'Antoine'}, {'name': 'Lepoire'}, {'name': 'Schoumacker'}] \n",
      "\n",
      "2 . Paper index =  11771\n",
      "Similarity score:  1.0\n",
      "Title : A Study of Special Functions in the Theory of Eclipsing Binary Systems\n",
      "FOS : ['Contact binary']\n",
      "Year : 1981\n",
      "Abstract : nan\n",
      "Authors : [{'name': 'Filaretti Zafiropoulos', 'org': 'University of Manchester'}] \n",
      "\n",
      "3 . Paper index =  11773\n",
      "Similarity score:  1.0\n",
      "Title : Studies of powder flow using a recording powder flowmeter and measurement of the dynamic angle of repose\n",
      "FOS : nan\n",
      "Year : 1985\n",
      "Abstract : This paper describes the utility of the dynamic measurement of the angle of repose for pharmaceutical systems, using a variable rotating cylinder to quantify powder flow. The dynamic angle of repose of sodium chloride powder sieve fractions was evaluated using a variable rotating cylinder. The relationship between the static and the dynamic angle of repose is discussed. The dynamic angle of repose of six lots of a multivitamin preparation were compared for inter- and intralot variation. In both cases, no significant differences (p> 0.05) were observed. In the multivitamin formulation, lubricants at lower concentration levels did not show a significant effect (p> 0.05) on the dynamic angle of repose when compared with flow rates. The effect of different hopper sizes and geometry has been evaluated using the recording powder flowmeter. The results indicate that although different hoppers affect the quantitative nature of the results, the same general trends are apparent. Thus, it appears possible to use a recording powder flowmeter with small quantities of material to predict the effect of formulation and processing variables on the flow of production scale quantities. This paper does not describe a comprehensive evaluation of the pharmaceutical utility of measuring the dynamic angle of repose. However, the results discussed are not encouraging and suggest that the recording powder flowmeter is more sensitive to the effects of formulation and production variables on powder flow.\n",
      "Authors : [{'name': 'Ramachandra P. Hegde', 'org': 'Department of Pharmacy, University of Rhode Island, Kingston, RI 02881'}, {'name': 'J.L. Rheingold', 'org': 'Formulation Research, Lederle Laboratories, Pearl River, NY 10965'}, {'name': 'S. Welch', 'org': 'Formulation Research, Lederle Laboratories, Pearl River, NY 10965'}, {'name': 'C. T. Rhodes', 'org': 'Department of Pharmacy, University of Rhode Island, Kingston, RI 02881|||Department of Pharmacy, University of Rhode Island, Kingston, RI 02881'}] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "paper_recommender(second_features, 2, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be honest, I don't think we are doing a very good job. Let's keep going to see if we can make this better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (3) abstract: stopwords, frequency based filtering (tf-idf?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# need to fill in NaN for sklearn\n",
    "filled_df = model_df.fillna('None')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    A system and method for maskless direct write ...\n",
       "1                                                 None\n",
       "2                                                 None\n",
       "5                                                 None\n",
       "7    Full textFull text is available as a scanned c...\n",
       "Name: abstract, dtype: object"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# abstract: stopwords, frequency based filtering (tf-idf?)\n",
    "filled_df['abstract'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<10399x48516 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 374055 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5,\n",
    "                                 stop_words='english')\n",
    "X_abstract = vectorizer.fit_transform(filled_df['abstract'])\n",
    "\n",
    "X_abstract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n_samples: 10399, n_features: 48516\n"
     ]
    }
   ],
   "source": [
    "print(\"n_samples: %d, n_features: %d\" % X_abstract.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's combine these new features with our last engineered features to see if we are on the right track."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "56139"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_yrs.shape[1] + X_fos.shape[1] + X_abstract.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 19.7 s, sys: 43.8 s, total: 1min 3s\n",
      "Wall time: 1min 55s\n"
     ]
    }
   ],
   "source": [
    "# now looking at 10399 x  356139 array for our feature space\n",
    "\n",
    "%time third_features = np.append(second_features, X_abstract.toarray(), axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Based on the paper:\n",
      "Paper index =  2\n",
      "Title : Should endometriosis be an indication for intracytoplasmic sperm injection (ICSI) in fresh IVF cycles\n",
      "FOS : nan\n",
      "Year : 2015\n",
      "Abstract : nan\n",
      "Authors : [{'name': 'Jovana P. Lekovich', 'org': 'Weill Cornell Medical College, New York, NY'}, {'name': 'G.D. Palermo', 'org': 'Weill Medical College of Cornell University, New York, NY'}, {'name': 'Nigel Pereira', 'org': 'The Ronald O. Perelman and Claudia Cohen Center, New York, NY'}, {'name': 'Zev Rosenwaks', 'org': 'Weill Cornell Medical College, New York, NY'}] \n",
      "\n",
      "\n",
      "Top 3 results: \n",
      "1 . Paper index =  10055\n",
      "Similarity score:  1.0\n",
      "Title : [Diagnosis of cerebral tumors; comparative studies on arteriography, ventriculography and electroencephalography].\n",
      "FOS : ['Radiology', 'Pathology', 'Surgery']\n",
      "Year : 1953\n",
      "Abstract : nan\n",
      "Authors : [{'name': 'Antoine'}, {'name': 'Lepoire'}, {'name': 'Schoumacker'}] \n",
      "\n",
      "2 . Paper index =  11773\n",
      "Similarity score:  1.0\n",
      "Title : Studies of powder flow using a recording powder flowmeter and measurement of the dynamic angle of repose\n",
      "FOS : nan\n",
      "Year : 1985\n",
      "Abstract : This paper describes the utility of the dynamic measurement of the angle of repose for pharmaceutical systems, using a variable rotating cylinder to quantify powder flow. The dynamic angle of repose of sodium chloride powder sieve fractions was evaluated using a variable rotating cylinder. The relationship between the static and the dynamic angle of repose is discussed. The dynamic angle of repose of six lots of a multivitamin preparation were compared for inter- and intralot variation. In both cases, no significant differences (p> 0.05) were observed. In the multivitamin formulation, lubricants at lower concentration levels did not show a significant effect (p> 0.05) on the dynamic angle of repose when compared with flow rates. The effect of different hopper sizes and geometry has been evaluated using the recording powder flowmeter. The results indicate that although different hoppers affect the quantitative nature of the results, the same general trends are apparent. Thus, it appears possible to use a recording powder flowmeter with small quantities of material to predict the effect of formulation and processing variables on the flow of production scale quantities. This paper does not describe a comprehensive evaluation of the pharmaceutical utility of measuring the dynamic angle of repose. However, the results discussed are not encouraging and suggest that the recording powder flowmeter is more sensitive to the effects of formulation and production variables on powder flow.\n",
      "Authors : [{'name': 'Ramachandra P. Hegde', 'org': 'Department of Pharmacy, University of Rhode Island, Kingston, RI 02881'}, {'name': 'J.L. Rheingold', 'org': 'Formulation Research, Lederle Laboratories, Pearl River, NY 10965'}, {'name': 'S. Welch', 'org': 'Formulation Research, Lederle Laboratories, Pearl River, NY 10965'}, {'name': 'C. T. Rhodes', 'org': 'Department of Pharmacy, University of Rhode Island, Kingston, RI 02881|||Department of Pharmacy, University of Rhode Island, Kingston, RI 02881'}] \n",
      "\n",
      "3 . Paper index =  11778\n",
      "Similarity score:  1.0\n",
      "Title : Direct antagonistic action of prostaglandins and phenylisopropyladenosine on the activity of pancreatic triglyceride lipase and the antagonistic effect of polyphloretin phosphate\n",
      "FOS : ['Endocrinology', 'Biochemistry', 'Diabetes mellitus']\n",
      "Year : 1974\n",
      "Abstract : nan\n",
      "Authors : [{'name': 'Mentz P'}, {'name': 'Förster W'}, {'name': 'Giessler C'}] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "paper_recommender(third_features, 2, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## authors: One-Hot Encoding using sklearn DictVectorizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>authors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[{'name': 'Ahmed M. Alluwaimi'}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[{'name': 'Jovana P. Lekovich', 'org': 'Weill ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[{'name': 'George C. Sponsler'}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[{'name': 'M. T. Richards'}]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             authors\n",
       "0                                               None\n",
       "1                   [{'name': 'Ahmed M. Alluwaimi'}]\n",
       "2  [{'name': 'Jovana P. Lekovich', 'org': 'Weill ...\n",
       "5                   [{'name': 'George C. Sponsler'}]\n",
       "7                       [{'name': 'M. T. Richards'}]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "authors_df = pd.DataFrame(filled_df.authors)\n",
    "authors_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "authors_list = []\n",
    "\n",
    "for row in authors_df.itertuples():\n",
    "    # create a dictionary from each Series index\n",
    "    if type(row.authors) is str:\n",
    "        y = {'None': row.Index}\n",
    "    if type(row.authors) is list:\n",
    "        # add these keys + values to our running dictionary    \n",
    "        y = dict.fromkeys(row.authors[0].values(), row.Index)\n",
    "    authors_list.append(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'None': 0},\n",
       " {'Ahmed M. Alluwaimi': 1},\n",
       " {'Jovana P. Lekovich': 2, 'Weill Cornell Medical College, New York, NY': 2},\n",
       " {'George C. Sponsler': 5},\n",
       " {'M. T. Richards': 7}]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "authors_list[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       ..., \n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.]])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction import DictVectorizer\n",
    "v = DictVectorizer(sparse=False)\n",
    "D = authors_list\n",
    "X_authors = v.fit_transform(D)\n",
    "\n",
    "X_authors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n_samples: 10399, n_features: 14028\n"
     ]
    }
   ],
   "source": [
    "print(\"n_samples: %d, n_features: %d\" % X_authors.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "70167"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_yrs.shape[1] + X_fos.shape[1] + X_abstract.shape[1] + X_authors.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 11.4 s, sys: 28.6 s, total: 40.1 s\n",
      "Wall time: 57.7 s\n"
     ]
    }
   ],
   "source": [
    "# now looking at 5167 x  38070 array for our feature space\n",
    "\n",
    "%time fourth_features = np.append(third_features, X_authors, axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Based on the paper:\n",
      "Paper index =  2\n",
      "Title : Should endometriosis be an indication for intracytoplasmic sperm injection (ICSI) in fresh IVF cycles\n",
      "FOS : nan\n",
      "Year : 2015\n",
      "Abstract : nan\n",
      "Authors : [{'name': 'Jovana P. Lekovich', 'org': 'Weill Cornell Medical College, New York, NY'}, {'name': 'G.D. Palermo', 'org': 'Weill Medical College of Cornell University, New York, NY'}, {'name': 'Nigel Pereira', 'org': 'The Ronald O. Perelman and Claudia Cohen Center, New York, NY'}, {'name': 'Zev Rosenwaks', 'org': 'Weill Cornell Medical College, New York, NY'}] \n",
      "\n",
      "\n",
      "Top 3 results: \n",
      "1 . Paper index =  10055\n",
      "Similarity score:  1.0\n",
      "Title : [Diagnosis of cerebral tumors; comparative studies on arteriography, ventriculography and electroencephalography].\n",
      "FOS : ['Radiology', 'Pathology', 'Surgery']\n",
      "Year : 1953\n",
      "Abstract : nan\n",
      "Authors : [{'name': 'Antoine'}, {'name': 'Lepoire'}, {'name': 'Schoumacker'}] \n",
      "\n",
      "2 . Paper index =  5601\n",
      "Similarity score:  1.0\n",
      "Title : 633 Survival after coronary revascularization, with and without mitral valve repair, in patients with ischemic mitral regurgitation. Importance of pre-operative myocardial viability\n",
      "FOS : ['Cardiology']\n",
      "Year : 2005\n",
      "Abstract : nan\n",
      "Authors : [{'name': 'J.B. Le Polain De Waroux'}, {'name': 'Anne-Catherine Pouleur'}, {'name': 'B. Beige'}, {'name': 'Agnes Pasquet'}, {'name': 'B. Gerbe'}, {'name': 'B. Noirhomme'}, {'name': 'G. El Khoury'}, {'name': 'Jean Louis Vanoverschelde'}] \n",
      "\n",
      "3 . Paper index =  12256\n",
      "Similarity score:  1.0\n",
      "Title : Nucleotide Sequence and Analysis of an Insertion Sequence from Bacillus thuringiensis Related to IS150\n",
      "FOS : ['Biology', 'Molecular biology', 'Insertion sequence', 'Nucleic acid sequence', 'Bioinformatics', 'Genetics']\n",
      "Year : 1994\n",
      "Abstract : A 5.8-kb DNA fragment encoding the  cryIC  gene from  Bacillus thuringiensis  (Bt) subsp.  aizawai  HD229 was subcloned into the pMex7 vector for expression in  Escherichia coli . In addition to the 135-kDa CryIC δ-endotoxin, this DNA fragment also encoded a 30-kDa polypeptide whose open reading frame ( orfX ) was located less than 200 bp upstream of  cryIC  . Nucleotide sequencing showed that  orfX  was truncated at the 5′ end, and full sequence was obtained from a second overlapping clone. Sequence analysis showed that  orfX  could encode a polypeptide closely related to the putative transposase from IS 150 . OrfX was flanked by a 17-bp imperfect inverted repeat, defining the length of the element as 998 bp. Southern blot analysis revealed that the novel insertion sequence was present in a single copy and located in an identical position immediately upstream of  cryIC  in plasmid DNA from both Bt subsp.  aizawai  and  entomocidus.\n",
      "Authors : [{'name': 'Geoffrey P. Smith'}, {'name': 'David J. Ellar'}, {'name': 'Sharon J. Keeler'}, {'name': 'Cynthia E. Seip'}] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "paper_recommender(fourth_features, 2, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a few more...does that help? results? performance?\n",
    "\n",
    "Yes! We are all in the medical field now. That's great."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## titles: noun phrases + chunking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_df['title'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## keywords: stemming?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_df['keywords'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}