labs/16-Similar_Users_Recommender_Lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Similar Users Lab\n",
    "\n",
    "BUT FIRST a quick word about strings, lists, and sets:\n",
    "\n",
    "## Working with sets\n",
    "\n",
    "In mathematics, a set is a collection of distinct objects.  In Python, \"Sets\" are lists with no duplicate entries. Set objects also support mathematical operations like union, intersection, difference, and symmetric difference.\n",
    "\n",
    "_Fun fact for your next party:  Techincally, Python sets are implemented using dictionaries (under the hood)._\n",
    "\n",
    "Here are two sets of colors:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "a = set([\"Red\", \"Green\", \"Blue\"])\n",
    "b = set([\"Black\", \"White\", \"Green\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find out which items are in both sets (**both sets only**), use the \"intersection\" method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "a.intersection(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find the items in a, but not b."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "a.difference(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find the items in b, but not a."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "b.difference(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find a list of all unique sets (aka: union):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "a.union(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many are different?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print \"Number of different items in b:  %d\" % len(b.difference(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## From Sets to Lists\n",
    "\n",
    "Now that we're experts with working with Python sets.  Let's get savvy working with lists and unstructured data.\n",
    "\n",
    "Using the split() method on a string, we can \"split\" it by a delimiter, to be used as a list.  By default, the .split() method can be applied to any string object, and will automatically split on spaces.  \n",
    "\n",
    "*You can pass a parameter to split to change which character it will split on, such as \",\", if you're trying to turn a comma seprated list of items into a list.*\n",
    "\n",
    "The following will turn a space delimited *string* into a **list**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"my name is dave my name is dave my name is dave\".split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's up with this though?  Well all know \"my name is dave\", but if we had many values, it would be hard to know which of them are unique.  That's when we use sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "set(\"my name is dave my name is dave my name is dave\".split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok so we should know enough to conquer our jaccard distance problem, and step into our real problem:\n",
    "\n",
    "## Who has similar tastes in music?\n",
    "\n",
    "What we will attempt, is building a small process that takes feedback from a survey, mapping a distance function to find similar users based on Jaccard.\n",
    "\n",
    "Along the way we will be:\n",
    "* Working with requests\n",
    "* Understanding Python fundamentals with sets and lists\n",
    "* Cleaning up bad data\n",
    "* Implementing Jaccard distance function\n",
    "* Finding similar users\n",
    "\n",
    "First, we will be taking a survey!  Let's all visit the survey posted in the channel before continuing.\n",
    "\n",
    "*[Check out #General]*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully everything goes smooothly.  It's possible that I may need to modify the permissions on the sheet or provide a CSV snapshot if we hit a snag.\n",
    "\n",
    "We will be loading our results via HTTP, then loading them into Pandas via StringIO which allows us to interoperate on strings as if they were file resources, then load them as a Dataframe.  This is setup for us now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import requests\n",
    "\n",
    "from StringIO import StringIO  \n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "spreadsheet = \"https://docs.google.com/spreadsheets/d/1cpUb7XbN-qOq4xbGdYfhY9FtrMqRd0izz4PmTPMejt0/export?format=csv&id=1cpUb7XbN-qOq4xbGdYfhY9FtrMqRd0izz4PmTPMejt0&gid=216538035\"\n",
    "http = requests.get(spreadsheet)\n",
    "csv_data = StringIO(http.content)\n",
    "df = pd.read_csv(csv_data, index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. Rename the genre feature**\n",
    "\n",
    "We get bad data from spreadsheets all the time.  This case, it's coming from a survey.  For ease of reference, rename the feature **\"Favorite Genres / Genres you like\"** to **\"genres\"**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. Select only your response from the new \"genre\" feature**\n",
    "\n",
    "Try printing out only the first value, where df[\"Name\"] == \"[Your name]\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. Take your survey response for \"genre\", and split it into a list, equal to the number of responses you chose**\n",
    "\n",
    "For example if you chose \"Blues, Reggae, Electronic Music\", convert it to a list that looks like [\"Blues\", \"Raggae\", \"Electronic Music\"]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4. Create a function that takes 2 lists, then calculate Jaccard distance**\n",
    "\n",
    "0-60 mph I know but you can do this!  Double check our slides, and refer to the set operations for how to calculate this.  \n",
    "\n",
    "Here is a boilerplate to get you going."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def jaccard(list1, list2):\n",
    "    print \"list1: \", list1\n",
    "    print \"list2: \", list2\n",
    "    # Update / your code here\n",
    "    \n",
    "list1 = ['blue', 'green', 'yellow']\n",
    "list2 = ['black', 'orange', 'yellow', 'green']\n",
    "\n",
    "jaccard(list1, list2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5.  Now for our final trick, calculate the distance between your genre preferences vs everyone else.**\n",
    "\n",
    "Loop through everyone in the dataframe, create a list out of their \"genre\" string, echo out their name, then finally the distance between you and their sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Optional 6. Try calculating the distance on the time of day feature.**\n",
    "\n",
    "Try to make a new dataframe, for just you vs everyone, using jaccard, and time of day.  Is there any interesting patterns you see?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Optional 7. What can you say about the selection of options for genre or time and what they mean?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}