Example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example of versioning ML experiments using DVC\n",
    "\n",
    "This notebook aims to be a guideline for versioning your ML projects using DVC, from a Jupyter notebook.\n",
    "\n",
    "This notebook allows you to experiment as much as you like, and when you are in a state that you would like to preserve for future reference as a git commit, use the DVC cells to version all your relevant files. \n",
    "\n",
    "The cells marked with a green markdown box are responsible for creating a snapshot of your raw data, processed data, and trained models.\n",
    "\n",
    "This snapshot is implemented as md5 hashes of the respective files saved as text in the `.dvc` files. The hashes in the .dvc files will be part of the git commit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports and global declarations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "import sklearn\n",
    "from sklearn import preprocessing\n",
    "from sklearn.externals import joblib\n",
    "from sklearn import metrics\n",
    "from sklearn import model_selection\n",
    "import numpy as np\n",
    "import pickle\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LinearRegression\n",
    "import json\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<h2>Download and version raw data</h2>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading Cal. housing from https://ndownloader.figshare.com/files/5976036 to data/raw\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving 'data/raw' to cache '.dvc/cache'.\n",
      "Linking directory 'data/raw'.\n",
      "Saving information to 'data/raw.dvc'.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "raw_data = datasets.fetch_california_housing(data_home=\"data/raw\")\n",
    "# Save the raw input data for reproducibility\n",
    "!dvc commit -f data/raw.dvc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_dataframe(X, y):\n",
    "    return pd.concat([\n",
    "            pd.DataFrame(data=X, columns=raw_data.feature_names),\n",
    "            pd.DataFrame(data=y, columns=['Value'])\n",
    "        ],\n",
    "        axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MedInc</th>\n",
       "      <th>HouseAge</th>\n",
       "      <th>AveRooms</th>\n",
       "      <th>AveBedrms</th>\n",
       "      <th>Population</th>\n",
       "      <th>AveOccup</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8.3252</td>\n",
       "      <td>41.0</td>\n",
       "      <td>6.984127</td>\n",
       "      <td>1.023810</td>\n",
       "      <td>322.0</td>\n",
       "      <td>2.555556</td>\n",
       "      <td>37.88</td>\n",
       "      <td>-122.23</td>\n",
       "      <td>4.526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8.3014</td>\n",
       "      <td>21.0</td>\n",
       "      <td>6.238137</td>\n",
       "      <td>0.971880</td>\n",
       "      <td>2401.0</td>\n",
       "      <td>2.109842</td>\n",
       "      <td>37.86</td>\n",
       "      <td>-122.22</td>\n",
       "      <td>3.585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7.2574</td>\n",
       "      <td>52.0</td>\n",
       "      <td>8.288136</td>\n",
       "      <td>1.073446</td>\n",
       "      <td>496.0</td>\n",
       "      <td>2.802260</td>\n",
       "      <td>37.85</td>\n",
       "      <td>-122.24</td>\n",
       "      <td>3.521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.6431</td>\n",
       "      <td>52.0</td>\n",
       "      <td>5.817352</td>\n",
       "      <td>1.073059</td>\n",
       "      <td>558.0</td>\n",
       "      <td>2.547945</td>\n",
       "      <td>37.85</td>\n",
       "      <td>-122.25</td>\n",
       "      <td>3.413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.8462</td>\n",
       "      <td>52.0</td>\n",
       "      <td>6.281853</td>\n",
       "      <td>1.081081</td>\n",
       "      <td>565.0</td>\n",
       "      <td>2.181467</td>\n",
       "      <td>37.85</td>\n",
       "      <td>-122.25</td>\n",
       "      <td>3.422</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MedInc  HouseAge  AveRooms  AveBedrms  Population  AveOccup  Latitude  \\\n",
       "0  8.3252      41.0  6.984127   1.023810       322.0  2.555556     37.88   \n",
       "1  8.3014      21.0  6.238137   0.971880      2401.0  2.109842     37.86   \n",
       "2  7.2574      52.0  8.288136   1.073446       496.0  2.802260     37.85   \n",
       "3  5.6431      52.0  5.817352   1.073059       558.0  2.547945     37.85   \n",
       "4  3.8462      52.0  6.281853   1.081081       565.0  2.181467     37.85   \n",
       "\n",
       "   Longitude  Value  \n",
       "0    -122.23  4.526  \n",
       "1    -122.22  3.585  \n",
       "2    -122.24  3.521  \n",
       "3    -122.25  3.413  \n",
       "4    -122.25  3.422  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_df = to_dataframe(raw_data.data, raw_data.target)\n",
    "raw_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MedInc</th>\n",
       "      <th>HouseAge</th>\n",
       "      <th>AveRooms</th>\n",
       "      <th>AveBedrms</th>\n",
       "      <th>Population</th>\n",
       "      <th>AveOccup</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "      <td>20640.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.870671</td>\n",
       "      <td>28.639486</td>\n",
       "      <td>5.429000</td>\n",
       "      <td>1.096675</td>\n",
       "      <td>1425.476744</td>\n",
       "      <td>3.070655</td>\n",
       "      <td>35.631861</td>\n",
       "      <td>-119.569704</td>\n",
       "      <td>2.068558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.899822</td>\n",
       "      <td>12.585558</td>\n",
       "      <td>2.474173</td>\n",
       "      <td>0.473911</td>\n",
       "      <td>1132.462122</td>\n",
       "      <td>10.386050</td>\n",
       "      <td>2.135952</td>\n",
       "      <td>2.003532</td>\n",
       "      <td>1.153956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.499900</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.846154</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.692308</td>\n",
       "      <td>32.540000</td>\n",
       "      <td>-124.350000</td>\n",
       "      <td>0.149990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2.563400</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>4.440716</td>\n",
       "      <td>1.006079</td>\n",
       "      <td>787.000000</td>\n",
       "      <td>2.429741</td>\n",
       "      <td>33.930000</td>\n",
       "      <td>-121.800000</td>\n",
       "      <td>1.196000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.534800</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>5.229129</td>\n",
       "      <td>1.048780</td>\n",
       "      <td>1166.000000</td>\n",
       "      <td>2.818116</td>\n",
       "      <td>34.260000</td>\n",
       "      <td>-118.490000</td>\n",
       "      <td>1.797000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>4.743250</td>\n",
       "      <td>37.000000</td>\n",
       "      <td>6.052381</td>\n",
       "      <td>1.099526</td>\n",
       "      <td>1725.000000</td>\n",
       "      <td>3.282261</td>\n",
       "      <td>37.710000</td>\n",
       "      <td>-118.010000</td>\n",
       "      <td>2.647250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>15.000100</td>\n",
       "      <td>52.000000</td>\n",
       "      <td>141.909091</td>\n",
       "      <td>34.066667</td>\n",
       "      <td>35682.000000</td>\n",
       "      <td>1243.333333</td>\n",
       "      <td>41.950000</td>\n",
       "      <td>-114.310000</td>\n",
       "      <td>5.000010</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             MedInc      HouseAge      AveRooms     AveBedrms    Population  \\\n",
       "count  20640.000000  20640.000000  20640.000000  20640.000000  20640.000000   \n",
       "mean       3.870671     28.639486      5.429000      1.096675   1425.476744   \n",
       "std        1.899822     12.585558      2.474173      0.473911   1132.462122   \n",
       "min        0.499900      1.000000      0.846154      0.333333      3.000000   \n",
       "25%        2.563400     18.000000      4.440716      1.006079    787.000000   \n",
       "50%        3.534800     29.000000      5.229129      1.048780   1166.000000   \n",
       "75%        4.743250     37.000000      6.052381      1.099526   1725.000000   \n",
       "max       15.000100     52.000000    141.909091     34.066667  35682.000000   \n",
       "\n",
       "           AveOccup      Latitude     Longitude         Value  \n",
       "count  20640.000000  20640.000000  20640.000000  20640.000000  \n",
       "mean       3.070655     35.631861   -119.569704      2.068558  \n",
       "std       10.386050      2.135952      2.003532      1.153956  \n",
       "min        0.692308     32.540000   -124.350000      0.149990  \n",
       "25%        2.429741     33.930000   -121.800000      1.196000  \n",
       "50%        2.818116     34.260000   -118.490000      1.797000  \n",
       "75%        3.282261     37.710000   -118.010000      2.647250  \n",
       "max     1243.333333     41.950000   -114.310000      5.000010  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test train split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_X, test_X, train_y, test_y = model_selection.train_test_split(raw_df[raw_df.columns[:-1]], raw_df['Value'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MedInc</th>\n",
       "      <th>HouseAge</th>\n",
       "      <th>AveRooms</th>\n",
       "      <th>AveBedrms</th>\n",
       "      <th>Population</th>\n",
       "      <th>AveOccup</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17584</th>\n",
       "      <td>2.6573</td>\n",
       "      <td>29.0</td>\n",
       "      <td>3.451128</td>\n",
       "      <td>1.077694</td>\n",
       "      <td>969.0</td>\n",
       "      <td>2.428571</td>\n",
       "      <td>37.31</td>\n",
       "      <td>-121.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11752</th>\n",
       "      <td>3.1583</td>\n",
       "      <td>16.0</td>\n",
       "      <td>5.622378</td>\n",
       "      <td>1.034965</td>\n",
       "      <td>792.0</td>\n",
       "      <td>2.769231</td>\n",
       "      <td>38.76</td>\n",
       "      <td>-121.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>909</th>\n",
       "      <td>4.9167</td>\n",
       "      <td>34.0</td>\n",
       "      <td>5.963675</td>\n",
       "      <td>1.057692</td>\n",
       "      <td>1276.0</td>\n",
       "      <td>2.726496</td>\n",
       "      <td>37.55</td>\n",
       "      <td>-122.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6418</th>\n",
       "      <td>2.5417</td>\n",
       "      <td>43.0</td>\n",
       "      <td>4.186386</td>\n",
       "      <td>1.025932</td>\n",
       "      <td>1544.0</td>\n",
       "      <td>2.502431</td>\n",
       "      <td>34.15</td>\n",
       "      <td>-118.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1631</th>\n",
       "      <td>7.6107</td>\n",
       "      <td>33.0</td>\n",
       "      <td>7.554167</td>\n",
       "      <td>1.045833</td>\n",
       "      <td>1348.0</td>\n",
       "      <td>2.808333</td>\n",
       "      <td>37.88</td>\n",
       "      <td>-122.17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       MedInc  HouseAge  AveRooms  AveBedrms  Population  AveOccup  Latitude  \\\n",
       "17584  2.6573      29.0  3.451128   1.077694       969.0  2.428571     37.31   \n",
       "11752  3.1583      16.0  5.622378   1.034965       792.0  2.769231     38.76   \n",
       "909    4.9167      34.0  5.963675   1.057692      1276.0  2.726496     37.55   \n",
       "6418   2.5417      43.0  4.186386   1.025932      1544.0  2.502431     34.15   \n",
       "1631   7.6107      33.0  7.554167   1.045833      1348.0  2.808333     37.88   \n",
       "\n",
       "       Longitude  \n",
       "17584    -121.93  \n",
       "11752    -121.21  \n",
       "909      -122.01  \n",
       "6418     -118.00  \n",
       "1631     -122.17  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalize feature columns by training data only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MedInc</th>\n",
       "      <th>HouseAge</th>\n",
       "      <th>AveRooms</th>\n",
       "      <th>AveBedrms</th>\n",
       "      <th>Population</th>\n",
       "      <th>AveOccup</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17584</th>\n",
       "      <td>-0.643187</td>\n",
       "      <td>0.029381</td>\n",
       "      <td>-0.774588</td>\n",
       "      <td>-0.034971</td>\n",
       "      <td>-0.402238</td>\n",
       "      <td>-0.057802</td>\n",
       "      <td>0.789598</td>\n",
       "      <td>-1.181518</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11752</th>\n",
       "      <td>-0.377902</td>\n",
       "      <td>-1.004661</td>\n",
       "      <td>0.079716</td>\n",
       "      <td>-0.120778</td>\n",
       "      <td>-0.556614</td>\n",
       "      <td>-0.027600</td>\n",
       "      <td>1.467986</td>\n",
       "      <td>-0.822341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>909</th>\n",
       "      <td>0.553191</td>\n",
       "      <td>0.427090</td>\n",
       "      <td>0.214004</td>\n",
       "      <td>-0.075138</td>\n",
       "      <td>-0.134478</td>\n",
       "      <td>-0.031389</td>\n",
       "      <td>0.901883</td>\n",
       "      <td>-1.221427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6418</th>\n",
       "      <td>-0.704399</td>\n",
       "      <td>1.142965</td>\n",
       "      <td>-0.485292</td>\n",
       "      <td>-0.138919</td>\n",
       "      <td>0.099267</td>\n",
       "      <td>-0.051254</td>\n",
       "      <td>-0.688818</td>\n",
       "      <td>0.778992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1631</th>\n",
       "      <td>1.979695</td>\n",
       "      <td>0.347548</td>\n",
       "      <td>0.839802</td>\n",
       "      <td>-0.098953</td>\n",
       "      <td>-0.071681</td>\n",
       "      <td>-0.024134</td>\n",
       "      <td>1.056275</td>\n",
       "      <td>-1.301244</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         MedInc  HouseAge  AveRooms  AveBedrms  Population  AveOccup  \\\n",
       "17584 -0.643187  0.029381 -0.774588  -0.034971   -0.402238 -0.057802   \n",
       "11752 -0.377902 -1.004661  0.079716  -0.120778   -0.556614 -0.027600   \n",
       "909    0.553191  0.427090  0.214004  -0.075138   -0.134478 -0.031389   \n",
       "6418  -0.704399  1.142965 -0.485292  -0.138919    0.099267 -0.051254   \n",
       "1631   1.979695  0.347548  0.839802  -0.098953   -0.071681 -0.024134   \n",
       "\n",
       "       Latitude  Longitude  \n",
       "17584  0.789598  -1.181518  \n",
       "11752  1.467986  -0.822341  \n",
       "909    0.901883  -1.221427  \n",
       "6418  -0.688818   0.778992  \n",
       "1631   1.056275  -1.301244  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler = preprocessing.StandardScaler()\n",
    "train_X_scaled = pd.DataFrame(scaler.fit_transform(train_X), index=train_X.index, columns=train_X.columns)\n",
    "train_X_scaled.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MedInc</th>\n",
       "      <th>HouseAge</th>\n",
       "      <th>AveRooms</th>\n",
       "      <th>AveBedrms</th>\n",
       "      <th>Population</th>\n",
       "      <th>AveOccup</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>186</th>\n",
       "      <td>-0.495771</td>\n",
       "      <td>1.858841</td>\n",
       "      <td>-0.429702</td>\n",
       "      <td>-0.273856</td>\n",
       "      <td>0.199568</td>\n",
       "      <td>0.083884</td>\n",
       "      <td>1.014168</td>\n",
       "      <td>-1.331175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4934</th>\n",
       "      <td>-1.069603</td>\n",
       "      <td>0.904340</td>\n",
       "      <td>-0.571272</td>\n",
       "      <td>0.074738</td>\n",
       "      <td>0.768230</td>\n",
       "      <td>0.114198</td>\n",
       "      <td>-0.758996</td>\n",
       "      <td>0.644301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>911</th>\n",
       "      <td>0.597299</td>\n",
       "      <td>0.268006</td>\n",
       "      <td>0.429510</td>\n",
       "      <td>-0.135071</td>\n",
       "      <td>-0.263561</td>\n",
       "      <td>-0.019934</td>\n",
       "      <td>0.897205</td>\n",
       "      <td>-1.221427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2359</th>\n",
       "      <td>0.557586</td>\n",
       "      <td>-0.527411</td>\n",
       "      <td>0.515851</td>\n",
       "      <td>-0.187719</td>\n",
       "      <td>0.318184</td>\n",
       "      <td>-0.013504</td>\n",
       "      <td>0.532279</td>\n",
       "      <td>-0.034146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16169</th>\n",
       "      <td>-0.504031</td>\n",
       "      <td>0.268006</td>\n",
       "      <td>-0.666624</td>\n",
       "      <td>-0.004922</td>\n",
       "      <td>-0.175470</td>\n",
       "      <td>-0.082624</td>\n",
       "      <td>1.014168</td>\n",
       "      <td>-1.490810</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         MedInc  HouseAge  AveRooms  AveBedrms  Population  AveOccup  \\\n",
       "186   -0.495771  1.858841 -0.429702  -0.273856    0.199568  0.083884   \n",
       "4934  -1.069603  0.904340 -0.571272   0.074738    0.768230  0.114198   \n",
       "911    0.597299  0.268006  0.429510  -0.135071   -0.263561 -0.019934   \n",
       "2359   0.557586 -0.527411  0.515851  -0.187719    0.318184 -0.013504   \n",
       "16169 -0.504031  0.268006 -0.666624  -0.004922   -0.175470 -0.082624   \n",
       "\n",
       "       Latitude  Longitude  \n",
       "186    1.014168  -1.331175  \n",
       "4934  -0.758996   0.644301  \n",
       "911    0.897205  -1.221427  \n",
       "2359   0.532279  -0.034146  \n",
       "16169  1.014168  -1.490810  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_X_scaled = pd.DataFrame(scaler.transform(test_X), index=test_X.index, columns=test_X.columns)\n",
    "test_X_scaled.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.concat([train_X_scaled, train_y], axis=1)\n",
    "test_df = pd.concat([test_X_scaled, test_y], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<h2>Optional: Version the processed data with DVC for efficiency and/or reproducibility</h2>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving 'data/processed' to cache '.dvc/cache'.\n",
      "Linking directory 'data/processed'.\n",
      "Saving information to 'process_data.dvc'.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "os.makedirs('data/processed/')\n",
    "train_df.to_csv('data/processed/california_households_train.csv', index_label='Index')\n",
    "test_df.to_csv('data/processed/california_households_test.csv', index_label='Index')\n",
    "joblib.dump(scaler, 'data/processed/california_households_scaler.pkl')\n",
    "!dvc commit -f process_data.dvc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use this cell to reload processed data, after switching branches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.read_csv('data/processed/california_households_train.csv', index_col=0)\n",
    "test_df = pd.read_csv('data/processed/california_households_train.csv', index_col=0)\n",
    "scaler = joblib.load('data/processed/california_households_scaler.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,\n",
       "         normalize=False)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = LinearRegression()\n",
    "X = train_df[train_df.columns[:-1]]\n",
    "y = train_df['Value']\n",
    "model.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<h2>Save the trained model for reproducibility</h2>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving 'models' to cache '.dvc/cache'.\n",
      "Linking directory 'models'.\n",
      "Saving information to 'models.dvc'.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "os.makedirs('models')\n",
    "joblib.dump(model, 'models/california_households.pkl')\n",
    "!dvc commit -f models.dvc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use this cell to reload the model, after switching branches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = joblib.load('models/california_households.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>R2</th>\n",
       "      <th>MAE</th>\n",
       "      <th>MSE</th>\n",
       "      <th>median_absolute_error</th>\n",
       "      <th>loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.609808</td>\n",
       "      <td>0.528453</td>\n",
       "      <td>0.515188</td>\n",
       "      <td>0.412521</td>\n",
       "      <td>0.515188</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         R2       MAE       MSE  median_absolute_error      loss\n",
       "0  0.609808  0.528453  0.515188               0.412521  0.515188"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions = model.predict(test_df[test_df.columns[:-1]])\n",
    "truth = test_df['Value']\n",
    "metrics_dict = {}\n",
    "metrics_dict['R2'] = metrics.r2_score(truth, predictions)\n",
    "metrics_dict['MAE'] = metrics.mean_absolute_error(truth, predictions)\n",
    "metrics_dict['MSE'] = metrics.mean_squared_error(truth, predictions)\n",
    "metrics_dict['median_absolute_error'] = metrics.median_absolute_error(truth, predictions)\n",
    "metrics_dict['loss'] = metrics_dict['MSE']\n",
    "pd.DataFrame(metrics_dict, index=[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<h2>Save the computed metrics for easy display in DVC and DAGsHub</h2>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output 'metrics/metrics.json' doesn't use cache. Skipping saving.\n",
      "Saving information to 'eval.dvc'.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "with open('metrics/metrics.json', 'w') as f:\n",
    "    json.dump(metrics_dict, f, indent=2)\n",
    "!dvc commit -f eval.dvc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<h2>Versioning section - use the following cells to create a full commit of your current state</h2>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make sure all data and models are committed to DVC\n",
    "The output of the following cell should be: `Pipeline is up to date. Nothing to reproduce.`\n",
    "\n",
    "If you get something else, then maybe you forgot to `dvc commit` earlier in the notebook.\n",
    "We recommend to make sure that the current contents in the data and models directories are to your liking,\n",
    "and if so, use the commit cell below to automatically commit all current files to DVC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline is up to date. Nothing to reproduce.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!dvc status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 427,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving 'models' to cache '.dvc/cache'.\n",
      "Linking directory 'models'.\n",
      "Saving information to 'models.dvc'.\n",
      "Saving 'data/processed' to cache '.dvc/cache'.\n",
      "Linking directory 'data/processed'.\n",
      "Saving information to 'process_data.dvc'.\n",
      "Saving information to 'eval.dvc'.\n",
      "Saving 'data/raw' to cache '.dvc/cache'.\n",
      "Linking directory 'data/raw'.\n",
      "Saving information to 'data/raw.dvc'.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "# Use this if dvc status is not up-to-date and you're sure the current state is OK.\n",
    "!dvc commit -f"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}