Handling NaN - Feature Engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0ed2d08e",
   "metadata": {},
   "source": [
    "# Handeling the missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5dfac6b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing necessary libraries\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5922fadc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>...</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>70</td>\n",
       "      <td>RL</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>WD</td>\n",
       "      <td>Abnorml</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 81 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
       "0   1          60       RL         65.0     8450   Pave   NaN      Reg   \n",
       "1   2          20       RL         80.0     9600   Pave   NaN      Reg   \n",
       "2   3          60       RL         68.0    11250   Pave   NaN      IR1   \n",
       "3   4          70       RL         60.0     9550   Pave   NaN      IR1   \n",
       "4   5          60       RL         84.0    14260   Pave   NaN      IR1   \n",
       "\n",
       "  LandContour Utilities  ... PoolArea PoolQC Fence MiscFeature MiscVal MoSold  \\\n",
       "0         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      2   \n",
       "1         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      5   \n",
       "2         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      9   \n",
       "3         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      2   \n",
       "4         Lvl    AllPub  ...        0    NaN   NaN         NaN       0     12   \n",
       "\n",
       "  YrSold  SaleType  SaleCondition  SalePrice  \n",
       "0   2008        WD         Normal     208500  \n",
       "1   2007        WD         Normal     181500  \n",
       "2   2008        WD         Normal     223500  \n",
       "3   2006        WD        Abnorml     140000  \n",
       "4   2008        WD         Normal     250000  \n",
       "\n",
       "[5 rows x 81 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# importing dataset with missing values\n",
    "\n",
    "df_train = pd.read_csv('datasets/train.csv')\n",
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c11b22c",
   "metadata": {},
   "source": [
    "As we can see in the above dataset that we have many null values. So, in this section we'll deal with that first."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad449c9e",
   "metadata": {},
   "source": [
    "There are hree ways we can do this:\n",
    "1. Dropping columns with missing values\n",
    "2. Imputation\n",
    "3. An extension to imputation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4fb5fc5",
   "metadata": {},
   "source": [
    "## 1. Dropping columns with missing values\n",
    "\n",
    "The simplest option is to drop columns with missing values.\n",
    "\n",
    "Unless most values in the dropped columns are missing, the model loses access to a lot of (potentially useful!) information with this approach. As an extreme example, consider a dataset with 10,000 rows, where one important column is missing a single entry. This approach would drop the column entirely!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b281324a",
   "metadata": {},
   "source": [
    "## 2. Imputation\n",
    "\n",
    "Imputation fills in the missing values with some number. For instance, we can fill in the mean value along each column.\n",
    "\n",
    "\n",
    "The imputed value won't be exactly right in most cases, but it usually leads to more accurate models than you would get from dropping the column entirely."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d3b2617",
   "metadata": {},
   "source": [
    "## 3. An extension to Imputation\n",
    "\n",
    "Imputation is the standard approach, and it usually works well. However, imputed values may be systematically above or below their actual values (which weren't collected in the dataset). Or rows with missing values may be unique in some other way. \n",
    "\n",
    "In this approach, we impute the missing values, as before. And, additionally, for each column with missing entries in the original dataset, we add a new column that shows the location of the imputed entries.\n",
    "\n",
    "In some cases, this will meaningfully improve results. In other cases, it doesn't help at all.\n",
    "                                                                           "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "166ff91a",
   "metadata": {},
   "source": [
    "## Droping all categorical columns from dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b295c923",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Id',\n",
       " 'MSSubClass',\n",
       " 'MSZoning',\n",
       " 'LotFrontage',\n",
       " 'LotArea',\n",
       " 'Street',\n",
       " 'Alley',\n",
       " 'LotShape',\n",
       " 'LandContour',\n",
       " 'Utilities',\n",
       " 'LotConfig',\n",
       " 'LandSlope',\n",
       " 'Neighborhood',\n",
       " 'Condition1',\n",
       " 'Condition2',\n",
       " 'BldgType',\n",
       " 'HouseStyle',\n",
       " 'OverallQual',\n",
       " 'OverallCond',\n",
       " 'YearBuilt',\n",
       " 'YearRemodAdd',\n",
       " 'RoofStyle',\n",
       " 'RoofMatl',\n",
       " 'Exterior1st',\n",
       " 'Exterior2nd',\n",
       " 'MasVnrType',\n",
       " 'MasVnrArea',\n",
       " 'ExterQual',\n",
       " 'ExterCond',\n",
       " 'Foundation',\n",
       " 'BsmtQual',\n",
       " 'BsmtCond',\n",
       " 'BsmtExposure',\n",
       " 'BsmtFinType1',\n",
       " 'BsmtFinSF1',\n",
       " 'BsmtFinType2',\n",
       " 'BsmtFinSF2',\n",
       " 'BsmtUnfSF',\n",
       " 'TotalBsmtSF',\n",
       " 'Heating',\n",
       " 'HeatingQC',\n",
       " 'CentralAir',\n",
       " 'Electrical',\n",
       " '1stFlrSF',\n",
       " '2ndFlrSF',\n",
       " 'LowQualFinSF',\n",
       " 'GrLivArea',\n",
       " 'BsmtFullBath',\n",
       " 'BsmtHalfBath',\n",
       " 'FullBath',\n",
       " 'HalfBath',\n",
       " 'BedroomAbvGr',\n",
       " 'KitchenAbvGr',\n",
       " 'KitchenQual',\n",
       " 'TotRmsAbvGrd',\n",
       " 'Functional',\n",
       " 'Fireplaces',\n",
       " 'FireplaceQu',\n",
       " 'GarageType',\n",
       " 'GarageYrBlt',\n",
       " 'GarageFinish',\n",
       " 'GarageCars',\n",
       " 'GarageArea',\n",
       " 'GarageQual',\n",
       " 'GarageCond',\n",
       " 'PavedDrive',\n",
       " 'WoodDeckSF',\n",
       " 'OpenPorchSF',\n",
       " 'EnclosedPorch',\n",
       " '3SsnPorch',\n",
       " 'ScreenPorch',\n",
       " 'PoolArea',\n",
       " 'PoolQC',\n",
       " 'Fence',\n",
       " 'MiscFeature',\n",
       " 'MiscVal',\n",
       " 'MoSold',\n",
       " 'YrSold',\n",
       " 'SaleType',\n",
       " 'SaleCondition',\n",
       " 'SalePrice']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get the names of all columns \n",
    "\n",
    "list_of_all_cols = [column for column in df_train.columns]\n",
    "list_of_all_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8ac0962c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>...</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1455</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1456</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1457</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1458</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1459</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1460 rows × 81 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Id  MSSubClass  MSZoning  LotFrontage  LotArea  Street  Alley  \\\n",
       "0     False       False     False        False    False   False   True   \n",
       "1     False       False     False        False    False   False   True   \n",
       "2     False       False     False        False    False   False   True   \n",
       "3     False       False     False        False    False   False   True   \n",
       "4     False       False     False        False    False   False   True   \n",
       "...     ...         ...       ...          ...      ...     ...    ...   \n",
       "1455  False       False     False        False    False   False   True   \n",
       "1456  False       False     False        False    False   False   True   \n",
       "1457  False       False     False        False    False   False   True   \n",
       "1458  False       False     False        False    False   False   True   \n",
       "1459  False       False     False        False    False   False   True   \n",
       "\n",
       "      LotShape  LandContour  Utilities  ...  PoolArea  PoolQC  Fence  \\\n",
       "0        False        False      False  ...     False    True   True   \n",
       "1        False        False      False  ...     False    True   True   \n",
       "2        False        False      False  ...     False    True   True   \n",
       "3        False        False      False  ...     False    True   True   \n",
       "4        False        False      False  ...     False    True   True   \n",
       "...        ...          ...        ...  ...       ...     ...    ...   \n",
       "1455     False        False      False  ...     False    True   True   \n",
       "1456     False        False      False  ...     False    True  False   \n",
       "1457     False        False      False  ...     False    True  False   \n",
       "1458     False        False      False  ...     False    True   True   \n",
       "1459     False        False      False  ...     False    True   True   \n",
       "\n",
       "      MiscFeature  MiscVal  MoSold  YrSold  SaleType  SaleCondition  SalePrice  \n",
       "0            True    False   False   False     False          False      False  \n",
       "1            True    False   False   False     False          False      False  \n",
       "2            True    False   False   False     False          False      False  \n",
       "3            True    False   False   False     False          False      False  \n",
       "4            True    False   False   False     False          False      False  \n",
       "...           ...      ...     ...     ...       ...            ...        ...  \n",
       "1455         True    False   False   False     False          False      False  \n",
       "1456         True    False   False   False     False          False      False  \n",
       "1457        False    False   False   False     False          False      False  \n",
       "1458         True    False   False   False     False          False      False  \n",
       "1459         True    False   False   False     False          False      False  \n",
       "\n",
       "[1460 rows x 81 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train[list_of_all_cols].isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1cd848a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id               False\n",
       "MSSubClass       False\n",
       "MSZoning         False\n",
       "LotFrontage       True\n",
       "LotArea          False\n",
       "                 ...  \n",
       "MoSold           False\n",
       "YrSold           False\n",
       "SaleType         False\n",
       "SaleCondition    False\n",
       "SalePrice        False\n",
       "Length: 81, dtype: bool"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train[list_of_all_cols].isnull().any()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "affeac53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['LotFrontage',\n",
       " 'Alley',\n",
       " 'MasVnrType',\n",
       " 'MasVnrArea',\n",
       " 'BsmtQual',\n",
       " 'BsmtCond',\n",
       " 'BsmtExposure',\n",
       " 'BsmtFinType1',\n",
       " 'BsmtFinType2',\n",
       " 'Electrical',\n",
       " 'FireplaceQu',\n",
       " 'GarageType',\n",
       " 'GarageYrBlt',\n",
       " 'GarageFinish',\n",
       " 'GarageQual',\n",
       " 'GarageCond',\n",
       " 'PoolQC',\n",
       " 'Fence',\n",
       " 'MiscFeature']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now getting the columns with missing values\n",
    "\n",
    "missing_cols = [column for column in df_train.columns\n",
    "               if df_train[column].isnull().any()]\n",
    "missing_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "54cfaaeb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "object\n"
     ]
    }
   ],
   "source": [
    "# As we can see below we also have categorical/object type columns\n",
    "\n",
    "print(df_train['MSZoning'].dtypes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "0d210f2c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No. of categorical features:  43\n"
     ]
    }
   ],
   "source": [
    "# getting only categorical columns\n",
    "\n",
    "cat_columns = [col for col in df_train.columns \n",
    "              if df_train[col].dtypes == 'object']\n",
    "print(\"No. of categorical features: \", len(cat_columns))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "16342233",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total no. of features:  81\n"
     ]
    }
   ],
   "source": [
    "print(\"Total no. of features: \", len(df_train.columns))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "e0e7d93c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# removing all the categorical features from our dataset\n",
    "\n",
    "df_train.drop(columns=cat_columns, axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "a4c7105d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1460 entries, 0 to 1459\n",
      "Data columns (total 38 columns):\n",
      " #   Column         Non-Null Count  Dtype  \n",
      "---  ------         --------------  -----  \n",
      " 0   Id             1460 non-null   int64  \n",
      " 1   MSSubClass     1460 non-null   int64  \n",
      " 2   LotFrontage    1201 non-null   float64\n",
      " 3   LotArea        1460 non-null   int64  \n",
      " 4   OverallQual    1460 non-null   int64  \n",
      " 5   OverallCond    1460 non-null   int64  \n",
      " 6   YearBuilt      1460 non-null   int64  \n",
      " 7   YearRemodAdd   1460 non-null   int64  \n",
      " 8   MasVnrArea     1452 non-null   float64\n",
      " 9   BsmtFinSF1     1460 non-null   int64  \n",
      " 10  BsmtFinSF2     1460 non-null   int64  \n",
      " 11  BsmtUnfSF      1460 non-null   int64  \n",
      " 12  TotalBsmtSF    1460 non-null   int64  \n",
      " 13  1stFlrSF       1460 non-null   int64  \n",
      " 14  2ndFlrSF       1460 non-null   int64  \n",
      " 15  LowQualFinSF   1460 non-null   int64  \n",
      " 16  GrLivArea      1460 non-null   int64  \n",
      " 17  BsmtFullBath   1460 non-null   int64  \n",
      " 18  BsmtHalfBath   1460 non-null   int64  \n",
      " 19  FullBath       1460 non-null   int64  \n",
      " 20  HalfBath       1460 non-null   int64  \n",
      " 21  BedroomAbvGr   1460 non-null   int64  \n",
      " 22  KitchenAbvGr   1460 non-null   int64  \n",
      " 23  TotRmsAbvGrd   1460 non-null   int64  \n",
      " 24  Fireplaces     1460 non-null   int64  \n",
      " 25  GarageYrBlt    1379 non-null   float64\n",
      " 26  GarageCars     1460 non-null   int64  \n",
      " 27  GarageArea     1460 non-null   int64  \n",
      " 28  WoodDeckSF     1460 non-null   int64  \n",
      " 29  OpenPorchSF    1460 non-null   int64  \n",
      " 30  EnclosedPorch  1460 non-null   int64  \n",
      " 31  3SsnPorch      1460 non-null   int64  \n",
      " 32  ScreenPorch    1460 non-null   int64  \n",
      " 33  PoolArea       1460 non-null   int64  \n",
      " 34  MiscVal        1460 non-null   int64  \n",
      " 35  MoSold         1460 non-null   int64  \n",
      " 36  YrSold         1460 non-null   int64  \n",
      " 37  SalePrice      1460 non-null   int64  \n",
      "dtypes: float64(3), int64(35)\n",
      "memory usage: 433.6 KB\n"
     ]
    }
   ],
   "source": [
    "df_train.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "509a0c0d",
   "metadata": {},
   "source": [
    "As we can see above that we have dropped all the categorical features from our dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "2aff7c21",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>...</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2003</td>\n",
       "      <td>2003</td>\n",
       "      <td>196.0</td>\n",
       "      <td>706</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>61</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>1976</td>\n",
       "      <td>1976</td>\n",
       "      <td>0.0</td>\n",
       "      <td>978</td>\n",
       "      <td>...</td>\n",
       "      <td>298</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2001</td>\n",
       "      <td>2002</td>\n",
       "      <td>162.0</td>\n",
       "      <td>486</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>70</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>1915</td>\n",
       "      <td>1970</td>\n",
       "      <td>0.0</td>\n",
       "      <td>216</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>272</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>2000</td>\n",
       "      <td>2000</td>\n",
       "      <td>350.0</td>\n",
       "      <td>655</td>\n",
       "      <td>...</td>\n",
       "      <td>192</td>\n",
       "      <td>84</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass  LotFrontage  LotArea  OverallQual  OverallCond  YearBuilt  \\\n",
       "0   1          60         65.0     8450            7            5       2003   \n",
       "1   2          20         80.0     9600            6            8       1976   \n",
       "2   3          60         68.0    11250            7            5       2001   \n",
       "3   4          70         60.0     9550            7            5       1915   \n",
       "4   5          60         84.0    14260            8            5       2000   \n",
       "\n",
       "   YearRemodAdd  MasVnrArea  BsmtFinSF1  ...  WoodDeckSF  OpenPorchSF  \\\n",
       "0          2003       196.0         706  ...           0           61   \n",
       "1          1976         0.0         978  ...         298            0   \n",
       "2          2002       162.0         486  ...           0           42   \n",
       "3          1970         0.0         216  ...           0           35   \n",
       "4          2000       350.0         655  ...         192           84   \n",
       "\n",
       "   EnclosedPorch  3SsnPorch  ScreenPorch  PoolArea  MiscVal  MoSold  YrSold  \\\n",
       "0              0          0            0         0        0       2    2008   \n",
       "1              0          0            0         0        0       5    2007   \n",
       "2              0          0            0         0        0       9    2008   \n",
       "3            272          0            0         0        0       2    2006   \n",
       "4              0          0            0         0        0      12    2008   \n",
       "\n",
       "   SalePrice  \n",
       "0     208500  \n",
       "1     181500  \n",
       "2     223500  \n",
       "3     140000  \n",
       "4     250000  \n",
       "\n",
       "[5 rows x 38 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "8e46b225",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                 0\n",
       "MSSubClass         0\n",
       "LotFrontage      259\n",
       "LotArea            0\n",
       "OverallQual        0\n",
       "OverallCond        0\n",
       "YearBuilt          0\n",
       "YearRemodAdd       0\n",
       "MasVnrArea         8\n",
       "BsmtFinSF1         0\n",
       "BsmtFinSF2         0\n",
       "BsmtUnfSF          0\n",
       "TotalBsmtSF        0\n",
       "1stFlrSF           0\n",
       "2ndFlrSF           0\n",
       "LowQualFinSF       0\n",
       "GrLivArea          0\n",
       "BsmtFullBath       0\n",
       "BsmtHalfBath       0\n",
       "FullBath           0\n",
       "HalfBath           0\n",
       "BedroomAbvGr       0\n",
       "KitchenAbvGr       0\n",
       "TotRmsAbvGrd       0\n",
       "Fireplaces         0\n",
       "GarageYrBlt       81\n",
       "GarageCars         0\n",
       "GarageArea         0\n",
       "WoodDeckSF         0\n",
       "OpenPorchSF        0\n",
       "EnclosedPorch      0\n",
       "3SsnPorch          0\n",
       "ScreenPorch        0\n",
       "PoolArea           0\n",
       "MiscVal            0\n",
       "MoSold             0\n",
       "YrSold             0\n",
       "SalePrice          0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b876cf57",
   "metadata": {},
   "source": [
    "## Approach-1 Dropping columns with missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "f1aa1a0e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['LotFrontage', 'MasVnrArea', 'GarageYrBlt']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# getting all columns with missing values\n",
    "\n",
    "missing_col = [col for col in df_train.columns\n",
    "              if df_train[col].isnull().any()]\n",
    "missing_col"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "d0e34ca9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>...</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>8450</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2003</td>\n",
       "      <td>2003</td>\n",
       "      <td>706</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>61</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>9600</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>1976</td>\n",
       "      <td>1976</td>\n",
       "      <td>978</td>\n",
       "      <td>0</td>\n",
       "      <td>284</td>\n",
       "      <td>...</td>\n",
       "      <td>298</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>11250</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2001</td>\n",
       "      <td>2002</td>\n",
       "      <td>486</td>\n",
       "      <td>0</td>\n",
       "      <td>434</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>70</td>\n",
       "      <td>9550</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>1915</td>\n",
       "      <td>1970</td>\n",
       "      <td>216</td>\n",
       "      <td>0</td>\n",
       "      <td>540</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>272</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>14260</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>2000</td>\n",
       "      <td>2000</td>\n",
       "      <td>655</td>\n",
       "      <td>0</td>\n",
       "      <td>490</td>\n",
       "      <td>...</td>\n",
       "      <td>192</td>\n",
       "      <td>84</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass  LotArea  OverallQual  OverallCond  YearBuilt  YearRemodAdd  \\\n",
       "0   1          60     8450            7            5       2003          2003   \n",
       "1   2          20     9600            6            8       1976          1976   \n",
       "2   3          60    11250            7            5       2001          2002   \n",
       "3   4          70     9550            7            5       1915          1970   \n",
       "4   5          60    14260            8            5       2000          2000   \n",
       "\n",
       "   BsmtFinSF1  BsmtFinSF2  BsmtUnfSF  ...  WoodDeckSF  OpenPorchSF  \\\n",
       "0         706           0        150  ...           0           61   \n",
       "1         978           0        284  ...         298            0   \n",
       "2         486           0        434  ...           0           42   \n",
       "3         216           0        540  ...           0           35   \n",
       "4         655           0        490  ...         192           84   \n",
       "\n",
       "   EnclosedPorch  3SsnPorch  ScreenPorch  PoolArea  MiscVal  MoSold  YrSold  \\\n",
       "0              0          0            0         0        0       2    2008   \n",
       "1              0          0            0         0        0       5    2007   \n",
       "2              0          0            0         0        0       9    2008   \n",
       "3            272          0            0         0        0       2    2006   \n",
       "4              0          0            0         0        0      12    2008   \n",
       "\n",
       "   SalePrice  \n",
       "0     208500  \n",
       "1     181500  \n",
       "2     223500  \n",
       "3     140000  \n",
       "4     250000  \n",
       "\n",
       "[5 rows x 35 columns]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# dropping all the columns with missing values\n",
    "\n",
    "df_dropped_cols = df_train.drop(missing_col, axis=1)\n",
    "df_dropped_cols.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3f4088c",
   "metadata": {},
   "source": [
    "As we can see that we have dropped 3 columns from our dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a041178",
   "metadata": {},
   "source": [
    "## Approach-2 Imputation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1eeb23b5",
   "metadata": {},
   "source": [
    "we use SimpleImputer to replace missing values with the mean value along each column.\n",
    "\n",
    "Although it's simple, filling in the mean value generally performs quite well (but this varies by dataset). While statisticians have experimented with more complex ways to determine imputed values (such as regression imputation, for instance). \n",
    "\n",
    "The complex strategies typically give no additional benefit once you plug the results into sophisticated machine learning models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "2f789d83",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.00000e+00 6.00000e+01 6.50000e+01 ... 2.00000e+00 2.00800e+03\n",
      "  2.08500e+05]\n",
      " [2.00000e+00 2.00000e+01 8.00000e+01 ... 5.00000e+00 2.00700e+03\n",
      "  1.81500e+05]\n",
      " [3.00000e+00 6.00000e+01 6.80000e+01 ... 9.00000e+00 2.00800e+03\n",
      "  2.23500e+05]\n",
      " ...\n",
      " [1.45800e+03 7.00000e+01 6.60000e+01 ... 5.00000e+00 2.01000e+03\n",
      "  2.66500e+05]\n",
      " [1.45900e+03 2.00000e+01 6.80000e+01 ... 4.00000e+00 2.01000e+03\n",
      "  1.42125e+05]\n",
      " [1.46000e+03 2.00000e+01 7.50000e+01 ... 6.00000e+00 2.00800e+03\n",
      "  1.47500e+05]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# code \n",
    "\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "# creating instance of imputer\n",
    "imputer = SimpleImputer()\n",
    "\n",
    "array = imputer.fit_transform(df_train)\n",
    "# imputed_df = pd.DataFrame(\n",
    "#     imputer.fit_transform()\n",
    "# )\n",
    "print(array)\n",
    "type(array)\n",
    "\n",
    "# Reference: https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "9b933a17",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>...</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>196.0</td>\n",
       "      <td>706.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2008.0</td>\n",
       "      <td>208500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1976.0</td>\n",
       "      <td>1976.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>978.0</td>\n",
       "      <td>...</td>\n",
       "      <td>298.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>181500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2001.0</td>\n",
       "      <td>2002.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>486.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2008.0</td>\n",
       "      <td>223500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1915.0</td>\n",
       "      <td>1970.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>216.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>272.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2006.0</td>\n",
       "      <td>140000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>350.0</td>\n",
       "      <td>655.0</td>\n",
       "      <td>...</td>\n",
       "      <td>192.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2008.0</td>\n",
       "      <td>250000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    Id  MSSubClass  LotFrontage  LotArea  OverallQual  OverallCond  YearBuilt  \\\n",
       "0  1.0        60.0         65.0   8450.0          7.0          5.0     2003.0   \n",
       "1  2.0        20.0         80.0   9600.0          6.0          8.0     1976.0   \n",
       "2  3.0        60.0         68.0  11250.0          7.0          5.0     2001.0   \n",
       "3  4.0        70.0         60.0   9550.0          7.0          5.0     1915.0   \n",
       "4  5.0        60.0         84.0  14260.0          8.0          5.0     2000.0   \n",
       "\n",
       "   YearRemodAdd  MasVnrArea  BsmtFinSF1  ...  WoodDeckSF  OpenPorchSF  \\\n",
       "0        2003.0       196.0       706.0  ...         0.0         61.0   \n",
       "1        1976.0         0.0       978.0  ...       298.0          0.0   \n",
       "2        2002.0       162.0       486.0  ...         0.0         42.0   \n",
       "3        1970.0         0.0       216.0  ...         0.0         35.0   \n",
       "4        2000.0       350.0       655.0  ...       192.0         84.0   \n",
       "\n",
       "   EnclosedPorch  3SsnPorch  ScreenPorch  PoolArea  MiscVal  MoSold  YrSold  \\\n",
       "0            0.0        0.0          0.0       0.0      0.0     2.0  2008.0   \n",
       "1            0.0        0.0          0.0       0.0      0.0     5.0  2007.0   \n",
       "2            0.0        0.0          0.0       0.0      0.0     9.0  2008.0   \n",
       "3          272.0        0.0          0.0       0.0      0.0     2.0  2006.0   \n",
       "4            0.0        0.0          0.0       0.0      0.0    12.0  2008.0   \n",
       "\n",
       "   SalePrice  \n",
       "0   208500.0  \n",
       "1   181500.0  \n",
       "2   223500.0  \n",
       "3   140000.0  \n",
       "4   250000.0  \n",
       "\n",
       "[5 rows x 38 columns]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# converting to dataframe\n",
    "\n",
    "df_imputed = pd.DataFrame(array, columns=df_train.columns)\n",
    "\n",
    "df_imputed.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "9339161e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id               0\n",
       "MSSubClass       0\n",
       "LotFrontage      0\n",
       "LotArea          0\n",
       "OverallQual      0\n",
       "OverallCond      0\n",
       "YearBuilt        0\n",
       "YearRemodAdd     0\n",
       "MasVnrArea       0\n",
       "BsmtFinSF1       0\n",
       "BsmtFinSF2       0\n",
       "BsmtUnfSF        0\n",
       "TotalBsmtSF      0\n",
       "1stFlrSF         0\n",
       "2ndFlrSF         0\n",
       "LowQualFinSF     0\n",
       "GrLivArea        0\n",
       "BsmtFullBath     0\n",
       "BsmtHalfBath     0\n",
       "FullBath         0\n",
       "HalfBath         0\n",
       "BedroomAbvGr     0\n",
       "KitchenAbvGr     0\n",
       "TotRmsAbvGrd     0\n",
       "Fireplaces       0\n",
       "GarageYrBlt      0\n",
       "GarageCars       0\n",
       "GarageArea       0\n",
       "WoodDeckSF       0\n",
       "OpenPorchSF      0\n",
       "EnclosedPorch    0\n",
       "3SsnPorch        0\n",
       "ScreenPorch      0\n",
       "PoolArea         0\n",
       "MiscVal          0\n",
       "MoSold           0\n",
       "YrSold           0\n",
       "SalePrice        0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_imputed.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "d32e0bb0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id               float64\n",
       "MSSubClass       float64\n",
       "LotFrontage      float64\n",
       "LotArea          float64\n",
       "OverallQual      float64\n",
       "OverallCond      float64\n",
       "YearBuilt        float64\n",
       "YearRemodAdd     float64\n",
       "MasVnrArea       float64\n",
       "BsmtFinSF1       float64\n",
       "BsmtFinSF2       float64\n",
       "BsmtUnfSF        float64\n",
       "TotalBsmtSF      float64\n",
       "1stFlrSF         float64\n",
       "2ndFlrSF         float64\n",
       "LowQualFinSF     float64\n",
       "GrLivArea        float64\n",
       "BsmtFullBath     float64\n",
       "BsmtHalfBath     float64\n",
       "FullBath         float64\n",
       "HalfBath         float64\n",
       "BedroomAbvGr     float64\n",
       "KitchenAbvGr     float64\n",
       "TotRmsAbvGrd     float64\n",
       "Fireplaces       float64\n",
       "GarageYrBlt      float64\n",
       "GarageCars       float64\n",
       "GarageArea       float64\n",
       "WoodDeckSF       float64\n",
       "OpenPorchSF      float64\n",
       "EnclosedPorch    float64\n",
       "3SsnPorch        float64\n",
       "ScreenPorch      float64\n",
       "PoolArea         float64\n",
       "MiscVal          float64\n",
       "MoSold           float64\n",
       "YrSold           float64\n",
       "SalePrice        float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_imputed.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bfc7de7",
   "metadata": {},
   "source": [
    "## Approach-3 An extension to Imputation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "2ca806a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Making a copy to avoid changing original dataframe\n",
    "\n",
    "df_train_copy = df_train.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "b2c54868",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['LotFrontage', 'MasVnrArea', 'GarageYrBlt']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using missing_col from before\n",
    "\n",
    "missing_col"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "d3af9c83",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       False\n",
       "1       False\n",
       "2       False\n",
       "3       False\n",
       "4       False\n",
       "        ...  \n",
       "1455    False\n",
       "1456    False\n",
       "1457    False\n",
       "1458    False\n",
       "1459    False\n",
       "Name: LotFrontage, Length: 1460, dtype: bool"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train_copy['LotFrontage'].isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "e14e62f0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       False\n",
       "1       False\n",
       "2       False\n",
       "3       False\n",
       "4       False\n",
       "        ...  \n",
       "1455    False\n",
       "1456    False\n",
       "1457    False\n",
       "1458    False\n",
       "1459    False\n",
       "Name: LotFrontage, Length: 1460, dtype: bool"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# column assignment\n",
    "\n",
    "new_column = df_train_copy['LotFrontage'].isnull()\n",
    "new_column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "5b937cd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Making new columns that will specify the place where imputation is \n",
    "# taking place\n",
    "\n",
    "for column in missing_col:\n",
    "    df_train_copy[column+'_was_missing'] = df_train_copy[column].isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "cbd0e52a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>...</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SalePrice</th>\n",
       "      <th>LotFrontage_was_missing</th>\n",
       "      <th>MasVnrArea_was_missing</th>\n",
       "      <th>GarageYrBlt_was_missing</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2003</td>\n",
       "      <td>2003</td>\n",
       "      <td>196.0</td>\n",
       "      <td>706</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>208500</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>1976</td>\n",
       "      <td>1976</td>\n",
       "      <td>0.0</td>\n",
       "      <td>978</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>181500</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2001</td>\n",
       "      <td>2002</td>\n",
       "      <td>162.0</td>\n",
       "      <td>486</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>223500</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>70</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>1915</td>\n",
       "      <td>1970</td>\n",
       "      <td>0.0</td>\n",
       "      <td>216</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>140000</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>2000</td>\n",
       "      <td>2000</td>\n",
       "      <td>350.0</td>\n",
       "      <td>655</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>250000</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass  LotFrontage  LotArea  OverallQual  OverallCond  YearBuilt  \\\n",
       "0   1          60         65.0     8450            7            5       2003   \n",
       "1   2          20         80.0     9600            6            8       1976   \n",
       "2   3          60         68.0    11250            7            5       2001   \n",
       "3   4          70         60.0     9550            7            5       1915   \n",
       "4   5          60         84.0    14260            8            5       2000   \n",
       "\n",
       "   YearRemodAdd  MasVnrArea  BsmtFinSF1  ...  3SsnPorch  ScreenPorch  \\\n",
       "0          2003       196.0         706  ...          0            0   \n",
       "1          1976         0.0         978  ...          0            0   \n",
       "2          2002       162.0         486  ...          0            0   \n",
       "3          1970         0.0         216  ...          0            0   \n",
       "4          2000       350.0         655  ...          0            0   \n",
       "\n",
       "   PoolArea  MiscVal  MoSold  YrSold  SalePrice  LotFrontage_was_missing  \\\n",
       "0         0        0       2    2008     208500                    False   \n",
       "1         0        0       5    2007     181500                    False   \n",
       "2         0        0       9    2008     223500                    False   \n",
       "3         0        0       2    2006     140000                    False   \n",
       "4         0        0      12    2008     250000                    False   \n",
       "\n",
       "   MasVnrArea_was_missing  GarageYrBlt_was_missing  \n",
       "0                   False                    False  \n",
       "1                   False                    False  \n",
       "2                   False                    False  \n",
       "3                   False                    False  \n",
       "4                   False                    False  \n",
       "\n",
       "[5 rows x 41 columns]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train_copy.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "5018e7e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# now doing the imputation \n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "imputer = SimpleImputer()\n",
    "\n",
    "df_train_copy_imputed = pd.DataFrame(\n",
    "    imputer.fit_transform(df_train_copy)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "c3fb11ce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     0\n",
       "1     0\n",
       "2     0\n",
       "3     0\n",
       "4     0\n",
       "5     0\n",
       "6     0\n",
       "7     0\n",
       "8     0\n",
       "9     0\n",
       "10    0\n",
       "11    0\n",
       "12    0\n",
       "13    0\n",
       "14    0\n",
       "15    0\n",
       "16    0\n",
       "17    0\n",
       "18    0\n",
       "19    0\n",
       "20    0\n",
       "21    0\n",
       "22    0\n",
       "23    0\n",
       "24    0\n",
       "25    0\n",
       "26    0\n",
       "27    0\n",
       "28    0\n",
       "29    0\n",
       "30    0\n",
       "31    0\n",
       "32    0\n",
       "33    0\n",
       "34    0\n",
       "35    0\n",
       "36    0\n",
       "37    0\n",
       "38    0\n",
       "39    0\n",
       "40    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train_copy_imputed.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "e0d9ae9f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>196.0</td>\n",
       "      <td>706.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2008.0</td>\n",
       "      <td>208500.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1976.0</td>\n",
       "      <td>1976.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>978.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>181500.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2001.0</td>\n",
       "      <td>2002.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>486.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2008.0</td>\n",
       "      <td>223500.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1915.0</td>\n",
       "      <td>1970.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>216.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2006.0</td>\n",
       "      <td>140000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>350.0</td>\n",
       "      <td>655.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2008.0</td>\n",
       "      <td>250000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    0     1     2        3    4    5       6       7      8      9   ...   31  \\\n",
       "0  1.0  60.0  65.0   8450.0  7.0  5.0  2003.0  2003.0  196.0  706.0  ...  0.0   \n",
       "1  2.0  20.0  80.0   9600.0  6.0  8.0  1976.0  1976.0    0.0  978.0  ...  0.0   \n",
       "2  3.0  60.0  68.0  11250.0  7.0  5.0  2001.0  2002.0  162.0  486.0  ...  0.0   \n",
       "3  4.0  70.0  60.0   9550.0  7.0  5.0  1915.0  1970.0    0.0  216.0  ...  0.0   \n",
       "4  5.0  60.0  84.0  14260.0  8.0  5.0  2000.0  2000.0  350.0  655.0  ...  0.0   \n",
       "\n",
       "    32   33   34    35      36        37   38   39   40  \n",
       "0  0.0  0.0  0.0   2.0  2008.0  208500.0  0.0  0.0  0.0  \n",
       "1  0.0  0.0  0.0   5.0  2007.0  181500.0  0.0  0.0  0.0  \n",
       "2  0.0  0.0  0.0   9.0  2008.0  223500.0  0.0  0.0  0.0  \n",
       "3  0.0  0.0  0.0   2.0  2006.0  140000.0  0.0  0.0  0.0  \n",
       "4  0.0  0.0  0.0  12.0  2008.0  250000.0  0.0  0.0  0.0  \n",
       "\n",
       "[5 rows x 41 columns]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train_copy_imputed.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eacba33",
   "metadata": {},
   "source": [
    "As we can see that imputer hasn't put back all the column names in our new dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e42fad4e",
   "metadata": {},
   "source": [
    "### So, why did imputation perform better than dropping the columns?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42cc5075",
   "metadata": {},
   "source": [
    "The training data has 10864 rows and 12 columns, where three columns contain missing data. For each column, less than half of the entries are missing. Thus, dropping the columns removes a lot of useful information, and so it makes sense that imputation would perform better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "acc55c81",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing_value_count_by_column = (df_train.isnull().sum())\n",
    "type(missing_value_count_by_column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "6eeb72c7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing_value_count_by_column = df_train.isnull().sum()\n",
    "type(missing_value_count_by_column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "6935fde8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LotFrontage    259\n",
      "MasVnrArea       8\n",
      "GarageYrBlt     81\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# columns having null values\n",
    "\n",
    "print(missing_value_count_by_column[missing_value_count_by_column>0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "d221ee8c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['LotFrontage', 'MasVnrArea', 'GarageYrBlt']"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing_val_count_by_col = [col for col in df_train.columns\n",
    "                           if df_train[col].isnull().any()]\n",
    "missing_val_count_by_col"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbd45829",
   "metadata": {},
   "source": [
    "### Imputation may not always perform better"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62eef6ac",
   "metadata": {},
   "source": [
    "Imputation may not always perform better in all datasets. \n",
    "\n",
    "Given that thre are so few missing values in the dataset, we'd expect imputation to perform better than dropping columns entirely. However, we see that dropping columns performs slightly better! \n",
    "\n",
    "While this can probably partially be attributed to noise in the dataset, another potential explanation is that the imputation method is not a great match to this dataset. That is, maybe instead of filling in the mean value, it makes more sense to set every missing value to a value of 0, to fill in the most frequently encountered value, or to use some other method. For instance, consider the GarageYrBlt column (which indicates the year that the garage was built). \n",
    "\n",
    "It's likely that in some cases, a missing value could indicate a house that does not have a garage. Does it make more sense to fill in the median value along each column in this case? Or could we get better results by filling in the minimum value along each column? It's not quite clear what's best in this case, but perhaps we can rule out some options immediately - for instance, setting missing values in this column to 0 is likely to yield horrible results!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48309061",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d8e7be9f",
   "metadata": {},
   "source": [
    "#### Author: Piyush Kumar\n",
    "[Github](https://github.com/styles3544)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "950b1a0f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}