memory_profiler.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Profile memory usage\n",
    "We notice that our data files are large and pandas dataframes approach the limit of what can be handled on a 16GiB RAM computer when reading in a whole year's worth of trip data.\n",
    "This notebook simply profiles memory usage and improvements possible by\n",
    "* Using smaller numeric types\n",
    "* Using `categorical` type instead of `object` (strings)\n",
    "* Using DateTime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initial Memory Usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"data/NY_2019.csv\")\n",
    "df.drop(\"Unnamed: 0\", axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# initial usage, no optimization. see column dtypes\n",
    "df.info(memory_usage=\"deep\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Total GiB (same as above)\n",
    "start_memory = df.memory_usage(index=False, deep=True).sum() / (2 ** 30)\n",
    "print(f\"{round(start_memory, 3)} GiB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use smaller numeric types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Max int values (we have no negative numbers in our data)\n",
    "print(np.iinfo(np.int64).max)\n",
    "print(np.iinfo(np.int32).max)\n",
    "print(np.iinfo(np.int16).max)\n",
    "print(np.iinfo(np.int8).max)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# What is the max for each numeric column?\n",
    "\n",
    "print(\"Max values:\")\n",
    "# okay who took a 1000+ hour (44 days) trip...should we drop outliers? perhaps not because then we can't determine bike rebalancing\n",
    "print(df.tripduration.max())\n",
    "print(df[\"startstationid\"].max())\n",
    "print(df[\"endstationid\"].max())\n",
    "print(df.bikeid.max())\n",
    "print(df.birthyear.max())\n",
    "\n",
    "# NOTE: we have no negative values (as expected) so can use unsigned ints when downcasting\n",
    "print(\"\\nMin values:\")\n",
    "print(df.tripduration.min())\n",
    "print(df[\"startstationid\"].min())\n",
    "print(df[\"endstationid\"].min())\n",
    "print(df.bikeid.min())\n",
    "print(df.birthyear.min())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Drop NAs before downcasting\n",
    "df.dropna(axis=0, inplace=True)\n",
    "\n",
    "# # Use smaller numeric types\n",
    "# df['tripduration'] = df['tripduration'].astype('int32')\n",
    "# df['startstationid'] = df['startstationid'].astype('int16')\n",
    "# df['endstationid'] = df['endstationid'].astype('int16')\n",
    "# df['bikeid'] = df['bikeid'].astype('int32')\n",
    "# df['birthyear'] = df['birthyear'].astype('int16')\n",
    "# df['gender'] = df['gender'].astype('int8')\n",
    "\n",
    "# actually, let's downcast automatically instead of manually...\n",
    "# NOTE: we might lose precision, but not sure if that matters based on the operations we perform on these columns\n",
    "# E.g., float32 gives 6 digits of precision as opposed to 15 for float64\n",
    "for column in df:\n",
    "    if df[column].dtype == \"float64\":\n",
    "        df[column] = pd.to_numeric(df[column], downcast=\"float\")\n",
    "    if df[column].dtype == \"int64\":\n",
    "        df[column] = pd.to_numeric(df[column], downcast=\"unsigned\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# profile memory again\n",
    "downcasted_memory = df.memory_usage(index=False, deep=True).sum() / (2 ** 30)\n",
    "print(f\"{round(downcasted_memory, 3)} GiB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use categorical type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "df[\"usertype\"] = df[\"usertype\"].astype(\"category\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# profile memory again\n",
    "categorical_memory = df.memory_usage(index=False, deep=True).sum() / (2 ** 30)\n",
    "print(f\"{round(categorical_memory, 3)} GiB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DateTime\n",
    "Not sure if this will reduce or increase size. But it's necessary to do for our time series analysis anyways, so let's see"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "df[\"starttime\"] = pd.to_datetime(df[\"starttime\"])\n",
    "df[\"stoptime\"] = pd.to_datetime(df[\"stoptime\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# profile memory again\n",
    "datetime_memory = df.memory_usage(index=False, deep=True).sum() / (2 ** 30)\n",
    "print(f\"{round(datetime_memory, 3)} GiB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outcome\n",
    "Wow! Using DateTime helps a lot and we get significant gains in memory reduction from using a categorical type. Smaller numeric types give a smaller percentage reduction, but still useful"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "print(\n",
    "    f\"Reduced dataframe size by {round(100*(start_memory - datetime_memory)/start_memory, 2)}%\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}