1.2. Capestone Data Cleaning_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting Prescriber induced Overdose"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DATA SCIENCE IMMERSIVE FINAL CAPE STONE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This capestone was initaited with an idea that why are so much more physicians presecribing medications that can lead to overdose. I have worked with patients for more than a year in a clinic as well as a hospital and saw significant amount of patients requesting pain relief medications again and again. I have seen people i know been affected with the chronic and life threatening overdose.\n",
    "\n",
    "The objective of this capestone is to **identify types of prescribers that are a high-risk for opioid related fatalities and predict most influential opioids leading to deaths.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The public health data was extracted from [CDC](https://wonder.cdc.gov/). Downloading data from wonder requires the signing f online conscent and querring the needed information separately and the system will generate the text versions and i have extracting individually for the text versions the deaths associated with opiods. I noticed some anomalies when extracting data from Wonder in large due to the system or functionality of wonder tool they have to group the needed data based on the querry.  The opiods related with Drug enforcement Agency (DEA) licensed specialities connected with the opiods were downloaded from FDA and cross matched with the 2013-2016 Opiods drusglist for the year 2013-2016 can also be accessed [here](https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/Medicare-Provider-Charge-Data/OpioidMap.html). The Part D Prescriber PUF file was sas version and in texti downloaded into individual tab separated text files as dowloading the 3.2GB data at once and extracting needs significant processing power besides I noticed some anomalies when extracting data from Wonder in large due to the system or functionality of wonder tool they have to group the needed data. The otherData was extraced from CMS Medicare [Part D Opioid Prescriber Data](\thttps://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/Medicare-Provider-Charge-Data/Part-D-Prescriber.html),[cms](https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/Medicare-Provider-Charge-Data/OpioidMap.html) and  [NHE](https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/NationalHealthExpendData/NationalHealthAccountsStateHealthAccountsProvider.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, i am going to analyze the datasets i would use for the prediction. In particular, in this note book i will use the Exploratory Data Analysis(eda) in order to understand the statistical structure of the data. Furthermore, in the following notebooks i will build a predictive machine learning model to predict a certain target variable (prescribers) from other attributes that are linked with the features that i will learn from the eda. The datasets are collected from a real world datasets currently available  and they contains a substantial amount of missing values. In this notebook, i will try to set up  my datasets in note book 1.1 and in 1.2 i will cean all the datasets that i will use in the future and try to understand their relationships and in the next notebook. In Notebook 1.3  i will visualize the most important features and associated ones based on the data collected  that will be used for the next work. All the datasets were collected from different sources and validated. Somedatasets are official querried from respective government institutions using private usercode and adjusted to the subject after cleaning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Author : Kiros Gebremariam**\n",
    "\n",
    "Cohorts of the Data Science Immersive, General Assembly @ Washington DC campus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Data Cleaning,munging and EDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Loadding Necessary Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os                                                                    # os library\n",
    "import numpy as np                                                           # numpy library\n",
    "import pandas as pd                                                          # pandas library\n",
    "import matplotlib.pylab as plt                                               # plotting and visualization library\n",
    "import seaborn as sns                                                        # plotting\n",
    "import statsmodels.api as sm                                                 # for running regressions\n",
    "np.random.seed(2018)                                                         # Random seed for replication \n",
    "\n",
    "# Configure visual settings:\n",
    "plt.style.use('fivethirtyeight')\n",
    "sns.set(font_scale=2)\n",
    "cmap = sns.diverging_palette(220, 10, as_cmap=True) # one of the many color mappings\n",
    "sns.set_style('whitegrid')\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) \n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['.DS_Store', '.git', '.ipynb_checkpoints', '1.1. Capestone Data setup_1.ipynb', '1.2. Capestone Data Cleaning_1.ipynb', '1.3. Capestone  Data visualization_1.ipynb', '1.4. Modeling-Feature importance and  Evaluation_another model testing.ipynb', '1.4.1. Models and feature importance .ipynb', '1.5. Modeling-Feature importance and  Evaluation_another model testing.ipynb', '1.6 .Visualisation and identifying the most prescribed drug by Speciality.ipynb', '1.6. Modeling-Feature importance and  Evaluation_another model testing.ipynb', 'inputs', 'readme.md']\n"
     ]
    }
   ],
   "source": [
    "import os                      # just ensuring the files using as working Directory(pwd)\n",
    "print(os.listdir())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prescriber related datasets disaggregated by state,county,zip and rate \n",
    "pres_info_df = pd.read_csv('https://data.cms.gov/api/views/6wg9-kwip/rows.csv?accessType=DOWNLOAD')\n",
    "Prescriber_state_df =pd.read_csv('./inputs/Prescriber_state.csv')  # prescribers by state\n",
    "Prescriber_county_df =pd.read_csv('./inputs/Prescriber_county.csv')   # prescribers by county\n",
    "Prescriber_zip_df =pd.read_csv('./inputs/Prescriber_zip.csv')       # prescribers by zip\n",
    "psr_df= pd.read_csv('./inputs/prescriber_state_rate.csv')             # this is the  state wide rate of change in op prescription per 100,000 people for years 2013-2016\n",
    "pcr_df = pd.read_csv('./inputs/prescriber_county_rate.csv')\n",
    "pzr_df= pd.read_csv('./inputs/prescriber_zip_rate.csv')\n",
    "#Drugs, Mental health and Deaths datasets different datasources\n",
    "KFFMentalHealthspendState_df =pd.read_csv('./inputs/KFFMentalHealthspendState.csv')  # mental health spending by state\n",
    "pop_urban_df =pd.read_csv('./inputs/pop-urban-pct-historical_cleaned_up.csv')     # urban population based on census tract of 2010\n",
    "rawdataKFFopioiddeath_df = pd.read_csv('./inputs/rawdataKFFopioiddeath.csv', engine =\"python\")  # deaths due to opiods overdose\n",
    "Wonder_Drug_Death_Data1_df = pd.read_csv('./inputs/Wonder_Drug_Death_Data_1.csv')     # drug related deaths  extracted from CDC/Wonder part1\n",
    "Wonder_Drug_Death_Data2_df = pd.read_csv('./inputs/Wonder_Drug_Death_Data_2.csv')     # # drug related deaths  extracted from CDC/Wonder part2\n",
    "drugs_df = pd.read_csv('./inputs/list_opioids.csv')\n",
    "# crime related data both property and nonviolent\n",
    "violentcrime_df =pd.read_csv('./inputs/violentcrime19992014.csv')    # violent crime extracted by querring the UCR of DOJ\n",
    "propertycrime_df=pd.read_csv('./inputs/propertycrime19992014.csv')   # property crime extracted by querring the UCR of DOJ\n",
    "tcrime_df =pd.read_csv('./inputs/TotalVncrime.csv')                 # total crime  by state\n",
    "\n",
    "# National Health Expenditure from 2009-2016\n",
    "NHE2016_df = pd.read_csv('./inputs/NHE2016.csv',engine='python')     # National health expenditure by state  collected from CMS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Checking Data Integrity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Checking data Integrity and Quality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NPI</th>\n",
       "      <th>NPPES Provider Last Name</th>\n",
       "      <th>NPPES Provider First Name</th>\n",
       "      <th>NPPES Provider ZIP Code</th>\n",
       "      <th>NPPES Provider State</th>\n",
       "      <th>Specialty Description</th>\n",
       "      <th>Total Claim Count</th>\n",
       "      <th>Opioid Claim Count</th>\n",
       "      <th>Opioid Prescribing Rate</th>\n",
       "      <th>Extended-Release Opioid Claims</th>\n",
       "      <th>Extended-Release Opioid Prescribing Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1003000126</td>\n",
       "      <td>ENKESHAFI</td>\n",
       "      <td>ARDALAN</td>\n",
       "      <td>21502.0</td>\n",
       "      <td>MD</td>\n",
       "      <td>Internal Medicine</td>\n",
       "      <td>545</td>\n",
       "      <td>23.0</td>\n",
       "      <td>4.22%</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1003000142</td>\n",
       "      <td>KHALIL</td>\n",
       "      <td>RASHID</td>\n",
       "      <td>43623.0</td>\n",
       "      <td>OH</td>\n",
       "      <td>Anesthesiology</td>\n",
       "      <td>1733</td>\n",
       "      <td>1004.0</td>\n",
       "      <td>57.93%</td>\n",
       "      <td>63.0</td>\n",
       "      <td>6.27%</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          NPI NPPES Provider Last Name NPPES Provider First Name  \\\n",
       "0  1003000126                ENKESHAFI                   ARDALAN   \n",
       "1  1003000142                   KHALIL                    RASHID   \n",
       "\n",
       "   NPPES Provider ZIP Code NPPES Provider State Specialty Description  \\\n",
       "0                  21502.0                   MD     Internal Medicine   \n",
       "1                  43623.0                   OH        Anesthesiology   \n",
       "\n",
       "   Total Claim Count  Opioid Claim Count Opioid Prescribing Rate  \\\n",
       "0                545                23.0                   4.22%   \n",
       "1               1733              1004.0                  57.93%   \n",
       "\n",
       "   Extended-Release Opioid Claims Extended-Release Opioid Prescribing Rate  \n",
       "0                             NaN                                      NaN  \n",
       "1                            63.0                                    6.27%  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pres_info_df.head()[:2]  # the data is about the Medicare part D prescriber summary extracted from cms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below i am going to see the top `20`,`40` and `60`  `speciality prescribers of opiods across the nation` based on the dataset from cms and the dataset has `total claims` both `opiods` and `others` and gives the opiods claims by speciality. As a matter of confidentiality i will not `focus on the name and last name` of the specailities, even though i have access to the information from cms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "Extended-Release Opioid Prescribing Rate    0.656609\n",
      "Opioid Prescribing Rate                     0.281917\n",
      "Opioid Claim Count                          0.281917\n",
      "Extended-Release Opioid Claims              0.113573\n",
      "NPPES Provider ZIP Code                     0.000047\n",
      "NPPES Provider First Name                   0.000028\n",
      "NPPES Provider Last Name                    0.000028\n",
      "Total Claim Count                           0.000000\n",
      "Specialty Description                       0.000000\n",
      "NPPES Provider State                        0.000000\n",
      "NPI                                         0.000000\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((pres_info_df.isnull().sum()/pres_info_df.shape[0]).sort_values(ascending=False) )  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['NPI',\n",
       " 'NPPES Provider Last Name',\n",
       " 'NPPES Provider First Name',\n",
       " 'NPPES Provider ZIP Code',\n",
       " 'NPPES Provider State',\n",
       " 'Specialty Description',\n",
       " 'Total Claim Count',\n",
       " 'Opioid Claim Count',\n",
       " 'Opioid Prescribing Rate',\n",
       " 'Extended-Release Opioid Claims',\n",
       " 'Extended-Release Opioid Prescribing Rate']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(pres_info_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============<Speciality description: this shows the licensed specialities count======================\n",
      "Nurse Practitioner                                                137822\n",
      "Internal Medicine                                                 133174\n",
      "Dentist                                                           129650\n",
      "Family Practice                                                   109334\n",
      "Physician Assistant                                                86733\n",
      "Student in an Organized Health Care Education/Training Program     51615\n",
      "Emergency Medicine                                                 47229\n",
      "Obstetrics & Gynecology                                            29713\n",
      "Optometry                                                          28570\n",
      "Psychiatry                                                         25528\n",
      "General Surgery                                                    22944\n",
      "Orthopedic Surgery                                                 21302\n",
      "Cardiovascular Disease (Cardiology)                                20370\n",
      "Ophthalmology                                                      19763\n",
      "Podiatry                                                           15285\n",
      "Psychiatry & Neurology                                             13959\n",
      "Neurology                                                          13656\n",
      "Gastroenterology                                                   13350\n",
      "Dermatology                                                        13031\n",
      "Pediatric Medicine                                                 11744\n",
      "Urology                                                            10710\n",
      "Otolaryngology                                                     10440\n",
      "General Practice                                                   10171\n",
      "Pulmonary Disease                                                   9277\n",
      "Nephrology                                                          8485\n",
      "Physical Medicine and Rehabilitation                                8330\n",
      "Hematology-Oncology                                                 8324\n",
      "Anesthesiology                                                      7180\n",
      "Pharmacist                                                          6133\n",
      "Obstetrics/Gynecology                                               5985\n",
      "                                                                   ...  \n",
      "Respiratory Therapist, Certified                                       2\n",
      "Residential Treatment Facility, Emotionally Disturbed Children         2\n",
      "Orthotic Fitter                                                        2\n",
      "Prosthetist                                                            2\n",
      "Dietetic Technician, Registered                                        1\n",
      "Technician, Cardiology                                                 1\n",
      "Dietary Manager                                                        1\n",
      "Spec/Tech, Cardiovascular                                              1\n",
      "Sleep Specialist, PhD                                                  1\n",
      "Dental Therapist                                                       1\n",
      "Mechanotherapist                                                       1\n",
      "Radiation Therapy                                                      1\n",
      "In Home Supportive Care                                                1\n",
      "Day Training/Habilitation Specialist                                   1\n",
      "Individual Certified Prosthetist-Orthotist                             1\n",
      "Midwife, Lay                                                           1\n",
      "Spec/Tech, Health Info                                                 1\n",
      "Doula                                                                  1\n",
      "Voluntary Health or Charitable Agencies                                1\n",
      "Respite Care                                                           1\n",
      "Public Health or Welfare Agency                                        1\n",
      "Lodging                                                                1\n",
      "Alzheimer Center (Dementia Center)                                     1\n",
      "Prosthetic/Orthotic Supplier                                           1\n",
      "Perfusionist                                                           1\n",
      "Private Vehicle                                                        1\n",
      "Dental Laboratory Technician                                           1\n",
      "Nursing Facility, Other (Dmercs Only)                                  1\n",
      "Clinical Laboratory                                                    1\n",
      "Psychiatric Residential Treatment Facility                             1\n",
      "Name: Specialty Description, Length: 223, dtype: int64\n",
      "-----------------< total instance count of speciality description>------------------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"============<Speciality description: this shows the licensed specialities count======================\")\n",
    "print(pres_info_df['Specialty Description'].value_counts())\n",
    "print(\"-----------------< total instance count of speciality description>------------------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============<Speciality description: this shows the licensed specialities count======================\n",
      "Nurse Practitioner                                                1.217993e-01\n",
      "Internal Medicine                                                 1.176917e-01\n",
      "Dentist                                                           1.145773e-01\n",
      "Family Practice                                                   9.662322e-02\n",
      "Physician Assistant                                               7.664973e-02\n",
      "Student in an Organized Health Care Education/Training Program    4.561442e-02\n",
      "Emergency Medicine                                                4.173832e-02\n",
      "Obstetrics & Gynecology                                           2.625867e-02\n",
      "Optometry                                                         2.524855e-02\n",
      "Psychiatry                                                        2.256021e-02\n",
      "General Surgery                                                   2.027661e-02\n",
      "Orthopedic Surgery                                                1.882550e-02\n",
      "Cardiovascular Disease (Cardiology)                               1.800186e-02\n",
      "Ophthalmology                                                     1.746542e-02\n",
      "Podiatry                                                          1.350802e-02\n",
      "Psychiatry & Neurology                                            1.233618e-02\n",
      "Neurology                                                         1.206840e-02\n",
      "Gastroenterology                                                  1.179798e-02\n",
      "Dermatology                                                       1.151606e-02\n",
      "Pediatric Medicine                                                1.037868e-02\n",
      "Urology                                                           9.464893e-03\n",
      "Otolaryngology                                                    9.226283e-03\n",
      "General Practice                                                  8.988556e-03\n",
      "Pulmonary Disease                                                 8.198489e-03\n",
      "Nephrology                                                        7.498564e-03\n",
      "Physical Medicine and Rehabilitation                              7.361584e-03\n",
      "Hematology-Oncology                                               7.356281e-03\n",
      "Anesthesiology                                                    6.345279e-03\n",
      "Pharmacist                                                        5.419999e-03\n",
      "Obstetrics/Gynecology                                             5.289205e-03\n",
      "                                                                      ...     \n",
      "Respiratory Therapist, Certified                                  1.767487e-06\n",
      "Residential Treatment Facility, Emotionally Disturbed Children    1.767487e-06\n",
      "Orthotic Fitter                                                   1.767487e-06\n",
      "Prosthetist                                                       1.767487e-06\n",
      "Dietetic Technician, Registered                                   8.837435e-07\n",
      "Technician, Cardiology                                            8.837435e-07\n",
      "Dietary Manager                                                   8.837435e-07\n",
      "Spec/Tech, Cardiovascular                                         8.837435e-07\n",
      "Sleep Specialist, PhD                                             8.837435e-07\n",
      "Dental Therapist                                                  8.837435e-07\n",
      "Mechanotherapist                                                  8.837435e-07\n",
      "Radiation Therapy                                                 8.837435e-07\n",
      "In Home Supportive Care                                           8.837435e-07\n",
      "Day Training/Habilitation Specialist                              8.837435e-07\n",
      "Individual Certified Prosthetist-Orthotist                        8.837435e-07\n",
      "Midwife, Lay                                                      8.837435e-07\n",
      "Spec/Tech, Health Info                                            8.837435e-07\n",
      "Doula                                                             8.837435e-07\n",
      "Voluntary Health or Charitable Agencies                           8.837435e-07\n",
      "Respite Care                                                      8.837435e-07\n",
      "Public Health or Welfare Agency                                   8.837435e-07\n",
      "Lodging                                                           8.837435e-07\n",
      "Alzheimer Center (Dementia Center)                                8.837435e-07\n",
      "Prosthetic/Orthotic Supplier                                      8.837435e-07\n",
      "Perfusionist                                                      8.837435e-07\n",
      "Private Vehicle                                                   8.837435e-07\n",
      "Dental Laboratory Technician                                      8.837435e-07\n",
      "Nursing Facility, Other (Dmercs Only)                             8.837435e-07\n",
      "Clinical Laboratory                                               8.837435e-07\n",
      "Psychiatric Residential Treatment Facility                        8.837435e-07\n",
      "Name: Specialty Description, Length: 223, dtype: float64\n",
      "-----------------< total instance count of speciality description normalized>-------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"============<Speciality description: this shows the licensed specialities count======================\")\n",
    "print(pres_info_df['Specialty Description'].value_counts(normalize =True))\n",
    "print(\"-----------------< total instance count of speciality description normalized>-------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NPI                                              0\n",
       "NPPES Provider Last Name                        32\n",
       "NPPES Provider First Name                       32\n",
       "NPPES Provider ZIP Code                         53\n",
       "NPPES Provider State                             0\n",
       "Specialty Description                            0\n",
       "Total Claim Count                                0\n",
       "Opioid Claim Count                          319003\n",
       "Opioid Prescribing Rate                     319003\n",
       "Extended-Release Opioid Claims              128514\n",
       "Extended-Release Opioid Prescribing Rate    742986\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pres_info_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "pres_info_df.fillna(0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pres_info_df.isnull().sum().any()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "pres_info_df['zipcode'] = pres_info_df['NPPES Provider ZIP Code'].fillna(0).astype('int').astype('category')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp =pres_info_df['Specialty Description'].value_counts()\n",
    "misc_spec =tmp[tmp<100].index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Hospital (Dmercs Only)', 'Acupuncturist', 'Colon & Rectal Surgery',\n",
       "       'Cardiac Electrophysiology', 'Physical Therapist',\n",
       "       'Multispecialty Clinic/Group Practice', 'Occupational therapist',\n",
       "       'Speech Language Pathologist',\n",
       "       'Neuromusculoskeletal Medicine, Sports Medicine',\n",
       "       'Peripheral Vascular Disease',\n",
       "       ...\n",
       "       'Public Health or Welfare Agency', 'Lodging',\n",
       "       'Alzheimer Center (Dementia Center)', 'Prosthetic/Orthotic Supplier',\n",
       "       'Perfusionist', 'Private Vehicle', 'Dental Laboratory Technician',\n",
       "       'Nursing Facility, Other (Dmercs Only)', 'Clinical Laboratory',\n",
       "       'Psychiatric Residential Treatment Facility'],\n",
       "      dtype='object', length=128)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "misc_spec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>NPI</th>\n",
       "      <td>1131550.0</td>\n",
       "      <td>1.499842e+09</td>\n",
       "      <td>2.879109e+08</td>\n",
       "      <td>1.003000e+09</td>\n",
       "      <td>1.245689e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NPPES Provider ZIP Code</th>\n",
       "      <td>1131550.0</td>\n",
       "      <td>4.806590e+04</td>\n",
       "      <td>3.029656e+04</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>2.120100e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Total Claim Count</th>\n",
       "      <td>1131550.0</td>\n",
       "      <td>1.308800e+03</td>\n",
       "      <td>3.147395e+03</td>\n",
       "      <td>1.100000e+01</td>\n",
       "      <td>5.400000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Opioid Claim Count</th>\n",
       "      <td>1131550.0</td>\n",
       "      <td>6.906041e+01</td>\n",
       "      <td>2.708184e+02</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Extended-Release Opioid Claims</th>\n",
       "      <td>1131550.0</td>\n",
       "      <td>4.757531e+00</td>\n",
       "      <td>4.409088e+01</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    count          mean           std  \\\n",
       "NPI                             1131550.0  1.499842e+09  2.879109e+08   \n",
       "NPPES Provider ZIP Code         1131550.0  4.806590e+04  3.029656e+04   \n",
       "Total Claim Count               1131550.0  1.308800e+03  3.147395e+03   \n",
       "Opioid Claim Count              1131550.0  6.906041e+01  2.708184e+02   \n",
       "Extended-Release Opioid Claims  1131550.0  4.757531e+00  4.409088e+01   \n",
       "\n",
       "                                         min           25%  \n",
       "NPI                             1.003000e+09  1.245689e+09  \n",
       "NPPES Provider ZIP Code         0.000000e+00  2.120100e+04  \n",
       "Total Claim Count               1.100000e+01  5.400000e+01  \n",
       "Opioid Claim Count              0.000000e+00  0.000000e+00  \n",
       "Extended-Release Opioid Claims  0.000000e+00  0.000000e+00  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pres_info_df.describe()[:5].T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NPI</th>\n",
       "      <th>NPPES Provider ZIP Code</th>\n",
       "      <th>Total Claim Count</th>\n",
       "      <th>Opioid Claim Count</th>\n",
       "      <th>Extended-Release Opioid Claims</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>NPI</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000973</td>\n",
       "      <td>-0.000598</td>\n",
       "      <td>0.001135</td>\n",
       "      <td>0.000630</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NPPES Provider ZIP Code</th>\n",
       "      <td>0.000973</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.011660</td>\n",
       "      <td>0.019057</td>\n",
       "      <td>0.012177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Total Claim Count</th>\n",
       "      <td>-0.000598</td>\n",
       "      <td>-0.011660</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.510209</td>\n",
       "      <td>0.200623</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Opioid Claim Count</th>\n",
       "      <td>0.001135</td>\n",
       "      <td>0.019057</td>\n",
       "      <td>0.510209</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.810273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Extended-Release Opioid Claims</th>\n",
       "      <td>0.000630</td>\n",
       "      <td>0.012177</td>\n",
       "      <td>0.200623</td>\n",
       "      <td>0.810273</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     NPI  NPPES Provider ZIP Code  \\\n",
       "NPI                             1.000000                 0.000973   \n",
       "NPPES Provider ZIP Code         0.000973                 1.000000   \n",
       "Total Claim Count              -0.000598                -0.011660   \n",
       "Opioid Claim Count              0.001135                 0.019057   \n",
       "Extended-Release Opioid Claims  0.000630                 0.012177   \n",
       "\n",
       "                                Total Claim Count  Opioid Claim Count  \\\n",
       "NPI                                     -0.000598            0.001135   \n",
       "NPPES Provider ZIP Code                 -0.011660            0.019057   \n",
       "Total Claim Count                        1.000000            0.510209   \n",
       "Opioid Claim Count                       0.510209            1.000000   \n",
       "Extended-Release Opioid Claims           0.200623            0.810273   \n",
       "\n",
       "                                Extended-Release Opioid Claims  \n",
       "NPI                                                   0.000630  \n",
       "NPPES Provider ZIP Code                               0.012177  \n",
       "Total Claim Count                                     0.200623  \n",
       "Opioid Claim Count                                    0.810273  \n",
       "Extended-Release Opioid Claims                        1.000000  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pres_info_df.corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['NPI', 'NPPES Provider Last Name', 'NPPES Provider First Name',\n",
       "       'NPPES Provider ZIP Code', 'NPPES Provider State',\n",
       "       'Specialty Description', 'Total Claim Count', 'Opioid Claim Count',\n",
       "       'Opioid Prescribing Rate', 'Extended-Release Opioid Claims',\n",
       "       'Extended-Release Opioid Prescribing Rate', 'zipcode'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pres_info_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NPI                                            int64\n",
       "NPPES Provider Last Name                      object\n",
       "NPPES Provider First Name                     object\n",
       "NPPES Provider ZIP Code                      float64\n",
       "NPPES Provider State                          object\n",
       "Specialty Description                         object\n",
       "Total Claim Count                              int64\n",
       "Opioid Claim Count                           float64\n",
       "Opioid Prescribing Rate                       object\n",
       "Extended-Release Opioid Claims               float64\n",
       "Extended-Release Opioid Prescribing Rate      object\n",
       "zipcode                                     category\n",
       "dtype: object"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pres_info_df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp =pres_info_df['Opioid Claim Count'].groupby(pres_info_df['Specialty Description']).describe().unstack()\n",
    "# misc =tmp.loc[pres_info_df['Total Claim Count']<10,:].index\n",
    "pres_info_df=pres_info_df.loc[-pres_info_df['Specialty Description'].isin(tmp),:]\n",
    "grouped_spec =pres_info_df['Opioid Claim Count'].groupby(pres_info_df['Specialty Description'])\n",
    "grouped_spec_mean =grouped_spec.mean().sort_values(ascending=False)\n",
    "top20=grouped_spec_mean.index[:20]\n",
    "top40 = grouped_spec_mean.index[:40]\n",
    "top60 =grouped_spec_mean.index[:60]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_spec =pres_info_df['Total Claim Count'].groupby(pres_info_df['Specialty Description']).sum().sort_values(ascending= False)\n",
    "# total_spec/total_spec.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "Prescription=pd.read_csv('./inputs/prescriber-info.csv',index_col=0, parse_dates=True)\n",
    "\n",
    "#Reading the csv file that contains the prescriber's info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>State</th>\n",
       "      <th>Credentials</th>\n",
       "      <th>Specialty</th>\n",
       "      <th>ABILIFY</th>\n",
       "      <th>ACETAMINOPHEN.CODEINE</th>\n",
       "      <th>ACYCLOVIR</th>\n",
       "      <th>ADVAIR.DISKUS</th>\n",
       "      <th>AGGRENOX</th>\n",
       "      <th>ALENDRONATE.SODIUM</th>\n",
       "      <th>...</th>\n",
       "      <th>VERAPAMIL.ER</th>\n",
       "      <th>VESICARE</th>\n",
       "      <th>VOLTAREN</th>\n",
       "      <th>VYTORIN</th>\n",
       "      <th>WARFARIN.SODIUM</th>\n",
       "      <th>XARELTO</th>\n",
       "      <th>ZETIA</th>\n",
       "      <th>ZIPRASIDONE.HCL</th>\n",
       "      <th>ZOLPIDEM.TARTRATE</th>\n",
       "      <th>Opioid.Prescriber</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NPI</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1710982582</th>\n",
       "      <td>M</td>\n",
       "      <td>TX</td>\n",
       "      <td>DDS</td>\n",
       "      <td>Dentist</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1245278100</th>\n",
       "      <td>F</td>\n",
       "      <td>AL</td>\n",
       "      <td>MD</td>\n",
       "      <td>General Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 255 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           Gender State Credentials        Specialty  ABILIFY  \\\n",
       "NPI                                                             \n",
       "1710982582      M    TX         DDS          Dentist        0   \n",
       "1245278100      F    AL          MD  General Surgery        0   \n",
       "\n",
       "            ACETAMINOPHEN.CODEINE  ACYCLOVIR  ADVAIR.DISKUS  AGGRENOX  \\\n",
       "NPI                                                                     \n",
       "1710982582                      0          0              0         0   \n",
       "1245278100                      0          0              0         0   \n",
       "\n",
       "            ALENDRONATE.SODIUM        ...          VERAPAMIL.ER  VESICARE  \\\n",
       "NPI                                   ...                                   \n",
       "1710982582                   0        ...                     0         0   \n",
       "1245278100                   0        ...                     0         0   \n",
       "\n",
       "            VOLTAREN  VYTORIN  WARFARIN.SODIUM  XARELTO  ZETIA  \\\n",
       "NPI                                                              \n",
       "1710982582         0        0                0        0      0   \n",
       "1245278100         0        0                0        0      0   \n",
       "\n",
       "            ZIPRASIDONE.HCL  ZOLPIDEM.TARTRATE  Opioid.Prescriber  \n",
       "NPI                                                                \n",
       "1710982582                0                  0                  1  \n",
       "1245278100                0                 35                  1  \n",
       "\n",
       "[2 rows x 255 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Prescription.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 255)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Prescription.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "s='.'\n",
    "NewColumn=list()\n",
    "for i in Prescription.columns:\n",
    "    NewColumn.append(i.split(s,1)[0])\n",
    "\n",
    "# Rename the columns by eliminating periods from the original column names\n",
    "\n",
    "Prescription.columns=NewColumn\n",
    "\n",
    "#displaying the column names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Gender', 'State', 'Credentials', 'Specialty', 'ABILIFY',\n",
      "       'ACETAMINOPHEN', 'ACYCLOVIR', 'ADVAIR', 'AGGRENOX', 'ALENDRONATE',\n",
      "       ...\n",
      "       'VERAPAMIL', 'VESICARE', 'VOLTAREN', 'VYTORIN', 'WARFARIN', 'XARELTO',\n",
      "       'ZETIA', 'ZIPRASIDONE', 'ZOLPIDEM', 'Opioid'],\n",
      "      dtype='object', length=255)\n"
     ]
    }
   ],
   "source": [
    "print (Prescription.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ABSTRAL',\n",
       " 'ACETAMINOPHEN-CODEINE',\n",
       " 'ACTIQ',\n",
       " 'ASA-BUTALB-CAFFEINE-CODEINE',\n",
       " 'ASCOMP WITH CODEINE',\n",
       " 'ASPIRIN-CAFFEINE-DIHYDROCODEIN',\n",
       " 'AVINZA',\n",
       " 'BELLADONNA-OPIUM',\n",
       " 'BUPRENORPHINE HCL',\n",
       " 'BUTALB-ACETAMINOPH-CAFF-CODEIN',\n",
       " 'BUTALB-CAFF-ACETAMINOPH-CODEIN',\n",
       " 'BUTALBITAL COMPOUND-CODEINE',\n",
       " 'BUTORPHANOL TARTRATE',\n",
       " 'BUTRANS',\n",
       " 'CAPITAL W-CODEINE',\n",
       " 'CARISOPRODOL COMPOUND-CODEINE',\n",
       " 'CARISOPRODOL-ASPIRIN-CODEINE',\n",
       " 'CODEINE SULFATE',\n",
       " 'CO-GESIC',\n",
       " 'CONZIP',\n",
       " 'DEMEROL',\n",
       " 'DEMEROL',\n",
       " 'DILAUDID',\n",
       " 'DILAUDID',\n",
       " 'DILAUDID-HP',\n",
       " 'DISKETS',\n",
       " 'DOLOPHINE HCL',\n",
       " 'DURAGESIC',\n",
       " 'DURAMORPH',\n",
       " 'ENDOCET',\n",
       " 'ENDODAN',\n",
       " 'EXALGO',\n",
       " 'FENTANYL',\n",
       " 'FENTANYL CITRATE',\n",
       " 'FENTORA',\n",
       " 'FIORICET WITH CODEINE',\n",
       " 'FIORINAL WITH CODEINE #3',\n",
       " 'HYCET',\n",
       " 'HYDROCODONE-ACETAMINOPHEN',\n",
       " 'HYDROCODONE-IBUPROFEN',\n",
       " 'HYDROMORPHONE ER',\n",
       " 'HYDROMORPHONE HCL',\n",
       " 'HYDROMORPHONE HCL',\n",
       " 'IBUDONE',\n",
       " 'INFUMORPH',\n",
       " 'KADIAN',\n",
       " 'LAZANDA',\n",
       " 'LEVORPHANOL TARTRATE',\n",
       " 'LORCET',\n",
       " 'LORCET 10-650',\n",
       " 'LORCET HD',\n",
       " 'LORCET PLUS',\n",
       " 'LORTAB',\n",
       " 'MAGNACET',\n",
       " 'MEPERIDINE HCL',\n",
       " 'MEPERIDINE HCL',\n",
       " 'MEPERITAB',\n",
       " 'METHADONE HCL',\n",
       " 'METHADONE INTENSOL',\n",
       " 'METHADOSE',\n",
       " 'MORPHINE SULFATE',\n",
       " 'MORPHINE SULFATE',\n",
       " 'MORPHINE SULFATE ER',\n",
       " 'MS CONTIN',\n",
       " 'NALBUPHINE HCL',\n",
       " 'NORCO',\n",
       " 'NUCYNTA',\n",
       " 'NUCYNTA ER',\n",
       " 'OPANA',\n",
       " 'OPANA ER',\n",
       " 'OPIUM TINCTURE',\n",
       " 'OXECTA',\n",
       " 'OXYCODONE HCL',\n",
       " 'OXYCODONE HCL ER',\n",
       " 'OXYCODONE HCL-ASPIRIN',\n",
       " 'OXYCODONE HCL-IBUPROFEN',\n",
       " 'OXYCODONE-ACETAMINOPHEN',\n",
       " 'OXYCONTIN',\n",
       " 'OXYMORPHONE HCL',\n",
       " 'OXYMORPHONE HCL ER',\n",
       " 'PENTAZOCINE-ACETAMINOPHEN',\n",
       " 'PENTAZOCINE-NALOXONE HCL',\n",
       " 'PERCOCET',\n",
       " 'PERCODAN',\n",
       " 'PRIMLEV',\n",
       " 'REPREXAIN',\n",
       " 'ROXICET',\n",
       " 'ROXICODONE',\n",
       " 'RYBIX ODT',\n",
       " 'STAGESIC',\n",
       " 'SUBSYS',\n",
       " 'SYNALGOS-DC',\n",
       " 'TALWIN',\n",
       " 'TRAMADOL HCL',\n",
       " 'TRAMADOL HCL ER',\n",
       " 'TRAMADOL HCL-ACETAMINOPHEN',\n",
       " 'TREZIX',\n",
       " 'TYLENOL-CODEINE NO.3',\n",
       " 'TYLENOL-CODEINE NO.4',\n",
       " 'ULTRACET',\n",
       " 'ULTRAM',\n",
       " 'ULTRAM ER',\n",
       " 'VICODIN',\n",
       " 'VICODIN ES',\n",
       " 'VICODIN HP',\n",
       " 'VICOPROFEN',\n",
       " 'XARTEMIS XR',\n",
       " 'XODOL 10-300',\n",
       " 'XODOL 5-300',\n",
       " 'XODOL 7.5-300',\n",
       " 'XTAMPZA ER',\n",
       " 'XYLON 10',\n",
       " 'ZAMICET',\n",
       " 'ZOHYDRO ER',\n",
       " 'ZOLVIT',\n",
       " 'Part D Prescriber PUF Summary - CY 2016: Opioids',\n",
       " 'Part D Prescriber PUF Summary - CY 2016: Extended-Release Opioids',\n",
       " 'Note: **Drug name includes NDC identifiers from the Prescriber Drug Category List for opioids',\n",
       " 'as well as NDC identifiers not included in the Prescriber Drug Category List for opioids.']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# %matplotlib inline # command for plotting graphs\n",
    "\n",
    "opName=list()\n",
    "\n",
    "import csv\n",
    "with open('./inputs/list_opioids.csv', newline='') as f:\n",
    "    reader = csv.reader(f)\n",
    "    for row in reader:\n",
    "        opName.append(row[0])\n",
    "\n",
    "#creating a list and adding durg names from the fie opioids.csv\n",
    "\n",
    "opName.pop(0)\n",
    "\n",
    "#Removing the first element i.e. a lablel\n",
    "\n",
    "opName\n",
    "\n",
    "#display the list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ABSTRAL', 'ACETAMINOPHEN', 'ACTIQ', 'ASA', 'ASCOMP', 'ASPIRIN', 'AVINZA', 'BELLADONNA', 'BUPRENORPHINE', 'BUTALB', 'BUTALB', 'BUTALBITAL', 'BUTORPHANOL', 'BUTRANS', 'CAPITAL', 'CARISOPRODOL', 'CARISOPRODOL', 'CODEINE', 'CO', 'CONZIP', 'DEMEROL', 'DEMEROL', 'DILAUDID', 'DILAUDID', 'DILAUDID', 'DISKETS', 'DOLOPHINE', 'DURAGESIC', 'DURAMORPH', 'ENDOCET', 'ENDODAN', 'EXALGO', 'FENTANYL', 'FENTANYL', 'FENTORA', 'FIORICET', 'FIORINAL', 'HYCET', 'HYDROCODONE', 'HYDROCODONE', 'HYDROMORPHONE', 'HYDROMORPHONE', 'HYDROMORPHONE', 'IBUDONE', 'INFUMORPH', 'KADIAN', 'LAZANDA', 'LEVORPHANOL', 'LORCET', 'LORCET', 'LORCET', 'LORCET', 'LORTAB', 'MAGNACET', 'MEPERIDINE', 'MEPERIDINE', 'MEPERITAB', 'METHADONE', 'METHADONE', 'METHADOSE', 'MORPHINE', 'MORPHINE', 'MORPHINE', 'MS', 'NALBUPHINE', 'NORCO', 'NUCYNTA', 'NUCYNTA', 'OPANA', 'OPANA', 'OPIUM', 'OXECTA', 'OXYCODONE', 'OXYCODONE', 'OXYCODONE', 'OXYCODONE', 'OXYCODONE', 'OXYCONTIN', 'OXYMORPHONE', 'OXYMORPHONE', 'PENTAZOCINE', 'PENTAZOCINE', 'PERCOCET', 'PERCODAN', 'PRIMLEV', 'REPREXAIN', 'ROXICET', 'ROXICODONE', 'RYBIX', 'STAGESIC', 'SUBSYS', 'SYNALGOS', 'TALWIN', 'TRAMADOL', 'TRAMADOL', 'TRAMADOL', 'TREZIX', 'TYLENOL', 'TYLENOL', 'ULTRACET', 'ULTRAM', 'ULTRAM', 'VICODIN', 'VICODIN', 'VICODIN', 'VICOPROFEN', 'XARTEMIS', 'XODOL', 'XODOL', 'XODOL', 'XTAMPZA', 'XYLON', 'ZAMICET', 'ZOHYDRO', 'ZOLVIT', 'Part', 'Part', 'Note:', 'as']\n"
     ]
    }
   ],
   "source": [
    "Opioids=list()\n",
    "\n",
    "#creating a new list\n",
    "\n",
    "import re\n",
    "for i in opName:\n",
    "    j=re.split(' ',i)[0]\n",
    "    Opioids.append(j.split('-')[0])\n",
    "    \n",
    "#Selecting the first element from every drug in the opName list\n",
    "\n",
    "print(Opioids)\n",
    "\n",
    "#Display the list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "238"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "RemoveCol=list()\n",
    "\n",
    "#Creating a list\n",
    "\n",
    "for Pcol in Prescription.columns[4:254]:\n",
    "    if Pcol not in Opioids:\n",
    "        RemoveCol.append(Pcol)\n",
    "\n",
    "# Adding the drug names in the list which are not opioid drugs\n",
    "\n",
    "len(RemoveCol)\n",
    "\n",
    "#Displaying the number of drigs that are not an opioid drug"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>State</th>\n",
       "      <th>Credentials</th>\n",
       "      <th>Specialty</th>\n",
       "      <th>ACETAMINOPHEN</th>\n",
       "      <th>CARISOPRODOL</th>\n",
       "      <th>FENTANYL</th>\n",
       "      <th>HYDROCODONE</th>\n",
       "      <th>HYDROMORPHONE</th>\n",
       "      <th>METHADONE</th>\n",
       "      <th>MORPHINE</th>\n",
       "      <th>MORPHINE</th>\n",
       "      <th>OXYCODONE</th>\n",
       "      <th>OXYCODONE</th>\n",
       "      <th>OXYCONTIN</th>\n",
       "      <th>TRAMADOL</th>\n",
       "      <th>Opioid</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NPI</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1710982582</th>\n",
       "      <td>M</td>\n",
       "      <td>TX</td>\n",
       "      <td>DDS</td>\n",
       "      <td>Dentist</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1245278100</th>\n",
       "      <td>F</td>\n",
       "      <td>AL</td>\n",
       "      <td>MD</td>\n",
       "      <td>General Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "      <td>182</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>24</td>\n",
       "      <td>33</td>\n",
       "      <td>87</td>\n",
       "      <td>0</td>\n",
       "      <td>36</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1669567541</th>\n",
       "      <td>M</td>\n",
       "      <td>AZ</td>\n",
       "      <td>MD</td>\n",
       "      <td>Internal Medicine</td>\n",
       "      <td>43</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "      <td>146</td>\n",
       "      <td>229</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>90</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1679650949</th>\n",
       "      <td>M</td>\n",
       "      <td>NV</td>\n",
       "      <td>M.D.</td>\n",
       "      <td>Hematology/Oncology</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1548580897</th>\n",
       "      <td>M</td>\n",
       "      <td>PA</td>\n",
       "      <td>DO</td>\n",
       "      <td>General Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Gender State Credentials            Specialty  ACETAMINOPHEN  \\\n",
       "NPI                                                                       \n",
       "1710982582      M    TX         DDS              Dentist              0   \n",
       "1245278100      F    AL          MD      General Surgery              0   \n",
       "1669567541      M    AZ          MD    Internal Medicine             43   \n",
       "1679650949      M    NV        M.D.  Hematology/Oncology              0   \n",
       "1548580897      M    PA          DO      General Surgery              0   \n",
       "\n",
       "            CARISOPRODOL  FENTANYL  HYDROCODONE  HYDROMORPHONE  METHADONE  \\\n",
       "NPI                                                                         \n",
       "1710982582             0         0           17              0          0   \n",
       "1245278100            45         0          182              0          0   \n",
       "1669567541             0         0            0              0          0   \n",
       "1679650949             0        22           22              0          0   \n",
       "1548580897             0         0            0              0          0   \n",
       "\n",
       "            MORPHINE  MORPHINE  OXYCODONE  OXYCODONE  OXYCONTIN  TRAMADOL  \\\n",
       "NPI                                                                         \n",
       "1710982582         0         0          0          0          0         0   \n",
       "1245278100         0        24         33         87          0        36   \n",
       "1669567541        22       146        229         29          0        90   \n",
       "1679650949         0         0          0         22          0         0   \n",
       "1548580897         0         0         25          0          0         0   \n",
       "\n",
       "            Opioid  \n",
       "NPI                 \n",
       "1710982582       1  \n",
       "1245278100       1  \n",
       "1669567541       1  \n",
       "1679650949       1  \n",
       "1548580897       1  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Prescription_opioid=Prescription.drop(RemoveCol, axis=1)\n",
    "\n",
    "#Removing the column of drug names that are not opioids because they are not relevant to be considered\n",
    "\n",
    "Prescription_opioid = Prescription_opioid[Prescription_opioid.Opioid == 1]\n",
    "\n",
    "#Selecting a subset that has entries for which the prsecriber or specialist prescribed opioids\n",
    "# all pysicians are not entitled to prescribe abd those who will prescribe will get NPI from FDA as well as the DEA\n",
    "\n",
    "Prescription_opioid.head()[:5]\n",
    "\n",
    "#Displaying the subset as follows of sepecialists that are entitled  to prescribe  the listed opiods by the DEA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>State</th>\n",
       "      <th>Credentials</th>\n",
       "      <th>Specialty</th>\n",
       "      <th>ABILIFY</th>\n",
       "      <th>ACETAMINOPHEN</th>\n",
       "      <th>ACYCLOVIR</th>\n",
       "      <th>ADVAIR</th>\n",
       "      <th>AGGRENOX</th>\n",
       "      <th>ALENDRONATE</th>\n",
       "      <th>...</th>\n",
       "      <th>VERAPAMIL</th>\n",
       "      <th>VESICARE</th>\n",
       "      <th>VOLTAREN</th>\n",
       "      <th>VYTORIN</th>\n",
       "      <th>WARFARIN</th>\n",
       "      <th>XARELTO</th>\n",
       "      <th>ZETIA</th>\n",
       "      <th>ZIPRASIDONE</th>\n",
       "      <th>ZOLPIDEM</th>\n",
       "      <th>Opioid</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NPI</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1427182161</th>\n",
       "      <td>F</td>\n",
       "      <td>NY</td>\n",
       "      <td>M.D.</td>\n",
       "      <td>General Practice</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1407113988</th>\n",
       "      <td>F</td>\n",
       "      <td>PA</td>\n",
       "      <td>RN, MSN, ANP-BC</td>\n",
       "      <td>Nurse Practitioner</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1023260569</th>\n",
       "      <td>M</td>\n",
       "      <td>TX</td>\n",
       "      <td>O.D.</td>\n",
       "      <td>Optometry</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1548275050</th>\n",
       "      <td>M</td>\n",
       "      <td>OH</td>\n",
       "      <td>MD</td>\n",
       "      <td>Cardiology</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1356388011</th>\n",
       "      <td>F</td>\n",
       "      <td>MA</td>\n",
       "      <td>M.D.</td>\n",
       "      <td>Internal Medicine</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 255 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           Gender State      Credentials           Specialty  ABILIFY  \\\n",
       "NPI                                                                     \n",
       "1427182161      F    NY             M.D.    General Practice        0   \n",
       "1407113988      F    PA  RN, MSN, ANP-BC  Nurse Practitioner        0   \n",
       "1023260569      M    TX             O.D.           Optometry        0   \n",
       "1548275050      M    OH               MD          Cardiology        0   \n",
       "1356388011      F    MA             M.D.   Internal Medicine        0   \n",
       "\n",
       "            ACETAMINOPHEN  ACYCLOVIR  ADVAIR  AGGRENOX  ALENDRONATE   ...    \\\n",
       "NPI                                                                   ...     \n",
       "1427182161              0          0       0         0            0   ...     \n",
       "1407113988              0          0       0         0            0   ...     \n",
       "1023260569              0          0       0         0            0   ...     \n",
       "1548275050              0          0       0         0            0   ...     \n",
       "1356388011              0          0       0         0           14   ...     \n",
       "\n",
       "            VERAPAMIL  VESICARE  VOLTAREN  VYTORIN  WARFARIN  XARELTO  ZETIA  \\\n",
       "NPI                                                                            \n",
       "1427182161          0         0         0        0         0        0      0   \n",
       "1407113988          0         0         0        0         0        0      0   \n",
       "1023260569          0         0         0        0         0        0      0   \n",
       "1548275050          0         0         0        0         0        0      0   \n",
       "1356388011          0         0         0        0         0        0      0   \n",
       "\n",
       "            ZIPRASIDONE  ZOLPIDEM  Opioid  \n",
       "NPI                                        \n",
       "1427182161            0        25       0  \n",
       "1407113988            0         0       0  \n",
       "1023260569            0         0       0  \n",
       "1548275050            0         0       0  \n",
       "1356388011            0         0       0  \n",
       "\n",
       "[5 rows x 255 columns]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Prescription_notopioid = Prescription[Prescription.Opioid == 0]\n",
    "\n",
    "#Selecting a subset that has entries for which the prsecriber did not prescribed opioids\n",
    "\n",
    "Prescription_notopioid.head()[:5]\n",
    "\n",
    "# Displaying the subset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14688"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(Prescription_opioid)\n",
    "\n",
    "#The number of entries pertaining to the prescriber specialist prescribing opioids across all the 52 states and\n",
    "# other US locations including military facilities that people are entitled to get the medications from the  center for \n",
    "# medical services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10312"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(Prescription_notopioid)\n",
    "\n",
    "#The number of entries pertaining to the specialist or prescriber not entitled to prescribing opioids by the DEA and\n",
    "# these pysicians or specialists are not entitled to prescribe any opiods for the patient for the maximum 84 days\n",
    "# according to the DEA and FDA narcotics control and prescription guidelines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "Prescription= Prescription_opioid\n",
    "\n",
    "#Make a copy of the Prescription_opioid dataframe which will help to work on little data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NPI\n",
       "1710982582             DDS\n",
       "1245278100              MD\n",
       "1669567541              MD\n",
       "1679650949              MD\n",
       "1548580897              DO\n",
       "1437192002              MD\n",
       "1821106832              MD\n",
       "1609931914              MD\n",
       "1659334472              MD\n",
       "1144205303              MD\n",
       "1952598419              MD\n",
       "1780661793              MD\n",
       "1548238389              MD\n",
       "1366582587             DDS\n",
       "1083907059              MD\n",
       "1598748931              MD\n",
       "1063566727             DPM\n",
       "1629394754              MD\n",
       "1174555080              MD\n",
       "1871548537              MD\n",
       "1841349677       MSNAPRNBC\n",
       "1629038443              MD\n",
       "1386607315              DO\n",
       "1295852101              MD\n",
       "1467448381             PAC\n",
       "1285675637              MD\n",
       "1679630891    DNPAPRNBCFNP\n",
       "1346496197              MD\n",
       "1922136613             DDS\n",
       "1235426800            CRNP\n",
       "                  ...     \n",
       "1669782355             DMD\n",
       "1689618753              MD\n",
       "1376836916             DPM\n",
       "1407898505              MD\n",
       "1679914113            APRN\n",
       "1871541482              MD\n",
       "1710929898              MD\n",
       "1558395954              MD\n",
       "1710963681              MD\n",
       "1639337884              MD\n",
       "1407090921              MD\n",
       "1205977352              MD\n",
       "1023019064              MD\n",
       "1912068164            CRNP\n",
       "1811084726             DDS\n",
       "1134193352              MD\n",
       "1427157841              DO\n",
       "1356344964              MD\n",
       "1114958618              DO\n",
       "1811254113             PAC\n",
       "1447222344              PA\n",
       "1891735460              MD\n",
       "1073519963            CRNP\n",
       "1255646931              MD\n",
       "1114963212              MD\n",
       "1427008374              MD\n",
       "1619019353              MD\n",
       "1780693671              DO\n",
       "1346270956              MD\n",
       "1023116514              MD\n",
       "Name: Credentials, Length: 14688, dtype: object"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Prescription['Credentials'] = Prescription['Credentials'].str.replace('[^\\w\\s]','')\n",
    "Prescription['Credentials']=Prescription['Credentials'].str.replace(' ','')\n",
    "Prescription['Credentials']\n",
    "\n",
    "#Cleaning the Speciality column in the dataframe by removing the punctuations as some have white spaces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "Prescription\n",
    "df=Prescription\n",
    "\n",
    "#Make a copy of the Prescription dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NPI\n",
       "1710982582   NaN\n",
       "1245278100   NaN\n",
       "1669567541   NaN\n",
       "1679650949   NaN\n",
       "1548580897   NaN\n",
       "Name: Total_Opioids, dtype: float64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Total_Opioids']=pd.DataFrame({'Total_Opioids': [] })\n",
    "\n",
    "#Adding a new column that shows the total opiods prescribed by the specialists who are entitled to practice\n",
    "df['Total_Opioids'][:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "b=list()\n",
    "for i in range(0, len(df)):\n",
    "    a=list(df.iloc[i,4:16])\n",
    "    df.iloc[i,17]=sum(a)\n",
    "\n",
    "#Adding number of opioids prescribed by the Speciality for any opioid drugs \n",
    "# and assigning it to the new variable 'Total opioids' so that it will be easy \n",
    "# to work on with out refering back to the  previous version of the Dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>State</th>\n",
       "      <th>Credentials</th>\n",
       "      <th>Specialty</th>\n",
       "      <th>ACETAMINOPHEN</th>\n",
       "      <th>CARISOPRODOL</th>\n",
       "      <th>FENTANYL</th>\n",
       "      <th>HYDROCODONE</th>\n",
       "      <th>HYDROMORPHONE</th>\n",
       "      <th>METHADONE</th>\n",
       "      <th>MORPHINE</th>\n",
       "      <th>MORPHINE</th>\n",
       "      <th>OXYCODONE</th>\n",
       "      <th>OXYCODONE</th>\n",
       "      <th>OXYCONTIN</th>\n",
       "      <th>TRAMADOL</th>\n",
       "      <th>Opioid</th>\n",
       "      <th>Total_Opioids</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NPI</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1710982582</th>\n",
       "      <td>M</td>\n",
       "      <td>TX</td>\n",
       "      <td>DDS</td>\n",
       "      <td>Dentist</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1245278100</th>\n",
       "      <td>F</td>\n",
       "      <td>AL</td>\n",
       "      <td>MD</td>\n",
       "      <td>General Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "      <td>182</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>24</td>\n",
       "      <td>33</td>\n",
       "      <td>87</td>\n",
       "      <td>0</td>\n",
       "      <td>36</td>\n",
       "      <td>1</td>\n",
       "      <td>407.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1669567541</th>\n",
       "      <td>M</td>\n",
       "      <td>AZ</td>\n",
       "      <td>MD</td>\n",
       "      <td>Internal Medicine</td>\n",
       "      <td>43</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "      <td>146</td>\n",
       "      <td>229</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>90</td>\n",
       "      <td>1</td>\n",
       "      <td>559.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1679650949</th>\n",
       "      <td>M</td>\n",
       "      <td>NV</td>\n",
       "      <td>MD</td>\n",
       "      <td>Hematology/Oncology</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>66.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1548580897</th>\n",
       "      <td>M</td>\n",
       "      <td>PA</td>\n",
       "      <td>DO</td>\n",
       "      <td>General Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1437192002</th>\n",
       "      <td>M</td>\n",
       "      <td>NH</td>\n",
       "      <td>MD</td>\n",
       "      <td>Family Practice</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>90.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1821106832</th>\n",
       "      <td>F</td>\n",
       "      <td>WI</td>\n",
       "      <td>MD</td>\n",
       "      <td>Internal Medicine</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>28</td>\n",
       "      <td>1</td>\n",
       "      <td>64.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1609931914</th>\n",
       "      <td>F</td>\n",
       "      <td>PR</td>\n",
       "      <td>MD</td>\n",
       "      <td>General Practice</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>71</td>\n",
       "      <td>1</td>\n",
       "      <td>71.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1659334472</th>\n",
       "      <td>M</td>\n",
       "      <td>TX</td>\n",
       "      <td>MD</td>\n",
       "      <td>General Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1144205303</th>\n",
       "      <td>M</td>\n",
       "      <td>CO</td>\n",
       "      <td>MD</td>\n",
       "      <td>Family Practice</td>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>109</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>22</td>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>1</td>\n",
       "      <td>247.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1952598419</th>\n",
       "      <td>F</td>\n",
       "      <td>TX</td>\n",
       "      <td>MD</td>\n",
       "      <td>Hematology/Oncology</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>69</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>37</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>36</td>\n",
       "      <td>1</td>\n",
       "      <td>155.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1780661793</th>\n",
       "      <td>M</td>\n",
       "      <td>OH</td>\n",
       "      <td>MD</td>\n",
       "      <td>General Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1548238389</th>\n",
       "      <td>M</td>\n",
       "      <td>FL</td>\n",
       "      <td>MD</td>\n",
       "      <td>General Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>41.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1366582587</th>\n",
       "      <td>M</td>\n",
       "      <td>MN</td>\n",
       "      <td>DDS</td>\n",
       "      <td>Dentist</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1083907059</th>\n",
       "      <td>M</td>\n",
       "      <td>UT</td>\n",
       "      <td>MD</td>\n",
       "      <td>Obstetrics/Gynecology</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1598748931</th>\n",
       "      <td>F</td>\n",
       "      <td>AL</td>\n",
       "      <td>MD</td>\n",
       "      <td>Family Practice</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>86</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>1</td>\n",
       "      <td>139.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1063566727</th>\n",
       "      <td>F</td>\n",
       "      <td>IL</td>\n",
       "      <td>DPM</td>\n",
       "      <td>Podiatry</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>48</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>48.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1629394754</th>\n",
       "      <td>M</td>\n",
       "      <td>NY</td>\n",
       "      <td>MD</td>\n",
       "      <td>Student in an Organized Health Care Education/...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1174555080</th>\n",
       "      <td>F</td>\n",
       "      <td>FL</td>\n",
       "      <td>MD</td>\n",
       "      <td>Internal Medicine</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>20.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1871548537</th>\n",
       "      <td>F</td>\n",
       "      <td>MT</td>\n",
       "      <td>MD</td>\n",
       "      <td>Internal Medicine</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>31.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1841349677</th>\n",
       "      <td>F</td>\n",
       "      <td>IN</td>\n",
       "      <td>MSNAPRNBC</td>\n",
       "      <td>Nurse Practitioner</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>24</td>\n",
       "      <td>37</td>\n",
       "      <td>12</td>\n",
       "      <td>152</td>\n",
       "      <td>56</td>\n",
       "      <td>93</td>\n",
       "      <td>0</td>\n",
       "      <td>131</td>\n",
       "      <td>82</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>587.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1629038443</th>\n",
       "      <td>M</td>\n",
       "      <td>VA</td>\n",
       "      <td>MD</td>\n",
       "      <td>General Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1386607315</th>\n",
       "      <td>F</td>\n",
       "      <td>FL</td>\n",
       "      <td>DO</td>\n",
       "      <td>Neurology</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1295852101</th>\n",
       "      <td>M</td>\n",
       "      <td>IN</td>\n",
       "      <td>MD</td>\n",
       "      <td>General Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>53</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>53.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1467448381</th>\n",
       "      <td>F</td>\n",
       "      <td>OR</td>\n",
       "      <td>PAC</td>\n",
       "      <td>Physician Assistant</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>55</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>19</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>116.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1285675637</th>\n",
       "      <td>M</td>\n",
       "      <td>NE</td>\n",
       "      <td>MD</td>\n",
       "      <td>Family Practice</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>90</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>153</td>\n",
       "      <td>1</td>\n",
       "      <td>256.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1679630891</th>\n",
       "      <td>F</td>\n",
       "      <td>TN</td>\n",
       "      <td>DNPAPRNBCFNP</td>\n",
       "      <td>Nurse Practitioner</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>104</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>36</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>161.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1346496197</th>\n",
       "      <td>F</td>\n",
       "      <td>IL</td>\n",
       "      <td>MD</td>\n",
       "      <td>Rheumatology</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1922136613</th>\n",
       "      <td>M</td>\n",
       "      <td>CT</td>\n",
       "      <td>DDS</td>\n",
       "      <td>Dentist</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1235426800</th>\n",
       "      <td>F</td>\n",
       "      <td>PA</td>\n",
       "      <td>CRNP</td>\n",
       "      <td>Nurse Practitioner</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>33.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1669782355</th>\n",
       "      <td>M</td>\n",
       "      <td>AZ</td>\n",
       "      <td>DMD</td>\n",
       "      <td>Dentist</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>19.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1689618753</th>\n",
       "      <td>M</td>\n",
       "      <td>SC</td>\n",
       "      <td>MD</td>\n",
       "      <td>Gastroenterology</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>22.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1376836916</th>\n",
       "      <td>M</td>\n",
       "      <td>OH</td>\n",
       "      <td>DPM</td>\n",
       "      <td>Podiatry</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1407898505</th>\n",
       "      <td>M</td>\n",
       "      <td>TX</td>\n",
       "      <td>MD</td>\n",
       "      <td>Hematology/Oncology</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>69</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>80.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1679914113</th>\n",
       "      <td>F</td>\n",
       "      <td>AR</td>\n",
       "      <td>APRN</td>\n",
       "      <td>Nurse Practitioner</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>49</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>63.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1871541482</th>\n",
       "      <td>M</td>\n",
       "      <td>AL</td>\n",
       "      <td>MD</td>\n",
       "      <td>Internal Medicine</td>\n",
       "      <td>28</td>\n",
       "      <td>20</td>\n",
       "      <td>22</td>\n",
       "      <td>610</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>36</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>196</td>\n",
       "      <td>1</td>\n",
       "      <td>923.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1710929898</th>\n",
       "      <td>F</td>\n",
       "      <td>NJ</td>\n",
       "      <td>MD</td>\n",
       "      <td>Internal Medicine</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>53.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1558395954</th>\n",
       "      <td>F</td>\n",
       "      <td>MI</td>\n",
       "      <td>MD</td>\n",
       "      <td>Family Practice</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "      <td>77</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>116.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1710963681</th>\n",
       "      <td>M</td>\n",
       "      <td>AK</td>\n",
       "      <td>MD</td>\n",
       "      <td>Anesthesiology</td>\n",
       "      <td>0</td>\n",
       "      <td>29</td>\n",
       "      <td>28</td>\n",
       "      <td>82</td>\n",
       "      <td>84</td>\n",
       "      <td>25</td>\n",
       "      <td>11</td>\n",
       "      <td>74</td>\n",
       "      <td>0</td>\n",
       "      <td>132</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>496.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1639337884</th>\n",
       "      <td>M</td>\n",
       "      <td>NJ</td>\n",
       "      <td>MD</td>\n",
       "      <td>Orthopedic Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1407090921</th>\n",
       "      <td>M</td>\n",
       "      <td>FL</td>\n",
       "      <td>MD</td>\n",
       "      <td>Internal Medicine</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>90</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>68</td>\n",
       "      <td>1</td>\n",
       "      <td>182.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1205977352</th>\n",
       "      <td>F</td>\n",
       "      <td>OH</td>\n",
       "      <td>MD</td>\n",
       "      <td>Family Practice</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>37</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>62.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1023019064</th>\n",
       "      <td>M</td>\n",
       "      <td>LA</td>\n",
       "      <td>MD</td>\n",
       "      <td>Hematology/Oncology</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>74.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1912068164</th>\n",
       "      <td>F</td>\n",
       "      <td>MD</td>\n",
       "      <td>CRNP</td>\n",
       "      <td>Nurse Practitioner</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1811084726</th>\n",
       "      <td>M</td>\n",
       "      <td>MA</td>\n",
       "      <td>DDS</td>\n",
       "      <td>Maxillofacial Surgery</td>\n",
       "      <td>78</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>140</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>218.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1134193352</th>\n",
       "      <td>M</td>\n",
       "      <td>NM</td>\n",
       "      <td>MD</td>\n",
       "      <td>Emergency Medicine</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1427157841</th>\n",
       "      <td>M</td>\n",
       "      <td>IL</td>\n",
       "      <td>DO</td>\n",
       "      <td>Emergency Medicine</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>33.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1356344964</th>\n",
       "      <td>F</td>\n",
       "      <td>VA</td>\n",
       "      <td>MD</td>\n",
       "      <td>Emergency Medicine</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>11</td>\n",
       "      <td>162</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>69</td>\n",
       "      <td>44</td>\n",
       "      <td>0</td>\n",
       "      <td>127</td>\n",
       "      <td>1</td>\n",
       "      <td>453.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1114958618</th>\n",
       "      <td>M</td>\n",
       "      <td>PA</td>\n",
       "      <td>DO</td>\n",
       "      <td>Physical Medicine and Rehabilitation</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>100</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>45</td>\n",
       "      <td>47</td>\n",
       "      <td>12</td>\n",
       "      <td>21</td>\n",
       "      <td>1</td>\n",
       "      <td>253.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1811254113</th>\n",
       "      <td>F</td>\n",
       "      <td>CO</td>\n",
       "      <td>PAC</td>\n",
       "      <td>Physician Assistant</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>27</td>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>41.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1447222344</th>\n",
       "      <td>F</td>\n",
       "      <td>IA</td>\n",
       "      <td>PA</td>\n",
       "      <td>Physician Assistant</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>37</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>54</td>\n",
       "      <td>1</td>\n",
       "      <td>91.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1891735460</th>\n",
       "      <td>M</td>\n",
       "      <td>OH</td>\n",
       "      <td>MD</td>\n",
       "      <td>Family Practice</td>\n",
       "      <td>14</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "      <td>136</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>237</td>\n",
       "      <td>1</td>\n",
       "      <td>420.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1073519963</th>\n",
       "      <td>F</td>\n",
       "      <td>MD</td>\n",
       "      <td>CRNP</td>\n",
       "      <td>Nurse Practitioner</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>22</td>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>53</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>188.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1255646931</th>\n",
       "      <td>F</td>\n",
       "      <td>AZ</td>\n",
       "      <td>MD</td>\n",
       "      <td>Internal Medicine</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1114963212</th>\n",
       "      <td>F</td>\n",
       "      <td>IL</td>\n",
       "      <td>MD</td>\n",
       "      <td>Internal Medicine</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>20.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1427008374</th>\n",
       "      <td>M</td>\n",
       "      <td>NC</td>\n",
       "      <td>MD</td>\n",
       "      <td>General Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>46</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>69.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1619019353</th>\n",
       "      <td>F</td>\n",
       "      <td>WA</td>\n",
       "      <td>MD</td>\n",
       "      <td>Family Practice</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>19</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>92.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1780693671</th>\n",
       "      <td>F</td>\n",
       "      <td>MI</td>\n",
       "      <td>DO</td>\n",
       "      <td>Internal Medicine</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>58</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "      <td>102.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1346270956</th>\n",
       "      <td>M</td>\n",
       "      <td>AZ</td>\n",
       "      <td>MD</td>\n",
       "      <td>Emergency Medicine</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>42.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1023116514</th>\n",
       "      <td>F</td>\n",
       "      <td>IN</td>\n",
       "      <td>MD</td>\n",
       "      <td>Family Practice</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>14688 rows × 18 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           Gender State   Credentials  \\\n",
       "NPI                                     \n",
       "1710982582      M    TX           DDS   \n",
       "1245278100      F    AL            MD   \n",
       "1669567541      M    AZ            MD   \n",
       "1679650949      M    NV            MD   \n",
       "1548580897      M    PA            DO   \n",
       "1437192002      M    NH            MD   \n",
       "1821106832      F    WI            MD   \n",
       "1609931914      F    PR            MD   \n",
       "1659334472      M    TX            MD   \n",
       "1144205303      M    CO            MD   \n",
       "1952598419      F    TX            MD   \n",
       "1780661793      M    OH            MD   \n",
       "1548238389      M    FL            MD   \n",
       "1366582587      M    MN           DDS   \n",
       "1083907059      M    UT            MD   \n",
       "1598748931      F    AL            MD   \n",
       "1063566727      F    IL           DPM   \n",
       "1629394754      M    NY            MD   \n",
       "1174555080      F    FL            MD   \n",
       "1871548537      F    MT            MD   \n",
       "1841349677      F    IN     MSNAPRNBC   \n",
       "1629038443      M    VA            MD   \n",
       "1386607315      F    FL            DO   \n",
       "1295852101      M    IN            MD   \n",
       "1467448381      F    OR           PAC   \n",
       "1285675637      M    NE            MD   \n",
       "1679630891      F    TN  DNPAPRNBCFNP   \n",
       "1346496197      F    IL            MD   \n",
       "1922136613      M    CT           DDS   \n",
       "1235426800      F    PA          CRNP   \n",
       "...           ...   ...           ...   \n",
       "1669782355      M    AZ           DMD   \n",
       "1689618753      M    SC            MD   \n",
       "1376836916      M    OH           DPM   \n",
       "1407898505      M    TX            MD   \n",
       "1679914113      F    AR          APRN   \n",
       "1871541482      M    AL            MD   \n",
       "1710929898      F    NJ            MD   \n",
       "1558395954      F    MI            MD   \n",
       "1710963681      M    AK            MD   \n",
       "1639337884      M    NJ            MD   \n",
       "1407090921      M    FL            MD   \n",
       "1205977352      F    OH            MD   \n",
       "1023019064      M    LA            MD   \n",
       "1912068164      F    MD          CRNP   \n",
       "1811084726      M    MA           DDS   \n",
       "1134193352      M    NM            MD   \n",
       "1427157841      M    IL            DO   \n",
       "1356344964      F    VA            MD   \n",
       "1114958618      M    PA            DO   \n",
       "1811254113      F    CO           PAC   \n",
       "1447222344      F    IA            PA   \n",
       "1891735460      M    OH            MD   \n",
       "1073519963      F    MD          CRNP   \n",
       "1255646931      F    AZ            MD   \n",
       "1114963212      F    IL            MD   \n",
       "1427008374      M    NC            MD   \n",
       "1619019353      F    WA            MD   \n",
       "1780693671      F    MI            DO   \n",
       "1346270956      M    AZ            MD   \n",
       "1023116514      F    IN            MD   \n",
       "\n",
       "                                                    Specialty  ACETAMINOPHEN  \\\n",
       "NPI                                                                            \n",
       "1710982582                                            Dentist              0   \n",
       "1245278100                                    General Surgery              0   \n",
       "1669567541                                  Internal Medicine             43   \n",
       "1679650949                                Hematology/Oncology              0   \n",
       "1548580897                                    General Surgery              0   \n",
       "1437192002                                    Family Practice              0   \n",
       "1821106832                                  Internal Medicine              0   \n",
       "1609931914                                   General Practice              0   \n",
       "1659334472                                    General Surgery              0   \n",
       "1144205303                                    Family Practice             14   \n",
       "1952598419                                Hematology/Oncology              0   \n",
       "1780661793                                    General Surgery              0   \n",
       "1548238389                                    General Surgery              0   \n",
       "1366582587                                            Dentist              0   \n",
       "1083907059                              Obstetrics/Gynecology              0   \n",
       "1598748931                                    Family Practice              0   \n",
       "1063566727                                           Podiatry              0   \n",
       "1629394754  Student in an Organized Health Care Education/...              0   \n",
       "1174555080                                  Internal Medicine              0   \n",
       "1871548537                                  Internal Medicine              0   \n",
       "1841349677                                 Nurse Practitioner              0   \n",
       "1629038443                                    General Surgery              0   \n",
       "1386607315                                          Neurology              0   \n",
       "1295852101                                    General Surgery              0   \n",
       "1467448381                                Physician Assistant              0   \n",
       "1285675637                                    Family Practice              0   \n",
       "1679630891                                 Nurse Practitioner              0   \n",
       "1346496197                                       Rheumatology              0   \n",
       "1922136613                                            Dentist             15   \n",
       "1235426800                                 Nurse Practitioner              0   \n",
       "...                                                       ...            ...   \n",
       "1669782355                                            Dentist              0   \n",
       "1689618753                                   Gastroenterology              0   \n",
       "1376836916                                           Podiatry              0   \n",
       "1407898505                                Hematology/Oncology              0   \n",
       "1679914113                                 Nurse Practitioner              0   \n",
       "1871541482                                  Internal Medicine             28   \n",
       "1710929898                                  Internal Medicine             21   \n",
       "1558395954                                    Family Practice              0   \n",
       "1710963681                                     Anesthesiology              0   \n",
       "1639337884                                 Orthopedic Surgery              0   \n",
       "1407090921                                  Internal Medicine              0   \n",
       "1205977352                                    Family Practice              0   \n",
       "1023019064                                Hematology/Oncology              0   \n",
       "1912068164                                 Nurse Practitioner              0   \n",
       "1811084726                              Maxillofacial Surgery             78   \n",
       "1134193352                                 Emergency Medicine              0   \n",
       "1427157841                                 Emergency Medicine              0   \n",
       "1356344964                                 Emergency Medicine              0   \n",
       "1114958618               Physical Medicine and Rehabilitation              0   \n",
       "1811254113                                Physician Assistant              0   \n",
       "1447222344                                Physician Assistant              0   \n",
       "1891735460                                    Family Practice             14   \n",
       "1073519963                                 Nurse Practitioner              0   \n",
       "1255646931                                  Internal Medicine              0   \n",
       "1114963212                                  Internal Medicine              0   \n",
       "1427008374                                    General Surgery              0   \n",
       "1619019353                                    Family Practice             12   \n",
       "1780693671                                  Internal Medicine             15   \n",
       "1346270956                                 Emergency Medicine              0   \n",
       "1023116514                                    Family Practice              0   \n",
       "\n",
       "            CARISOPRODOL  FENTANYL  HYDROCODONE  HYDROMORPHONE  METHADONE  \\\n",
       "NPI                                                                         \n",
       "1710982582             0         0           17              0          0   \n",
       "1245278100            45         0          182              0          0   \n",
       "1669567541             0         0            0              0          0   \n",
       "1679650949             0        22           22              0          0   \n",
       "1548580897             0         0            0              0          0   \n",
       "1437192002             0         0           24              0          0   \n",
       "1821106832             0        25            0              0          0   \n",
       "1609931914             0         0            0              0          0   \n",
       "1659334472             0         0           17              0          0   \n",
       "1144205303             0        14          109              0          0   \n",
       "1952598419             0         0           69             13          0   \n",
       "1780661793             0         0            0              0          0   \n",
       "1548238389             0         0           28              0          0   \n",
       "1366582587             0         0           14              0          0   \n",
       "1083907059             0         0            0              0          0   \n",
       "1598748931             0         0           86              0          0   \n",
       "1063566727             0         0           48              0          0   \n",
       "1629394754             0         0            0              0          0   \n",
       "1174555080             0         0            0              0          0   \n",
       "1871548537             0         0           13              0          0   \n",
       "1841349677             0        24           37             12        152   \n",
       "1629038443             0         0            0              0          0   \n",
       "1386607315             0         0            0              0          0   \n",
       "1295852101             0         0           53              0          0   \n",
       "1467448381            12         0           55              0          0   \n",
       "1285675637             0         0           90              0          0   \n",
       "1679630891             0         0          104              0          0   \n",
       "1346496197             0         0           25              0          0   \n",
       "1922136613             0         0            0              0          0   \n",
       "1235426800             0         0            0              0          0   \n",
       "...                  ...       ...          ...            ...        ...   \n",
       "1669782355             0         0            0              0          0   \n",
       "1689618753             0         0           22              0          0   \n",
       "1376836916             0         0           14              0          0   \n",
       "1407898505             0        11           69              0          0   \n",
       "1679914113             0         0           49              0          0   \n",
       "1871541482            20        22          610              0          0   \n",
       "1710929898             0        32            0              0          0   \n",
       "1558395954             0        22           77              0          0   \n",
       "1710963681            29        28           82             84         25   \n",
       "1639337884             0         0            0              0          0   \n",
       "1407090921             0         0           90              0          0   \n",
       "1205977352             0         0           37              0         13   \n",
       "1023019064             0         0           29              0          0   \n",
       "1912068164             0         0           18              0          0   \n",
       "1811084726             0         0          140              0          0   \n",
       "1134193352             0         0            0              0          0   \n",
       "1427157841             0         0           22              0          0   \n",
       "1356344964            16        11          162             13          0   \n",
       "1114958618             0        12          100              0          0   \n",
       "1811254113             0         0            0              0          0   \n",
       "1447222344             0         0           37              0          0   \n",
       "1891735460            18         0          136              0          0   \n",
       "1073519963             0        14           22             17          0   \n",
       "1255646931             0         0            0              0          0   \n",
       "1114963212             0         0           20              0          0   \n",
       "1427008374             0         0           46              0          0   \n",
       "1619019353             0         0           40              0          0   \n",
       "1780693671             0         0           58              0          0   \n",
       "1346270956             0         0           42              0          0   \n",
       "1023116514             0         0            0              0          0   \n",
       "\n",
       "            MORPHINE  MORPHINE  OXYCODONE  OXYCODONE  OXYCONTIN  TRAMADOL  \\\n",
       "NPI                                                                         \n",
       "1710982582         0         0          0          0          0         0   \n",
       "1245278100         0        24         33         87          0        36   \n",
       "1669567541        22       146        229         29          0        90   \n",
       "1679650949         0         0          0         22          0         0   \n",
       "1548580897         0         0         25          0          0         0   \n",
       "1437192002         0         0         30         21          0        15   \n",
       "1821106832         0         0          0         11          0        28   \n",
       "1609931914         0         0          0          0          0        71   \n",
       "1659334472         0         0          0          0          0         0   \n",
       "1144205303         0        11         22         17          0        60   \n",
       "1952598419         0        37          0          0          0        36   \n",
       "1780661793         0         0         25          0          0         0   \n",
       "1548238389         0         0         13          0          0         0   \n",
       "1366582587         0         0          0          0          0         0   \n",
       "1083907059         0         0          0         11          0         0   \n",
       "1598748931         0         0         13          0          0        40   \n",
       "1063566727         0         0          0          0          0         0   \n",
       "1629394754         0         0          0          0          0         0   \n",
       "1174555080         0         0          0         20          0         0   \n",
       "1871548537         0         0          0          0          0        18   \n",
       "1841349677        56        93          0        131         82         0   \n",
       "1629038443         0         0          0          0          0         0   \n",
       "1386607315         0         0          0          0          0         0   \n",
       "1295852101         0         0          0          0          0         0   \n",
       "1467448381         0         0         19         15          0        15   \n",
       "1285675637         0         0          0         13          0       153   \n",
       "1679630891         0         0         21         36          0         0   \n",
       "1346496197         0         0          0          0          0         0   \n",
       "1922136613         0         0          0          0          0         0   \n",
       "1235426800         0         0          0         33          0         0   \n",
       "...              ...       ...        ...        ...        ...       ...   \n",
       "1669782355         0         0         19          0          0         0   \n",
       "1689618753         0         0          0          0          0         0   \n",
       "1376836916         0         0          0          0          0         0   \n",
       "1407898505         0         0          0          0          0         0   \n",
       "1679914113         0         0          0          0          0        14   \n",
       "1871541482         0        11         36          0          0       196   \n",
       "1710929898         0         0          0          0          0         0   \n",
       "1558395954         0         0          0          0          0        17   \n",
       "1710963681        11        74          0        132         31         0   \n",
       "1639337884         0         0         11          0          0         0   \n",
       "1407090921         0         0         24          0          0        68   \n",
       "1205977352         0         0          0         12          0         0   \n",
       "1023019064        13        17          0         15          0         0   \n",
       "1912068164         0         0          0          0          0         0   \n",
       "1811084726         0         0          0          0          0         0   \n",
       "1134193352         0         0          0          0          0         0   \n",
       "1427157841         0         0          0          0          0        11   \n",
       "1356344964        11         0         69         44          0       127   \n",
       "1114958618         0        16         45         47         12        21   \n",
       "1811254113         0         0          0         27         14         0   \n",
       "1447222344         0         0          0          0          0        54   \n",
       "1891735460         0         0         15          0          0       237   \n",
       "1073519963         0         0         53         57          0        25   \n",
       "1255646931         0         0          0          0          0         0   \n",
       "1114963212         0         0          0          0          0         0   \n",
       "1427008374         0         0         23          0          0         0   \n",
       "1619019353         0         0         19         21          0         0   \n",
       "1780693671         0         0          0          0          0        29   \n",
       "1346270956         0         0          0          0          0         0   \n",
       "1023116514         0         0          0          0          0         0   \n",
       "\n",
       "            Opioid  Total_Opioids  \n",
       "NPI                                \n",
       "1710982582       1           17.0  \n",
       "1245278100       1          407.0  \n",
       "1669567541       1          559.0  \n",
       "1679650949       1           66.0  \n",
       "1548580897       1           25.0  \n",
       "1437192002       1           90.0  \n",
       "1821106832       1           64.0  \n",
       "1609931914       1           71.0  \n",
       "1659334472       1           17.0  \n",
       "1144205303       1          247.0  \n",
       "1952598419       1          155.0  \n",
       "1780661793       1           25.0  \n",
       "1548238389       1           41.0  \n",
       "1366582587       1           14.0  \n",
       "1083907059       1           11.0  \n",
       "1598748931       1          139.0  \n",
       "1063566727       1           48.0  \n",
       "1629394754       1            0.0  \n",
       "1174555080       1           20.0  \n",
       "1871548537       1           31.0  \n",
       "1841349677       1          587.0  \n",
       "1629038443       1            0.0  \n",
       "1386607315       1            0.0  \n",
       "1295852101       1           53.0  \n",
       "1467448381       1          116.0  \n",
       "1285675637       1          256.0  \n",
       "1679630891       1          161.0  \n",
       "1346496197       1           25.0  \n",
       "1922136613       1           15.0  \n",
       "1235426800       1           33.0  \n",
       "...            ...            ...  \n",
       "1669782355       1           19.0  \n",
       "1689618753       1           22.0  \n",
       "1376836916       1           14.0  \n",
       "1407898505       1           80.0  \n",
       "1679914113       1           63.0  \n",
       "1871541482       1          923.0  \n",
       "1710929898       1           53.0  \n",
       "1558395954       1          116.0  \n",
       "1710963681       1          496.0  \n",
       "1639337884       1           11.0  \n",
       "1407090921       1          182.0  \n",
       "1205977352       1           62.0  \n",
       "1023019064       1           74.0  \n",
       "1912068164       1           18.0  \n",
       "1811084726       1          218.0  \n",
       "1134193352       1            0.0  \n",
       "1427157841       1           33.0  \n",
       "1356344964       1          453.0  \n",
       "1114958618       1          253.0  \n",
       "1811254113       1           41.0  \n",
       "1447222344       1           91.0  \n",
       "1891735460       1          420.0  \n",
       "1073519963       1          188.0  \n",
       "1255646931       1            0.0  \n",
       "1114963212       1           20.0  \n",
       "1427008374       1           69.0  \n",
       "1619019353       1           92.0  \n",
       "1780693671       1          102.0  \n",
       "1346270956       1           42.0  \n",
       "1023116514       1            0.0  \n",
       "\n",
       "[14688 rows x 18 columns]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df #Displaying the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df.to_csv('./inputs/TotalOpioidsvisualization.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_df = pd.read_csv('./inputs/prescribers_clean832.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>ABILIFY</th>\n",
       "      <th>ACETAMINOPHEN.CODEINE</th>\n",
       "      <th>ACYCLOVIR</th>\n",
       "      <th>ADVAIR.DISKUS</th>\n",
       "      <th>AGGRENOX</th>\n",
       "      <th>ALENDRONATE.SODIUM</th>\n",
       "      <th>ALLOPURINOL</th>\n",
       "      <th>ALPRAZOLAM</th>\n",
       "      <th>AMIODARONE.HCL</th>\n",
       "      <th>...</th>\n",
       "      <th>State_TX</th>\n",
       "      <th>State_UT</th>\n",
       "      <th>State_VA</th>\n",
       "      <th>State_VI</th>\n",
       "      <th>State_VT</th>\n",
       "      <th>State_WA</th>\n",
       "      <th>State_WI</th>\n",
       "      <th>State_WV</th>\n",
       "      <th>State_WY</th>\n",
       "      <th>State_ZZ</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>134</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 355 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  ABILIFY  ACETAMINOPHEN.CODEINE  ACYCLOVIR  ADVAIR.DISKUS  \\\n",
       "0           0        0                      0          0              0   \n",
       "1           1        0                      0          0              0   \n",
       "\n",
       "   AGGRENOX  ALENDRONATE.SODIUM  ALLOPURINOL  ALPRAZOLAM  AMIODARONE.HCL  \\\n",
       "0         0                   0            0           0               0   \n",
       "1         0                   0            0         134               0   \n",
       "\n",
       "     ...     State_TX  State_UT  State_VA  State_VI  State_VT  State_WA  \\\n",
       "0    ...            1         0         0         0         0         0   \n",
       "1    ...            0         0         0         0         0         0   \n",
       "\n",
       "   State_WI  State_WV  State_WY  State_ZZ  \n",
       "0         0         0         0         0  \n",
       "1         0         0         0         0  \n",
       "\n",
       "[2 rows x 355 columns]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_df.head()[:2]  # the total deaths by state merged with the opiods by speciality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_df.drop(['Unnamed: 0'], axis =1, inplace =True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(24237, 354)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State_Name</th>\n",
       "      <th>State_Abbreviation</th>\n",
       "      <th>State_FIPS</th>\n",
       "      <th>2013_Part_ D_Prescribers</th>\n",
       "      <th>2013_Part_Opioid_Prescribers</th>\n",
       "      <th>2013_Opioid Claims</th>\n",
       "      <th>2013_Extended_Release_Opioid_Claims</th>\n",
       "      <th>2013_Overall_Claims</th>\n",
       "      <th>2013_Opioid_Prescribing_Rate</th>\n",
       "      <th>2013_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>...</th>\n",
       "      <th>2016_Part D Opioid Prescribers</th>\n",
       "      <th>2016_Opioid Claims</th>\n",
       "      <th>2016_Extended Release Opioid Claims</th>\n",
       "      <th>2016_Overall_Claims</th>\n",
       "      <th>2016_Opioid_Prescribing_Rate</th>\n",
       "      <th>2016_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2013_2016_Change in Opioid Prescribing Rate</th>\n",
       "      <th>2013_2016_Change in Extended Release Opioid Prescribing Rate</th>\n",
       "      <th>2013_2015_Change_in_Opioid_Prescribing_Rate</th>\n",
       "      <th>2013_2015_Change_in_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>National</td>\n",
       "      <td>National</td>\n",
       "      <td>National</td>\n",
       "      <td>1,037,770</td>\n",
       "      <td>491,500</td>\n",
       "      <td>78,045,683</td>\n",
       "      <td>4,773,286</td>\n",
       "      <td>1,342,096,654</td>\n",
       "      <td>5.82</td>\n",
       "      <td>6.12</td>\n",
       "      <td>...</td>\n",
       "      <td>496350</td>\n",
       "      <td>77716608</td>\n",
       "      <td>5381275</td>\n",
       "      <td>1452978228</td>\n",
       "      <td>5.34</td>\n",
       "      <td>6.92</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>0.80</td>\n",
       "      <td>-0.30</td>\n",
       "      <td>0.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>AL</td>\n",
       "      <td>1</td>\n",
       "      <td>12,820</td>\n",
       "      <td>7,422</td>\n",
       "      <td>2,260,284</td>\n",
       "      <td>118,523</td>\n",
       "      <td>29,160,952</td>\n",
       "      <td>7.75</td>\n",
       "      <td>5.24</td>\n",
       "      <td>...</td>\n",
       "      <td>7,004</td>\n",
       "      <td>2,156,504</td>\n",
       "      <td>120,579</td>\n",
       "      <td>28,882,148</td>\n",
       "      <td>7.47</td>\n",
       "      <td>5.59</td>\n",
       "      <td>-0.28</td>\n",
       "      <td>0.35</td>\n",
       "      <td>-0.21</td>\n",
       "      <td>0.31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  State_Name State_Abbreviation State_FIPS 2013_Part_ D_Prescribers  \\\n",
       "0   National           National   National                1,037,770   \n",
       "1    Alabama                 AL          1                   12,820   \n",
       "\n",
       "  2013_Part_Opioid_Prescribers 2013_Opioid Claims  \\\n",
       "0                      491,500         78,045,683   \n",
       "1                        7,422          2,260,284   \n",
       "\n",
       "  2013_Extended_Release_Opioid_Claims 2013_Overall_Claims  \\\n",
       "0                           4,773,286       1,342,096,654   \n",
       "1                             118,523          29,160,952   \n",
       "\n",
       "   2013_Opioid_Prescribing_Rate  \\\n",
       "0                          5.82   \n",
       "1                          7.75   \n",
       "\n",
       "   2013_Extended_Release_Opioid_Prescribing_Rate  \\\n",
       "0                                           6.12   \n",
       "1                                           5.24   \n",
       "\n",
       "                               ...                               \\\n",
       "0                              ...                                \n",
       "1                              ...                                \n",
       "\n",
       "  2016_Part D Opioid Prescribers 2016_Opioid Claims  \\\n",
       "0                         496350           77716608   \n",
       "1                          7,004          2,156,504   \n",
       "\n",
       "  2016_Extended Release Opioid Claims 2016_Overall_Claims  \\\n",
       "0                             5381275          1452978228   \n",
       "1                             120,579          28,882,148   \n",
       "\n",
       "  2016_Opioid_Prescribing_Rate  \\\n",
       "0                         5.34   \n",
       "1                         7.47   \n",
       "\n",
       "   2016_Extended_Release_Opioid_Prescribing_ Rate  \\\n",
       "0                                            6.92   \n",
       "1                                            5.59   \n",
       "\n",
       "   2013_2016_Change in Opioid Prescribing Rate  \\\n",
       "0                                        -0.47   \n",
       "1                                        -0.28   \n",
       "\n",
       "  2013_2016_Change in Extended Release Opioid Prescribing Rate   \\\n",
       "0                                               0.80              \n",
       "1                                               0.35              \n",
       "\n",
       "  2013_2015_Change_in_Opioid_Prescribing_Rate  \\\n",
       "0                                       -0.30   \n",
       "1                                       -0.21   \n",
       "\n",
       "  2013_2015_Change_in_Extended_Release_Opioid_Prescribing_Rate   \n",
       "0                                               0.59             \n",
       "1                                               0.31             \n",
       "\n",
       "[2 rows x 35 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#  extracted from from CMS Medicare Part D Opioid Prescribing Geographic 2013-2016 Excel file and \n",
    "# created CSV files for the three taps 1) state 2) county and 3) zip\n",
    "# Stored presciber-state file in a variable(df2)\n",
    "Prescriber_state_df.fillna('National')[:2]   # the data is about the prescriber/Physician_Locator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "State_Abbreviation                                               0.019231\n",
      "State_FIPS                                                       0.019231\n",
      "2013_2015_Change_in_Extended_Release_Opioid_Prescribing_Rate     0.000000\n",
      "2013_Extended_Release_Opioid_Prescribing_Rate                    0.000000\n",
      "2014_Overall_Claims                                              0.000000\n",
      "2014_Extended_Release_Opioid_Claims                              0.000000\n",
      "2014_Opioid_Claims                                               0.000000\n",
      "2014_Part D_Opioid_Prescribers                                   0.000000\n",
      "2014_Part_ D_Prescribers                                         0.000000\n",
      "2013_Overall_Claims                                              0.000000\n",
      "2013_Opioid_Prescribing_Rate                                     0.000000\n",
      "2014_Extended_Release_Opioid_Prescribing_Rate                    0.000000\n",
      "2013_Extended_Release_Opioid_Claims                              0.000000\n",
      "2013_Opioid Claims                                               0.000000\n",
      "2013_Part_Opioid_Prescribers                                     0.000000\n",
      "2013_Part_ D_Prescribers                                         0.000000\n",
      "2014_Opioid_Prescribing_Rate                                     0.000000\n",
      "2015_Part D Prescribers                                          0.000000\n",
      "2013_2015_Change_in_Opioid_Prescribing_Rate                      0.000000\n",
      "2016_Opioid Claims                                               0.000000\n",
      "2013_2016_Change in Extended Release Opioid Prescribing Rate     0.000000\n",
      "2013_2016_Change in Opioid Prescribing Rate                      0.000000\n",
      "2016_Extended_Release_Opioid_Prescribing_ Rate                   0.000000\n",
      "2016_Opioid_Prescribing_Rate                                     0.000000\n",
      "2016_Overall_Claims                                              0.000000\n",
      "2016_Extended Release Opioid Claims                              0.000000\n",
      "2016_Part D Opioid Prescribers                                   0.000000\n",
      "2015_Part D Opioid Prescribers                                   0.000000\n",
      "2016_Part D Prescribers                                          0.000000\n",
      "2015_Extended_Release_Opioid_Prescribing_ Rate                   0.000000\n",
      "2015_Opioid_Prescribing_Rate                                     0.000000\n",
      "2015_Overall_Claims                                              0.000000\n",
      "2015_Extended Release Opioid Claims                              0.000000\n",
      "2015_Opioid Claims                                               0.000000\n",
      "State_Name                                                       0.000000\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((Prescriber_state_df.isnull().sum()/Prescriber_state_df.shape[0]).sort_values(ascending=False) )  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State_Name</th>\n",
       "      <th>State_Abbreviation</th>\n",
       "      <th>State_FIPS</th>\n",
       "      <th>2013_Part_ D_Prescribers</th>\n",
       "      <th>2013_Part_Opioid_Prescribers</th>\n",
       "      <th>2013_Opioid Claims</th>\n",
       "      <th>2013_Extended_Release_Opioid_Claims</th>\n",
       "      <th>2013_Overall_Claims</th>\n",
       "      <th>2013_Opioid_Prescribing_Rate</th>\n",
       "      <th>2013_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>...</th>\n",
       "      <th>2016_Part D Opioid Prescribers</th>\n",
       "      <th>2016_Opioid Claims</th>\n",
       "      <th>2016_Extended Release Opioid Claims</th>\n",
       "      <th>2016_Overall_Claims</th>\n",
       "      <th>2016_Opioid_Prescribing_Rate</th>\n",
       "      <th>2016_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2013_2016_Change in Opioid Prescribing Rate</th>\n",
       "      <th>2013_2016_Change in Extended Release Opioid Prescribing Rate</th>\n",
       "      <th>2013_2015_Change_in_Opioid_Prescribing_Rate</th>\n",
       "      <th>2013_2015_Change_in_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>National</td>\n",
       "      <td>National</td>\n",
       "      <td>National</td>\n",
       "      <td>1,037,770</td>\n",
       "      <td>491,500</td>\n",
       "      <td>78,045,683</td>\n",
       "      <td>4,773,286</td>\n",
       "      <td>1,342,096,654</td>\n",
       "      <td>5.82</td>\n",
       "      <td>6.12</td>\n",
       "      <td>...</td>\n",
       "      <td>496350</td>\n",
       "      <td>77716608</td>\n",
       "      <td>5381275</td>\n",
       "      <td>1452978228</td>\n",
       "      <td>5.34</td>\n",
       "      <td>6.92</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>0.80</td>\n",
       "      <td>-0.30</td>\n",
       "      <td>0.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>AL</td>\n",
       "      <td>1</td>\n",
       "      <td>12,820</td>\n",
       "      <td>7,422</td>\n",
       "      <td>2,260,284</td>\n",
       "      <td>118,523</td>\n",
       "      <td>29,160,952</td>\n",
       "      <td>7.75</td>\n",
       "      <td>5.24</td>\n",
       "      <td>...</td>\n",
       "      <td>7,004</td>\n",
       "      <td>2,156,504</td>\n",
       "      <td>120,579</td>\n",
       "      <td>28,882,148</td>\n",
       "      <td>7.47</td>\n",
       "      <td>5.59</td>\n",
       "      <td>-0.28</td>\n",
       "      <td>0.35</td>\n",
       "      <td>-0.21</td>\n",
       "      <td>0.31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Alaska</td>\n",
       "      <td>AK</td>\n",
       "      <td>2</td>\n",
       "      <td>2,275</td>\n",
       "      <td>1,099</td>\n",
       "      <td>86,517</td>\n",
       "      <td>8,602</td>\n",
       "      <td>1,281,057</td>\n",
       "      <td>6.75</td>\n",
       "      <td>9.94</td>\n",
       "      <td>...</td>\n",
       "      <td>1,043</td>\n",
       "      <td>82,557</td>\n",
       "      <td>9,074</td>\n",
       "      <td>1,225,327</td>\n",
       "      <td>6.74</td>\n",
       "      <td>10.99</td>\n",
       "      <td>-0.01</td>\n",
       "      <td>1.05</td>\n",
       "      <td>-0.09</td>\n",
       "      <td>0.71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Arizona</td>\n",
       "      <td>AZ</td>\n",
       "      <td>4</td>\n",
       "      <td>20,542</td>\n",
       "      <td>10,510</td>\n",
       "      <td>1,545,138</td>\n",
       "      <td>151,086</td>\n",
       "      <td>22,126,421</td>\n",
       "      <td>6.98</td>\n",
       "      <td>9.78</td>\n",
       "      <td>...</td>\n",
       "      <td>11,026</td>\n",
       "      <td>1,621,223</td>\n",
       "      <td>173,171</td>\n",
       "      <td>24,891,001</td>\n",
       "      <td>6.51</td>\n",
       "      <td>10.68</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>0.90</td>\n",
       "      <td>-0.20</td>\n",
       "      <td>0.48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Arkansas</td>\n",
       "      <td>AR</td>\n",
       "      <td>5</td>\n",
       "      <td>7,909</td>\n",
       "      <td>4,700</td>\n",
       "      <td>1,128,356</td>\n",
       "      <td>57,519</td>\n",
       "      <td>16,759,116</td>\n",
       "      <td>6.73</td>\n",
       "      <td>5.10</td>\n",
       "      <td>...</td>\n",
       "      <td>4,854</td>\n",
       "      <td>1,155,171</td>\n",
       "      <td>65,819</td>\n",
       "      <td>18,139,502</td>\n",
       "      <td>6.37</td>\n",
       "      <td>5.70</td>\n",
       "      <td>-0.36</td>\n",
       "      <td>0.60</td>\n",
       "      <td>-0.36</td>\n",
       "      <td>0.49</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  State_Name State_Abbreviation State_FIPS 2013_Part_ D_Prescribers  \\\n",
       "0   National           National   National                1,037,770   \n",
       "1    Alabama                 AL          1                   12,820   \n",
       "2     Alaska                 AK          2                    2,275   \n",
       "3    Arizona                 AZ          4                   20,542   \n",
       "4   Arkansas                 AR          5                    7,909   \n",
       "\n",
       "  2013_Part_Opioid_Prescribers 2013_Opioid Claims  \\\n",
       "0                      491,500         78,045,683   \n",
       "1                        7,422          2,260,284   \n",
       "2                        1,099             86,517   \n",
       "3                       10,510          1,545,138   \n",
       "4                        4,700          1,128,356   \n",
       "\n",
       "  2013_Extended_Release_Opioid_Claims 2013_Overall_Claims  \\\n",
       "0                           4,773,286       1,342,096,654   \n",
       "1                             118,523          29,160,952   \n",
       "2                               8,602           1,281,057   \n",
       "3                             151,086          22,126,421   \n",
       "4                              57,519          16,759,116   \n",
       "\n",
       "   2013_Opioid_Prescribing_Rate  \\\n",
       "0                          5.82   \n",
       "1                          7.75   \n",
       "2                          6.75   \n",
       "3                          6.98   \n",
       "4                          6.73   \n",
       "\n",
       "   2013_Extended_Release_Opioid_Prescribing_Rate  \\\n",
       "0                                           6.12   \n",
       "1                                           5.24   \n",
       "2                                           9.94   \n",
       "3                                           9.78   \n",
       "4                                           5.10   \n",
       "\n",
       "                               ...                               \\\n",
       "0                              ...                                \n",
       "1                              ...                                \n",
       "2                              ...                                \n",
       "3                              ...                                \n",
       "4                              ...                                \n",
       "\n",
       "  2016_Part D Opioid Prescribers 2016_Opioid Claims  \\\n",
       "0                         496350           77716608   \n",
       "1                          7,004          2,156,504   \n",
       "2                          1,043             82,557   \n",
       "3                         11,026          1,621,223   \n",
       "4                          4,854          1,155,171   \n",
       "\n",
       "  2016_Extended Release Opioid Claims 2016_Overall_Claims  \\\n",
       "0                             5381275          1452978228   \n",
       "1                             120,579          28,882,148   \n",
       "2                               9,074           1,225,327   \n",
       "3                             173,171          24,891,001   \n",
       "4                              65,819          18,139,502   \n",
       "\n",
       "  2016_Opioid_Prescribing_Rate  \\\n",
       "0                         5.34   \n",
       "1                         7.47   \n",
       "2                         6.74   \n",
       "3                         6.51   \n",
       "4                         6.37   \n",
       "\n",
       "   2016_Extended_Release_Opioid_Prescribing_ Rate  \\\n",
       "0                                            6.92   \n",
       "1                                            5.59   \n",
       "2                                           10.99   \n",
       "3                                           10.68   \n",
       "4                                            5.70   \n",
       "\n",
       "   2013_2016_Change in Opioid Prescribing Rate  \\\n",
       "0                                        -0.47   \n",
       "1                                        -0.28   \n",
       "2                                        -0.01   \n",
       "3                                        -0.47   \n",
       "4                                        -0.36   \n",
       "\n",
       "  2013_2016_Change in Extended Release Opioid Prescribing Rate   \\\n",
       "0                                               0.80              \n",
       "1                                               0.35              \n",
       "2                                               1.05              \n",
       "3                                               0.90              \n",
       "4                                               0.60              \n",
       "\n",
       "  2013_2015_Change_in_Opioid_Prescribing_Rate  \\\n",
       "0                                       -0.30   \n",
       "1                                       -0.21   \n",
       "2                                       -0.09   \n",
       "3                                       -0.20   \n",
       "4                                       -0.36   \n",
       "\n",
       "  2013_2015_Change_in_Extended_Release_Opioid_Prescribing_Rate   \n",
       "0                                               0.59             \n",
       "1                                               0.31             \n",
       "2                                               0.71             \n",
       "3                                               0.48             \n",
       "4                                               0.49             \n",
       "\n",
       "[5 rows x 35 columns]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# here just filling the NaN value due to the columns mismatach as there is no need fo data on \n",
    "# state abrevation and state state FIPs\n",
    "Prescriber_state_df.fillna(\"National\")[:5] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['State_Name',\n",
       " 'State_Abbreviation',\n",
       " 'State_FIPS',\n",
       " '2013_Part_ D_Prescribers',\n",
       " '2013_Part_Opioid_Prescribers',\n",
       " '2013_Opioid Claims',\n",
       " '2013_Extended_Release_Opioid_Claims',\n",
       " '2013_Overall_Claims',\n",
       " '2013_Opioid_Prescribing_Rate',\n",
       " '2013_Extended_Release_Opioid_Prescribing_Rate',\n",
       " '2014_Part_ D_Prescribers',\n",
       " '2014_Part D_Opioid_Prescribers',\n",
       " '2014_Opioid_Claims',\n",
       " '2014_Extended_Release_Opioid_Claims',\n",
       " '2014_Overall_Claims',\n",
       " '2014_Opioid_Prescribing_Rate',\n",
       " '2014_Extended_Release_Opioid_Prescribing_Rate',\n",
       " '2015_Part D Prescribers',\n",
       " '2015_Part D Opioid Prescribers',\n",
       " '2015_Opioid Claims',\n",
       " '2015_Extended Release Opioid Claims',\n",
       " '2015_Overall_Claims',\n",
       " '2015_Opioid_Prescribing_Rate',\n",
       " '2015_Extended_Release_Opioid_Prescribing_ Rate',\n",
       " '2016_Part D Prescribers',\n",
       " '2016_Part D Opioid Prescribers',\n",
       " '2016_Opioid Claims',\n",
       " '2016_Extended Release Opioid Claims',\n",
       " '2016_Overall_Claims',\n",
       " '2016_Opioid_Prescribing_Rate',\n",
       " '2016_Extended_Release_Opioid_Prescribing_ Rate',\n",
       " '2013_2016_Change in Opioid Prescribing Rate',\n",
       " '2013_2016_Change in Extended Release Opioid Prescribing Rate ',\n",
       " '2013_2015_Change_in_Opioid_Prescribing_Rate',\n",
       " '2013_2015_Change_in_Extended_Release_Opioid_Prescribing_Rate ']"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(Prescriber_state_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State_Name</th>\n",
       "      <th>State_Abbreviation</th>\n",
       "      <th>2013_Opioid_Prescribing_Rate</th>\n",
       "      <th>2013_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>2014_Opioid_Prescribing_Rate</th>\n",
       "      <th>2014_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2016_Opioid_Prescribing_Rate</th>\n",
       "      <th>2016_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2013_2016_Change in Opioid Prescribing Rate</th>\n",
       "      <th>2013_2016_Change in Extended Release Opioid Prescribing Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>National</td>\n",
       "      <td>National</td>\n",
       "      <td>5.82</td>\n",
       "      <td>6.12</td>\n",
       "      <td>5.74</td>\n",
       "      <td>6.27</td>\n",
       "      <td>5.52</td>\n",
       "      <td>6.71</td>\n",
       "      <td>5.34</td>\n",
       "      <td>6.92</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>0.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>AL</td>\n",
       "      <td>7.75</td>\n",
       "      <td>5.24</td>\n",
       "      <td>7.86</td>\n",
       "      <td>5.24</td>\n",
       "      <td>7.54</td>\n",
       "      <td>5.55</td>\n",
       "      <td>7.47</td>\n",
       "      <td>5.59</td>\n",
       "      <td>-0.28</td>\n",
       "      <td>0.35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  State_Name State_Abbreviation  2013_Opioid_Prescribing_Rate  \\\n",
       "0   National           National                          5.82   \n",
       "1    Alabama                 AL                          7.75   \n",
       "\n",
       "   2013_Extended_Release_Opioid_Prescribing_Rate  \\\n",
       "0                                           6.12   \n",
       "1                                           5.24   \n",
       "\n",
       "   2014_Opioid_Prescribing_Rate  \\\n",
       "0                          5.74   \n",
       "1                          7.86   \n",
       "\n",
       "   2014_Extended_Release_Opioid_Prescribing_Rate  \\\n",
       "0                                           6.27   \n",
       "1                                           5.24   \n",
       "\n",
       "   2015_Opioid_Prescribing_Rate  \\\n",
       "0                          5.52   \n",
       "1                          7.54   \n",
       "\n",
       "   2015_Extended_Release_Opioid_Prescribing_ Rate  \\\n",
       "0                                            6.71   \n",
       "1                                            5.55   \n",
       "\n",
       "   2016_Opioid_Prescribing_Rate  \\\n",
       "0                          5.34   \n",
       "1                          7.47   \n",
       "\n",
       "   2016_Extended_Release_Opioid_Prescribing_ Rate  \\\n",
       "0                                            6.92   \n",
       "1                                            5.59   \n",
       "\n",
       "   2013_2016_Change in Opioid Prescribing Rate  \\\n",
       "0                                        -0.47   \n",
       "1                                        -0.28   \n",
       "\n",
       "   2013_2016_Change in Extended Release Opioid Prescribing Rate   \n",
       "0                                               0.80              \n",
       "1                                               0.35              "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Selected the opioid prescribe rate columsn for years 2013-2016 by prescriber state(psr)\n",
    "# this was extracted from df2 so that it will be important to see for features in the next stages\n",
    "psr_df.head()[:2]\n",
    "psr_df.fillna(\"National\")[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "State_Abbreviation                                               0.019231\n",
      "2013_2016_Change in Extended Release Opioid Prescribing Rate     0.000000\n",
      "2013_2016_Change in Opioid Prescribing Rate                      0.000000\n",
      "2016_Extended_Release_Opioid_Prescribing_ Rate                   0.000000\n",
      "2016_Opioid_Prescribing_Rate                                     0.000000\n",
      "2015_Extended_Release_Opioid_Prescribing_ Rate                   0.000000\n",
      "2015_Opioid_Prescribing_Rate                                     0.000000\n",
      "2014_Extended_Release_Opioid_Prescribing_Rate                    0.000000\n",
      "2014_Opioid_Prescribing_Rate                                     0.000000\n",
      "2013_Extended_Release_Opioid_Prescribing_Rate                    0.000000\n",
      "2013_Opioid_Prescribing_Rate                                     0.000000\n",
      "State_Name                                                       0.000000\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((psr_df.isnull().sum()/psr_df.shape[0]).sort_values(ascending=False))  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State_Name</th>\n",
       "      <th>State_Abbreviation</th>\n",
       "      <th>2013_Opioid_Prescribing_Rate</th>\n",
       "      <th>2013_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>2014_Opioid_Prescribing_Rate</th>\n",
       "      <th>2014_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2016_Opioid_Prescribing_Rate</th>\n",
       "      <th>2016_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2013_2016_Change in Opioid Prescribing Rate</th>\n",
       "      <th>2013_2016_Change in Extended Release Opioid Prescribing Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>National</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.82</td>\n",
       "      <td>6.12</td>\n",
       "      <td>5.74</td>\n",
       "      <td>6.27</td>\n",
       "      <td>5.52</td>\n",
       "      <td>6.71</td>\n",
       "      <td>5.34</td>\n",
       "      <td>6.92</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>0.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>AL</td>\n",
       "      <td>7.75</td>\n",
       "      <td>5.24</td>\n",
       "      <td>7.86</td>\n",
       "      <td>5.24</td>\n",
       "      <td>7.54</td>\n",
       "      <td>5.55</td>\n",
       "      <td>7.47</td>\n",
       "      <td>5.59</td>\n",
       "      <td>-0.28</td>\n",
       "      <td>0.35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  State_Name State_Abbreviation  2013_Opioid_Prescribing_Rate  \\\n",
       "0   National                NaN                          5.82   \n",
       "1    Alabama                 AL                          7.75   \n",
       "\n",
       "   2013_Extended_Release_Opioid_Prescribing_Rate  \\\n",
       "0                                           6.12   \n",
       "1                                           5.24   \n",
       "\n",
       "   2014_Opioid_Prescribing_Rate  \\\n",
       "0                          5.74   \n",
       "1                          7.86   \n",
       "\n",
       "   2014_Extended_Release_Opioid_Prescribing_Rate  \\\n",
       "0                                           6.27   \n",
       "1                                           5.24   \n",
       "\n",
       "   2015_Opioid_Prescribing_Rate  \\\n",
       "0                          5.52   \n",
       "1                          7.54   \n",
       "\n",
       "   2015_Extended_Release_Opioid_Prescribing_ Rate  \\\n",
       "0                                            6.71   \n",
       "1                                            5.55   \n",
       "\n",
       "   2016_Opioid_Prescribing_Rate  \\\n",
       "0                          5.34   \n",
       "1                          7.47   \n",
       "\n",
       "   2016_Extended_Release_Opioid_Prescribing_ Rate  \\\n",
       "0                                            6.92   \n",
       "1                                            5.59   \n",
       "\n",
       "   2013_2016_Change in Opioid Prescribing Rate  \\\n",
       "0                                        -0.47   \n",
       "1                                        -0.28   \n",
       "\n",
       "   2013_2016_Change in Extended Release Opioid Prescribing Rate   \n",
       "0                                               0.80              \n",
       "1                                               0.35              "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "psr_df.fillna(\"National\")\n",
    "psr_df.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State_Name</th>\n",
       "      <th>2013_2016_Change in Opioid Prescribing Rate</th>\n",
       "      <th>2013_2016_Change in Extended Release Opioid Prescribing Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>National</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>0.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>-0.28</td>\n",
       "      <td>0.35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  State_Name  2013_2016_Change in Opioid Prescribing Rate  \\\n",
       "0   National                                        -0.47   \n",
       "1    Alabama                                        -0.28   \n",
       "\n",
       "   2013_2016_Change in Extended Release Opioid Prescribing Rate   \n",
       "0                                               0.80              \n",
       "1                                               0.35              "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "psr_changeex_df = psr_df[['State_Name','2013_2016_Change in Opioid Prescribing Rate', '2013_2016_Change in Extended Release Opioid Prescribing Rate ']]\n",
    "psr_changeex_df .head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State_Name</th>\n",
       "      <th>State_Abbreviation</th>\n",
       "      <th>2013_Opioid_Prescribing_Rate</th>\n",
       "      <th>2013_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>2014_Opioid_Prescribing_Rate</th>\n",
       "      <th>2014_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2016_Opioid_Prescribing_Rate</th>\n",
       "      <th>2016_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2013_2016_Change in Opioid Prescribing Rate</th>\n",
       "      <th>2013_2016_Change in Extended Release Opioid Prescribing Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>National</td>\n",
       "      <td>National</td>\n",
       "      <td>5.82</td>\n",
       "      <td>6.12</td>\n",
       "      <td>5.74</td>\n",
       "      <td>6.27</td>\n",
       "      <td>5.52</td>\n",
       "      <td>6.71</td>\n",
       "      <td>5.34</td>\n",
       "      <td>6.92</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>0.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>AL</td>\n",
       "      <td>7.75</td>\n",
       "      <td>5.24</td>\n",
       "      <td>7.86</td>\n",
       "      <td>5.24</td>\n",
       "      <td>7.54</td>\n",
       "      <td>5.55</td>\n",
       "      <td>7.47</td>\n",
       "      <td>5.59</td>\n",
       "      <td>-0.28</td>\n",
       "      <td>0.35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  State_Name State_Abbreviation  2013_Opioid_Prescribing_Rate  \\\n",
       "0   National           National                          5.82   \n",
       "1    Alabama                 AL                          7.75   \n",
       "\n",
       "   2013_Extended_Release_Opioid_Prescribing_Rate  \\\n",
       "0                                           6.12   \n",
       "1                                           5.24   \n",
       "\n",
       "   2014_Opioid_Prescribing_Rate  \\\n",
       "0                          5.74   \n",
       "1                          7.86   \n",
       "\n",
       "   2014_Extended_Release_Opioid_Prescribing_Rate  \\\n",
       "0                                           6.27   \n",
       "1                                           5.24   \n",
       "\n",
       "   2015_Opioid_Prescribing_Rate  \\\n",
       "0                          5.52   \n",
       "1                          7.54   \n",
       "\n",
       "   2015_Extended_Release_Opioid_Prescribing_ Rate  \\\n",
       "0                                            6.71   \n",
       "1                                            5.55   \n",
       "\n",
       "   2016_Opioid_Prescribing_Rate  \\\n",
       "0                          5.34   \n",
       "1                          7.47   \n",
       "\n",
       "   2016_Extended_Release_Opioid_Prescribing_ Rate  \\\n",
       "0                                            6.92   \n",
       "1                                            5.59   \n",
       "\n",
       "   2013_2016_Change in Opioid Prescribing Rate  \\\n",
       "0                                        -0.47   \n",
       "1                                        -0.28   \n",
       "\n",
       "   2013_2016_Change in Extended Release Opioid Prescribing Rate   \n",
       "0                                               0.80              \n",
       "1                                               0.35              "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Selected the opioid prescribe rate columsn for years 2013-2016 by prescriber county\n",
    "# pcr_df.head()[:2]\n",
    "psr_df.fillna(\"National\")[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "2013_2016_Change in Extended Release Opioid Prescribing Rate     0.005509\n",
      "2013_2016_Change in Opioid Prescribing Rate                      0.005509\n",
      "2015_Extended_Release_Opioid_Prescribing_ Rate                   0.002592\n",
      "2015_Opioid_Prescribing_Rate                                     0.002592\n",
      "2016_Extended_Release_Opioid_Prescribing_ Rate                   0.002268\n",
      "2016_Opioid_Prescribing_Rate                                     0.002268\n",
      "2014_Extended_Release_Opioid_Prescribing_Rate                    0.001944\n",
      "2014_Opioid_Prescribing_Rate                                     0.001944\n",
      "2013_Extended_Release_Opioid_Prescribing_Rate                    0.001944\n",
      "2013_Opioid_Prescribing_Rate                                     0.001944\n",
      "State_FIPS                                                       0.000324\n",
      "County Name                                                      0.000324\n",
      "State_Abbreviation                                               0.000324\n",
      "State_Name                                                       0.000000\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((pcr_df.isnull().sum()/pcr_df.shape[0]).sort_values(ascending=False)  )  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State_Name</th>\n",
       "      <th>State_Abbreviation</th>\n",
       "      <th>County Name</th>\n",
       "      <th>State_FIPS</th>\n",
       "      <th>2013_Opioid_Prescribing_Rate</th>\n",
       "      <th>2013_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>2014_Opioid_Prescribing_Rate</th>\n",
       "      <th>2014_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2016_Opioid_Prescribing_Rate</th>\n",
       "      <th>2016_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2013_2016_Change in Opioid Prescribing Rate</th>\n",
       "      <th>2013_2016_Change in Extended Release Opioid Prescribing Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>National</td>\n",
       "      <td>National</td>\n",
       "      <td>National</td>\n",
       "      <td>National</td>\n",
       "      <td>5.82</td>\n",
       "      <td>6.12</td>\n",
       "      <td>5.74</td>\n",
       "      <td>6.27</td>\n",
       "      <td>5.52</td>\n",
       "      <td>6.71</td>\n",
       "      <td>5.34</td>\n",
       "      <td>6.92</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>0.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>AL</td>\n",
       "      <td>Autauga</td>\n",
       "      <td>1001</td>\n",
       "      <td>9.8</td>\n",
       "      <td>4.78</td>\n",
       "      <td>9.72</td>\n",
       "      <td>5.85</td>\n",
       "      <td>8.07</td>\n",
       "      <td>7.24</td>\n",
       "      <td>11.47</td>\n",
       "      <td>9.39</td>\n",
       "      <td>1.67</td>\n",
       "      <td>4.61</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  State_Name State_Abbreviation County Name State_FIPS  \\\n",
       "0   National           National    National   National   \n",
       "1    Alabama                 AL     Autauga       1001   \n",
       "\n",
       "  2013_Opioid_Prescribing_Rate 2013_Extended_Release_Opioid_Prescribing_Rate  \\\n",
       "0                         5.82                                          6.12   \n",
       "1                          9.8                                          4.78   \n",
       "\n",
       "  2014_Opioid_Prescribing_Rate 2014_Extended_Release_Opioid_Prescribing_Rate  \\\n",
       "0                         5.74                                          6.27   \n",
       "1                         9.72                                          5.85   \n",
       "\n",
       "  2015_Opioid_Prescribing_Rate 2015_Extended_Release_Opioid_Prescribing_ Rate  \\\n",
       "0                         5.52                                           6.71   \n",
       "1                         8.07                                           7.24   \n",
       "\n",
       "  2016_Opioid_Prescribing_Rate 2016_Extended_Release_Opioid_Prescribing_ Rate  \\\n",
       "0                         5.34                                           6.92   \n",
       "1                        11.47                                           9.39   \n",
       "\n",
       "  2013_2016_Change in Opioid Prescribing Rate  \\\n",
       "0                                       -0.47   \n",
       "1                                        1.67   \n",
       "\n",
       "  2013_2016_Change in Extended Release Opioid Prescribing Rate   \n",
       "0                                                0.8             \n",
       "1                                               4.61             "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pcr_df.fillna(\"National\")[:2]\n",
    "# pcr_df.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['State_Name',\n",
       " 'State_Abbreviation',\n",
       " 'County Name',\n",
       " 'State_FIPS',\n",
       " '2013_Opioid_Prescribing_Rate',\n",
       " '2013_Extended_Release_Opioid_Prescribing_Rate',\n",
       " '2014_Opioid_Prescribing_Rate',\n",
       " '2014_Extended_Release_Opioid_Prescribing_Rate',\n",
       " '2015_Opioid_Prescribing_Rate',\n",
       " '2015_Extended_Release_Opioid_Prescribing_ Rate',\n",
       " '2016_Opioid_Prescribing_Rate',\n",
       " '2016_Extended_Release_Opioid_Prescribing_ Rate',\n",
       " '2013_2016_Change in Opioid Prescribing Rate',\n",
       " '2013_2016_Change in Extended Release Opioid Prescribing Rate ']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(pcr_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State_Name</th>\n",
       "      <th>2013_2016_Change in Opioid Prescribing Rate</th>\n",
       "      <th>2013_2016_Change in Extended Release Opioid Prescribing Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>National</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>0.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>1.67</td>\n",
       "      <td>4.61</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  State_Name  2013_2016_Change in Opioid Prescribing Rate  \\\n",
       "0   National                                        -0.47   \n",
       "1    Alabama                                         1.67   \n",
       "\n",
       "   2013_2016_Change in Extended Release Opioid Prescribing Rate   \n",
       "0                                               0.80              \n",
       "1                                               4.61              "
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pcr_changeex_df = pcr_df[['State_Name','2013_2016_Change in Opioid Prescribing Rate', '2013_2016_Change in Extended Release Opioid Prescribing Rate ']]\n",
    "pcr_changeex_df .head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "2013_2015_Change_in_Extended_Release_Opioid_Prescribing_Rate     0.004537\n",
      "2013_2015_Change_in_Opioid_Prescribing_Rate                      0.004537\n",
      "2015_Extended_Release_Opioid_Prescribing_ Rate                   0.002592\n",
      "2015_Opioid_Prescribing_Rate                                     0.002592\n",
      "2015_Overall_Claims                                              0.002592\n",
      "2015_Extended Release Opioid Claims                              0.002592\n",
      "2015_Opioid Claims                                               0.002592\n",
      "2015_Part D Opioid Prescribers                                   0.002592\n",
      "2015_Part D Prescribers                                          0.002592\n",
      "2013_Extended_Release_Opioid_Prescribing_Rate                    0.001944\n",
      "2013_Part_ D_Prescribers                                         0.001944\n",
      "2013_Part_Opioid_Prescribers                                     0.001944\n",
      "2013_Opioid Claims                                               0.001944\n",
      "2013_Extended_Release_Opioid_Claims                              0.001944\n",
      "2013_Overall_Claims                                              0.001944\n",
      "2013_Opioid_Prescribing_Rate                                     0.001944\n",
      "2014_Opioid_Claims                                               0.001944\n",
      "2014_Part_ D_Prescribers                                         0.001944\n",
      "2014_Part D_Opioid_Prescribers                                   0.001944\n",
      "2014_Extended_Release_Opioid_Claims                              0.001944\n",
      "2014_Overall_Claims                                              0.001944\n",
      "2014_Opioid_Prescribing_Rate                                     0.001944\n",
      "2014_Extended_Release_Opioid_Prescribing_Rate                    0.001944\n",
      "State_FIPS                                                       0.000324\n",
      "County Name                                                      0.000324\n",
      "State_Abbreviation                                               0.000324\n",
      "State_Name                                                       0.000000\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((Prescriber_county_df.isnull().sum()/Prescriber_county_df.shape[0]).sort_values(ascending=False)   )  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['State_Name',\n",
       " 'State_Abbreviation',\n",
       " 'County Name',\n",
       " 'State_FIPS',\n",
       " '2013_Part_ D_Prescribers',\n",
       " '2013_Part_Opioid_Prescribers',\n",
       " '2013_Opioid Claims',\n",
       " '2013_Extended_Release_Opioid_Claims',\n",
       " '2013_Overall_Claims',\n",
       " '2013_Opioid_Prescribing_Rate',\n",
       " '2013_Extended_Release_Opioid_Prescribing_Rate',\n",
       " '2014_Part_ D_Prescribers',\n",
       " '2014_Part D_Opioid_Prescribers',\n",
       " '2014_Opioid_Claims',\n",
       " '2014_Extended_Release_Opioid_Claims',\n",
       " '2014_Overall_Claims',\n",
       " '2014_Opioid_Prescribing_Rate',\n",
       " '2014_Extended_Release_Opioid_Prescribing_Rate',\n",
       " '2015_Part D Prescribers',\n",
       " '2015_Part D Opioid Prescribers',\n",
       " '2015_Opioid Claims',\n",
       " '2015_Extended Release Opioid Claims',\n",
       " '2015_Overall_Claims',\n",
       " '2015_Opioid_Prescribing_Rate',\n",
       " '2015_Extended_Release_Opioid_Prescribing_ Rate',\n",
       " '2013_2015_Change_in_Opioid_Prescribing_Rate',\n",
       " '2013_2015_Change_in_Extended_Release_Opioid_Prescribing_Rate ']"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(Prescriber_county_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State_Name</th>\n",
       "      <th>State_Abbreviation</th>\n",
       "      <th>County Name</th>\n",
       "      <th>State_FIPS</th>\n",
       "      <th>2013_Part_ D_Prescribers</th>\n",
       "      <th>2013_Part_Opioid_Prescribers</th>\n",
       "      <th>2013_Opioid Claims</th>\n",
       "      <th>2013_Extended_Release_Opioid_Claims</th>\n",
       "      <th>2013_Overall_Claims</th>\n",
       "      <th>2013_Opioid_Prescribing_Rate</th>\n",
       "      <th>...</th>\n",
       "      <th>2014_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Part D Prescribers</th>\n",
       "      <th>2015_Part D Opioid Prescribers</th>\n",
       "      <th>2015_Opioid Claims</th>\n",
       "      <th>2015_Extended Release Opioid Claims</th>\n",
       "      <th>2015_Overall_Claims</th>\n",
       "      <th>2015_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2013_2015_Change_in_Opioid_Prescribing_Rate</th>\n",
       "      <th>2013_2015_Change_in_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>National</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1,037,770</td>\n",
       "      <td>491,500</td>\n",
       "      <td>78,045,683</td>\n",
       "      <td>4,773,286</td>\n",
       "      <td>1,342,096,654</td>\n",
       "      <td>5.82</td>\n",
       "      <td>...</td>\n",
       "      <td>6.27</td>\n",
       "      <td>1,090,979</td>\n",
       "      <td>496,383</td>\n",
       "      <td>78,372,855</td>\n",
       "      <td>5,257,481</td>\n",
       "      <td>1,418,618,875</td>\n",
       "      <td>5.52</td>\n",
       "      <td>6.71</td>\n",
       "      <td>-0.30</td>\n",
       "      <td>0.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>AL</td>\n",
       "      <td>Autauga</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>67</td>\n",
       "      <td>44</td>\n",
       "      <td>20,482</td>\n",
       "      <td>979</td>\n",
       "      <td>208,936</td>\n",
       "      <td>9.80</td>\n",
       "      <td>...</td>\n",
       "      <td>5.85</td>\n",
       "      <td>68</td>\n",
       "      <td>43</td>\n",
       "      <td>15,998</td>\n",
       "      <td>1,159</td>\n",
       "      <td>198,347</td>\n",
       "      <td>8.07</td>\n",
       "      <td>7.24</td>\n",
       "      <td>-1.73</td>\n",
       "      <td>2.46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  State_Name State_Abbreviation County Name  State_FIPS  \\\n",
       "0   National                NaN         NaN         NaN   \n",
       "1    Alabama                 AL     Autauga      1001.0   \n",
       "\n",
       "  2013_Part_ D_Prescribers 2013_Part_Opioid_Prescribers 2013_Opioid Claims  \\\n",
       "0                1,037,770                      491,500         78,045,683   \n",
       "1                       67                           44             20,482   \n",
       "\n",
       "  2013_Extended_Release_Opioid_Claims 2013_Overall_Claims  \\\n",
       "0                           4,773,286       1,342,096,654   \n",
       "1                                 979             208,936   \n",
       "\n",
       "   2013_Opioid_Prescribing_Rate  \\\n",
       "0                          5.82   \n",
       "1                          9.80   \n",
       "\n",
       "                               ...                               \\\n",
       "0                              ...                                \n",
       "1                              ...                                \n",
       "\n",
       "   2014_Extended_Release_Opioid_Prescribing_Rate 2015_Part D Prescribers  \\\n",
       "0                                           6.27               1,090,979   \n",
       "1                                           5.85                      68   \n",
       "\n",
       "  2015_Part D Opioid Prescribers 2015_Opioid Claims  \\\n",
       "0                        496,383         78,372,855   \n",
       "1                             43             15,998   \n",
       "\n",
       "  2015_Extended Release Opioid Claims 2015_Overall_Claims  \\\n",
       "0                           5,257,481       1,418,618,875   \n",
       "1                               1,159             198,347   \n",
       "\n",
       "   2015_Opioid_Prescribing_Rate  \\\n",
       "0                          5.52   \n",
       "1                          8.07   \n",
       "\n",
       "   2015_Extended_Release_Opioid_Prescribing_ Rate  \\\n",
       "0                                            6.71   \n",
       "1                                            7.24   \n",
       "\n",
       "  2013_2015_Change_in_Opioid_Prescribing_Rate  \\\n",
       "0                                       -0.30   \n",
       "1                                       -1.73   \n",
       "\n",
       "  2013_2015_Change_in_Extended_Release_Opioid_Prescribing_Rate   \n",
       "0                                               0.59             \n",
       "1                                               2.46             \n",
       "\n",
       "[2 rows x 27 columns]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Prescriber_county_df.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "2013_2015_Change_in_Extended_Release_Opioid_Prescribing_Rate     0.004537\n",
      "2013_2015_Change_in_Opioid_Prescribing_Rate                      0.004537\n",
      "2015_Extended_Release_Opioid_Prescribing_ Rate                   0.002592\n",
      "2015_Opioid_Prescribing_Rate                                     0.002592\n",
      "2015_Overall_Claims                                              0.002592\n",
      "2015_Extended Release Opioid Claims                              0.002592\n",
      "2015_Opioid Claims                                               0.002592\n",
      "2015_Part D Opioid Prescribers                                   0.002592\n",
      "2015_Part D Prescribers                                          0.002592\n",
      "2013_Extended_Release_Opioid_Prescribing_Rate                    0.001944\n",
      "2013_Part_ D_Prescribers                                         0.001944\n",
      "2013_Part_Opioid_Prescribers                                     0.001944\n",
      "2013_Opioid Claims                                               0.001944\n",
      "2013_Extended_Release_Opioid_Claims                              0.001944\n",
      "2013_Overall_Claims                                              0.001944\n",
      "2013_Opioid_Prescribing_Rate                                     0.001944\n",
      "2014_Opioid_Claims                                               0.001944\n",
      "2014_Part_ D_Prescribers                                         0.001944\n",
      "2014_Part D_Opioid_Prescribers                                   0.001944\n",
      "2014_Extended_Release_Opioid_Claims                              0.001944\n",
      "2014_Overall_Claims                                              0.001944\n",
      "2014_Opioid_Prescribing_Rate                                     0.001944\n",
      "2014_Extended_Release_Opioid_Prescribing_Rate                    0.001944\n",
      "State_FIPS                                                       0.000324\n",
      "County Name                                                      0.000324\n",
      "State_Abbreviation                                               0.000324\n",
      "State_Name                                                       0.000000\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((Prescriber_county_df.isnull().sum()/Prescriber_county_df.shape[0]).sort_values(ascending=False))  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['State_Name',\n",
       " 'State_Abbreviation',\n",
       " 'County Name',\n",
       " 'State_FIPS',\n",
       " '2013_Part_ D_Prescribers',\n",
       " '2013_Part_Opioid_Prescribers',\n",
       " '2013_Opioid Claims',\n",
       " '2013_Extended_Release_Opioid_Claims',\n",
       " '2013_Overall_Claims',\n",
       " '2013_Opioid_Prescribing_Rate',\n",
       " '2013_Extended_Release_Opioid_Prescribing_Rate',\n",
       " '2014_Part_ D_Prescribers',\n",
       " '2014_Part D_Opioid_Prescribers',\n",
       " '2014_Opioid_Claims',\n",
       " '2014_Extended_Release_Opioid_Claims',\n",
       " '2014_Overall_Claims',\n",
       " '2014_Opioid_Prescribing_Rate',\n",
       " '2014_Extended_Release_Opioid_Prescribing_Rate',\n",
       " '2015_Part D Prescribers',\n",
       " '2015_Part D Opioid Prescribers',\n",
       " '2015_Opioid Claims',\n",
       " '2015_Extended Release Opioid Claims',\n",
       " '2015_Overall_Claims',\n",
       " '2015_Opioid_Prescribing_Rate',\n",
       " '2015_Extended_Release_Opioid_Prescribing_ Rate',\n",
       " '2013_2015_Change_in_Opioid_Prescribing_Rate',\n",
       " '2013_2015_Change_in_Extended_Release_Opioid_Prescribing_Rate ']"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(Prescriber_county_df)   # this list is for prescriber county information based on cms datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State_Name</th>\n",
       "      <th>State_Abbreviation</th>\n",
       "      <th>State_FIPS</th>\n",
       "      <th>2013_Part_ D_Prescribers</th>\n",
       "      <th>2013_Part_Opioid_Prescribers</th>\n",
       "      <th>2013_Opioid Claims</th>\n",
       "      <th>2013_Extended_Release_Opioid_Claims</th>\n",
       "      <th>2013_Overall_Claims</th>\n",
       "      <th>2013_Opioid_Prescribing_Rate</th>\n",
       "      <th>2013_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>...</th>\n",
       "      <th>2014_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Part D Prescribers</th>\n",
       "      <th>2015_Part D Opioid Prescribers</th>\n",
       "      <th>2015_Opioid Claims</th>\n",
       "      <th>2015_Extended Release Opioid Claims</th>\n",
       "      <th>2015_Overall_Claims</th>\n",
       "      <th>2015_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2013_2015_Change_in_Opioid_Prescribing_Rate</th>\n",
       "      <th>2013_2015_Change_in_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>National</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1,037,770</td>\n",
       "      <td>491,500</td>\n",
       "      <td>78,045,683</td>\n",
       "      <td>4,773,286</td>\n",
       "      <td>1,342,096,654</td>\n",
       "      <td>5.82</td>\n",
       "      <td>6.12</td>\n",
       "      <td>...</td>\n",
       "      <td>6.27</td>\n",
       "      <td>1,090,979</td>\n",
       "      <td>496,383</td>\n",
       "      <td>78,372,855</td>\n",
       "      <td>5,257,481</td>\n",
       "      <td>1,418,618,875</td>\n",
       "      <td>5.52</td>\n",
       "      <td>6.71</td>\n",
       "      <td>-0.30</td>\n",
       "      <td>0.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>AL</td>\n",
       "      <td>35004.0</td>\n",
       "      <td>18</td>\n",
       "      <td>9</td>\n",
       "      <td>1,340</td>\n",
       "      <td>19</td>\n",
       "      <td>18,632</td>\n",
       "      <td>7.19</td>\n",
       "      <td>1.42</td>\n",
       "      <td>...</td>\n",
       "      <td>6.30</td>\n",
       "      <td>17</td>\n",
       "      <td>7</td>\n",
       "      <td>1,340</td>\n",
       "      <td>39</td>\n",
       "      <td>16,830</td>\n",
       "      <td>7.96</td>\n",
       "      <td>2.91</td>\n",
       "      <td>0.77</td>\n",
       "      <td>1.49</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  State_Name State_Abbreviation  State_FIPS 2013_Part_ D_Prescribers  \\\n",
       "0   National                NaN         NaN                1,037,770   \n",
       "1    Alabama                 AL     35004.0                       18   \n",
       "\n",
       "  2013_Part_Opioid_Prescribers 2013_Opioid Claims  \\\n",
       "0                      491,500         78,045,683   \n",
       "1                            9              1,340   \n",
       "\n",
       "  2013_Extended_Release_Opioid_Claims 2013_Overall_Claims  \\\n",
       "0                           4,773,286       1,342,096,654   \n",
       "1                                  19              18,632   \n",
       "\n",
       "   2013_Opioid_Prescribing_Rate  \\\n",
       "0                          5.82   \n",
       "1                          7.19   \n",
       "\n",
       "   2013_Extended_Release_Opioid_Prescribing_Rate  \\\n",
       "0                                           6.12   \n",
       "1                                           1.42   \n",
       "\n",
       "                               ...                               \\\n",
       "0                              ...                                \n",
       "1                              ...                                \n",
       "\n",
       "  2014_Extended_Release_Opioid_Prescribing_Rate 2015_Part D Prescribers  \\\n",
       "0                                          6.27               1,090,979   \n",
       "1                                          6.30                      17   \n",
       "\n",
       "  2015_Part D Opioid Prescribers 2015_Opioid Claims  \\\n",
       "0                        496,383         78,372,855   \n",
       "1                              7              1,340   \n",
       "\n",
       "  2015_Extended Release Opioid Claims  2015_Overall_Claims  \\\n",
       "0                           5,257,481        1,418,618,875   \n",
       "1                                  39               16,830   \n",
       "\n",
       "   2015_Opioid_Prescribing_Rate  \\\n",
       "0                          5.52   \n",
       "1                          7.96   \n",
       "\n",
       "  2015_Extended_Release_Opioid_Prescribing_ Rate  \\\n",
       "0                                           6.71   \n",
       "1                                           2.91   \n",
       "\n",
       "  2013_2015_Change_in_Opioid_Prescribing_Rate  \\\n",
       "0                                       -0.30   \n",
       "1                                        0.77   \n",
       "\n",
       "  2013_2015_Change_in_Extended_Release_Opioid_Prescribing_Rate   \n",
       "0                                               0.59             \n",
       "1                                               1.49             \n",
       "\n",
       "[2 rows x 26 columns]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Prescriber_zip_df.head()[:2] # this is the dataset for the prescriber zip which is one of the dataclassifications by cms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State_Name</th>\n",
       "      <th>State_Abbreviation</th>\n",
       "      <th>County Name</th>\n",
       "      <th>State_FIPS</th>\n",
       "      <th>2013_Opioid_Prescribing_Rate</th>\n",
       "      <th>2013_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>2014_Opioid_Prescribing_Rate</th>\n",
       "      <th>2014_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2016_Opioid_Prescribing_Rate</th>\n",
       "      <th>2016_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2013_2016_Change in Opioid Prescribing Rate</th>\n",
       "      <th>2013_2016_Change in Extended Release Opioid Prescribing Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>National</td>\n",
       "      <td>National</td>\n",
       "      <td>National</td>\n",
       "      <td>National</td>\n",
       "      <td>5.82</td>\n",
       "      <td>6.12</td>\n",
       "      <td>5.74</td>\n",
       "      <td>6.27</td>\n",
       "      <td>5.52</td>\n",
       "      <td>6.71</td>\n",
       "      <td>5.34</td>\n",
       "      <td>6.92</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>0.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>AL</td>\n",
       "      <td>Autauga</td>\n",
       "      <td>1001</td>\n",
       "      <td>9.8</td>\n",
       "      <td>4.78</td>\n",
       "      <td>9.72</td>\n",
       "      <td>5.85</td>\n",
       "      <td>8.07</td>\n",
       "      <td>7.24</td>\n",
       "      <td>11.47</td>\n",
       "      <td>9.39</td>\n",
       "      <td>1.67</td>\n",
       "      <td>4.61</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  State_Name State_Abbreviation County Name State_FIPS  \\\n",
       "0   National           National    National   National   \n",
       "1    Alabama                 AL     Autauga       1001   \n",
       "\n",
       "  2013_Opioid_Prescribing_Rate 2013_Extended_Release_Opioid_Prescribing_Rate  \\\n",
       "0                         5.82                                          6.12   \n",
       "1                          9.8                                          4.78   \n",
       "\n",
       "  2014_Opioid_Prescribing_Rate 2014_Extended_Release_Opioid_Prescribing_Rate  \\\n",
       "0                         5.74                                          6.27   \n",
       "1                         9.72                                          5.85   \n",
       "\n",
       "  2015_Opioid_Prescribing_Rate 2015_Extended_Release_Opioid_Prescribing_ Rate  \\\n",
       "0                         5.52                                           6.71   \n",
       "1                         8.07                                           7.24   \n",
       "\n",
       "  2016_Opioid_Prescribing_Rate 2016_Extended_Release_Opioid_Prescribing_ Rate  \\\n",
       "0                         5.34                                           6.92   \n",
       "1                        11.47                                           9.39   \n",
       "\n",
       "  2013_2016_Change in Opioid Prescribing Rate  \\\n",
       "0                                       -0.47   \n",
       "1                                        1.67   \n",
       "\n",
       "  2013_2016_Change in Extended Release Opioid Prescribing Rate   \n",
       "0                                                0.8             \n",
       "1                                               4.61             "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Selected the opioid prescribe rate columsn for years 2013-2016 by prescriber county_rate\n",
    "# pcr_df.head()[:2]\n",
    "pcr_df.fillna(\"National\")[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State_Name</th>\n",
       "      <th>2013_2016_Change in Opioid Prescribing Rate</th>\n",
       "      <th>2013_2016_Change in Extended Release Opioid Prescribing Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>National</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>0.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>1.67</td>\n",
       "      <td>4.61</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  State_Name  2013_2016_Change in Opioid Prescribing Rate  \\\n",
       "0   National                                        -0.47   \n",
       "1    Alabama                                         1.67   \n",
       "\n",
       "   2013_2016_Change in Extended Release Opioid Prescribing Rate   \n",
       "0                                               0.80              \n",
       "1                                               4.61              "
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pcr_changeex_df = pcr_df[['State_Name','2013_2016_Change in Opioid Prescribing Rate', '2013_2016_Change in Extended Release Opioid Prescribing Rate ']]\n",
    "pcr_changeex_df .head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "2013_2016_Change in Extended Release Opioid Prescribing Rate     0.060631\n",
      "2013_2016_Change in Opioid Prescribing Rate                      0.060631\n",
      "2015_Extended_Release_Opioid_Prescribing_ Rate                   0.029482\n",
      "2015_Opioid_Prescribing_Rate                                     0.029482\n",
      "2016_Extended Release Opioid Prescribing Rate                    0.029013\n",
      "2016_Opioid Prescribing Rate                                     0.029013\n",
      "2014_Extended_Release_Opioid_Prescribing_Rate                    0.028597\n",
      "2014_Opioid_Prescribing_Rate                                     0.028597\n",
      "2013_Extended_Release_Opioid_Prescribing_Rate                    0.027659\n",
      "2013_Opioid_Prescribing_Rate                                     0.027659\n",
      "State_FIPS                                                       0.000052\n",
      "State_Abbreviation                                               0.000052\n",
      "State_Name                                                       0.000000\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((pzr_df.isnull().sum()/pzr_df.shape[0]).sort_values(ascending=False) )  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State_Name</th>\n",
       "      <th>State_Abbreviation</th>\n",
       "      <th>State_FIPS</th>\n",
       "      <th>2013_Opioid_Prescribing_Rate</th>\n",
       "      <th>2013_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>2014_Opioid_Prescribing_Rate</th>\n",
       "      <th>2014_Extended_Release_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Opioid_Prescribing_Rate</th>\n",
       "      <th>2015_Extended_Release_Opioid_Prescribing_ Rate</th>\n",
       "      <th>2016_Opioid Prescribing Rate</th>\n",
       "      <th>2016_Extended Release Opioid Prescribing Rate</th>\n",
       "      <th>2013_2016_Change in Opioid Prescribing Rate</th>\n",
       "      <th>2013_2016_Change in Extended Release Opioid Prescribing Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>National</td>\n",
       "      <td>National</td>\n",
       "      <td>National</td>\n",
       "      <td>5.82</td>\n",
       "      <td>6.12</td>\n",
       "      <td>5.74</td>\n",
       "      <td>6.27</td>\n",
       "      <td>5.52</td>\n",
       "      <td>6.71</td>\n",
       "      <td>5.34</td>\n",
       "      <td>6.92</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>0.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>AL</td>\n",
       "      <td>35004</td>\n",
       "      <td>7.19</td>\n",
       "      <td>1.42</td>\n",
       "      <td>9.71</td>\n",
       "      <td>6.3</td>\n",
       "      <td>7.96</td>\n",
       "      <td>2.91</td>\n",
       "      <td>7.47</td>\n",
       "      <td>5.61</td>\n",
       "      <td>0.28</td>\n",
       "      <td>4.19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  State_Name State_Abbreviation State_FIPS 2013_Opioid_Prescribing_Rate  \\\n",
       "0   National           National   National                         5.82   \n",
       "1    Alabama                 AL      35004                         7.19   \n",
       "\n",
       "  2013_Extended_Release_Opioid_Prescribing_Rate 2014_Opioid_Prescribing_Rate  \\\n",
       "0                                          6.12                         5.74   \n",
       "1                                          1.42                         9.71   \n",
       "\n",
       "  2014_Extended_Release_Opioid_Prescribing_Rate 2015_Opioid_Prescribing_Rate  \\\n",
       "0                                          6.27                         5.52   \n",
       "1                                           6.3                         7.96   \n",
       "\n",
       "  2015_Extended_Release_Opioid_Prescribing_ Rate 2016_Opioid Prescribing Rate  \\\n",
       "0                                           6.71                         5.34   \n",
       "1                                           2.91                         7.47   \n",
       "\n",
       "  2016_Extended Release Opioid Prescribing Rate  \\\n",
       "0                                          6.92   \n",
       "1                                          5.61   \n",
       "\n",
       "  2013_2016_Change in Opioid Prescribing Rate  \\\n",
       "0                                       -0.47   \n",
       "1                                        0.28   \n",
       "\n",
       "  2013_2016_Change in Extended Release Opioid Prescribing Rate   \n",
       "0                                                0.8             \n",
       "1                                               4.19             "
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pzr_df.fillna(\"National\")[:2]\n",
    "# pzr_df.head()[:2] here the data is by the prescriber zib which is prescriber zip rate\n",
    "# flling the NaN values under the stateabbrevation and stateFips with National"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['State_Name',\n",
       " 'State_Abbreviation',\n",
       " 'State_FIPS',\n",
       " '2013_Opioid_Prescribing_Rate',\n",
       " '2013_Extended_Release_Opioid_Prescribing_Rate',\n",
       " '2014_Opioid_Prescribing_Rate',\n",
       " '2014_Extended_Release_Opioid_Prescribing_Rate',\n",
       " '2015_Opioid_Prescribing_Rate',\n",
       " '2015_Extended_Release_Opioid_Prescribing_ Rate',\n",
       " '2016_Opioid Prescribing Rate',\n",
       " '2016_Extended Release Opioid Prescribing Rate',\n",
       " '2013_2016_Change in Opioid Prescribing Rate',\n",
       " '2013_2016_Change in Extended Release Opioid Prescribing Rate ']"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(pzr_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State_Name</th>\n",
       "      <th>2013_2016_Change in Opioid Prescribing Rate</th>\n",
       "      <th>2013_2016_Change in Extended Release Opioid Prescribing Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>National</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>0.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>0.28</td>\n",
       "      <td>4.19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  State_Name  2013_2016_Change in Opioid Prescribing Rate  \\\n",
       "0   National                                        -0.47   \n",
       "1    Alabama                                         0.28   \n",
       "\n",
       "   2013_2016_Change in Extended Release Opioid Prescribing Rate   \n",
       "0                                               0.80              \n",
       "1                                               4.19              "
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pzr_changeex_df = pzr_df[['State_Name','2013_2016_Change in Opioid Prescribing Rate', '2013_2016_Change in Extended Release Opioid Prescribing Rate ']]\n",
    "pzr_changeex_df .head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Location</th>\n",
       "      <th>FY2004__SMHA Expenditures Per Capita</th>\n",
       "      <th>FY2005__SMHA Expenditures Per Capita</th>\n",
       "      <th>FY2006__SMHA Expenditures Per Capita</th>\n",
       "      <th>FY2007__SMHA Expenditures Per Capita</th>\n",
       "      <th>FY2008__SMHA Expenditures Per Capita</th>\n",
       "      <th>FY2009__SMHA Expenditures Per Capita</th>\n",
       "      <th>FY2010__SMHA Expenditures Per Capita</th>\n",
       "      <th>FY2011__SMHA Expenditures Per Capita</th>\n",
       "      <th>FY2012__SMHA Expenditures Per Capita</th>\n",
       "      <th>FY2013__SMHA Expenditures Per Capita</th>\n",
       "      <th>Footnotes</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>United States</td>\n",
       "      <td>93.04</td>\n",
       "      <td>99.55</td>\n",
       "      <td>103.53</td>\n",
       "      <td>113.27</td>\n",
       "      <td>121.13</td>\n",
       "      <td>122.90</td>\n",
       "      <td>120.56</td>\n",
       "      <td>123.93</td>\n",
       "      <td>124.99</td>\n",
       "      <td>119.62</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>58.78</td>\n",
       "      <td>60.31</td>\n",
       "      <td>64.43</td>\n",
       "      <td>74.03</td>\n",
       "      <td>79.39</td>\n",
       "      <td>77.89</td>\n",
       "      <td>78.19</td>\n",
       "      <td>78.33</td>\n",
       "      <td>76.27</td>\n",
       "      <td>72.64</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Location  FY2004__SMHA Expenditures Per Capita  \\\n",
       "0  United States                                 93.04   \n",
       "1        Alabama                                 58.78   \n",
       "\n",
       "   FY2005__SMHA Expenditures Per Capita  FY2006__SMHA Expenditures Per Capita  \\\n",
       "0                                 99.55                                103.53   \n",
       "1                                 60.31                                 64.43   \n",
       "\n",
       "   FY2007__SMHA Expenditures Per Capita  FY2008__SMHA Expenditures Per Capita  \\\n",
       "0                                113.27                                121.13   \n",
       "1                                 74.03                                 79.39   \n",
       "\n",
       "   FY2009__SMHA Expenditures Per Capita  FY2010__SMHA Expenditures Per Capita  \\\n",
       "0                                122.90                                120.56   \n",
       "1                                 77.89                                 78.19   \n",
       "\n",
       "  FY2011__SMHA Expenditures Per Capita  FY2012__SMHA Expenditures Per Capita  \\\n",
       "0                               123.93                                124.99   \n",
       "1                                78.33                                 76.27   \n",
       "\n",
       "  FY2013__SMHA Expenditures Per Capita Footnotes  \n",
       "0                               119.62       NaN  \n",
       "1                                72.64       NaN  "
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "KFFMentalHealthspendState_df.head()[:2]  # this is the mental health state data collected from KFF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "Footnotes                               0.584416\n",
      "FY2007__SMHA Expenditures Per Capita    0.337662\n",
      "FY2010__SMHA Expenditures Per Capita    0.324675\n",
      "FY2009__SMHA Expenditures Per Capita    0.324675\n",
      "FY2008__SMHA Expenditures Per Capita    0.324675\n",
      "FY2006__SMHA Expenditures Per Capita    0.324675\n",
      "FY2005__SMHA Expenditures Per Capita    0.324675\n",
      "FY2004__SMHA Expenditures Per Capita    0.324675\n",
      "FY2013__SMHA Expenditures Per Capita    0.311688\n",
      "FY2012__SMHA Expenditures Per Capita    0.311688\n",
      "FY2011__SMHA Expenditures Per Capita    0.311688\n",
      "Location                                0.103896\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((KFFMentalHealthspendState_df.isnull().sum()/KFFMentalHealthspendState_df.shape[0]).sort_values(ascending=False))  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Location',\n",
       " 'FY2004__SMHA Expenditures Per Capita',\n",
       " 'FY2005__SMHA Expenditures Per Capita',\n",
       " 'FY2006__SMHA Expenditures Per Capita',\n",
       " 'FY2007__SMHA Expenditures Per Capita',\n",
       " 'FY2008__SMHA Expenditures Per Capita',\n",
       " 'FY2009__SMHA Expenditures Per Capita',\n",
       " 'FY2010__SMHA Expenditures Per Capita',\n",
       " 'FY2011__SMHA Expenditures Per Capita',\n",
       " 'FY2012__SMHA Expenditures Per Capita',\n",
       " 'FY2013__SMHA Expenditures Per Capita',\n",
       " 'Footnotes']"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(KFFMentalHealthspendState_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FIPS</th>\n",
       "      <th>Area Name</th>\n",
       "      <th>1900</th>\n",
       "      <th>1910</th>\n",
       "      <th>1920</th>\n",
       "      <th>1930</th>\n",
       "      <th>1940</th>\n",
       "      <th>1950</th>\n",
       "      <th>1960</th>\n",
       "      <th>1970</th>\n",
       "      <th>1980</th>\n",
       "      <th>1990</th>\n",
       "      <th>2000</th>\n",
       "      <th>2010</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>United States</td>\n",
       "      <td>39.6</td>\n",
       "      <td>45.6</td>\n",
       "      <td>51.2</td>\n",
       "      <td>56.1</td>\n",
       "      <td>56.5</td>\n",
       "      <td>64.0</td>\n",
       "      <td>69.9</td>\n",
       "      <td>73.6</td>\n",
       "      <td>73.7</td>\n",
       "      <td>75.2</td>\n",
       "      <td>79.0</td>\n",
       "      <td>80.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>11.9</td>\n",
       "      <td>17.3</td>\n",
       "      <td>21.7</td>\n",
       "      <td>28.1</td>\n",
       "      <td>30.2</td>\n",
       "      <td>43.8</td>\n",
       "      <td>54.8</td>\n",
       "      <td>58.6</td>\n",
       "      <td>60.0</td>\n",
       "      <td>60.4</td>\n",
       "      <td>55.4</td>\n",
       "      <td>59.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  FIPS      Area Name  1900  1910  1920  1930  1940  1950  1960  1970  1980  \\\n",
       "0    0  United States  39.6  45.6  51.2  56.1  56.5  64.0  69.9  73.6  73.7   \n",
       "1    1        Alabama  11.9  17.3  21.7  28.1  30.2  43.8  54.8  58.6  60.0   \n",
       "\n",
       "   1990  2000  2010  \n",
       "0  75.2  79.0  80.7  \n",
       "1  60.4  55.4  59.0  "
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pop_urban_df.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['FIPS',\n",
       " 'Area Name',\n",
       " '1900',\n",
       " '1910',\n",
       " '1920',\n",
       " '1930',\n",
       " '1940',\n",
       " '1950',\n",
       " '1960',\n",
       " '1970',\n",
       " '1980',\n",
       " '1990',\n",
       " '2000',\n",
       " '2010']"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(pop_urban_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "2010         0.118644\n",
      "2000         0.118644\n",
      "1990         0.118644\n",
      "1980         0.118644\n",
      "1970         0.118644\n",
      "1960         0.118644\n",
      "1950         0.118644\n",
      "1940         0.118644\n",
      "1930         0.118644\n",
      "1920         0.118644\n",
      "1910         0.118644\n",
      "1900         0.118644\n",
      "Area Name    0.118644\n",
      "FIPS         0.033898\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((pop_urban_df.isnull().sum()/pop_urban_df.shape[0]).sort_values(ascending=False) )  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['FIPS',\n",
       " 'Area Name',\n",
       " '1900',\n",
       " '1910',\n",
       " '1920',\n",
       " '1930',\n",
       " '1940',\n",
       " '1950',\n",
       " '1960',\n",
       " '1970',\n",
       " '1980',\n",
       " '1990',\n",
       " '2000',\n",
       " '2010']"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(pop_urban_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extra: Mental health and opioids: is the opioid epidemic really a mental health crisis?\n",
    "\n",
    "This was added to see if there is any relationship  of any kind with the mental health  of the person and opiods as well as to see if its connected with any of the crimes as the US is becoming a hub of gun related deaths across the nation,  in different locations like what has  happened recently the Capital Gazette shooting in Annapolis, MD on June 28, 2018, at the offices of newspaper publisher Capital Gazette Communications. The perpetrator killed five Capital Gazette employees and gravely wounded several in the attack on the newsroom[Capital Gazette ](https://www.cnn.com/us/live-news/maryland-newspaper-shooting/index.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Notes</th>\n",
       "      <th>State</th>\n",
       "      <th>State Code</th>\n",
       "      <th>Year</th>\n",
       "      <th>Year Code</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Gender Code</th>\n",
       "      <th>Deaths</th>\n",
       "      <th>Population</th>\n",
       "      <th>Crude Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>F</td>\n",
       "      <td>84.0</td>\n",
       "      <td>2293259.0</td>\n",
       "      <td>3.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>M</td>\n",
       "      <td>111.0</td>\n",
       "      <td>2136882.0</td>\n",
       "      <td>5.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Notes    State  State Code    Year  Year Code  Gender Gender Code  Deaths  \\\n",
       "0   NaN  Alabama         1.0  1999.0     1999.0  Female           F    84.0   \n",
       "1   NaN  Alabama         1.0  1999.0     1999.0    Male           M   111.0   \n",
       "\n",
       "   Population  Crude Rate  \n",
       "0   2293259.0         3.7  \n",
       "1   2136882.0         5.2  "
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Wonder_Drug_Death_Data1_df.head()[:2]  # looking on the first two rows of the Dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Notes</th>\n",
       "      <th>State</th>\n",
       "      <th>State Code</th>\n",
       "      <th>Year</th>\n",
       "      <th>Year Code</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Gender Code</th>\n",
       "      <th>Deaths</th>\n",
       "      <th>Population</th>\n",
       "      <th>Crude Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>F</td>\n",
       "      <td>84.0</td>\n",
       "      <td>2293259.0</td>\n",
       "      <td>3.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>M</td>\n",
       "      <td>111.0</td>\n",
       "      <td>2136882.0</td>\n",
       "      <td>5.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Notes    State  State Code    Year  Year Code  Gender Gender Code  Deaths  \\\n",
       "0   NaN  Alabama         1.0  1999.0     1999.0  Female           F    84.0   \n",
       "1   NaN  Alabama         1.0  1999.0     1999.0    Male           M   111.0   \n",
       "\n",
       "   Population  Crude Rate  \n",
       "0   2293259.0         3.7  \n",
       "1   2136882.0         5.2  "
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Wonder_Drug_Death_Data1_df.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Notes', 'State', 'State Code', 'Year', 'Year Code', 'Gender',\n",
       "       'Gender Code', 'Deaths', 'Population', 'Crude Rate'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Wonder_Drug_Death_Data1_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "Notes          0.961905\n",
      "Crude Rate     0.063305\n",
      "Population     0.038095\n",
      "Deaths         0.038095\n",
      "Gender Code    0.038095\n",
      "Gender         0.038095\n",
      "Year Code      0.038095\n",
      "Year           0.038095\n",
      "State Code     0.038095\n",
      "State          0.038095\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((Wonder_Drug_Death_Data1_df.isnull().sum()/Wonder_Drug_Death_Data1_df.shape[0]).sort_values(ascending=False))  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "Wonder_Drug_Death_Data1_df.drop(['Notes'], axis=1, inplace=True)         # drops Notes column\n",
    "Wonder_Drug_Death_Data1_df.dropna(axis=0, how='any', inplace=True)      # drops rows with NaN\n",
    "Wonder_Drug_Death_Data1_df['Year'] = Wonder_Drug_Death_Data1_df['Year'].astype(int)     # casts datatypes for columns\n",
    "Wonder_Drug_Death_Data1_df['Year Code'] = Wonder_Drug_Death_Data1_df['Year Code'].astype(int)\n",
    "Wonder_Drug_Death_Data1_df['Year Code'] = Wonder_Drug_Death_Data1_df['Year Code'].astype(str)\n",
    "Wonder_Drug_Death_Data1_df['Deaths'] = Wonder_Drug_Death_Data1_df['Deaths'].astype(int)\n",
    "Wonder_Drug_Death_Data1_df['Population'] = Wonder_Drug_Death_Data1_df['Population'].astype(int)\n",
    "\n",
    "# Calculates death rate per 100,000 of population (Deaths / Population) * 100,000\n",
    "Wonder_Drug_Death_Data1_df['Death Rate'] = Wonder_Drug_Death_Data1_df['Deaths'] / Wonder_Drug_Death_Data1_df['Population'] * 100000\n",
    "Wonder_Drug_Death_Data1_df.drop('Crude Rate', axis=1, inplace=True)             # drops Crude Rate column\n",
    "Wonder_Drug_Death_Data1_df.set_index('Year', inplace=True)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>State Code</th>\n",
       "      <th>Year Code</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Gender Code</th>\n",
       "      <th>Deaths</th>\n",
       "      <th>Population</th>\n",
       "      <th>Death Rate</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1999</td>\n",
       "      <td>Female</td>\n",
       "      <td>F</td>\n",
       "      <td>84</td>\n",
       "      <td>2293259</td>\n",
       "      <td>3.662909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1999</td>\n",
       "      <td>Male</td>\n",
       "      <td>M</td>\n",
       "      <td>111</td>\n",
       "      <td>2136882</td>\n",
       "      <td>5.194484</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        State  State Code Year Code  Gender Gender Code  Deaths  Population  \\\n",
       "Year                                                                          \n",
       "1999  Alabama         1.0      1999  Female           F      84     2293259   \n",
       "1999  Alabama         1.0      1999    Male           M     111     2136882   \n",
       "\n",
       "      Death Rate  \n",
       "Year              \n",
       "1999    3.662909  \n",
       "1999    5.194484  "
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Wonder_Drug_Death_Data1_df.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "Wonder_Drug_Death_Data1_df.to_csv('./inputs/Wonder_Drug_Death_Data1V_df.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Notes</th>\n",
       "      <th>Year</th>\n",
       "      <th>Year Code</th>\n",
       "      <th>State</th>\n",
       "      <th>State Code</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Gender Code</th>\n",
       "      <th>Ten-Year Age Groups</th>\n",
       "      <th>Ten-Year Age Groups Code</th>\n",
       "      <th>Deaths</th>\n",
       "      <th>Population</th>\n",
       "      <th>Crude Rate</th>\n",
       "      <th>Crude Rate Standard Error</th>\n",
       "      <th>% of Total Deaths</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Total</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>F</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>21.0</td>\n",
       "      <td>2293259.0</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>M</td>\n",
       "      <td>25-34 years</td>\n",
       "      <td>25-34</td>\n",
       "      <td>11.0</td>\n",
       "      <td>300073.0</td>\n",
       "      <td>Unreliable</td>\n",
       "      <td>1.1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Notes    Year  Year Code    State  State Code  Gender Gender Code  \\\n",
       "0  Total  1999.0     1999.0  Alabama         1.0  Female           F   \n",
       "1    NaN  1999.0     1999.0  Alabama         1.0    Male           M   \n",
       "\n",
       "  Ten-Year Age Groups Ten-Year Age Groups Code  Deaths  Population  \\\n",
       "0                 NaN                      NaN    21.0   2293259.0   \n",
       "1         25-34 years                    25-34    11.0    300073.0   \n",
       "\n",
       "   Crude Rate  Crude Rate Standard Error  % of Total Deaths  \n",
       "0         0.9                        0.2                0.0  \n",
       "1  Unreliable                        1.1                0.0  "
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Wonder_Drug_Death_Data2_df.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "Notes                        0.694154\n",
      "Ten-Year Age Groups Code     0.305846\n",
      "Ten-Year Age Groups          0.305846\n",
      "Gender Code                  0.111634\n",
      "Gender                       0.111634\n",
      "State Code                   0.011175\n",
      "State                        0.011175\n",
      "Year Code                    0.009175\n",
      "Year                         0.009175\n",
      "% of Total Deaths            0.009058\n",
      "Crude Rate Standard Error    0.009058\n",
      "Crude Rate                   0.009058\n",
      "Population                   0.009058\n",
      "Deaths                       0.009058\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((Wonder_Drug_Death_Data2_df.isnull().sum()/Wonder_Drug_Death_Data2_df.shape[0]).sort_values(ascending=False))  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Notes', 'Year', 'Year Code', 'State', 'State Code', 'Gender',\n",
       "       'Gender Code', 'Ten-Year Age Groups', 'Ten-Year Age Groups Code',\n",
       "       'Deaths', 'Population', 'Crude Rate', 'Crude Rate Standard Error',\n",
       "       '% of Total Deaths'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Wonder_Drug_Death_Data2_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "Wonder_Drug_Death_Data2_df.drop(['Notes'], axis=1, inplace=True)                          # drops Notes column\n",
    "Wonder_Drug_Death_Data2_df.dropna(axis=0, how='any', inplace=True)                          # drops rows with NaN\n",
    "Wonder_Drug_Death_Data2_df['Year'] = Wonder_Drug_Death_Data2_df['Year'].astype(int)             # casts datatypes for columns\n",
    "Wonder_Drug_Death_Data2_df['Year Code'] = Wonder_Drug_Death_Data2_df['Year Code'].astype(int)\n",
    "Wonder_Drug_Death_Data2_df['Year Code'] = Wonder_Drug_Death_Data2_df['Year Code'].astype(str)\n",
    "Wonder_Drug_Death_Data2_df['Deaths'] = Wonder_Drug_Death_Data2_df['Deaths'].astype(int)\n",
    "Wonder_Drug_Death_Data2_df['Population'] = Wonder_Drug_Death_Data2_df['Population'].astype(int)\n",
    "\n",
    "# Calculates death rate per 100,000 of population (Deaths / Population) * 100,000\n",
    "Wonder_Drug_Death_Data2_df['Death Rate'] = Wonder_Drug_Death_Data2_df['Deaths'] / Wonder_Drug_Death_Data2_df['Population'] * 100000\n",
    "Wonder_Drug_Death_Data2_df.drop('Crude Rate', axis=1, inplace=True)                            # drops Crude Rate column\n",
    "Wonder_Drug_Death_Data2_df.set_index('Year', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year Code</th>\n",
       "      <th>State</th>\n",
       "      <th>State Code</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Gender Code</th>\n",
       "      <th>Ten-Year Age Groups</th>\n",
       "      <th>Ten-Year Age Groups Code</th>\n",
       "      <th>Deaths</th>\n",
       "      <th>Population</th>\n",
       "      <th>Crude Rate Standard Error</th>\n",
       "      <th>% of Total Deaths</th>\n",
       "      <th>Death Rate</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>1999</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>M</td>\n",
       "      <td>25-34 years</td>\n",
       "      <td>25-34</td>\n",
       "      <td>11</td>\n",
       "      <td>300073</td>\n",
       "      <td>1.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.665775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>1999</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>M</td>\n",
       "      <td>35-44 years</td>\n",
       "      <td>35-44</td>\n",
       "      <td>12</td>\n",
       "      <td>335631</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.575355</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Year Code    State  State Code Gender Gender Code Ten-Year Age Groups  \\\n",
       "Year                                                                         \n",
       "1999      1999  Alabama         1.0   Male           M         25-34 years   \n",
       "1999      1999  Alabama         1.0   Male           M         35-44 years   \n",
       "\n",
       "     Ten-Year Age Groups Code  Deaths  Population  Crude Rate Standard Error  \\\n",
       "Year                                                                           \n",
       "1999                    25-34      11      300073                        1.1   \n",
       "1999                    35-44      12      335631                        1.0   \n",
       "\n",
       "      % of Total Deaths  Death Rate  \n",
       "Year                                 \n",
       "1999                0.0    3.665775  \n",
       "1999                0.0    3.575355  "
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Wonder_Drug_Death_Data2_df.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Year Code', 'State', 'State Code', 'Gender', 'Gender Code',\n",
       "       'Ten-Year Age Groups', 'Ten-Year Age Groups Code', 'Deaths',\n",
       "       'Population', 'Crude Rate Standard Error', '% of Total Deaths',\n",
       "       'Death Rate'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Wonder_Drug_Death_Data2_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "Wonder_Drug_Death_Data2_df.to_csv('./inputs/Wonder_Drug_Death_Data2V_df.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Expenditure Amount (Millions)</th>\n",
       "      <th>1960</th>\n",
       "      <th>1961</th>\n",
       "      <th>1962</th>\n",
       "      <th>1963</th>\n",
       "      <th>1964</th>\n",
       "      <th>1965</th>\n",
       "      <th>1966</th>\n",
       "      <th>1967</th>\n",
       "      <th>1968</th>\n",
       "      <th>...</th>\n",
       "      <th>2007</th>\n",
       "      <th>2008</th>\n",
       "      <th>2009</th>\n",
       "      <th>2010</th>\n",
       "      <th>2011</th>\n",
       "      <th>2012</th>\n",
       "      <th>2013</th>\n",
       "      <th>2014</th>\n",
       "      <th>2015</th>\n",
       "      <th>2016</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Total National Health Expenditures</td>\n",
       "      <td>27,214</td>\n",
       "      <td>29,138</td>\n",
       "      <td>31,842</td>\n",
       "      <td>34,595</td>\n",
       "      <td>38,394</td>\n",
       "      <td>41,852</td>\n",
       "      <td>46,081</td>\n",
       "      <td>51,565</td>\n",
       "      <td>58,402</td>\n",
       "      <td>...</td>\n",
       "      <td>2,295,307</td>\n",
       "      <td>2,399,121</td>\n",
       "      <td>2,495,414</td>\n",
       "      <td>2,598,823</td>\n",
       "      <td>2,689,349</td>\n",
       "      <td>2,797,260</td>\n",
       "      <td>2,879,008</td>\n",
       "      <td>3,026,157</td>\n",
       "      <td>3,200,815</td>\n",
       "      <td>3,337,248</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Out of pocket</td>\n",
       "      <td>12,949</td>\n",
       "      <td>13,357</td>\n",
       "      <td>14,255</td>\n",
       "      <td>15,311</td>\n",
       "      <td>16,928</td>\n",
       "      <td>18,209</td>\n",
       "      <td>18,594</td>\n",
       "      <td>18,537</td>\n",
       "      <td>20,500</td>\n",
       "      <td>...</td>\n",
       "      <td>290,000</td>\n",
       "      <td>295,184</td>\n",
       "      <td>293,723</td>\n",
       "      <td>299,742</td>\n",
       "      <td>309,973</td>\n",
       "      <td>318,330</td>\n",
       "      <td>325,189</td>\n",
       "      <td>330,068</td>\n",
       "      <td>339,315</td>\n",
       "      <td>352,537</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 58 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        Expenditure Amount (Millions)    1960    1961    1962    1963    1964  \\\n",
       "0  Total National Health Expenditures  27,214  29,138  31,842  34,595  38,394   \n",
       "1                       Out of pocket  12,949  13,357  14,255  15,311  16,928   \n",
       "\n",
       "     1965    1966    1967    1968    ...           2007       2008       2009  \\\n",
       "0  41,852  46,081  51,565  58,402    ...      2,295,307  2,399,121  2,495,414   \n",
       "1  18,209  18,594  18,537  20,500    ...        290,000    295,184    293,723   \n",
       "\n",
       "        2010       2011       2012       2013       2014       2015       2016  \n",
       "0  2,598,823  2,689,349  2,797,260  2,879,008  3,026,157  3,200,815  3,337,248  \n",
       "1    299,742    309,973    318,330    325,189    330,068    339,315    352,537  \n",
       "\n",
       "[2 rows x 58 columns]"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NHE2016_df.head()[:2]   # USD in millions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "1967                             0.090239\n",
      "1960                             0.090239\n",
      "1961                             0.090239\n",
      "1962                             0.090239\n",
      "1963                             0.090239\n",
      "1964                             0.090239\n",
      "1965                             0.090239\n",
      "1966                             0.090239\n",
      "1968                             0.090239\n",
      "1969                             0.090239\n",
      "1983                             0.086556\n",
      "1980                             0.086556\n",
      "1981                             0.086556\n",
      "1982                             0.086556\n",
      "2016                             0.086556\n",
      "1984                             0.086556\n",
      "1985                             0.086556\n",
      "1978                             0.086556\n",
      "1979                             0.086556\n",
      "1972                             0.086556\n",
      "1977                             0.086556\n",
      "1976                             0.086556\n",
      "1975                             0.086556\n",
      "1974                             0.086556\n",
      "1973                             0.086556\n",
      "2015                             0.086556\n",
      "1971                             0.086556\n",
      "1970                             0.086556\n",
      "1986                             0.086556\n",
      "1987                             0.086556\n",
      "1988                             0.086556\n",
      "1989                             0.086556\n",
      "2014                             0.086556\n",
      "2013                             0.086556\n",
      "2012                             0.086556\n",
      "2011                             0.086556\n",
      "2010                             0.086556\n",
      "2009                             0.086556\n",
      "2008                             0.086556\n",
      "2007                             0.086556\n",
      "2006                             0.086556\n",
      "2005                             0.086556\n",
      "2004                             0.086556\n",
      "2003                             0.086556\n",
      "2002                             0.086556\n",
      "2001                             0.086556\n",
      "2000                             0.086556\n",
      "1999                             0.086556\n",
      "1998                             0.086556\n",
      "1997                             0.086556\n",
      "1996                             0.086556\n",
      "1995                             0.086556\n",
      "1994                             0.086556\n",
      "1993                             0.086556\n",
      "1992                             0.086556\n",
      "1991                             0.086556\n",
      "1990                             0.086556\n",
      "Expenditure Amount (Millions)    0.000000\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((NHE2016_df.isnull().sum()/NHE2016_df.shape[0]).sort_values(ascending=False))  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Expenditure Amount (Millions)',\n",
       " '1960',\n",
       " '1961',\n",
       " '1962',\n",
       " '1963',\n",
       " '1964',\n",
       " '1965',\n",
       " '1966',\n",
       " '1967',\n",
       " '1968',\n",
       " '1969',\n",
       " '1970',\n",
       " '1971',\n",
       " '1972',\n",
       " '1973',\n",
       " '1974',\n",
       " '1975',\n",
       " '1976',\n",
       " '1977',\n",
       " '1978',\n",
       " '1979',\n",
       " '1980',\n",
       " '1981',\n",
       " '1982',\n",
       " '1983',\n",
       " '1984',\n",
       " '1985',\n",
       " '1986',\n",
       " '1987',\n",
       " '1988',\n",
       " '1989',\n",
       " '1990',\n",
       " '1991',\n",
       " '1992',\n",
       " '1993',\n",
       " '1994',\n",
       " '1995',\n",
       " '1996',\n",
       " '1997',\n",
       " '1998',\n",
       " '1999',\n",
       " '2000',\n",
       " '2001',\n",
       " '2002',\n",
       " '2003',\n",
       " '2004',\n",
       " '2005',\n",
       " '2006',\n",
       " '2007',\n",
       " '2008',\n",
       " '2009',\n",
       " '2010',\n",
       " '2011',\n",
       " '2012',\n",
       " '2013',\n",
       " '2014',\n",
       " '2015',\n",
       " '2016']"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(NHE2016_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1960</th>\n",
       "      <th>1961</th>\n",
       "      <th>1962</th>\n",
       "      <th>1963</th>\n",
       "      <th>1964</th>\n",
       "      <th>1965</th>\n",
       "      <th>1966</th>\n",
       "      <th>1967</th>\n",
       "      <th>1968</th>\n",
       "      <th>1969</th>\n",
       "      <th>...</th>\n",
       "      <th>2007</th>\n",
       "      <th>2008</th>\n",
       "      <th>2009</th>\n",
       "      <th>2010</th>\n",
       "      <th>2011</th>\n",
       "      <th>2012</th>\n",
       "      <th>2013</th>\n",
       "      <th>2014</th>\n",
       "      <th>2015</th>\n",
       "      <th>2016</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Expenditure Amount (Millions)</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Total National Health Expenditures</th>\n",
       "      <td>27,214</td>\n",
       "      <td>29,138</td>\n",
       "      <td>31,842</td>\n",
       "      <td>34,595</td>\n",
       "      <td>38,394</td>\n",
       "      <td>41,852</td>\n",
       "      <td>46,081</td>\n",
       "      <td>51,565</td>\n",
       "      <td>58,402</td>\n",
       "      <td>65,923</td>\n",
       "      <td>...</td>\n",
       "      <td>2,295,307</td>\n",
       "      <td>2,399,121</td>\n",
       "      <td>2,495,414</td>\n",
       "      <td>2,598,823</td>\n",
       "      <td>2,689,349</td>\n",
       "      <td>2,797,260</td>\n",
       "      <td>2,879,008</td>\n",
       "      <td>3,026,157</td>\n",
       "      <td>3,200,815</td>\n",
       "      <td>3,337,248</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Total CMS Programs (Medicaid, CHIP and Medicare)</th>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>3,146</td>\n",
       "      <td>8,065</td>\n",
       "      <td>9,759</td>\n",
       "      <td>11,220</td>\n",
       "      <td>...</td>\n",
       "      <td>767,613</td>\n",
       "      <td>821,394</td>\n",
       "      <td>884,386</td>\n",
       "      <td>928,553</td>\n",
       "      <td>963,412</td>\n",
       "      <td>1,004,951</td>\n",
       "      <td>1,049,032</td>\n",
       "      <td>1,128,749</td>\n",
       "      <td>1,207,734</td>\n",
       "      <td>1,254,525</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 57 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    1960    1961    1962  \\\n",
       "Expenditure Amount (Millions)                                              \n",
       "Total National Health Expenditures                27,214  29,138  31,842   \n",
       "Total CMS Programs (Medicaid, CHIP and Medicare)    -       -       -      \n",
       "\n",
       "                                                    1963    1964    1965  \\\n",
       "Expenditure Amount (Millions)                                              \n",
       "Total National Health Expenditures                34,595  38,394  41,852   \n",
       "Total CMS Programs (Medicaid, CHIP and Medicare)    -       -       -      \n",
       "\n",
       "                                                    1966    1967    1968  \\\n",
       "Expenditure Amount (Millions)                                              \n",
       "Total National Health Expenditures                46,081  51,565  58,402   \n",
       "Total CMS Programs (Medicaid, CHIP and Medicare)   3,146   8,065   9,759   \n",
       "\n",
       "                                                    1969    ...      \\\n",
       "Expenditure Amount (Millions)                               ...       \n",
       "Total National Health Expenditures                65,923    ...       \n",
       "Total CMS Programs (Medicaid, CHIP and Medicare)  11,220    ...       \n",
       "\n",
       "                                                       2007       2008  \\\n",
       "Expenditure Amount (Millions)                                            \n",
       "Total National Health Expenditures                2,295,307  2,399,121   \n",
       "Total CMS Programs (Medicaid, CHIP and Medicare)    767,613    821,394   \n",
       "\n",
       "                                                       2009       2010  \\\n",
       "Expenditure Amount (Millions)                                            \n",
       "Total National Health Expenditures                2,495,414  2,598,823   \n",
       "Total CMS Programs (Medicaid, CHIP and Medicare)    884,386    928,553   \n",
       "\n",
       "                                                       2011       2012  \\\n",
       "Expenditure Amount (Millions)                                            \n",
       "Total National Health Expenditures                2,689,349  2,797,260   \n",
       "Total CMS Programs (Medicaid, CHIP and Medicare)    963,412  1,004,951   \n",
       "\n",
       "                                                       2013       2014  \\\n",
       "Expenditure Amount (Millions)                                            \n",
       "Total National Health Expenditures                2,879,008  3,026,157   \n",
       "Total CMS Programs (Medicaid, CHIP and Medicare)  1,049,032  1,128,749   \n",
       "\n",
       "                                                       2015       2016  \n",
       "Expenditure Amount (Millions)                                           \n",
       "Total National Health Expenditures                3,200,815  3,337,248  \n",
       "Total CMS Programs (Medicaid, CHIP and Medicare)  1,207,734  1,254,525  \n",
       "\n",
       "[2 rows x 57 columns]"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# extract Total NHE and Total CMS Program rows\n",
    "NHE2016_df = NHE2016_df.loc[[0,35,36],]\n",
    "NHE2016_df = NHE2016_df.set_index(\"Expenditure Amount (Millions)\")\n",
    "\n",
    "# set to numeric for plotting purposes\n",
    "NHE2016_df = NHE2016_df.apply(pd.to_numeric, errors='ignore')\n",
    "\n",
    "\n",
    "NHE2016_df.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract series of spending data 1999 - 2016 to compare against opioid deaths\n",
    "cms_spending = NHE2016_df.loc['Total CMS Programs (Medicaid, CHIP and Medicare)','1999':'2016']\n",
    "nhe_spending = NHE2016_df.loc['Total National Health Expenditures','1999':'2016']\n",
    "# percentage = cms_spending / nhe_spending\n",
    "\n",
    "# percentage.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "cms_spending = NHE2016_df.loc['Total CMS Programs (Medicaid, CHIP and Medicare)','1999':'2016'].str.replace(',', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "cms_spending = NHE2016_df.loc['Total CMS Programs (Medicaid, CHIP and Medicare)','1999':'2016'].str.replace(' ', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1999    398,350\n",
       "2000    428,187\n",
       "Name: Total CMS Programs (Medicaid, CHIP and Medicare), dtype: object"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cms_spending.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1999    1,277,700\n",
       "2000    1,369,125\n",
       "Name: Total National Health Expenditures, dtype: object"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nhe_spending.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1999    1,277,700\n",
       "2000    1,369,125\n",
       "2001    1,486,158\n",
       "2002    1,628,624\n",
       "2003    1,767,567\n",
       "2004    1,895,724\n",
       "2005    2,023,744\n",
       "2006    2,156,158\n",
       "2007    2,295,307\n",
       "2008    2,399,121\n",
       "2009    2,495,414\n",
       "2010    2,598,823\n",
       "2011    2,689,349\n",
       "2012    2,797,260\n",
       "2013    2,879,008\n",
       "2014    3,026,157\n",
       "2015    3,200,815\n",
       "2016    3,337,248\n",
       "Name: Total National Health Expenditures, dtype: object"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nhe_spending"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a function to convert the percentage to a fraction \n",
    "def p2f(x):\n",
    "    return float(x.strip('%'))/100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>State</th>\n",
       "      <th>Population_1999</th>\n",
       "      <th>Violent Crime rate_1999</th>\n",
       "      <th>Murder and nonnegligent manslaughter rate_1999</th>\n",
       "      <th>Legacy rape rate /1_1999</th>\n",
       "      <th>Revised rape rate /2_1999</th>\n",
       "      <th>Robbery rate_1999</th>\n",
       "      <th>Aggravated assault rate_1999</th>\n",
       "      <th>Unnamed: 8_1999</th>\n",
       "      <th>Population_2014</th>\n",
       "      <th>Violent Crime rate_2014</th>\n",
       "      <th>Murder and nonnegligent manslaughter rate_2014</th>\n",
       "      <th>Legacy rape rate /1_2014</th>\n",
       "      <th>Revised rape rate /2_2014</th>\n",
       "      <th>Robbery rate_2014</th>\n",
       "      <th>Aggravated assault rate_2014</th>\n",
       "      <th>Unnamed: 8_2014</th>\n",
       "      <th>Violent Crime rate_change</th>\n",
       "      <th>Robbery rate_change</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>4369862</td>\n",
       "      <td>490.2</td>\n",
       "      <td>7.9</td>\n",
       "      <td>34.6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>121.2</td>\n",
       "      <td>326.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4849377</td>\n",
       "      <td>427.4</td>\n",
       "      <td>5.7</td>\n",
       "      <td>29.6</td>\n",
       "      <td>41.3</td>\n",
       "      <td>96.9</td>\n",
       "      <td>283.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-62.8</td>\n",
       "      <td>-24.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Alaska</td>\n",
       "      <td>619500</td>\n",
       "      <td>630.8</td>\n",
       "      <td>8.4</td>\n",
       "      <td>83.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>91.4</td>\n",
       "      <td>447.6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>736732</td>\n",
       "      <td>635.8</td>\n",
       "      <td>5.6</td>\n",
       "      <td>75.3</td>\n",
       "      <td>104.7</td>\n",
       "      <td>85.4</td>\n",
       "      <td>440.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>-6.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0    State  Population_1999  Violent Crime rate_1999  \\\n",
       "0           0  Alabama          4369862                    490.2   \n",
       "1           1   Alaska           619500                    630.8   \n",
       "\n",
       "   Murder and nonnegligent manslaughter rate_1999  Legacy rape rate /1_1999  \\\n",
       "0                                             7.9                      34.6   \n",
       "1                                             8.4                      83.5   \n",
       "\n",
       "   Revised rape rate /2_1999  Robbery rate_1999  Aggravated assault rate_1999  \\\n",
       "0                        NaN              121.2                         326.5   \n",
       "1                        NaN               91.4                         447.6   \n",
       "\n",
       "   Unnamed: 8_1999  Population_2014  Violent Crime rate_2014  \\\n",
       "0              NaN          4849377                    427.4   \n",
       "1              NaN           736732                    635.8   \n",
       "\n",
       "   Murder and nonnegligent manslaughter rate_2014  Legacy rape rate /1_2014  \\\n",
       "0                                             5.7                      29.6   \n",
       "1                                             5.6                      75.3   \n",
       "\n",
       "   Revised rape rate /2_2014  Robbery rate_2014  Aggravated assault rate_2014  \\\n",
       "0                       41.3               96.9                         283.4   \n",
       "1                      104.7               85.4                         440.2   \n",
       "\n",
       "   Unnamed: 8_2014  Violent Crime rate_change  Robbery rate_change  \n",
       "0              NaN                      -62.8                -24.3  \n",
       "1              NaN                        5.0                 -6.0  "
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "violentcrime_df.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "violentcrime_df.drop(['Unnamed: 0','Unnamed: 8_1999','Unnamed: 8_2014'], axis =1, inplace =True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "Revised rape rate /2_1999                         1.0\n",
      "Robbery rate_change                               0.0\n",
      "Aggravated assault rate_1999                      0.0\n",
      "Population_1999                                   0.0\n",
      "Violent Crime rate_1999                           0.0\n",
      "Murder and nonnegligent manslaughter rate_1999    0.0\n",
      "Legacy rape rate /1_1999                          0.0\n",
      "Robbery rate_1999                                 0.0\n",
      "Population_2014                                   0.0\n",
      "Violent Crime rate_change                         0.0\n",
      "Violent Crime rate_2014                           0.0\n",
      "Murder and nonnegligent manslaughter rate_2014    0.0\n",
      "Legacy rape rate /1_2014                          0.0\n",
      "Revised rape rate /2_2014                         0.0\n",
      "Robbery rate_2014                                 0.0\n",
      "Aggravated assault rate_2014                      0.0\n",
      "State                                             0.0\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((violentcrime_df.isnull().sum()/violentcrime_df.shape[0]).sort_values(ascending=False))  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['State', 'Population_1999', 'Violent Crime rate_1999',\n",
       "       'Murder and nonnegligent manslaughter rate_1999',\n",
       "       'Legacy rape rate /1_1999', 'Revised rape rate /2_1999',\n",
       "       'Robbery rate_1999', 'Aggravated assault rate_1999', 'Population_2014',\n",
       "       'Violent Crime rate_2014',\n",
       "       'Murder and nonnegligent manslaughter rate_2014',\n",
       "       'Legacy rape rate /1_2014', 'Revised rape rate /2_2014',\n",
       "       'Robbery rate_2014', 'Aggravated assault rate_2014',\n",
       "       'Violent Crime rate_change', 'Robbery rate_change'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "violentcrime_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>State</th>\n",
       "      <th>Population_1999</th>\n",
       "      <th>Property crime rate_1999</th>\n",
       "      <th>Burglary rate_1999</th>\n",
       "      <th>Larceny-theft rate_1999</th>\n",
       "      <th>Motor vehicle theft rate_1999</th>\n",
       "      <th>Unnamed: 6_1999</th>\n",
       "      <th>Population_2014</th>\n",
       "      <th>Property crime rate_2014</th>\n",
       "      <th>Burglary rate_2014</th>\n",
       "      <th>Larceny-theft rate_2014</th>\n",
       "      <th>Motor vehicle theft rate_2014</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>4369862</td>\n",
       "      <td>3922.3</td>\n",
       "      <td>884.4</td>\n",
       "      <td>2737.3</td>\n",
       "      <td>300.6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4849377</td>\n",
       "      <td>3177.6</td>\n",
       "      <td>819.0</td>\n",
       "      <td>2149.5</td>\n",
       "      <td>209.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Alaska</td>\n",
       "      <td>619500</td>\n",
       "      <td>3728.7</td>\n",
       "      <td>611.3</td>\n",
       "      <td>2688.3</td>\n",
       "      <td>429.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>736732</td>\n",
       "      <td>2760.0</td>\n",
       "      <td>427.6</td>\n",
       "      <td>2096.4</td>\n",
       "      <td>236.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0    State  Population_1999  Property crime rate_1999  \\\n",
       "0           0  Alabama          4369862                    3922.3   \n",
       "1           1   Alaska           619500                    3728.7   \n",
       "\n",
       "   Burglary rate_1999  Larceny-theft rate_1999  Motor vehicle theft rate_1999  \\\n",
       "0               884.4                   2737.3                          300.6   \n",
       "1               611.3                   2688.3                          429.1   \n",
       "\n",
       "   Unnamed: 6_1999  Population_2014  Property crime rate_2014  \\\n",
       "0              NaN          4849377                    3177.6   \n",
       "1              NaN           736732                    2760.0   \n",
       "\n",
       "   Burglary rate_2014  Larceny-theft rate_2014  Motor vehicle theft rate_2014  \n",
       "0               819.0                   2149.5                          209.1  \n",
       "1               427.6                   2096.4                          236.0  "
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "propertycrime_df.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "propertycrime_df.drop(['Unnamed: 0','Unnamed: 6_1999'], axis =1, inplace =True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "Motor vehicle theft rate_2014    0.0\n",
      "Larceny-theft rate_2014          0.0\n",
      "Burglary rate_2014               0.0\n",
      "Property crime rate_2014         0.0\n",
      "Population_2014                  0.0\n",
      "Motor vehicle theft rate_1999    0.0\n",
      "Larceny-theft rate_1999          0.0\n",
      "Burglary rate_1999               0.0\n",
      "Property crime rate_1999         0.0\n",
      "Population_1999                  0.0\n",
      "State                            0.0\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((propertycrime_df.isnull().sum()/propertycrime_df.shape[0]).sort_values(ascending=False))  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['State', 'Population_1999', 'Property crime rate_1999',\n",
       "       'Burglary rate_1999', 'Larceny-theft rate_1999',\n",
       "       'Motor vehicle theft rate_1999', 'Population_2014',\n",
       "       'Property crime rate_2014', 'Burglary rate_2014',\n",
       "       'Larceny-theft rate_2014', 'Motor vehicle theft rate_2014'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "propertycrime_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>State</th>\n",
       "      <th>Population_1999_x</th>\n",
       "      <th>Violent Crime rate_1999</th>\n",
       "      <th>Murder and nonnegligent manslaughter rate_1999</th>\n",
       "      <th>Legacy rape rate /1_1999</th>\n",
       "      <th>Revised rape rate /2_1999</th>\n",
       "      <th>Robbery rate_1999</th>\n",
       "      <th>Aggravated assault rate_1999</th>\n",
       "      <th>Unnamed: 8_1999</th>\n",
       "      <th>...</th>\n",
       "      <th>Burglary rate_1999</th>\n",
       "      <th>Larceny-theft rate_1999</th>\n",
       "      <th>Motor vehicle theft rate_1999</th>\n",
       "      <th>Unnamed: 6_1999</th>\n",
       "      <th>Population_2014_y</th>\n",
       "      <th>Property crime rate_2014</th>\n",
       "      <th>Burglary rate_2014</th>\n",
       "      <th>Larceny-theft rate_2014</th>\n",
       "      <th>Motor vehicle theft rate_2014</th>\n",
       "      <th>Property crime rate_change</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>4369862</td>\n",
       "      <td>490.2</td>\n",
       "      <td>7.9</td>\n",
       "      <td>34.6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>121.2</td>\n",
       "      <td>326.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>884.4</td>\n",
       "      <td>2737.3</td>\n",
       "      <td>300.6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4849377</td>\n",
       "      <td>3177.6</td>\n",
       "      <td>819.0</td>\n",
       "      <td>2149.5</td>\n",
       "      <td>209.1</td>\n",
       "      <td>-744.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Alaska</td>\n",
       "      <td>619500</td>\n",
       "      <td>630.8</td>\n",
       "      <td>8.4</td>\n",
       "      <td>83.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>91.4</td>\n",
       "      <td>447.6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>611.3</td>\n",
       "      <td>2688.3</td>\n",
       "      <td>429.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>736732</td>\n",
       "      <td>2760.0</td>\n",
       "      <td>427.6</td>\n",
       "      <td>2096.4</td>\n",
       "      <td>236.0</td>\n",
       "      <td>-968.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0    State  Population_1999_x  Violent Crime rate_1999  \\\n",
       "0           0  Alabama            4369862                    490.2   \n",
       "1           1   Alaska             619500                    630.8   \n",
       "\n",
       "   Murder and nonnegligent manslaughter rate_1999  Legacy rape rate /1_1999  \\\n",
       "0                                             7.9                      34.6   \n",
       "1                                             8.4                      83.5   \n",
       "\n",
       "   Revised rape rate /2_1999  Robbery rate_1999  Aggravated assault rate_1999  \\\n",
       "0                        NaN              121.2                         326.5   \n",
       "1                        NaN               91.4                         447.6   \n",
       "\n",
       "   Unnamed: 8_1999             ...              Burglary rate_1999  \\\n",
       "0              NaN             ...                           884.4   \n",
       "1              NaN             ...                           611.3   \n",
       "\n",
       "   Larceny-theft rate_1999  Motor vehicle theft rate_1999  Unnamed: 6_1999  \\\n",
       "0                   2737.3                          300.6              NaN   \n",
       "1                   2688.3                          429.1              NaN   \n",
       "\n",
       "   Population_2014_y  Property crime rate_2014  Burglary rate_2014  \\\n",
       "0            4849377                    3177.6               819.0   \n",
       "1             736732                    2760.0               427.6   \n",
       "\n",
       "   Larceny-theft rate_2014  Motor vehicle theft rate_2014  \\\n",
       "0                   2149.5                          209.1   \n",
       "1                   2096.4                          236.0   \n",
       "\n",
       "   Property crime rate_change  \n",
       "0                      -744.7  \n",
       "1                      -968.7  \n",
       "\n",
       "[2 rows x 32 columns]"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tcrime_df.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "tcrime_df.drop(['Unnamed: 0','Unnamed: 8_1999','Unnamed: 6_1999'], axis =1, inplace =True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\n",
      "Revised rape rate /2_1999                         1.0\n",
      "Unnamed: 8_2014                                   1.0\n",
      "Property crime rate_change                        0.0\n",
      "Robbery rate_2014                                 0.0\n",
      "Population_1999_x                                 0.0\n",
      "Violent Crime rate_1999                           0.0\n",
      "Murder and nonnegligent manslaughter rate_1999    0.0\n",
      "Legacy rape rate /1_1999                          0.0\n",
      "Robbery rate_1999                                 0.0\n",
      "Aggravated assault rate_1999                      0.0\n",
      "Population_2014_x                                 0.0\n",
      "Violent Crime rate_2014                           0.0\n",
      "Murder and nonnegligent manslaughter rate_2014    0.0\n",
      "Legacy rape rate /1_2014                          0.0\n",
      "Revised rape rate /2_2014                         0.0\n",
      "Aggravated assault rate_2014                      0.0\n",
      "Motor vehicle theft rate_2014                     0.0\n",
      "Violent Crime rate_change                         0.0\n",
      "Robbery rate_change                               0.0\n",
      "Population_1999_y                                 0.0\n",
      "Property crime rate_1999                          0.0\n",
      "Burglary rate_1999                                0.0\n",
      "Larceny-theft rate_1999                           0.0\n",
      "Motor vehicle theft rate_1999                     0.0\n",
      "Population_2014_y                                 0.0\n",
      "Property crime rate_2014                          0.0\n",
      "Burglary rate_2014                                0.0\n",
      "Larceny-theft rate_2014                           0.0\n",
      "State                                             0.0\n",
      "dtype: float64\n",
      "----------------------------< For better display of the missiing values>-----------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"==<checking the null values of the dataframe and sorting them as percentage of the dataframe shape>==\")\n",
    "print((tcrime_df.isnull().sum()/tcrime_df.shape[0]).sort_values(ascending=False))  \n",
    "print(\"----------------------------< For better display of the missiing values>-----------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['State', 'Population_1999_x', 'Violent Crime rate_1999',\n",
       "       'Murder and nonnegligent manslaughter rate_1999',\n",
       "       'Legacy rape rate /1_1999', 'Revised rape rate /2_1999',\n",
       "       'Robbery rate_1999', 'Aggravated assault rate_1999',\n",
       "       'Population_2014_x', 'Violent Crime rate_2014',\n",
       "       'Murder and nonnegligent manslaughter rate_2014',\n",
       "       'Legacy rape rate /1_2014', 'Revised rape rate /2_2014',\n",
       "       'Robbery rate_2014', 'Aggravated assault rate_2014', 'Unnamed: 8_2014',\n",
       "       'Violent Crime rate_change', 'Robbery rate_change', 'Population_1999_y',\n",
       "       'Property crime rate_1999', 'Burglary rate_1999',\n",
       "       'Larceny-theft rate_1999', 'Motor vehicle theft rate_1999',\n",
       "       'Population_2014_y', 'Property crime rate_2014', 'Burglary rate_2014',\n",
       "       'Larceny-theft rate_2014', 'Motor vehicle theft rate_2014',\n",
       "       'Property crime rate_change'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tcrime_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>Population_1999_x</th>\n",
       "      <th>Violent Crime rate_1999</th>\n",
       "      <th>Murder and nonnegligent manslaughter rate_1999</th>\n",
       "      <th>Legacy rape rate /1_1999</th>\n",
       "      <th>Revised rape rate /2_1999</th>\n",
       "      <th>Robbery rate_1999</th>\n",
       "      <th>Aggravated assault rate_1999</th>\n",
       "      <th>Population_2014_x</th>\n",
       "      <th>Violent Crime rate_2014</th>\n",
       "      <th>...</th>\n",
       "      <th>Property crime rate_1999</th>\n",
       "      <th>Burglary rate_1999</th>\n",
       "      <th>Larceny-theft rate_1999</th>\n",
       "      <th>Motor vehicle theft rate_1999</th>\n",
       "      <th>Population_2014_y</th>\n",
       "      <th>Property crime rate_2014</th>\n",
       "      <th>Burglary rate_2014</th>\n",
       "      <th>Larceny-theft rate_2014</th>\n",
       "      <th>Motor vehicle theft rate_2014</th>\n",
       "      <th>Property crime rate_change</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>4369862</td>\n",
       "      <td>490.2</td>\n",
       "      <td>7.9</td>\n",
       "      <td>34.6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>121.2</td>\n",
       "      <td>326.5</td>\n",
       "      <td>4849377</td>\n",
       "      <td>427.4</td>\n",
       "      <td>...</td>\n",
       "      <td>3922.3</td>\n",
       "      <td>884.4</td>\n",
       "      <td>2737.3</td>\n",
       "      <td>300.6</td>\n",
       "      <td>4849377</td>\n",
       "      <td>3177.6</td>\n",
       "      <td>819.0</td>\n",
       "      <td>2149.5</td>\n",
       "      <td>209.1</td>\n",
       "      <td>-744.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alaska</td>\n",
       "      <td>619500</td>\n",
       "      <td>630.8</td>\n",
       "      <td>8.4</td>\n",
       "      <td>83.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>91.4</td>\n",
       "      <td>447.6</td>\n",
       "      <td>736732</td>\n",
       "      <td>635.8</td>\n",
       "      <td>...</td>\n",
       "      <td>3728.7</td>\n",
       "      <td>611.3</td>\n",
       "      <td>2688.3</td>\n",
       "      <td>429.1</td>\n",
       "      <td>736732</td>\n",
       "      <td>2760.0</td>\n",
       "      <td>427.6</td>\n",
       "      <td>2096.4</td>\n",
       "      <td>236.0</td>\n",
       "      <td>-968.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 29 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     State  Population_1999_x  Violent Crime rate_1999  \\\n",
       "0  Alabama            4369862                    490.2   \n",
       "1   Alaska             619500                    630.8   \n",
       "\n",
       "   Murder and nonnegligent manslaughter rate_1999  Legacy rape rate /1_1999  \\\n",
       "0                                             7.9                      34.6   \n",
       "1                                             8.4                      83.5   \n",
       "\n",
       "   Revised rape rate /2_1999  Robbery rate_1999  Aggravated assault rate_1999  \\\n",
       "0                        NaN              121.2                         326.5   \n",
       "1                        NaN               91.4                         447.6   \n",
       "\n",
       "   Population_2014_x  Violent Crime rate_2014             ...              \\\n",
       "0            4849377                    427.4             ...               \n",
       "1             736732                    635.8             ...               \n",
       "\n",
       "   Property crime rate_1999  Burglary rate_1999  Larceny-theft rate_1999  \\\n",
       "0                    3922.3               884.4                   2737.3   \n",
       "1                    3728.7               611.3                   2688.3   \n",
       "\n",
       "   Motor vehicle theft rate_1999  Population_2014_y  Property crime rate_2014  \\\n",
       "0                          300.6            4849377                    3177.6   \n",
       "1                          429.1             736732                    2760.0   \n",
       "\n",
       "   Burglary rate_2014  Larceny-theft rate_2014  Motor vehicle theft rate_2014  \\\n",
       "0               819.0                   2149.5                          209.1   \n",
       "1               427.6                   2096.4                          236.0   \n",
       "\n",
       "   Property crime rate_change  \n",
       "0                      -744.7  \n",
       "1                      -968.7  \n",
       "\n",
       "[2 rows x 29 columns]"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tcrime_df.drop_duplicates()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "tcrime_df.drop(['Population_2014_y','Population_1999_y'],axis =1, inplace =True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "tcrime_df.rename(columns={'Population_1999_x': 'Population_1999', 'Population_2014_x': 'Population_2014'}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>Population_1999</th>\n",
       "      <th>Violent Crime rate_1999</th>\n",
       "      <th>Murder and nonnegligent manslaughter rate_1999</th>\n",
       "      <th>Legacy rape rate /1_1999</th>\n",
       "      <th>Revised rape rate /2_1999</th>\n",
       "      <th>Robbery rate_1999</th>\n",
       "      <th>Aggravated assault rate_1999</th>\n",
       "      <th>Population_2014</th>\n",
       "      <th>Violent Crime rate_2014</th>\n",
       "      <th>...</th>\n",
       "      <th>Robbery rate_change</th>\n",
       "      <th>Property crime rate_1999</th>\n",
       "      <th>Burglary rate_1999</th>\n",
       "      <th>Larceny-theft rate_1999</th>\n",
       "      <th>Motor vehicle theft rate_1999</th>\n",
       "      <th>Property crime rate_2014</th>\n",
       "      <th>Burglary rate_2014</th>\n",
       "      <th>Larceny-theft rate_2014</th>\n",
       "      <th>Motor vehicle theft rate_2014</th>\n",
       "      <th>Property crime rate_change</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>4369862</td>\n",
       "      <td>490.2</td>\n",
       "      <td>7.9</td>\n",
       "      <td>34.6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>121.2</td>\n",
       "      <td>326.5</td>\n",
       "      <td>4849377</td>\n",
       "      <td>427.4</td>\n",
       "      <td>...</td>\n",
       "      <td>-24.3</td>\n",
       "      <td>3922.3</td>\n",
       "      <td>884.4</td>\n",
       "      <td>2737.3</td>\n",
       "      <td>300.6</td>\n",
       "      <td>3177.6</td>\n",
       "      <td>819.0</td>\n",
       "      <td>2149.5</td>\n",
       "      <td>209.1</td>\n",
       "      <td>-744.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alaska</td>\n",
       "      <td>619500</td>\n",
       "      <td>630.8</td>\n",
       "      <td>8.4</td>\n",
       "      <td>83.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>91.4</td>\n",
       "      <td>447.6</td>\n",
       "      <td>736732</td>\n",
       "      <td>635.8</td>\n",
       "      <td>...</td>\n",
       "      <td>-6.0</td>\n",
       "      <td>3728.7</td>\n",
       "      <td>611.3</td>\n",
       "      <td>2688.3</td>\n",
       "      <td>429.1</td>\n",
       "      <td>2760.0</td>\n",
       "      <td>427.6</td>\n",
       "      <td>2096.4</td>\n",
       "      <td>236.0</td>\n",
       "      <td>-968.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     State  Population_1999  Violent Crime rate_1999  \\\n",
       "0  Alabama          4369862                    490.2   \n",
       "1   Alaska           619500                    630.8   \n",
       "\n",
       "   Murder and nonnegligent manslaughter rate_1999  Legacy rape rate /1_1999  \\\n",
       "0                                             7.9                      34.6   \n",
       "1                                             8.4                      83.5   \n",
       "\n",
       "   Revised rape rate /2_1999  Robbery rate_1999  Aggravated assault rate_1999  \\\n",
       "0                        NaN              121.2                         326.5   \n",
       "1                        NaN               91.4                         447.6   \n",
       "\n",
       "   Population_2014  Violent Crime rate_2014             ...              \\\n",
       "0          4849377                    427.4             ...               \n",
       "1           736732                    635.8             ...               \n",
       "\n",
       "   Robbery rate_change  Property crime rate_1999  Burglary rate_1999  \\\n",
       "0                -24.3                    3922.3               884.4   \n",
       "1                 -6.0                    3728.7               611.3   \n",
       "\n",
       "   Larceny-theft rate_1999  Motor vehicle theft rate_1999  \\\n",
       "0                   2737.3                          300.6   \n",
       "1                   2688.3                          429.1   \n",
       "\n",
       "   Property crime rate_2014  Burglary rate_2014  Larceny-theft rate_2014  \\\n",
       "0                    3177.6               819.0                   2149.5   \n",
       "1                    2760.0               427.6                   2096.4   \n",
       "\n",
       "   Motor vehicle theft rate_2014  Property crime rate_change  \n",
       "0                          209.1                      -744.7  \n",
       "1                          236.0                      -968.7  \n",
       "\n",
       "[2 rows x 27 columns]"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tcrime_df.head()[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_prescribers = pd.read_csv('./inputs/prescriber-info.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NPI</th>\n",
       "      <th>Gender</th>\n",
       "      <th>State</th>\n",
       "      <th>Credentials</th>\n",
       "      <th>Specialty</th>\n",
       "      <th>ABILIFY</th>\n",
       "      <th>ACETAMINOPHEN.CODEINE</th>\n",
       "      <th>ACYCLOVIR</th>\n",
       "      <th>ADVAIR.DISKUS</th>\n",
       "      <th>AGGRENOX</th>\n",
       "      <th>...</th>\n",
       "      <th>VERAPAMIL.ER</th>\n",
       "      <th>VESICARE</th>\n",
       "      <th>VOLTAREN</th>\n",
       "      <th>VYTORIN</th>\n",
       "      <th>WARFARIN.SODIUM</th>\n",
       "      <th>XARELTO</th>\n",
       "      <th>ZETIA</th>\n",
       "      <th>ZIPRASIDONE.HCL</th>\n",
       "      <th>ZOLPIDEM.TARTRATE</th>\n",
       "      <th>Opioid.Prescriber</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1710982582</td>\n",
       "      <td>M</td>\n",
       "      <td>TX</td>\n",
       "      <td>DDS</td>\n",
       "      <td>Dentist</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1245278100</td>\n",
       "      <td>F</td>\n",
       "      <td>AL</td>\n",
       "      <td>MD</td>\n",
       "      <td>General Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 256 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          NPI Gender State Credentials        Specialty  ABILIFY  \\\n",
       "0  1710982582      M    TX         DDS          Dentist        0   \n",
       "1  1245278100      F    AL          MD  General Surgery        0   \n",
       "\n",
       "   ACETAMINOPHEN.CODEINE  ACYCLOVIR  ADVAIR.DISKUS  AGGRENOX  \\\n",
       "0                      0          0              0         0   \n",
       "1                      0          0              0         0   \n",
       "\n",
       "         ...          VERAPAMIL.ER  VESICARE  VOLTAREN  VYTORIN  \\\n",
       "0        ...                     0         0         0        0   \n",
       "1        ...                     0         0         0        0   \n",
       "\n",
       "   WARFARIN.SODIUM  XARELTO  ZETIA  ZIPRASIDONE.HCL  ZOLPIDEM.TARTRATE  \\\n",
       "0                0        0      0                0                  0   \n",
       "1                0        0      0                0                 35   \n",
       "\n",
       "   Opioid.Prescriber  \n",
       "0                  1  \n",
       "1                  1  \n",
       "\n",
       "[2 rows x 256 columns]"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_prescribers.head()[:2]  # calling the function  head to see the first two rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 256)"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_prescribers.shape # checking the dimension of the data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['NPI', 'Gender', 'State', 'Credentials', 'Specialty', 'ABILIFY',\n",
       "       'ACETAMINOPHEN.CODEINE', 'ACYCLOVIR', 'ADVAIR.DISKUS', 'AGGRENOX',\n",
       "       ...\n",
       "       'VERAPAMIL.ER', 'VESICARE', 'VOLTAREN', 'VYTORIN', 'WARFARIN.SODIUM',\n",
       "       'XARELTO', 'ZETIA', 'ZIPRASIDONE.HCL', 'ZOLPIDEM.TARTRATE',\n",
       "       'Opioid.Prescriber'],\n",
       "      dtype='object', length=256)"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_prescribers.columns  # what are the variables/features of the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NPI</th>\n",
       "      <th>Gender</th>\n",
       "      <th>State</th>\n",
       "      <th>Credentials</th>\n",
       "      <th>Specialty</th>\n",
       "      <th>ABILIFY</th>\n",
       "      <th>ACETAMINOPHEN.CODEINE</th>\n",
       "      <th>ACYCLOVIR</th>\n",
       "      <th>ADVAIR.DISKUS</th>\n",
       "      <th>AGGRENOX</th>\n",
       "      <th>...</th>\n",
       "      <th>VERAPAMIL.ER</th>\n",
       "      <th>VESICARE</th>\n",
       "      <th>VOLTAREN</th>\n",
       "      <th>VYTORIN</th>\n",
       "      <th>WARFARIN.SODIUM</th>\n",
       "      <th>XARELTO</th>\n",
       "      <th>ZETIA</th>\n",
       "      <th>ZIPRASIDONE.HCL</th>\n",
       "      <th>ZOLPIDEM.TARTRATE</th>\n",
       "      <th>Opioid.Prescriber</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1710982582</td>\n",
       "      <td>M</td>\n",
       "      <td>TX</td>\n",
       "      <td>DDS</td>\n",
       "      <td>Dentist</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1245278100</td>\n",
       "      <td>F</td>\n",
       "      <td>AL</td>\n",
       "      <td>MD</td>\n",
       "      <td>General Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 256 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          NPI Gender State Credentials        Specialty  ABILIFY  \\\n",
       "0  1710982582      M    TX         DDS          Dentist        0   \n",
       "1  1245278100      F    AL          MD  General Surgery        0   \n",
       "\n",
       "   ACETAMINOPHEN.CODEINE  ACYCLOVIR  ADVAIR.DISKUS  AGGRENOX  \\\n",
       "0                      0          0              0         0   \n",
       "1                      0          0              0         0   \n",
       "\n",
       "         ...          VERAPAMIL.ER  VESICARE  VOLTAREN  VYTORIN  \\\n",
       "0        ...                     0         0         0        0   \n",
       "1        ...                     0         0         0        0   \n",
       "\n",
       "   WARFARIN.SODIUM  XARELTO  ZETIA  ZIPRASIDONE.HCL  ZOLPIDEM.TARTRATE  \\\n",
       "0                0        0      0                0                  0   \n",
       "1                0        0      0                0                 35   \n",
       "\n",
       "   Opioid.Prescriber  \n",
       "0                  1  \n",
       "1                  1  \n",
       "\n",
       "[2 rows x 256 columns]"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_prescribers[:2] # Exploring the dataframe, specifically the NaN values in the Credentials column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NPI                                 0\n",
       "Gender                              0\n",
       "State                               0\n",
       "Credentials                       763\n",
       "Specialty                           0\n",
       "ABILIFY                             0\n",
       "ACETAMINOPHEN.CODEINE               0\n",
       "ACYCLOVIR                           0\n",
       "ADVAIR.DISKUS                       0\n",
       "AGGRENOX                            0\n",
       "ALENDRONATE.SODIUM                  0\n",
       "ALLOPURINOL                         0\n",
       "ALPRAZOLAM                          0\n",
       "AMIODARONE.HCL                      0\n",
       "AMITRIPTYLINE.HCL                   0\n",
       "AMLODIPINE.BESYLATE                 0\n",
       "AMLODIPINE.BESYLATE.BENAZEPRIL      0\n",
       "AMOXICILLIN                         0\n",
       "AMOX.TR.POTASSIUM.CLAVULANATE       0\n",
       "AMPHETAMINE.SALT.COMBO              0\n",
       "ATENOLOL                            0\n",
       "ATORVASTATIN.CALCIUM                0\n",
       "AVODART                             0\n",
       "AZITHROMYCIN                        0\n",
       "BACLOFEN                            0\n",
       "BD.ULTRA.FINE.PEN.NEEDLE            0\n",
       "BENAZEPRIL.HCL                      0\n",
       "BENICAR                             0\n",
       "BENICAR.HCT                         0\n",
       "BENZTROPINE.MESYLATE                0\n",
       "                                 ... \n",
       "TAMSULOSIN.HCL                      0\n",
       "TEMAZEPAM                           0\n",
       "TERAZOSIN.HCL                       0\n",
       "TIMOLOL.MALEATE                     0\n",
       "TIZANIDINE.HCL                      0\n",
       "TOLTERODINE.TARTRATE.ER             0\n",
       "TOPIRAMATE                          0\n",
       "TOPROL.XL                           0\n",
       "TORSEMIDE                           0\n",
       "TRAMADOL.HCL                        0\n",
       "TRAVATAN.Z                          0\n",
       "TRAZODONE.HCL                       0\n",
       "TRIAMCINOLONE.ACETONIDE             0\n",
       "TRIAMTERENE.HYDROCHLOROTHIAZID      0\n",
       "VALACYCLOVIR                        0\n",
       "VALSARTAN                           0\n",
       "VALSARTAN.HYDROCHLOROTHIAZIDE       0\n",
       "VENLAFAXINE.HCL                     0\n",
       "VENLAFAXINE.HCL.ER                  0\n",
       "VENTOLIN.HFA                        0\n",
       "VERAPAMIL.ER                        0\n",
       "VESICARE                            0\n",
       "VOLTAREN                            0\n",
       "VYTORIN                             0\n",
       "WARFARIN.SODIUM                     0\n",
       "XARELTO                             0\n",
       "ZETIA                               0\n",
       "ZIPRASIDONE.HCL                     0\n",
       "ZOLPIDEM.TARTRATE                   0\n",
       "Opioid.Prescriber                   0\n",
       "Length: 256, dtype: int64"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_prescribers.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    DDS\n",
       "1     MD\n",
       "Name: Credentials, dtype: object"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_prescribers['Credentials'].head()[:2] #  looking the feature Credentials column for the specialists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    DDS\n",
       "1     MD\n",
       "Name: Credentials, dtype: object"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_prescribers['Credentials'][:2] #  checking if there are any NaN values in the  feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MD      7034\n",
       "M.D.    6772\n",
       "Name: Credentials, dtype: int64"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dropping NaN values \n",
    "df_prescribers.Credentials.value_counts()[:2]  #  counting as its categorical variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dumming the categorical variable/feature \n",
    "df_prescribers = pd.get_dummies(df_prescribers, columns=['Gender'], drop_first=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_prescribers.dropna(how='any', inplace=True)   # here drping any NaN values using the boolean method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NPI      0\n",
       "State    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_prescribers.isnull().sum().head()[:2]  # NaN values  in the prescribers information does not have any value as they "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "108"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df_prescribers['Specialty'].unique())   # counting the number of unique values in the speciality category"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "specialty = pd.DataFrame(df_prescribers.groupby(['Specialty']).count()['NPI']).sort_values('NPI')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(63, 1)"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "specialty.loc[specialty['NPI']< 40].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above length of the data frame for prescriber shows that there are  108 unique Specialities, and 63 of them have a length of less than 40, so adjusting those so they can be a bit more balanced and sorted into specific specialities will help in the next step while  working which speciality has the maximum or minumum over prescrption."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "otherSpecialty = list(specialty.loc[specialty['NPI']<40].index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to the FDA and DEA all specialities are not entitled to prescribe opiods and its important to  divde the  specialities in the dataset in to two main groups like other and surgery "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_prescribers.loc[df_prescribers['Specialty'].isin(otherSpecialty), 'Specialty'] = df_prescribers.loc[df_prescribers['Specialty'].isin(otherSpecialty), 'Specialty'].apply(lambda x: 'Surgery' if 'Surgery' in list(x.split( )) else 'Other')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we split in to two its important to see what credntials they have in the two groups  using the categorical variable credentials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "Credentials = pd.DataFrame(df_prescribers.groupby(['Credentials']).count()['NPI']).reset_index(False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Credentials</th>\n",
       "      <th>NPI</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(DMD)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A.N.P.</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A.P.N.</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A.P.R.N.</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A.P.R.N. N.P.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>A.P.R.N., B.C.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>A.P.R.N., W.H.N.P.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>A.R.N.P</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>A.R.N.P.</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>A.R.N.P. , PMHNP-BC</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>A.R.N.P.-B.C.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>A.T.,C. , C.S.C.S.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>ACNP</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>ACNP-BC</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>ACNP-BC, CCNS</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>ACNP-BC, MSN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>ACNP/FNP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>ACNS-BC</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>ADULT NP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>AGACNP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>AGNP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>AGPCNP-BC</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>AHCNS</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>ANNA TAVAKKOLI M.D.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>ANP B-C</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>ANP,BC</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>ANP-BC, GNP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>ANP-C</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>ANP-FNP-BC</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>ANP.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>856</th>\n",
       "      <td>RN,FNP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>857</th>\n",
       "      <td>RN,MS,CNS</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>858</th>\n",
       "      <td>RN,MSN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>859</th>\n",
       "      <td>RN,MSN,CNP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>860</th>\n",
       "      <td>RN,MSN,FNP-C</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>861</th>\n",
       "      <td>RN,NP</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>862</th>\n",
       "      <td>RN-BC, MSN, CRNP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>863</th>\n",
       "      <td>RN-C, FNP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>864</th>\n",
       "      <td>RN/BSN, MSN ,FNP-BC</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>865</th>\n",
       "      <td>RN/PC, PMHCNS-BC</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>866</th>\n",
       "      <td>RNC WHNP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>867</th>\n",
       "      <td>RNC-ANP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>868</th>\n",
       "      <td>RNC-OB</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>869</th>\n",
       "      <td>RNCNP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>870</th>\n",
       "      <td>RNCS</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>871</th>\n",
       "      <td>RNCS FNP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>872</th>\n",
       "      <td>RNP</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>873</th>\n",
       "      <td>RPA</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>874</th>\n",
       "      <td>RPA C</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>876</th>\n",
       "      <td>RPA-C MPAS</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>877</th>\n",
       "      <td>RPAC</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>878</th>\n",
       "      <td>RPH</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>879</th>\n",
       "      <td>RPH,BCOP, BCPS</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>880</th>\n",
       "      <td>RPH.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>881</th>\n",
       "      <td>RXN, NP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>882</th>\n",
       "      <td>WHCNP</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>883</th>\n",
       "      <td>WHCNP/ANP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>884</th>\n",
       "      <td>WHNP</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>885</th>\n",
       "      <td>WHNP, CFNP</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>886</th>\n",
       "      <td>WHNP-BC</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>842 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             Credentials  NPI\n",
       "0                  (DMD)    1\n",
       "1                 A.N.P.    6\n",
       "2                 A.P.N.    4\n",
       "3               A.P.R.N.    5\n",
       "4          A.P.R.N. N.P.    1\n",
       "5         A.P.R.N., B.C.    1\n",
       "6     A.P.R.N., W.H.N.P.    1\n",
       "7                A.R.N.P    2\n",
       "8               A.R.N.P.   15\n",
       "9    A.R.N.P. , PMHNP-BC    1\n",
       "10         A.R.N.P.-B.C.    1\n",
       "11    A.T.,C. , C.S.C.S.    1\n",
       "12                  ACNP   12\n",
       "13               ACNP-BC   10\n",
       "14         ACNP-BC, CCNS    1\n",
       "15          ACNP-BC, MSN    1\n",
       "16              ACNP/FNP    1\n",
       "17               ACNS-BC    1\n",
       "18              ADULT NP    1\n",
       "19                AGACNP    1\n",
       "20                  AGNP    1\n",
       "21             AGPCNP-BC    1\n",
       "22                 AHCNS    1\n",
       "23   ANNA TAVAKKOLI M.D.    1\n",
       "25               ANP B-C    1\n",
       "26                ANP,BC    1\n",
       "28           ANP-BC, GNP    1\n",
       "29                 ANP-C    7\n",
       "30            ANP-FNP-BC    1\n",
       "31                  ANP.    1\n",
       "..                   ...  ...\n",
       "856               RN,FNP    1\n",
       "857            RN,MS,CNS    1\n",
       "858               RN,MSN    1\n",
       "859           RN,MSN,CNP    1\n",
       "860         RN,MSN,FNP-C    1\n",
       "861                RN,NP    2\n",
       "862     RN-BC, MSN, CRNP    1\n",
       "863            RN-C, FNP    1\n",
       "864  RN/BSN, MSN ,FNP-BC    1\n",
       "865     RN/PC, PMHCNS-BC    1\n",
       "866             RNC WHNP    1\n",
       "867              RNC-ANP    1\n",
       "868               RNC-OB    1\n",
       "869                RNCNP    1\n",
       "870                 RNCS    2\n",
       "871             RNCS FNP    1\n",
       "872                  RNP    5\n",
       "873                  RPA    1\n",
       "874                RPA C    1\n",
       "876           RPA-C MPAS    1\n",
       "877                 RPAC   12\n",
       "878                  RPH    1\n",
       "879       RPH,BCOP, BCPS    1\n",
       "880                 RPH.    1\n",
       "881              RXN, NP    1\n",
       "882                WHCNP    2\n",
       "883            WHCNP/ANP    1\n",
       "884                 WHNP    4\n",
       "885           WHNP, CFNP    1\n",
       "886              WHNP-BC    3\n",
       "\n",
       "[842 rows x 2 columns]"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Credentials[Credentials['NPI'] < 20]  # checking those with less than 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the above credentials list ,  it shows the existance of multiple unique credentials. Those are not the category that they are not going to predict and its important to let the data clean by droping them. NPI is unique identification for physicians and since i cleaned my data, i will also drop NPI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_df = df_prescribers.drop(['NPI', 'Credentials'], axis=1)  # droping the redundunt variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>Specialty</th>\n",
       "      <th>ABILIFY</th>\n",
       "      <th>ACETAMINOPHEN.CODEINE</th>\n",
       "      <th>ACYCLOVIR</th>\n",
       "      <th>ADVAIR.DISKUS</th>\n",
       "      <th>AGGRENOX</th>\n",
       "      <th>ALENDRONATE.SODIUM</th>\n",
       "      <th>ALLOPURINOL</th>\n",
       "      <th>ALPRAZOLAM</th>\n",
       "      <th>...</th>\n",
       "      <th>VESICARE</th>\n",
       "      <th>VOLTAREN</th>\n",
       "      <th>VYTORIN</th>\n",
       "      <th>WARFARIN.SODIUM</th>\n",
       "      <th>XARELTO</th>\n",
       "      <th>ZETIA</th>\n",
       "      <th>ZIPRASIDONE.HCL</th>\n",
       "      <th>ZOLPIDEM.TARTRATE</th>\n",
       "      <th>Opioid.Prescriber</th>\n",
       "      <th>Gender_M</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>TX</td>\n",
       "      <td>Dentist</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AL</td>\n",
       "      <td>General Surgery</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>134</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NY</td>\n",
       "      <td>General Practice</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AZ</td>\n",
       "      <td>Internal Medicine</td>\n",
       "      <td>0</td>\n",
       "      <td>43</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NV</td>\n",
       "      <td>Hematology/Oncology</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 254 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  State            Specialty  ABILIFY  ACETAMINOPHEN.CODEINE  ACYCLOVIR  \\\n",
       "0    TX              Dentist        0                      0          0   \n",
       "1    AL      General Surgery        0                      0          0   \n",
       "2    NY     General Practice        0                      0          0   \n",
       "3    AZ    Internal Medicine        0                     43          0   \n",
       "4    NV  Hematology/Oncology        0                      0          0   \n",
       "\n",
       "   ADVAIR.DISKUS  AGGRENOX  ALENDRONATE.SODIUM  ALLOPURINOL  ALPRAZOLAM  \\\n",
       "0              0         0                   0            0           0   \n",
       "1              0         0                   0            0         134   \n",
       "2              0         0                   0            0           0   \n",
       "3              0         0                  21            0           0   \n",
       "4              0         0                   0            0           0   \n",
       "\n",
       "     ...     VESICARE  VOLTAREN  VYTORIN  WARFARIN.SODIUM  XARELTO  ZETIA  \\\n",
       "0    ...            0         0        0                0        0      0   \n",
       "1    ...            0         0        0                0        0      0   \n",
       "2    ...            0         0        0                0        0      0   \n",
       "3    ...            0         0        0                0        0      0   \n",
       "4    ...            0         0        0               17       28      0   \n",
       "\n",
       "   ZIPRASIDONE.HCL  ZOLPIDEM.TARTRATE  Opioid.Prescriber  Gender_M  \n",
       "0                0                  0                  1         1  \n",
       "1                0                 35                  1         0  \n",
       "2                0                 25                  0         0  \n",
       "3                0                  0                  1         1  \n",
       "4                0                  0                  1         1  \n",
       "\n",
       "[5 rows x 254 columns]"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_df.head() # reading the first five rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dumming the categorical variables and Gender_M is from previous dumming above and included here.\n",
    "clean_df = pd.get_dummies(clean_df, columns = ['Gender_M', 'Specialty', 'State'], drop_first = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kirosgebremariam/anaconda3/envs/dsi/lib/python3.6/site-packages/matplotlib/axes/_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.\n",
      "  warnings.warn(\"The 'normed' kwarg is deprecated, and has been \"\n"
     ]
    },
    {
     "ename": "ValueError",
     "evalue": "color kwarg must have one color per dataset",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-139-0657da1d463e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0msns\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdistplot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclean_df\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/envs/dsi/lib/python3.6/site-packages/seaborn/distributions.py\u001b[0m in \u001b[0;36mdistplot\u001b[0;34m(a, bins, hist, kde, rug, fit, hist_kws, kde_kws, rug_kws, fit_kws, color, vertical, norm_hist, axlabel, label, ax)\u001b[0m\n\u001b[1;32m    216\u001b[0m         \u001b[0mhist_color\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhist_kws\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"color\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcolor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    217\u001b[0m         ax.hist(a, bins, orientation=orientation,\n\u001b[0;32m--> 218\u001b[0;31m                 color=hist_color, **hist_kws)\n\u001b[0m\u001b[1;32m    219\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mhist_color\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mcolor\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    220\u001b[0m             \u001b[0mhist_kws\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"color\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhist_color\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/dsi/lib/python3.6/site-packages/matplotlib/__init__.py\u001b[0m in \u001b[0;36minner\u001b[0;34m(ax, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1853\u001b[0m                         \u001b[0;34m\"the Matplotlib list!)\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlabel_namer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1854\u001b[0m                         RuntimeWarning, stacklevel=2)\n\u001b[0;32m-> 1855\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0max\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1856\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1857\u001b[0m         inner.__doc__ = _add_data_doc(inner.__doc__,\n",
      "\u001b[0;32m~/anaconda3/envs/dsi/lib/python3.6/site-packages/matplotlib/axes/_axes.py\u001b[0m in \u001b[0;36mhist\u001b[0;34m(***failed resolving arguments***)\u001b[0m\n\u001b[1;32m   6502\u001b[0m             \u001b[0mcolor\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmcolors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_rgba_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcolor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   6503\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcolor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mnx\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 6504\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"color kwarg must have one color per dataset\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   6505\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   6506\u001b[0m         \u001b[0;31m# If bins are not specified either explicitly or via range,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: color kwarg must have one color per dataset"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA5UAAAIBCAYAAADZKJsyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xm4ZFV59/1vA80gNAjITIfB6A1BE7FliIKiBB7jhBDwkQQFjJpEn9dAjEoeAY0kTlGJGjW+ifqKgkEZNMQRVJA4kLYdofVGUZSxGURQkKGh3z/2LnrV7jrnVK1TbZ/ifD/XdS72Xmftte9TXf/8WHuvtWDVqlVIkiRJklRjvXVdgCRJkiRpchkqJUmSJEnVDJWSJEmSpGqGSkmSJElSNUOlJEmSJKmaoVKSJEmSVM1QKUmSJEmqZqiUJEmSJFUzVEqSJEmSqhkqJUmSJEnVDJWSJEmSpGqGSkmSJElStQ3WdQF66Fq2bNmqdV2DJEmSpOEtWbJkwajXOFMpSZIkSarmTKXWuiVLlqzrEgQsW7YM8N9Dk8PvrCaJ31dNGr+z6up9J2o4UylJkiRJqmaolCRJkiRVM1RKkiRJkqoZKiVJkiRJ1SZyoZ6I2BA4EjgKWAJsA9wDXA8sBc4ELszMtbalRUQcBBwL/CGwE7A+cAPwA+Bs4NzMvGuE8T4NPKOilDMz85ghxt8ZeDFwCLAHsAi4BbgG+Azwkcz8ScX9JUmSJM1jEzdTGRF7A9+hCY7PBRYDGwNbAHsCLwQ+D1wYEYvXwv23jogLgC8DxwEBbAZsAuwOPBM4A/h+RBwwwtCPH3OpD4qIE4ArgdcBTwS2AhYCOwD7Aq8HfhgRJ0fE+murDkmSJEkPPRMVKiNif+ASmvDYswq4Ebit0/1g4OsRsdMY7781cCnwrM6vbmtrKGdGdwe+HBGHDDHu9sD2RdM1wFVD/tw4w9hvBU6nCb099wDXtv/tWQicBnxgpnolSZIkqWdiHn9tA915NI9tAqwE3gK8KzNvavv8HvAG4E/aPjsBF0TEPpl5/xjKOIv+QHsucGpmLm/vvy3wCuAkmsdhNwDOiYg/yMyrpxm3u0HQ3pl562yLjYijgVcVTT8DTgT+KzPvi4gNgGfThM5d2j7HRsR3M/P02d5fkiRJ0kPfJM1UnkbzuGbPMZl5ci9QAmTm8sw8Enhr0W9v4M9ne/OIeB5waNH0vsw8shco2/vflJknA4cDvRC7OfDGGYYvH339+ZgC5SKasNjzU2C/zDw/M+9r612ZmefThNofFH1fHxGPmG0NkiRJkh76JiJURsQ2wIuKpjMz8+xpLjmJ5jHVnlMiYrZ/60nF8U+AE6bqmJkXAG8qmp4fETHN2GWo/FZdeWs4HtiuOH9pZq4Y1LENsUfQzP5CE4RPHFMdkiRJkh7CJiJUAkcDGxXn0z6a2a76+paiaWfgybU3j4jfp5nx7HlvZt47w2WnA70+C2j+hqmUoXLZ6BUOdFxxfHlmXjRd58z8IfDJoulPx1SHJEmSpIewSQmVTy+Or83MYYLXhUC5pcdRY7o/9IevgTLzF/TPlg68f/uu6O8UTbOeqYyI7YDHFU0z1tv6VHG8a0TsM9taJEmSJD20TUqoLMPN14e5oJ1J/HbR9KRZ3H/f4vjmzLxqyOu+URzvGRFbDujT3UpkHI+/7kMzO9oz1GdGf70wu89MkiRJ0jww51d/bWfdykVjlk/Vd4BkdTCKiFjQPho7qr1mcf+eBcAerBnwylB5PbAiIv6I5vHTJ9I8ursesAL4H5rZxI9n5kqmtlfnfNiafwrcR7O9CPSvdCtJkiRJa5iEmcrFnfNrRrj2uuJ4Y/oXrqmtofb+sHrbjlIZKu+kCY4X0iy0E8CmNHtM7go8DzgTuCIiDhiy3lUD6hio3Xal3PdyUL2SJEmS9KBJCJXbds5vGeHa7tYcW41684h4GE2wW1v3L0Plo4AnFOe30wTC+zrXPBr4UkQcM8V9y8/s9t4WIkMqax7585IkSZI0v8z5x1+BzTrnvx7h2m7fRXPp/hGxOfDITp9bgTcDH+ltARIRGwEHAa8FDmz7LQQ+GBE/z8yvTFPzKPV2+9d8XmtYtmxcC9pqHPz30KTxO6tJ4vdVk8bvrMZhEmYqN+ycjzLr1u27cGCvdXf/velfUOfbwO9n5tvKPSUz857M/DxNsHxnZ7x/i4ju/xwoax6l3m7/ms9LkiRJ0jwyCTOVCzrnoyy00712rt3/+8AzgN1o3l98Z2ZeP1XnzHwgIk4EHgs8rW1+NM27lmdNcd9RFyYax2fWZ8mSJeMeUhV6/yfSfw9NCr+zmiR+XzVp/M6qazaz1pMQKu/tnI8ye9b9++6eS/dv97L87CjFZOaqiHg9q0MlwLPpD5VlzaPONpY113xekiRJkuaRSXj89Y7O+aYDew3WfSfwzor7376O7z/IV4FfFuf7dH5ffmaj1Av9NY+rXkmSJEkPUZMQKm/unG89wrXd1UtXDOw1jcy8m/5w9Vu9/yCZ+QBwddHU3Sql/MweHhHrjzB8WfNY6pUkSZL00DUJofLqzvmOI1y7U3F8F3DbGGqovT/AtZX3H+Q3xXE3NF5dHK/HkPtztuFz+6JpnPVKkiRJegia86EyM28Bbiqa9hjh8rLvFZk56qI1D147hvuvApZ3O0TEehGxbUTsPGJNjyiOu3tnXtE5H7bm3el/B/PyEWuSJEmSNM/M+VDZuqw43m+YCyJiQ5otO3qWjun+iyNihyGv2784/kFm9u0ZGRFJs4XHCuDzwxYTEQ+nf3/L73a6/A/9q74O9ZnRXy/M7jOTJEmSNA9MSqj8XHH86IgYZubtUGCTKcaYzf0BDpvpgojYCjhghvvfwOp/gz0jYtch6/nf9P/bXVT+MjNvBb5ZNM1Y74B+twDuhitJkiRpWpMSKs+hf5uM10zXOSIWAK8umlYwi1CZmcuB7xRNJ7QzodM5ESj7fHhAn08VxwuAv52pljasvrZouhM4Y0DXcouR/SLiKTOMuwf9ofIj7YJAkiRJkjSliQiVmXkT/cHpuIh46TSXvBk4sDh/R2beN8sy3lYcB/CBNryuISKeDfxd0fTZzPzegK5n0L81yF9FxJFTFRARmwPnAYuL5rdk5qAFiD4I3FqcnxURu0wx7lbtuL09Ku8G3jlVHZIkSZLUMxGhsnUK/QvS/GtEvCciHgxYEbFHRHyc/lnKK5giIEXEQRGxqvi5eJr7nwV8pTg/BrgoIp5QjLdNRLyBJqD1VmS9Ezhh0IDtY6qvLJrWA86OiH+OiN2KcR8WEc8HvgWUM45fBd40xdh30D+juyOwNCJeGBEbteOuHxGH0TzmumfR9x8y82eDxpUkSZKk0sSEysy8ETgC6C12swB4GfDziFgREbcAPwCOKi67ETg8M+8Zw/1XAc8Hflg0P40mqP0yIq5v73cKq2f8VgIvyMwrpxn3g8BpRdN6wF8DP4mImyPiGprZzI/RvzjP14FnZObKacb+APDPRdM2NI/h/qod99fAJ4Fdiz6fAN441ZiSJEmSVJqYUAmQmZfSBLnulhnbAlt32i4DnpSZPxrj/W+gmSn8dOdXWwA70P95Xg88MzPPH2LcU4Hn0b91CjTbhuxM/zYf9wFvAf6onY2caewTgVfR7NPZs7Add+Oi7X6aR3yPnsXWK5IkSZLmmQ1m7jK3ZObSiHgczQqoRwCPB7ajmbm8kWY7jf8APrU2Fppp3+98VkQcBBwNPJkmUD4M+AXN9h6folno5lcjjPuJiLigHfMZwBKasLwBzUJDV9OE2Y9n5tUj1vy2iPgYcBzwTJr9KLemCZpXAV8G/i0zfzjlIJIkSZI0wMSFSoD2kc8z25/ZjHMxTRitvfbi2dx/wJh3Ax9qf8YqM68D/rH9kSRJkqSxmKjHXyVJkiRJc4uhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdU2WNcFjCoiNgSOBI4ClgDbAPcA1wNLgTOBCzNz1Vqs4SDgWOAPgZ2A9YEbgB8AZwPnZuZdI465APhj4LnA/sCOwBbAr4AVwDeA/wQ+lZkPDDnmUcDHR6mjsDAzV1ZeK0mSJGmemKiZyojYG/gOTXB8LrAY2JgmfO0JvBD4PHBhRCxeC/ffOiIuAL4MHAcEsBmwCbA78EzgDOD7EXHACOM+Afg+8GngJcBjga1pQv+WwB7t/c4DroiIfYYc+vHD1iBJkiRJNSYmVEbE/sAlNOGxZxVwI3Bbp/vBwNcjYqcx3n9r4FLgWZ1f3dbWUM6M7g58OSIOGWLcZwNfA/bq/OpXwM/b/5b2AP47Ig4bouwyVN4OXDXCz1qb6ZUkSZL00DERj7+2ge48YFHbtBJ4C/CuzLyp7fN7wBuAP2n77ARcEBH7ZOb9YyjjLPoD7bnAqZm5vL3/tsArgJNoHofdADgnIv4gM6+e4u8KmsdTFxbN/w78c2ZeUfT7feBvgRe0TRsCH4uIfTPz8mlq3rs4Pi0z3z7jXylJkiRJI5iUmcrTgB2K82My8+ReoATIzOWZeSTw1qLf3sCfz/bmEfE84NCi6X2ZeWQvULb3vykzTwYOB3ohdnPgjdMM/Q6ax3d7jsvMl5SBsh37e5n5QuDlRfMm7fVT1byY5n3Tnm9NU4ckSZIkVZnzoTIitgFeVDSdmZlnT3PJSTSPqfacEhGz/TtPKo5/ApwwVcfMvAB4U9H0/HZGsk9E7EyzME/PRzPzw9MVkZnvpZkh7TlkmndHu+9Tfnu6sSVJkiSpxpwPlcDRwEbF+enTdW5XfX1L0bQz8OTam7ePnpaPkb43M++d4bLTgV6fBTR/Q9cz29/1vGfIkv69c/6UKfqVofKqzPzlkONLkiRJ0tAmIVQ+vTi+NjOXDXHNhUC5pcdRY7o/wCdnuiAzf0H/bOmg+z+2OH4A+OaQ9fyoc77jFP3KUOmjr5IkSZLWikkIleX2GV8f5oJ2JrF83PNJs7j/vsXxzZl51ZDXfaM43jMituz8/l0071++HHjdCHtCdse5Z4p+hkpJkiRJa92cXv01IrYDHlE0LZ+q7wDJ6jAZEbGgfTR2VOVWH6Pev2cBzVYgD4bizLwSuLKinj/qnF/d7dCuRFvOYC6LiF2B42m2W9mTZiXdW4Ef0+zt+YHMvKGiHkmSJEnz2FyfqewuQnPNCNdeVxxvDGw3hhpq7w+wS+X9HxQRG9O/Auy9wJcGdF3SOX8JTXg8lSZob0Wzjcn2wAE0q+teFRGnjmFRI0mSJEnzyFwPENt2zm8Z4dpbO+dbjXrziHgYsOm6uv8Ap9EsPNTzicz81YB+3ZVfj6LZOxOaIHoN0L1uE+DvgfMjYiGSJEmSNIQ5/fgrsFnn/NcjXNvtu2gC7/+gdq/MVxZN9wCvnaJ7N1QCnAH8C/DN3mPA7VYnfwH8H5qZS4DnAO8G/nI29ZaWLRtmbSX9tvjvoUnjd1aTxO+rJo3fWY3DXA+VG3bO7xvh2m7fmtm3dX1/ACLiGTShsNyC5K8z82dTXFKGynuBozLzP7udMjOBv4mI84HPsnpW9i8i4szMvLR7jSRJkiSV5nqoXNA5H2Whne61k3h/IuJw4GP079X575n5/mkuOwbYHdgN+P6gQFnKzEsj4mXAh4vmU4BD66rut2RJ9xVPrQu9/xPpv4cmhd9ZTRK/r5o0fmfVNZtZ67keKu/tnI8y29f92+6etPu3Qe/d9L/7ei4zPJqamV8Fvjri7T4KnAw8qj0/KCIWTfHOpiRJkiQBc3+hnjs655sO7DVY9x3GOyvuf/u6uH9ErB8R7wTeQ/+/0VnA8zPz/hHqGEpmPgB8rmhaCPzBuO8jSZIk6aFlrofKmzvnW49wbXe11RWj3jwz76Y/DK71+0fEljTh7hWdX70LOCYzV45Qw6iu6pzXbsMiSZIkaZ6Y64+/Xt0533GEa3cqju8CbptFDXvN8v4A1850QUTsAVwA/G7R/ADw6sx8+wj3rvWbzvn6A3tJkiRJUmtOz1Rm5i3ATUXTHiNcXva9oreNRoUrxnD/VcDy6TpHxIHA1+gPlHcBR9YEyohYFBGPjIhRguEjOuej7MspSZIkaR6a06GydVlxvN8wF0TEhsDeRdPSMd1/cUTsMOR1+xfHP8jMKfe4jIg/Br4AbFk0XwccmJnnD11pM9apEfEbmvdRf8zgPSun8oTieBXwvVHuLUmSJGn+mYRQWS4e8+j2EdGZHApsMsUYs7k/wGEzXRARWwEHDHP/iHgacD6wcdH8HWC/zPzWCHX2/Lwz1jOGuaitudxC5LvtTLEkSZIkTWkSQuU59G/t8ZrpOkfEAuDVRdMKZhEqM3M5TcjrOaGdCZ3OiUDZ58ODOkXELjR/X7kH5ReBJ2fmdRXlAnwauK84/8uI6K5EO8hp9K9u+97K+0uSJEmaR+Z8qMzMm4AziqbjIuKl01zyZuDA4vwdmXnfVJ2H9LbiOIAPtOF1DRHxbODviqbPZuYaj5G27zp+jP5HXr8CPGs2e0Nm5s3AfxRN2wMfiogpF2WKiFcALyualtP/mUuSJEnSQHM+VLZOoX/RmH+NiPdExOJeQ0TsEREfp3+W8grgnYMGjIiDImJV8XPxNPc/iybw9RwDXBQRD76DGBHbRMQbgPNYvWrqncAJU4x5NPCHxfl1wBHtNiaz9Sr6P68/Af47Ig4uF+6JiCUR8Qn6P6O7gBdk5j1jqEOSJEnSQ9xc31IEgMy8MSKOAD4DbAYsoJlZe1lE3EQT4rp7SN4IHD6OcJSZqyLi+cCXWL2q69OApRFxO00Q247+kL6SJpxdOcWwr+2cbwZcFhGjlnduZvY9EpyZKyLif9E8Svvwtnk/4CLgroi4hWYfzc06Y90JPLPyXU5JkiRJ89CkzFSSmZfSBLkrOr/aljUD5WXAkzLzR2O8/w3AU2jeWSxtAexA/2d5PU04G7hya0Q8hjW3J9kCeGTFz3ZT1PstmhVwL+786mHA77BmoPwKzeJAlwwaT5IkSZIGmYiZyp7MXBoRjwP+N3AEzXYZ29HMXN4I/A/N+4SfyswH1sL9bwKeFREH0Ty++mSaQPkw4BfAd4FPAR+Z4b3IUfa7rJaZVwNPjYgn0tR7AE2g3By4DbgBuBQ4LzO/9NuoSZIkSdJDy0SFSoDMXAmc2f7MZpyLacJo7bUXz+Le59Teu/J+XwO+9tu6nyRJkqT5Y2Ief5UkSZIkzT2GSkmSJElSNUOlJEmSJKmaoVKSJEmSVM1QKUmSJEmqZqiUJEmSJFUzVEqSJEmSqhkqJUmSJEnVDJWSJEmSpGqGSkmSJElSNUOlJEmSJKmaoVKSJEmSVM1QKUmSJEmqZqiUJEmSJFUzVEqSJEmSqhkqJUmSJEnVDJWSJEmSpGqGSkmSJElSNUOlJEmSJKmaoVKSJEmSVM1QKUmSJEmqZqiUJEmSJFUzVEqSJEmSqhkqJUmSJEnVDJWSJEmSpGqGSkmSJElSNUOlJEmSJKmaoVKSJEmSVM1QKUmSJEmqZqiUJEmSJFUzVEqSJEmSqhkqJUmSJEnVDJWSJEmSpGqGSkmSJElSNUOlJEmSJKmaoVKSJEmSVM1QKUmSJEmqZqiUJEmSJFUzVEqSJEmSqhkqJUmSJEnVDJWSJEmSpGqGSkmSJElSNUOlJEmSJKmaoVKSJEmSVM1QKUmSJEmqZqiUJEmSJFUzVEqSJEmSqhkqJUmSJEnVDJWSJEmSpGqGSkmSJElSNUOlJEmSJKmaoVKSJEmSVM1QKUmSJEmqZqiUJEmSJFUzVEqSJEmSqhkqJUmSJEnVDJWSJEmSpGqGSkmSJElSNUOlJEmSJKmaoVKSJEmSVM1QKUmSJEmqZqiUJEmSJFUzVEqSJEmSqhkqJUmSJEnVDJWSJEmSpGqGSkmSJElSNUOlJEmSJKmaoVKSJEmSVM1QKUmSJEmqZqiUJEmSJFXbYF0XUCMiNgSOBI4ClgDbAPcA1wNLgTOBCzNz1Vqs4SDgWOAPgZ2A9YEbgB8AZwPnZuZdFeM+Hvhz4EBgF2Aj4Ebgx8A5wNmZeVvFuI8GXgI8FXgksCmwArga+CRwVmbeMOq4kiRJkua3iZupjIi9ge/QBMfnAouBjYEtgD2BFwKfBy6MiMVr4f5bR8QFwJeB44AANgM2AXYHngmcAXw/Ig4YYdxNIuKDwDLgZcBjgc1pQuUuwMHA+4CMiOeOMO56EfEmYDnwtzQh/OHAQmBn4ADgbcCVEfHSYceVJEmSJJiwUBkR+wOX0ITHnlU0M3nd2buDga9HxE5jvP/WwKXAszq/uq2toZwZ3R34ckQcMsS4G9ME4eM7v7qDZvb1gaJtG+D8iOj2HTTuAuBjwEk0M6k9dwHXASuLts2A90fE62caV5IkSZJ6JiZUtoHuPGBR27QS+Edg+8zcITO3AvYCzi0u2wm4ICLWZzzOoj/QngvslZlbZeYOwPZtTfe3v98AOCcidp1h3HfRPO7acwmwb2ZukZk7AVsBr6QJgz3vj4j9Zhj374DnFeffA/4IWJSZO9PMhL4IuLXo87qIOGKGcSVJkiQJmKBQCZwG7FCcH5OZJ2fmTb2GzFyemUcCby367U3zjuKsRMTzgEOLpvdl5pGZuby4/02ZeTJwOKuD5ebAG6cZdz/gxUXTBcDBmbm0GPf2zHwHcBBwZ9u8EHjHNOP+DnBK0bQUeGJmfjEzH2jH/U1mfgjYl2amteft7XurkiRJkjStiQiVEbENzYxaz5mZefY0l5xE85hqzykRMdu/9aTi+CfACVN1zMwLgDcVTc+PiJii+2uABe3xHcDxmXn/oI5t0PzroumJ0zxeewLNu6bQBNwXZOadgzpm5k+AY4qmXWkWIZIkSZKkaU1EqASOplmwpuf06Tq3q76+pWjaGXhy7c0j4vdpZjx73puZ985w2elAr88Cmr+hO+6WwHOKpjMy89Zuv44P0z+r+GcDxl0PeEHR9NnMzOkGzcwvAt+cblxJkiRJ6pqUUPn04vjazFw2xDUX0v8O4lFjuj80W3BMKzN/Qf9s6aD7H0L/AjrDjLsS+HTRdHhEdLeGeQLwiFHGbX2qOD4wIrYb8jpJkiRJ89SkhMp9iuOvD3NBO5P47aLpSbO4/77F8c2ZedWQ132jON6znZmcatxVwGUV424OPGaacWHIz6wz7no0e3BKkiRJ0pTmfKhsZ8vKWbflU/UdoHzkM9otNmrsNYb7LwD2mGbcn2fmryvGhf4VabvjrgSuHNO4kiRJktRnzodKYHHn/JoRrr2uON4YqH2cs6yh9v4Au6yDcW9sH5kdxg3074nZHVeSJEmS+kxCqNy2c37LCNd2F73ZatSbR8TDgE3X0v3Lv22dj9uGzzumGVeSJEmS+nQXeJmLNuucD/uI6KC+i+bY/cux58K4vf4Pn2LcKsuWDbOukn5b/PfQpPE7q0ni91WTxu+sxmESZio37JzfN8K13b4L59j9y7GHHrfdx3LVuMcd0L/m85IkSZI0j0zCTGV3cZ1VA3sNd+1cu3/5+1HGHeW+o447js+sz5IlS8Y9pCr0/k+k/x6aFH5nNUn8vmrS+J1V12xmrSdhpvLezvkos2fd0Hz3HLt/OfbQ40bE+vSHv7GM2yprrvm8JEmSJM0jkxAq7+icbzqw12DddwLvrLj/7Wvx/uXfNhfG7Y5d83lJkiRJmkcmIVTe3DnfeoRru6uXrhj15pl5N/3hapz3L/+2dT5uRGxAf6gc+fOSJEmSNL9MQqi8unO+4wjX7lQc3wXcNoYaau8PcO06GHf7iBj233kH+r8T3XElSZIkqc+cD5WZeQtwU9G0xwiXl32vyMzaxXCuGMP9VwHLpxl3t4jYqGJcgMunGXcjYNcxjStJkiRJfeZ8qGxdVhzvN8wFEbEhsHfRtHRM918cETsMed3+xfEPMrO7Z2Q57gbA4yvGvQO4cppxYcjPrDPuKsCNiyRJkiRNa1JC5eeK40dHxDCzhYcCm0wxxmzuD3DYTBdExFbAATPc/4vAyhHH3QB4RtF0UWau7HT7Nv3vQ8447oB+385M36mUJEmSNK1JCZXn0L9Nxmum6xwRC4BXF00rmEWozMzlwHeKphPamdDpnAiUfT48YNzbgU8XTS9uw+h0jgW2n2HcVcB/FE2HR8Sjphs0Ig4Gyo2K1hhXkiRJkromIlRm5k3AGUXTcRHx0mkueTNwYHH+jsy8b5ZlvK04DuADbXhdQ0Q8G/i7oumzmfm9Kcb9p+J4a+ATU71bGRH7AO8smi4H/muKcd/J6iC+IXBeRGw5xbi7AR8tmlYAH5piXEmSJEl60ESEytYpwC3F+b9GxHsiYnGvISL2iIiP0z9LeQX9QYyi/0ERsar4uXia+58FfKU4Pwa4KCKeUIy3TUS8ATgPWL9tvhM4YapBM/Or9Ae6pwHfiIin9kJrRCyKiBOAi1m97+T9wMsz84Epxv0p8Nai6THqsvwoAAAgAElEQVTAsog4LCLWb8fdOCKOpXnftJz9fGVm/mqqmiVJkiSpZ4N1XcCwMvPGiDgC+AywGbAAeBnwsoi4iSbEdfdkvBE4PDPvGcP9V0XE84EvsXqV1KcBSyPidpotS7ajP6ivBF6Qmd2FdLr+imaF1t47mI9r73NnRPySJvCt37nmxMz8CtP7+7bWI9vz3YBPAndHxC1tvQs717w9M8+cYVxJkiRJAiZrppLMvJQmyF3R+dW2rBkoLwOelJk/GuP9bwCeQv97kABbsOYej9cDz8zM84cY99fA02neYyy3PdmUZk/KMlD+EjgmM989xLgrgefTPLpbPv67MbAz/YHybuBvM/NvZxpXkiRJknomKlQCZOZSmpm8Y2geM70a+A1NKLoa+DhwBPDEzPzJWrj/TZn5LOCpwP8L/BC4nSa0rQC+ALwc2CMzvzDCuHdm5nHAPsA/A98HfkEz23kLcAlwEvCoUWYSM/P+zHwV8FjgjTTbhNzcjnsb8A3gNCAy8+3DjitJkiRJMEGPv5baGbgz25/ZjHMxzWO0tddePJv7TzHuMtbC/pCZmcBr2x9JkiRJGouJm6mUJEmSJM0dhkpJkiRJUjVDpSRJkiSpmqFSkiRJklTNUClJkiRJqmaolCRJkiRVM1RKkiRJkqoZKiVJkiRJ1QyVkiRJkqRqhkpJkiRJUjVDpSRJkiSpmqFSkiRJklTNUClJkiRJqmaolCRJkiRVM1RKkiRJkqoZKiVJkiRJ1QyVkiRJkqRqhkpJkiRJUjVDpSRJkiSpmqFSkiRJklTNUClJkiRJqmaolCRJkiRVM1RKkiRJkqoZKiVJkiRJ1QyVkiRJkqRqhkpJkiRJUjVDpSRJkiSpmqFSkiRJklTNUClJkiRJqmaolCRJkiRVM1RKkiRJkqoZKiVJkiRJ1QyVkiRJkqRqhkpJkiRJUjVDpSRJkiSpmqFSkiRJklTNUClJkiRJqmaolCRJkiRVM1RKkiRJkqoZKiVJkiRJ1QyVkiRJkqRqhkpJkiRJUjVDpSRJkiSpmqFSkiRJklTNUClJkiRJqmaolCRJkiRVM1RKkiRJkqoZKiVJkiRJ1QyVkiRJkqRqhkpJkiRJUjVDpSRJkiSpmqFSkiRJklTNUClJkiRJqmaolCRJkiRVM1RKkiRJkqoZKiVJkiRJ1QyVkiRJkqRqhkpJkiRJUjVDpSRJkiSpmqFSkiRJklTNUClJkiRJqmaolCRJkiRVM1RKkiRJkqoZKiVJkiRJ1QyVkiRJkqRqhkpJkiRJUjVDpSRJkiSpmqFSkiRJklTNUClJkiRJqmaolCRJkiRV22BdFzCqiHg88OfAgcAuwEbAjcCPgXOAszPztrVcw87Ai4FDgD2ARcAtwDXAZ4CPZOZPKsZ9JHAszd/2aGAr4AHgVuBy4EvAhzLz1iHHWwDcAWw2ai3ASzLz3yuukyRJkjSPTEyojIhNgPcAxw/49S7tz8HAGyLipZn5ybVUxwnAG4FNOr/aof3ZF3htRLwBeFNm3j/EmIuAfwaOY/Ds8cOAxcAfA69vx/6nzFw1w9CPpi5QSpIkSdJQJiJURsTGwOdpZvBKdwC/BrZndRjbBjg/Il6UmR8acx1vBV7Vab4HuLm970Zt20LgNOB3aYLidGNuDVwC7NX51X3ATe1Y2wAL2vZNgbcAj4uIP5shWD6+OH4A+Ol0tXTcPkJfSZIkSfPURIRK4F30B8pLgFdl5lKAiNiC5pHY02hm9QDeHxHLM/OycRQQEUfTHyh/BpwI/Fdm3hcRGwDPBk6nmTUFODYivpuZp08z9CfoD5TfAU4FLszMu9t7bw38KfD3wJZtv6NpHvk9dZqxy1C5PDMfO01fSZIkSRrZnF+oJyL2o3l/secC4OBeoATIzNsz8x3AQcCdbfNC4B1jqmERTVjs+SmwX2aen5n3tTWszMzzgSXAD4q+r4+IR0wx7lHAU4umzwJ/mJkX9AJlO/atmfluYB/ghqL/qyNi12lKL0Plt6bpJ0mSJElV5nyoBF7D6kc/7wCOn+o9xTZo/nXR9MSIOGQMNRwPbFecvzQzV0xRw63AEcDKtmlzmhnNQV5aHN8BHFOGyQFjX9W5ZiPgBdPUvXdxbKiUJEmSNHZzOlRGxJbAc4qmM4ZY+fTDNKvB9vzZGEo5rji+PDMvmq5zZv4QKBcK+tNun4jYlP5Zyo9l5i+GqOXT9M9WPmVQp4jYjdWPygIsG2JsSZIkSRrJnA6VNFt2rF+cz7iia2aupAlePYe37ztWiYjtgMeNUkPrU8XxrhGxT+f3e9L/tw317me7MM9VRdOOU3TtLtLznWHGlyRJkqRRzPVQuW9xvIohgxfwjeJ4c+Axs6hhH1Y/fgvw9YoaAJ7UOf8R8L9oHq09GVjK8MoZyHum6FOGyh9l5q9HGF+SJEmShjLXV38tV0X9+QjBKDvne1I/U9fd6mP5kNf9lGZbkIVFDQ/KzNuBL4xaTETsCPxe0XT1FF3LULms3efzSJr3PZfQvCP6G+B6mtV0/yMzLxm1HkmSJEnz21yfqVxcHF8zwnXXdc53Gdhr9BpWDRh7oHYxofLdztnUUPob+mdOPztFv3KRnp1oth85A3guzd+0IbAFTdj9S+DiiPhCROw0pjolSZIkzQNzPVRuWxzfMsJ13cV8thpTDbf3thCpqGM2NQAQEY8DXlE03QGcO6DfTvSvVvsUVr97uQpYAdxE865l6RDgmxHx+7OtVZIkSdL8MJbHXyPiv1nzncFaj8rMH7fHmxXto7wT2O27aBb11NbQ7T+bGoiIbYDzWf04LcAbp1gN9/ED2i4H3gycn5l3tWNuCRwGvA7Yte23PXBBROyTmTfNpuaeZctceHYu8d9Dk8bvrCaJ31dNGr+zGoe5PlO5YXE89Axh++jpqqJp4VR911YNA/pX1xARW9O8f7lr0Xwp8PYpLumGyjOAJ2Tmmb1ACZCZt2Xm/0fzqOylRf/fAd5UW68kSZKk+WOuL9RTvju4aspec7eGBTN3mV5E7AB8DigfSb0WeF67fcogZwDfA3ajWf32H6bpS2b+MiKOoFmEaJu2+diIeH1mjvIu60BLliyZ7RAag97/ifTfQ5PC76wmid9XTRq/s+qazaz1uELlK+nf5mI2ri+O7wU2bo+HnumLiPXpD3R3z6Kee4vjUWcby8935BoiYg+aQFku8rMC+KPMvHHwVZCZP6VZfXZomXlLRLwbeEPbtD7wDOD9IxUtSZIkaV4ZS6jMzGH3jxzVHawOlZuOcF33/cU7Z1lDzyg1dOsYqYaIOBQ4G3h40XwtcHBmXjliHcP6NKtDJTR7dBoqJUmSJE1prr9TeXNxvPUI13VXWl0xphoe3s6C1tQxdA0RcQLwGfoD5XLgiWsxUAJc1TnfbmAvSZIkSWrN9Xcqrwb2ao93nKZfV3evxWtnWUPPejRB6/rBXVdrw+f2o9QQERsC7wNe1PnVJcDhmXnbTGPM0m8656MEaEmSJEnz0FyfqbyiON4tIjYa8ro9OueXj6mGQWNPZXf638GctoaI2Bz4LGsGyjOAQ0cNlBGxYUTs1K4cO6xHdM5H2RtUkiRJ0jw010Nl+a7mBgzef3GQ/YvjO4DZPDL6P/Sv+rpfRQ0AS6fq2Aa/LwNPK5pXAf83M4/NzHsHXzlwrIiIXwL30MyOvmGGS0pP6Jx/d4RrJUmSJM1Dcz1UfhEot8I4bKYLImIDmlVLey6abjuNmWTmrcA3R6lhQL9bgIFr9LYzlF+gPzD/BjgqM2v2ivwpsElx/scRMezWJkd3zr9YcX9JkiRJ88icDpWZeTvNiqQ9L46I7iI8XcfS/y7jh8dQylnF8X4R8ZTpOrdbgZSh8iOZ+cAU3T9Ef6C8jWbLkHNrCm1nNT9XNO0G/MlM10XE/sBRRdPXMvM7NTVIkiRJmj/mdKhs/VNxvDXwianerYyIfYB3Fk2XA/81hho+CNxanJ8VEbsM6tiG3vNYvQjS3Z2ayr5/BRxRNN1J8/7k12ZZb/d+742ImKpzROwJnMPqhXlWAf93ljVIkiRJmgfmfKjMzK8CHy2angZ8IyKe2nusMyIWtdtwXMzqvSTvB14+zQwhEXF1RKwqfnadooY7gNcUTTsCSyPihb2AGxHrR8RhNI+57ln0/YfM/NmAey8C/qHT/OLM/Ga376gy80vAR4qmbYDLIuIVEbFlUcN2EfFq4Bv0r5j71sy8ZLZ1SJIkSXrom+tbivT8FbArcEB7/jjgS8Cd7aI027Pm9hcnZuZXxlVAZn4gIh4DnNA2bUPzaO2/R8QKmpVTN+5c9gngjVMM+XL697F8APiHiOgGzWFq+90BzS+h+VwOac+3oJnBPD0ibqT5t98G6L5v+S+ZedKoNUiSJEman+b8TCVAZv4aeDpNiCtXYt2UZoatDJS/BI7JzHevhTpOBF4F3FU0LwR2pj9Q3g+8DTg6M8t6S0d2ztcDHln5M6jWe2gWLDqNZiXY8j47AtvSHyhvBl6Umf/PFPVKkiRJ0homZaaSzLwTOC4i3g0cAxxMEyg3pwmSV9Ds8/iBzFxr+ytm5tsi4mPAccAzafaj3JomaF5FszXIv2XmD2cYatj9Lqu1q96eGhHvAY6neXR4L5oZ0pXAjcBy4JPAee3CSJIkSZI0tIkJlT2ZuYwptueoGGvXyuuuA/6x/am992a111bcawXw5vZHkiRJksZmIh5/lSRJkiTNTYZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVTNUSpIkSZKqGSolSZIkSdUMlZIkSZKkaoZKSZIkSVI1Q6UkSZIkqZqhUpIkSZJUzVApSZIkSapmqJQkSZIkVdtgXRcwqoh4PPDnwIHALsBGwI3Aj4FzgLMz87a1XMPOwIuBQ4A9gEXALcA1wGeAj2TmT0YY7yjg45XlLMzMlTOMvyFwJHAUsATYBrgHuB5YCpwJXJiZqyprkCRJkjRPTcxMZURsEhEfBJYBLwMeC2xOEyp3AQ4G3gdkRDx3LdZxAnAl8DrgicBWwEJgB2Bf4PXADyPi5IhYf8hhH78WSgUgIvYGvkMTHJ8LLAY2BrYA9gReCHweuDAiFq+tOiRJkiQ9NE1EqIyIjWmCz/GdX91BM9v2QNG2DXB+RHT7jqOOtwKnA5sUzfcA17b/7VkInAZ8YMihy1B5O3DVCD9Tzi5GxP7AJTThsWcVzcxudzb3YODrEbHTkDVLkiRJ0sQ8/voumsddey4BXpWZSwEiYguaR2JPAx7W9nl/RCzPzMvGUUBEHA28qmj6GXAi8F+ZeV9EbAA8myZ07tL2OTYivpuZp88w/N7F8WmZ+fYx1Ls1cB7No7kAK4G3AO/KzJvaPr8HvAH4k7bPTsAFEbFPZt4/2xokSZIkPfTN+ZnKiNiP5v3FnguAg3uBEiAzb8/MdwAHAXe2zQuBd4yphkU0YbHnp8B+mXl+Zt7X1rAyM8+neWfxB0Xf10fEI6YZezHN7GrPt8ZRM03A3qE4PyYzT+4Fyrbm5Zl5JPDWot/eNAFdkiRJkmY050Ml8BpgQXt8B3D8VLNobdD866LpiRFxyBhqOB7Yrjh/aWaumKKGW4EjaGYGoXnv88Rpxu6+T/nt2iJ7ImIb4EVF05mZefY0l5wEXFqcnxIRk/DdkCRJkrSOzengEBFbAs8pms5oQ9t0PkzzzmDPn42hlOOK48sz86LpOmfmD4FPFk1/Ok33MlRelZm/HL28NRxNs4BRz7SP37arvr6laNoZePIY6pAkSZL0EDenQyXNlh3lCqqfnKpjT7u9xqeLpsPb9x2rRMR2wONGqaH1qeJ414jYZ4p+Zagc16OvTy+Or83MZUNccyFwV3F+1JhqkSRJkvQQNtdD5b7F8Spg2EV3vlEcbw48ZhY17MPqx28Bvl5RA8CTpui3NkJlGWCHqjcz76X/0dup6pUkSZKkB8311V/3Ko5/npm/HvK67JzvSbNX42xrAFg+5HU/Be6jWTCoV0OfiNgW2LFoWhYRu9K8w3lwe80i4FbgxzTbqnwgM2+Y6qbtzGq5MNCw9ULzufXCZETEgvbRWEmSJEkaaK7PVC4ujq8Z4brrOue7DOw1eg2rBow9ULuYUPlu56AalnTOX0ITHk+lCXdb0YTS7YEDaFZ0vSoiTp1mIZ3FnfPaz21j+hcnkiRJkqQ1zPVQuW1xfMsI13UX89lqTDXc3ttCpKKOQTV0V349itXvkN5LEwh/1emzCfD3wPkRsZA1bds5X1efmyRJkqR5YCyPv0bEfzO+d/AelZk/bo83K9qHffR1UN9Fs6intoZu/0E1dEMlwBnAvwDf7D16GhEB/AXwf1j9OO1zgHcDfzlNvaPWPM7P7UHLlg2zTpB+W/z30KTxO6tJ4vdVk8bvrMZhrs9UblgcDz1D2D56Wr4LOGhGb63WMKD/oBrKUHkvcFhmHpuZS8t3GbPxNzTvWd5ZXPMXEXHgNPWOWnO372w+N0mSJEnzwFxfqKdcdXVdLRgzmxoWzPD7Y4Ddgd2A72fmf07XOTMvjYiX0ezF2XMKcOg09xyl5pnqrbJkSffVUa0Lvf8T6b+HJoXfWU0Sv6+aNH5n1TWbWetxhcpXAluOaazri+N7aRaMgRFmzSJiffoD0t2zqOfe4njUmbvy812jhsz8KvDVEcf8KHAy8Kj2/KCIWJSZvXcv7+30H6Xm7vdhNp+bJEmSpHlgLKEyM4fdP3JUd7A6VG46wnXddwHvHNhr+Bp6RqmhW8dsanhQZj4QEZ9jdahcCPwB8N/t+R2dS9bV5yZJkiRpHpjr71TeXBxvPcJ13VVLV4yphoe3s6A1dcymhq6rOufl1h83d363rj43SZIkSfPAXA+VVxfHO45w3U6d82vHVMN6DLl3Yxs+tx9TDV2/6ZyXQffqzu9qP7e7gNtGuFaSJEnSPDTXQ+UVxfFuEbHRkNft0Tm/fEw1DBp7KrvT/z7jwBoiYlFEPHLEGdBHdM4f3IsyM28Bbip+N2y93b5XlCvQSpIkSdIgcz1Ulu9qbsDgfR0H2b84vgO4chY1/A/9K6juV1EDwNLyJCJOjYjf0NT3Y4b/2wCeUByvAr7X+X35uQ1Vb0RsCOxdNC2dqq8kSZIk9cz1UPlFYGVxfthMF0TEBsAziqaLMnPlVP1nkpm3At8cpYYB/W4Bumv0/pzVixBBf81Tioit6N9C5Lvt7GTpc8XxoyNimNnKQ4FNphhDkiRJkgaa06EyM28HPl00vbgNVdM5lv53Gf//9u48WK6qTuD4NwLDsEUgghCEREf9QTEqiAiiKCgOUUQ0gooCio6MosMiBUgNbqW4omBRoqUCChiUbUBBVJA1gyyGGRfAnwubMCREkEUCBELmj3t7crrz1n7d/bpfvp+qrrr39Ln3HKp+vPTvnnPP+d5wFcdhXnG8Q0S8ZqTKdRJXJpVnZObTLdUuBp4szj8YEa2rrw7lMzSv6HryEHXOpXlrkaNHumFETAOOKooWYVIpSZIkaQz6Oqmsfbk4ngGcM9y7lRGxPfC1ouh3wEUd6MOpwP3F+byImDVMHzYEzmfFdi2Pt/QJgMxcDPygKNoEOK0eaR1SRBwCHFwU3QKcPsS972spf29EHDTcfYEvADsX51/NzCeHqyxJkiRJDX2fVGbmfwFnFkWvBa6LiF3rEbbGYjeHAVeyYhRvGfDhIUYI/19E3BERy4vP7GH68DDNo30zgRsj4oBGghsRq0XEXlTTXLcq6n42M+8cpgtHUiyyA7wNmB8RrysX7omI7SLiHJqT0yXA/pn5xDD3/njLvb8ZEV+PiM2L+24ZEWfTPEp5M0MkwZIkSZI0lL5PKmsfAuYX59sAlwOPRMTdVFtfnACsXdQ5PDOv7lQHMvMU4MSiaCOqqbWPRMRfgL8DFwCzizrnAJ8b4Z6LgN2BB4viHYDLgIcj4s6IeITqnc69izqPAm/MzJtGuPdCYG7dL4BpVKOcd0XEooj4K3ArsE9x2ULgrSMkqpIkSZLUZCCSysz8OzCHKokrV2Jdh2pvxXI7jgeB/TLzpC7043Cq0cUlRfEawHNoXnRnGXA8sO9o23LUieG2VKOspbWBLYB1W8qvBnbIzKvG0N9rqEZ2W7dF2ZhqKnHpeuCVmfnH0e4rSZIkSQ3Dvr/XbzLzUap3A08C9gNeR5VQTqdKJG8GLgFOGWI11E724/iIOAt4L7AH1X6UM6gSzT8DVwDfzszfj+OedwC7RsROwL7Aq6gSyulUo7D3AtcA52fm5ePs740RsQ3wDqqRy5cCz6YauVxItWXKD4ALR5oqLEmSJElDGZiksiEzF7Dy9hzt3mt2m9fdAxxXfzomM68Fru3kPev7PgV8v/5IkiRJUscMxPRXSZIkSVJ/MqmUJEmSJLXNpFKSJEmS1LZpy5ePuDip1LYFCxYYXJIkSdIA2W677aaN9xpHKiVJkiRJbXOkUpIkSZLUNkcqJUmSJEltM6mUJEmSJLXNpFKSJEmS1DaTSkmSJElS20wqJUmSJEltM6mUJEmSJLXNpFKSJEmS1DaTSkmSJElS20wqJUmSJEltM6mUJEmSJLXNpFKSJEmS1DaTSkmSJElS21af7A5IGlpEPB94P7AL8AJgOvAAcA9wJXB2Zl7f5T7sArwL2AnYDFgHeAi4HbgGOC0zf9fNPmgwTHa8RsTqwDuAPYHtgWcDqwGLgF8B5wPnZubSbvVBU0NErAlcD7ykLjowM7/bpbZ2Ad4DvILqb+xqwL3ArcAPgfMyc0k32tbU0Mt4Hab9jwAn1afPzcw7etW2+su05cuXT3YfJBUiYm3gi8DBjD6b4BLg4E7/EY+IzYAzgF3HUP0M4COZ+XAn+6DB0Cfx+lrg28DzRql6J1WsXtTJ9jW1RMSJwKFFUcd/pEfEDOC7wJtGqXob8J7MnN/J9jV19CJeR2h7S+BGYN26yKRyFeb0V6mPRMS6wM+Aj9D8/+fjwF1UIz+lNwA3RcRWHezDbOAGVk4oH6n78FhL+f7AVRHxzE71QYOhT+L1IODnrJxQLgH+QnO8zgJ+HBGf6FT7mloiYg5wSJfbmEE106M1ofwbsBAon/Y/D7giIl7fzT5pMPUiXkdoeybVg8J1R6urVYNJpdRfvg68qjj/NfBGYHpmzsrMGVRTC79T1NkAuCQi1pto4xHxDOA8YGZRfCmwA/DMzJxFNa1xDvDbos42wKkTbV8DZ7LjdQ5wMtWUwYYrgNcA62XmFlTx+i9UT9MbPh0RB0+0fU0tEbEx1ejhtC43NQ8oH6ycB2ydmRtm5qbAJsBxwLL6+9WBc+sHfhLQ03gdqu3nUz0Ymd3rttW/TCqlPhERLwMOKIrmA6/IzEsy88lGYWb+KTM/ABxT1J1FZ55W7gO8tDifB+yemTdk5vK6/acy82fAjlQjmg1zI2LHDvRBA2Cy4zUi/gH4Js0J5QnA6zLz6sx8um7/qcy8lCr5Paeo+5WIGG26rFYtp1G9i9s1EfF2qoccDd/IzL0z85ZGQWbel5nHAm9lRWI5HfhcN/umgdP1eB1KROwDLGD01w20ijGplPrHh4vjp6nei2idalr6EtWCOQ37dKAPc4vjx4EPN5LJVvXiEa2jPXOHqqspabLj9V1UyWnDRcARI8TrUqoFUW6ri/4RcBqsAIiIQ6lG2QEWd7GpjxXHtwGHDVcxM38MfL4oemdERLc6psHRw3gt25wREd8CzqZ6yCE1MamU+sfLiuMrMvNPI1WuR2LKkcLnd6APLyiOf5OZD47ShwVU7wENdb2mtsmO1z1azo8aLqEs+vAY1Whmwz71e6FahUXEi6gWmwJ4EvhQl9p5MbBtUXTyGFYjPgFo1JkG7NuNvmlw9Cpei/bWiohjqB6CfKD46mHgrG62rcFiUin1icx8EfAsYGfgqDFeVr6Xdn8HurFmcbzWGK8pf8j7N2UV0Qfxul1xfEtm3jrG6y4tjtemev9Sq6iIWIvqh3Hjb9/Hqab2dcOclvMLRrsgMx+genetoRMzUjSgehyvDe+gmnpdjk7eQLXWws+73LYGiD8ApT6Smfdn5vzMvGm0uvUKguUP4l91oAvlaNPWEbHFKH14ObBhUfTb4epq6pnkeN24OL5l2For+0vL+csn2A8Ntq8AW9fHVwFf7mJbZawtzsw/j/G664rjrSJigw72SYOll/E6lL8BhwM7Zebve9y2+pxJpTSAIuJZVCsGrlMUf6kDty4XMnkGcEq9sfJQfVgf+FZRtJRqJTqpSZfitVygZ6R3OVu1TpHtxLRxDaCI2JMVUwcfBA5oLPDUJVsXx+N5EJLF8TRgy850R4NkEuK19FfgM1T7UJ6YmctGu0CrntUnuwOSRhcRq1FNHdwaeDNwELB+UeWIzLy+AzEQuVYAAAj0SURBVE3No3pn4tX1+W7ADRFxHHAl1b6DGwG7A5+keTnxI0Z7r06rhh7F62Jg8/p45kgVW7SOvm82wX5oAEXEpjRvg/ShzLyry81uXhy3jpiP5J6W81nALyfeHQ2KSYrXhp8AZ2XmEz1qTwPKpFLqcxGxCXDvMF/fBRyZmWd3oq3MfDoi3gycTpUMALwY+OEIl90DHJKZ53eiDxpsPYzXX7PiR/pOEbFeZj4yhut2azlff8hamrIiYhrwPap3ggHOzMwfdLnNtWkeqf/rOC5vff94wyFraUqajHgtZeZ9vWpLg83pr1L/mzXCd7cCy+p/dDoiMx/KzL2AvWle2XUoC4GDTChV6FW8XlwcrwX8x2gX1D/sP9pSPNYFqTR1HAG8vj6+g+btcbqldZXhv4/j2ta66w1ZS1PVZMSrNG4mlVL/m0W1D+AiqhGg8l2G3YFzgavrEaIJi4j1I+LrwPeBckGIB6mmbJXvr20CXBwRl0fEczrRvgZer+L19LqNhqMi4oPDVa5XTZzHyht2O2NnFRIR2wLH1afLgP0z8+EeNP0PLedPjuPa1rprTLAvGhCTGK/SuJlUSv3vF8D6mblJZs6kmvp0IHB3UedVwKURsc5QNxiriJgF3AQcTLVk+TKqfdJemJkbZOYWVE/JXwlcWFy6K3D9aKvFapXQk3jNzCVUT+wbC+9MA74RET+JiD3qjbrXiIjnRMSBwH8De1HFdLlqoe8JrSLqeDuLFQne5zNzfo+abx2dH3FP1VGu1SpgkuNVGjeTSqnP1ds2PFKcP5yZ3wVeSvMWHv/MGKYADiciVqdKFJ9bFz0BzMnMj2bmH4v2l2XmtZn5FpqnEs4EzurkVFwNnl7Fa33v86hisFwB8Q3ARVTvrC2lGl0/FYi63r/RvJ3JeFaO1WA7kSoOoNpn79M9bHtpy/l4RhtbR9Mfn2BfNBgmM16lcTOplAZUZi4G3kbz1KgPRkTrNKux2h94SXF+VGZeNkofTgC+UxTtxMobfEvdiNfGfU8E3gTcNkrVPwA7Z+YpNE/rXjyR9jUYIuJtwL/Wp48C787Mp3rYhYdazsczSt/6DuWjE+yL+lwfxKs0biaV0gCrRxDLaagb0JwYjse+xfEDwDfGeN2naZ7KtXeb7WuK63C8lve9hGrvvrdT7ZX6S6p9AK+keugxB9gqM6+tLym3IBlupVpNEfX73uWeuof2evujzHyc5mRwxjgub13tddGQtTQl9EO8Su1wgQJp8F1HcyI3C7ixjftsWxxfm5ljWkgiM++OiGTFhtwTThI0pXUqXpvU8XpO/RlWRKxB8yb0f5ho2+p776M5MTsmIo4ZoX7rb6MvRsSxxfm729xn9Q5WxN549lZt3Uv17iFraarol3iVxsWkUuoT9TTALajeaVyQmQ+M8dLW5ebbnYFQ7tc32lYirRazIqmc3mb7GiB9EK/t2p7mlThv6nH76r3WGPuncV6/cf1paHcbmptZkVRuOVLFFmXd5VSj8Jq6+iVepXFx+qvUByJiV6oFQ/4I/Jxqlcqxan2K3e50vgdHuOdoyqeqY00uNKD6IV4jYq2IeGFEvDoi3jqOS+cWx08A1w5XUeqwcrRo84jYdIzX7Vgc35qZ49njUpJ6wpFKqT/cSvNDnr2B08Z47RuL4yeotk5oxx+AZ9XHO0bE9LHshxURGwFbFUW/H66upox+iNe5wJmNk4iYlZl3jXRBvVflfkXRZeVKtZqaMvNTwKfGWj8iZgO3F0UH1isYT9RPga8U53sB3xylLxtSbcFT3kNTWB/FqzQujlRKfSAzF9I8YjInIrYZ7bqImAtsVxRdMoGn2D8ujtcGDhnjdR+j+W/JhcNV1NTQJ/F6Xcv5fkPWanYk8Ozi/KQ225bGLTNvAf6nKDpsDKsfH07zdO3vdbxjktQBJpVS/yifYD8DOLN+Sj2kiNiR5tGhZcCxw1Qfi2/T/C7lJyNixGmNEXEA1Y+eht8BP5pAHzQ4JjVeM/PPNE8nPDIinjtc/YjYB/hkUXRlZv6s3falNh1fHAdwynB7+0bEnkC5QMslmfmbbnZOktrl9FepT2Tm+RFxAfCWumhrYEFEHA38qF6SnoiYRbWB+0eBNYtbHJWZN7fed4ipMXdm5uwh2r8/Ig4FTq+LVgcuiIhTgZOBmzJzeX3Pl1Alk+8pbvE48IHMXDau/3ANpMmO19qxwKX18frANRFxOHBhZi6t77cVcBhwUHHdA6zYA06akIjYBbiiKLoqM3cZpvo8qlh8dX2+HzAzIo7OzF/V99sI+HeqhHK1ut6jVHEsTcg441UaM5NKqb+8m+qdmZ3r89nAD4GlEbGQasPs1v3NlgOfzcyvTrTxzDyjXjziC0Dj6fn76s8TEXEf1d6C67Zc+hjwrsxsnZKoqW2y4/WyiPgicHRdtBlwNlWsLqrbbt1k/m/AnvVIp9RTmbk8It4JXM6KVV1fC9wYEQ8BS6imaJczyZ4C9s9Mt7+R1Lec/ir1kcxcAuxGNUWq3CeysX1D6w/024G3ZOYnOtiHL1EtpnJ7y1drApuzckK5ANgxMy/oVB80GPokXj9G9a7k0qJ4zbr91oTyF8D2memKr5o0mXkv8Brg4pavnglsSvNvs/8F9sjM/+xR9ySpLY5USn2mnrZ3ZER8DXg/1VPsoNq24zFgEdW7ZD8BzsnMp7rQh59GxAuoVtjcA9iB6un5esBDwD3AfOCCzLx02BtpyuuTeD0+Is6imla4G9UI0HSqUZ87qBYV+n5mzu9021I7MvM+4E31VMR9qabDbkq1SNoDwK+pFj07wxWKJQ2CacuXL5/sPkiSJEmSBpTTXyVJkiRJbTOplCRJkiS1zaRSkiRJktQ2k0pJkiRJUttMKiVJkiRJbTOplCRJkiS1zaRSkiRJktQ2k0pJkiRJUttMKiVJkiRJbTOplCRJkiS1zaRSkiRJktQ2k0pJkiRJUttMKiVJkiRJbTOplCRJkiS1zaRSkiRJktQ2k0pJkiRJUttMKiVJkiRJbTOplCRJkiS1zaRSkiRJktQ2k0pJkiRJUttMKiVJkiRJbfs/PlNO4AHm2oUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 256,
       "width": 458
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.distplot(clean_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " # Now lets save the cleaned prescribers information to Csv for further work in notebook 1.3.\n",
    "# clean_df.to_csv('./inputs/prescribers_clean929.csv')"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:dsi]",
   "language": "python",
   "name": "conda-env-dsi-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}