FetalHH_Modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "933db124",
   "metadata": {},
   "source": [
    "# A Machine Learning Approach to Cardiotocography Interpretation \n",
    "## Milestone 4: Finalizing Your Results\n",
    "\n",
    "***Holly Figueroa  \n",
    "Karlie Schwartzwald  \n",
    "DSC 630 Winter 2022  \n",
    "Bellevue University***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de743c6e",
   "metadata": {},
   "source": [
    "### Milestone 2: Preliminary Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6038b127",
   "metadata": {},
   "source": [
    "***A Machine Learning Approach to Cardiotocography Interpretation*** \n",
    "\n",
    "Despite modern advances, childbirth remains a risky medical endeavor both for birthing people and the unborn. There exists no replacement for the expertise and direct involvement of a medical professional to manage care. Analytics, however, can offer significant aid in ways it could not before through advanced modeling. Cardiotocography (CTG) is widely used over the course of pregnancy and during labor (Fetal Heart Monitoring, 2019). CTG allows monitoring of fetal heart rate and uterine contractions to prevent fetal hypoxia, or oxygen loss in body tissues (Boudet et al., 2020). Creating effective modeling for CTG exam data can provide additional support to those dedicated preventing harm and loss of life.  The goal of this project will be to build an effective model to classify CTG data using supervised machine learning.  \n",
    "\n",
    "***Data Selection*** \n",
    "\n",
    "The dataset chosen for model training contains 2126 instances of heart-health-related features taken from cardiotocogram exams of fetuses. These were then classified by three expert obstetricians into 3 classes: 1 – Normal, 2 – Suspect, or 3- Pathological, making the target a multi-class categorical feature. There are a total of 22 features, both numerical and categorical. We found this dataset on Kaggle, but it was originally published in the Journal of Maternal-Fetal & Neonatal Medicine in the year 2000. This dataset should be useful for creating a model that can predict suspect and pathological health outcomes, with the assumption that some of the features provided will be correlated with fetal heart health.  \n",
    "\n",
    "***Model Selection & Evaluation***  \n",
    "\n",
    "We plan to implement two different models for this project, both of which will perform classification on the fetal heart health target feature. The first model will be a neural network. We think that this dataset provides a good opportunity to train a neural network because of the substantial number of instances in the data. Neural networks are also considered some of the most accurate models, and they are ideal in a medical setting like this where accuracy is important. We plan to use the percentage of accuracy to score and evaluate this model. In this context where we are discussing medical diagnostics, a false negative has more detrimental outcomes to the patient's life than a false positive. Thus, we will review the confusion matrices in addition to the accuracy percentage, so that we can calculate rates for false positives and false negatives.  \n",
    "\n",
    "The second model we plan to use is Naïve Bayes classification model.  Although the dataset contains over 2,000 instances, it is possible that this amount of data is not sufficient to train a competent neural network. The Naïve Bayes model is recommended as an alternative in that circumstance. It also has a reputation as an effective classifier and has been proven useful in medical diagnostics (Simfukwe et al., 2015). Since we are still doing classification, we will be using accuracy and confusion matrices to evaluate the results of our model. We can compare the accuracy score of this model to the neural network model in this proposal. \n",
    "\n",
    "To ensure quality and accuracy, we plan to implement a variety of techniques to improve both models. We will need to clean the data, do some feature scaling, and feature selection. If there are any categorical features that are relevant to the model, we can create dummy variables with them before training the model. We plan to use a grid search to perform hyperparameter tuning on our models as well. Since the target feature has imbalanced classes, we will need to perform oversampling on the underrepresented classes. \n",
    "\n",
    "***Risks and Ethical Implications***   \n",
    "\n",
    "Machine learning and artificial intelligence has already earned space in healthcare with the increased use of AI diagnostic systems. Some common examples involve the interpretation of x-rays, MRI scans, CT scans, and other image-related tools of measurement. While machine learning healthcare applications (ML-HCAs) and AI recommendation systems can offer powerful insights for the medical field, limits must be acknowledged. The most fundamental limit being correlation is not causation, therefore their role must be additive to care, not a replacement for expertise. A common hope is that the addition of ML-HCAs and AI diagnostics will circumvent bias within the healthcare system, however, these tools are just as suspectable (Naik, 2022). The data used to train a diagnostic model is critical to how accurately these systems perform for diverse populations. It is unknown how these concerns apply to CTG exams of the unborn. For example, we do not know if CTG exams, in practice, predict or protect all groups equally. We also do not know at this time if the anomalies associated with fetal heart issues present the same across groups. For this project, however, we will assume the data is sound towards its purpose. \n",
    "\n",
    "***Contingency Proposal***     \n",
    "\n",
    "In the case that our original proposal is unsuccessful, the contingency plan will be to use an alternative dataset to train a regression model. We found a dataset that is intended to train models to predict the insurance premium price of an individual given certain health history data points about them. This dataset has 10 predicting features about the health history of an individual and 986 instances with no missing values. Since our target is continuous and we want to include multiple features, we will use a multiple regression model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2b54145",
   "metadata": {},
   "source": [
    "### Milestone 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74a73c65",
   "metadata": {},
   "source": [
    "***Will I be able to answer the questions I want to answer with the data I have?*** \n",
    "\n",
    "Our preliminary analysis suggests no reason our data should not perform suitably for modeling with some adjustments for scaling and normalization. No missing data was found. Histograms and descriptive statistics were used to conduct univariate analysis on all variables. Multivariate analysis was conducted using correlations matrices and box plots to illustrate variable distributions across our 3 target classes for fetal health outcome: Normal(1), Suspect(2), and Pathological(3).  \n",
    "\n",
    "Our preliminary analysis with correlations showed that variation in many of our features are associated with variations in our target classes. The relationships are not necessarily strong, but combined, they will likely offer appropriate training for our predictive models. The strongest relationships to our target classes were related to CTG features involving deceleration at .50 Tau and abnormal variability, with those Tau’s ranging from .25 to .43  \n",
    "\n",
    "***What visualizations are especially useful for explaining my data?*** \n",
    "\n",
    "Histograms, while simple, were in many ways most effective in conveying data issues to consider. Knowing data is skewed, differently scaled, imbalanced, or lacking variation in measures, lends a lot for our preprocessing needs as mentioned above. Histograms related to heart decelerations per second were found to be highly imbalanced. But upon further inspection using value counts, the rare values represented cases with the most serious health outcome, ‘pathological’. Box were effective in allowing us to understand how our features distribute and relate to our target classes. Short-term abnormal variability appeared to have a positive linear relationship across our target classes as they ranged from normal health to pathological. \n",
    "\n",
    "***Do I need to adjust the data and/or driving questions?*** \n",
    "\n",
    "The analysis shows that some steps, such as scaling, and normalization for some variables will have to be taken prior to being used as input for our models. Strong correlations between variables were also shown between different central measures related to ETC histograms (mean, median, and mode), which led to the dropping of variables for median and mode. We also know that our target data will require us to use sampling techniques to balance our target classes when model training. So, while changes will have to made, the data itself and questions remain appropriate.  \n",
    "\n",
    "***Do I need to adjust my model/evaluation choices?*** \n",
    "\n",
    "At this time, our model choices appear reasonable. We have chosen both a neural network classifier and a naive Bayes classifier for modeling and comparison. It is possible that the independent treatment of features within a naive Bayes may not be as well suited. The features may share probabilistic qualities with each other given the specialized topic. Box plots also illustrate what appear to be nuanced difference in feature distributions across target classes. While we do not have the domain expertise to objectively know, our current model choices can be compared to offer some insights. The confusions matrices we have chosen to evaluate results will help illustrate how well our model discerns between classes.  \n",
    "\n",
    "***Are my original expectations still reasonable?*** \n",
    "\n",
    "Given the steps we can take to prepare data for modeling, our plans for modeling seem reasonable. We will have a neural network classifier and a naïve Bayes classifier planned to compare. Our preliminary analysis shows no indication an effective model cannot result from this dataset. The target classes may be imbalanced, but we can utilize over sampling techniques to give our model more proportionate data to work with. The data has no missing values, and there is no reason to believe any of the observations were made in error. We also already know that many of our features have some established relevance to predicting fetal outcomes as they represent output features from existing diagnostic techniques."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0d3863b",
   "metadata": {},
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd7ed31e",
   "metadata": {},
   "source": [
    "All the data preparation done for these models is described in detail in Milestone 3. In summary, data preparation consisted of removing highly correlated features with a threshold of correlation above 0.9. By this criterion there were two features that were removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9e437f3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.naive_bayes import ComplementNB\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.model_selection import cross_validate\n",
    "\n",
    "\n",
    "\n",
    "from imblearn.over_sampling import RandomOverSampler\n",
    "from sklearn.pipeline import Pipeline, FeatureUnion\n",
    "from sklearn import preprocessing\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.preprocessing import MinMaxScaler, StandardScaler, RobustScaler\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ce918434",
   "metadata": {},
   "outputs": [],
   "source": [
    "#    Title: get_tpr_fnr_fpr_tnr()\n",
    "#    Author: Md Abdul Bari\n",
    "#    Date: Oct 22, 2020\n",
    "#    Availability: https://stackoverflow.com/questions/45053238/how-to-get-all-confusion-matrix-terminologies-tpr-fpr-tnr-fnr-for-a-multi-c.\n",
    "     \n",
    "def get_tpr_fnr_fpr_tnr(cm):\n",
    "    \"\"\"\n",
    "    This function returns class-wise TPR, FNR, FPR & TNR\n",
    "    [[cm]]: a 2-D array of a multiclass confusion matrix\n",
    "            where horizontal axes represent actual classes\n",
    "            and vertical axes represent predicted classes\n",
    "    {output}: a dictionary of class-wise accuracy parameters\n",
    "    \"\"\"\n",
    "    dict_metric = dict()\n",
    "    n = len(cm[0])\n",
    "    row_sums = cm.sum(axis=1)\n",
    "    col_sums = cm.sum(axis=0)\n",
    "    array_sum = sum(sum(cm))\n",
    "    #initialize a blank nested dictionary\n",
    "    for i in range(1, n+1):\n",
    "        keys = str(i)\n",
    "        dict_metric[keys] = {\"TPR\":0, \"FNR\":0, \"FPR\":0, \"TNR\":0}\n",
    "    # calculate and store class-wise TPR, FNR, FPR, TNR\n",
    "    for i in range(n):\n",
    "        for j in range(n):\n",
    "            if i == j:\n",
    "                keys = str(i+1)\n",
    "                tp = cm[i, j]\n",
    "                fn = row_sums[i] - cm[i, j]\n",
    "                dict_metric[keys][\"TPR\"] = tp / (tp + fn)\n",
    "                dict_metric[keys][\"FNR\"] = fn / (tp + fn)\n",
    "                fp = col_sums[i] - cm[i, j]\n",
    "                tn = array_sum - tp - fn - fp\n",
    "                dict_metric[keys][\"FPR\"] = fp / (fp + tn)\n",
    "                dict_metric[keys][\"TNR\"] = tn / (fp + tn)\n",
    "    return dict_metric\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "abfb4456",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>baseline value</th>\n",
       "      <th>accelerations</th>\n",
       "      <th>fetal_movement</th>\n",
       "      <th>uterine_contractions</th>\n",
       "      <th>light_decelerations</th>\n",
       "      <th>severe_decelerations</th>\n",
       "      <th>prolongued_decelerations</th>\n",
       "      <th>abnormal_short_term_variability</th>\n",
       "      <th>mean_value_of_short_term_variability</th>\n",
       "      <th>percentage_of_time_with_abnormal_long_term_variability</th>\n",
       "      <th>...</th>\n",
       "      <th>histogram_min</th>\n",
       "      <th>histogram_max</th>\n",
       "      <th>histogram_number_of_peaks</th>\n",
       "      <th>histogram_number_of_zeroes</th>\n",
       "      <th>histogram_mode</th>\n",
       "      <th>histogram_mean</th>\n",
       "      <th>histogram_median</th>\n",
       "      <th>histogram_variance</th>\n",
       "      <th>histogram_tendency</th>\n",
       "      <th>fetal_health</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>120.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>73.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>43.0</td>\n",
       "      <td>...</td>\n",
       "      <td>62.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>121.0</td>\n",
       "      <td>73.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>132.0</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>2.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>68.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>141.0</td>\n",
       "      <td>136.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>133.0</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.008</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>68.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>141.0</td>\n",
       "      <td>135.0</td>\n",
       "      <td>138.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>134.0</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.008</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>53.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>132.0</td>\n",
       "      <td>0.007</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.008</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>53.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>136.0</td>\n",
       "      <td>138.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   baseline value  accelerations  fetal_movement  uterine_contractions  \\\n",
       "0           120.0          0.000             0.0                 0.000   \n",
       "1           132.0          0.006             0.0                 0.006   \n",
       "2           133.0          0.003             0.0                 0.008   \n",
       "3           134.0          0.003             0.0                 0.008   \n",
       "4           132.0          0.007             0.0                 0.008   \n",
       "\n",
       "   light_decelerations  severe_decelerations  prolongued_decelerations  \\\n",
       "0                0.000                   0.0                       0.0   \n",
       "1                0.003                   0.0                       0.0   \n",
       "2                0.003                   0.0                       0.0   \n",
       "3                0.003                   0.0                       0.0   \n",
       "4                0.000                   0.0                       0.0   \n",
       "\n",
       "   abnormal_short_term_variability  mean_value_of_short_term_variability  \\\n",
       "0                             73.0                                   0.5   \n",
       "1                             17.0                                   2.1   \n",
       "2                             16.0                                   2.1   \n",
       "3                             16.0                                   2.4   \n",
       "4                             16.0                                   2.4   \n",
       "\n",
       "   percentage_of_time_with_abnormal_long_term_variability  ...  histogram_min  \\\n",
       "0                                               43.0       ...           62.0   \n",
       "1                                                0.0       ...           68.0   \n",
       "2                                                0.0       ...           68.0   \n",
       "3                                                0.0       ...           53.0   \n",
       "4                                                0.0       ...           53.0   \n",
       "\n",
       "   histogram_max  histogram_number_of_peaks  histogram_number_of_zeroes  \\\n",
       "0          126.0                        2.0                         0.0   \n",
       "1          198.0                        6.0                         1.0   \n",
       "2          198.0                        5.0                         1.0   \n",
       "3          170.0                       11.0                         0.0   \n",
       "4          170.0                        9.0                         0.0   \n",
       "\n",
       "   histogram_mode  histogram_mean  histogram_median  histogram_variance  \\\n",
       "0           120.0           137.0             121.0                73.0   \n",
       "1           141.0           136.0             140.0                12.0   \n",
       "2           141.0           135.0             138.0                13.0   \n",
       "3           137.0           134.0             137.0                13.0   \n",
       "4           137.0           136.0             138.0                11.0   \n",
       "\n",
       "   histogram_tendency  fetal_health  \n",
       "0                 1.0           2.0  \n",
       "1                 0.0           1.0  \n",
       "2                 0.0           1.0  \n",
       "3                 1.0           1.0  \n",
       "4                 1.0           1.0  \n",
       "\n",
       "[5 rows x 22 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('fetal_health.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "73ca67f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop highly correlated variables\n",
    "# 0.9 Threshold as described in Milestone 3\n",
    "\n",
    "# Histogram mode and median highly correlate with hisogram mean\n",
    "df.drop(['histogram_median', 'histogram_mode', ], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f6a77aa8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# seperate target feature\n",
    "X = df.drop(['fetal_health'], axis=1)\n",
    "y = df['fetal_health']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "916efc64",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1700, 19) (1700,) (426, 19) (426,)\n"
     ]
    }
   ],
   "source": [
    "# train test split data using stratify to preserve the class ratios\n",
    "train_X, test_X, train_y, test_y = train_test_split(X,y, test_size = 0.2, stratify = y, random_state = 5)\n",
    "# Check shape\n",
    "print(train_X.shape, train_y.shape, test_X.shape, test_y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0601fe1",
   "metadata": {},
   "source": [
    "## Null Model for Comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39959648",
   "metadata": {},
   "source": [
    "Below we will calculate the accuracy of a model that simply always chooses the most likely outcome. In this case the model will predict that all fetal heart health outcomes are healthy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "15b338b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0    332\n",
       "2.0     59\n",
       "3.0     35\n",
       "Name: fetal_health, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check class distribution in test set\n",
    "test_y.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "dae79f29",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Null accuracy score: 0.7790\n"
     ]
    }
   ],
   "source": [
    "# check null accuracy score\n",
    "null_accuracy = (497/(497+94+47))\n",
    "print('Null accuracy score: {0:0.4f}'. format(null_accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "85ac1656",
   "metadata": {},
   "outputs": [],
   "source": [
    "# So we need to score above 78% to be a good model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33e0e36f",
   "metadata": {},
   "source": [
    "## Complement Naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfa3b2db",
   "metadata": {},
   "source": [
    "### Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e369794",
   "metadata": {},
   "source": [
    "For the Naïve Bayes classifier, we chose to use the Complement Naïve Bayes due to its ability to handle imbalanced classes in the target feature. We used a train-test-split ratio of 80/20 because it yielded the highest accuracy of any ratio we tested, without overfitting the model too much. We then applied a minmax scalar because the model only accepts positive values as input. After that we did a grid search to perform hyperparameter tuning on the only hyperparameter in a complement naïve Bayes model, using accuracy as our scoring metric. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "46775319",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scaling\n",
    "scaler = MinMaxScaler()\n",
    "# I had to use this particular scalar because of negative numbers error\n",
    "x_train = scaler.fit_transform(train_X)\n",
    "x_test = scaler.transform(test_X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f9c1145",
   "metadata": {},
   "source": [
    "### Hyperparameter Tuning for Model Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "870d3454",
   "metadata": {},
   "outputs": [],
   "source": [
    "# gridsearch values\n",
    "grid_vals = {\"alpha\": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, \n",
    "                       1, 2, 3, 4, 5, 6, 7, 8, 9, \n",
    "                      10, 11, 12, 13, 14, 15, 16, 17, 18, 19,\n",
    "                      20, 21, 22, 23, 24, 25, 26, 27, 28, 29,\n",
    "                      30, 31, 32, 33, 34, 35, 36, 37, 38, 39,\n",
    "                      40, 41, 42, 43, 44, 45, 46, 47, 48, 49,\n",
    "                      50, 51, 52, 53, 54, 55, 56, 57, 48, 59,\n",
    "                      60, 61, 62, 63, 64, 65, 66, 67, 68, 69,\n",
    "                      70, 71, 72, 73, 74, 75, 76, 77, 78, 79,\n",
    "                      80, 81, 82, 83, 84, 85, 86, 87, 88, 89,\n",
    "                      90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2ef181c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# gridsearch model\n",
    "grid_lr = GridSearchCV(estimator=ComplementNB(), param_grid=grid_vals, scoring='accuracy', \n",
    "                       cv=6, refit=True, return_train_score=True) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "30f7edc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Training and Prediction\n",
    "tuned_classifier = grid_lr.fit(x_train, train_y)\n",
    "tuned_preds = grid_lr.best_estimator_.predict(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b8c8c881",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuned Gaussian NB Parameter: {'alpha': 42}\n"
     ]
    }
   ],
   "source": [
    "# Print the tuned parameters and score\n",
    "print(\"Tuned Gaussian NB Parameter: {}\".format(grid_lr.best_params_))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fa17711",
   "metadata": {},
   "source": [
    "### Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "9fb27b37",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cross validate\n",
    "scores3 = cross_validate(tuned_classifier, x_train, train_y, return_train_score=True)\n",
    "scores_df3 = pd.DataFrame(scores3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "e7af2324",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       1.854373\n",
       "score_time     0.001412\n",
       "test_score     0.840588\n",
       "train_score    0.839559\n",
       "dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores_df3.mean()\n",
    "# Looks like we could be over fitting but the model is performing better than null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "b3ca5d73",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAEHCAYAAAA6U1oSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAAsTAAALEwEAmpwYAAAe80lEQVR4nO3deZQcZb3/8fdnJpNtshESYkhiAhKDESVA2K+RTdn0BrwKqBdQ0agsgsBF1PsTBBcUBBFFbwQPAVkE2YKyiAgGkC0JiyQhEAiQQCD7vs1Mf39/dA0MMEt30p3qrvm8zqmT6qe6q77TZ/KdZ6nnKUUEZmZZVJN2AGZm5eIEZ2aZ5QRnZpnlBGdmmeUEZ2aZ5QRnZpnVJe0AWhrQvzZGDKtLO4yK9cKsPmmHUPGioTHtECraetawMTZoc85x8P71sWRpU0HvnfbMhnsi4pDWjknqDkwBupHPRX+OiHMkbQfcAGwNTAOOjYiNkroBVwO7AUuAoyPi5fauX1EJbsSwOh6/Z1jaYVSsw3dr9ffEWmhc8EbaIVS0x+K+zT7H4qVNPHbP0ILeWzf4xQHtHN4AHBARqyXVAQ9Jugs4HbgkIm6Q9DvgBOC3yb/LImIHSccAPwOObu/6bqKaWZGCpsgVtLV7lrzVycu6ZAvgAODPSfkk4Ihkf3zymuT4gZLarY06wZlZUQLIEQVtwABJU1tsE1qeS1KtpKeAhcC9wIvA8oho7muYDwxJ9ocA8wCS4yvIN2PbVFFNVDOrDjnar521sDgixrZ1MCKagDGS+gG3AjtufnRvc4Izs6IEQUMHzc+izxmxXNL9wN5AP0ldklraUOC15G2vAcOA+ZK6AH3JDza0yU1UMytKAE1EQVt7JA1Mam5I6gF8ApgF3A98Nnnb8cDtyf7k5DXJ8X9EB6uFuAZnZkXLdZC8CjQYmCSplnxl68aI+IukmcANkn4EPAlcmbz/SuAaSXOApcAxHV3ACc7MihJAUwmWWYuIZ4BdWil/CdijlfL1wOeKuYYTnJkVrbQ9cOXjBGdmRYkC+tcqhROcmRUlAhqqI785wZlZsUQTmzWddYtxgjOzogSQcw3OzLLKNTgzy6T8jb5OcGaWQQE0RHVMgnKCM7OiBKKpSmZ5OsGZWdFy4SaqmWWQ++DMLMNEk/vgzCyL8iv6OsGZWQZFiI1Rm3YYBXGCM7Oi5dwHZ2ZZlB9kcBPVzDLJgwxmllEeZDCzTGvyjb5mlkWBaIjqSB3VEaWZVQwPMphZZgVyE9XMssuDDGaWSRH4NpFKt3G9OOMzO9CwsYamRvjY4Ss47n/e4I1Xu/KTbw5n5bIujPzIWs667FXqur69AP2Df+3Lj762HZfdNZsP7rwuxZ9gyxowaB1nnPdv+vXfSATcfeswJl8/nC9MmMPBR85n5bKuAEz6zUimPjww5WjTVdctxy9umUNd16C2S/DgX/txzUXvSzuskskPMniqFpIOAS4FaoErIuKCcl6vGHXdgp/f9CI96nM0NsDpR4xk9wNWcvPEgXzma4vY74jlXPqdodx9fX8+ffwSANauruG2Kway465rUo5+y2tqquGKS3bkxef60KNnI5f+8RGefHRrAG6/bji3XLNdyhFWjoYN4qzPfYD1a2up7RJcfNscnvhHb56bXp92aCVTLYMMZYtSUi3wG+BQYDTweUmjy3W9YknQoz7/fO7GBtHUICR4+qHefOxTywH4xOeW8sjdfd/6zKSfD+aokxbStVuVPFKohJYt7saLz/UBYN3aLsybW8/W26xPOapKJdavzddwutQFtXVBZOhXJhC5KGxLWznT8B7AnIh4KSI2AjcA48t4vaI1NcE3DxrF0R/diV3GrWLw8A3U922iNqnXDhjcwOI36gB44ZkeLHq9jj0PWplixJVhm8Hr2H7HVcx+th8AnzrqVX59w8Oc+oNn6dW7Id3gKkRNTXD5vbP50zMzeHJKL2Y/mZ3aG+RrcIVs7ZE0TNL9kmZKmiHp1KT8XEmvSXoq2Q5r8ZnvSpojabakgzuKs5wJbggwr8Xr+UnZO0iaIGmqpKmLljSVMZz3qq2F3/59NtdOm8nsp3oyb073Vt+Xy8HEHw5hwjmvb9H4KlH3Ho18/8Kn+P1FO7JuTRfu/PMwvjp+HKd8fh+WLe7GCd+enXaIFSGXEyd+YhRf3G00o8asZfio7PTX5p+LWlPQ1oFG4IyIGA3sBZzUopV3SUSMSbY7AZJjxwAfBg4BLk9aim1KvSEdERMjYmxEjB24dTodl736NrHzPquZNa0na1bU0tSYL1+8oI4B72tg3eoaXn6uO2f91w4ct8doZk3vyTlf2p7nn+6RSrxpqe2S43sXPsX9dw3mX/cPAmD50m7kciJC3H3rUD744RUpR1lZ1qys5el/9WL3/VelHUoJ5Z9sX8jWnohYEBHTk/1VwCxaqQS1MB64ISI2RMRcYA75lmKbypngXgOGtXg9NCmrCMuX1LJ6RT6hblgnpk/pzbCRG9h539U8+Jd+ANx7U3/2PngF9X1y3DTjWa5+fCZXPz6TD+26lh9e9VKnGkWF4NT/N4N5c+u57doRb5VuNWDDW/v77P8mr7zYK4XYKkvf/o3U98m3Rrp2z7HruNVttg6qUf6xgbUFbcCA5hZask1o7ZySRgC7AI8lRSdLekbSHyRtlZQV1CpsqZyjqE8AIyVtRz6xHQN8oYzXK8rSN+u46NT3k8uJXA7GfXo5e31iJcM/uJ6ffHM4V/18MDvstI6DP7807VArwugxyznwU68z94VeXHbdv4D8LSEfP3gB249aRQQsfL0Hl/3kwylHmr7+gxo489JXqamBmhqYckdfHvt7n7TDKpkIFdL8bLY4Isa29wZJvYCbgdMiYqWk3wLnk8+l5wO/AL6yKbGWLcFFRKOkk4F7yN8m8oeImFGu6xVr+9Hrufze599TPnj4Ri6784V2P3vhzXPKFVbFmvnUVhy+23v7dDv7PW+tmTurByd9clTaYZRVqW70lVRHPrldGxG3AETEmy2O/x74S/Ky6FZhWe+DSzoH7yznNcxsy8qvB7f5t4BIEnAlMCsiLm5RPjgiFiQvjwSeTfYnA9dJuhjYFhgJPN7eNTrtTAYz21QlW9F3X+BY4N+SnkrKvkf+ntkx5HPpy8DXASJihqQbgZnkR2BPioh2b71wgjOzouRvE9n8GlxEPAStVgXbbPVFxI+BHxd6DSc4MyuK56KaWaZ5uSQzy6T8cknpzzMthBOcmRWtEibSF8IJzsyKkl9NxE1UM8ug/FQtJzgzyyTX4Mwsw0oxk2FLcIIzs6J4FNXMMs1NVDPLpOZnMlQDJzgzK0oAja7BmVlWuYlqZtlUIY8ELIQTnJkVpVQLXm4JTnBmVjTX4Mwsk0q14OWW4ARnZkUJRGPOgwxmllHugzOzbAo3Uc0so9wHZ2aZ5gRnZpkUiCYPMphZVnmQwcwyKTzIYGZZFk5wZpZN1TPZvjp6Cs2sokSooK09koZJul/STEkzJJ2alPeXdK+kF5J/t0rKJelXkuZIekbSrh3FWVE1uBdm9OawD++fdhiVa5teaUdQ+RakHUD2RUBTriQ1uEbgjIiYLqk3ME3SvcCXgPsi4gJJZwNnA98BDgVGJtuewG+Tf9vkGpyZFS2HCtraExELImJ6sr8KmAUMAcYDk5K3TQKOSPbHA1dH3qNAP0mD27tGRdXgzKzyBUUNMgyQNLXF64kRMfHdb5I0AtgFeAwYFBHNdfE3gEHJ/hBgXouPzU/K2qy3O8GZWZGKGmRYHBFj2z2b1Au4GTgtIlZKb587IkJSbGqkbqKaWdEiCts6IqmOfHK7NiJuSYrfbG56Jv8uTMpfA4a1+PjQpKxNTnBmVrQSjaIKuBKYFREXtzg0GTg+2T8euL1F+XHJaOpewIoWTdlWuYlqZkXJj6KWpG60L3As8G9JTyVl3wMuAG6UdALwCnBUcuxO4DBgDrAW+HJHF3CCM7OiFdL87Pgc8RC0OdR6YCvvD+CkYq7hBGdmRfNULTPLpKDj/rVK4QRnZkUrQQt1i3CCM7PiBERppmqVnROcmRXNTVQzy6xSjKJuCW0mOEmX0U5TOyK+VZaIzKyiFTkXNVXt1eCmtnPMzDqrAKo9wUXEpJavJfWMiLXlD8nMKl21NFE7nG8haW9JM4Hnktc7S7q87JGZWYUSkStsS1shE8p+CRwMLAGIiKeBcWWMycwqXRS4paygUdSImNdyjSagqTzhmFnFi2wMMjSbJ2kfIJK1m04lv7SwmXVWFVA7K0QhTdRvkJ/BPwR4HRhDkTP6zSxrVOCWrg5rcBGxGPjiFojFzKpFLu0AClPIKOr2ku6QtEjSQkm3S9p+SwRnZhWo+T64QraUFdJEvQ64ERgMbAvcBFxfzqDMrLKV6pkM5VZIgusZEddERGOy/RHoXu7AzKyCVfttIpL6J7t3JU+XvoF8yEeTXxvdzDqrCmh+FqK9QYZp5BNa80/y9RbHAvhuuYIys8q26U8q3bLam4u63ZYMxMyqRAgqYBpWIQqaySBpJ2A0LfreIuLqcgVlZhWu2mtwzSSdA+xHPsHdCRwKPAQ4wZl1VlWS4AoZRf0s+WcUvhERXwZ2BvqWNSozq2zVPorawrqIyElqlNQHWAgMK3NcW9xp5z/HHh9fwvKldZx4xB4AbL/jKk7+wfPUdcuRaxS/+dEHef7ffVKONB11dU38/NJ/Utc1R21tjof+OZRrrxrNzrsu5ISv/xvVBOvXdeHiC8ay4PVeaYebqrpuOX5xyxzquga1XYIH/9qPay56X9phlU4WFrxsYaqkfsDvyY+srgYe6ehDkv4AfApYGBE7bU6QW8Lfb3sfd1w3hDN++vY6Al85/SWuu3wEUx/amrEfW8JXTn+Rs7+8S4pRpqehoYbvnj6O9eu7UFub46LLHmDqY4M4+bQnOe9/92beq304fPyLHHPsc1zys7Fph5uqhg3irM99gPVra6ntElx82xye+Edvnpten3ZoJVP1o6jNIuLEZPd3ku4G+kTEMwWc+yrg11RJX92z0/qxzbbr3lEWQM9e+ZWh6ns3snRRtxQiqxRi/fr8r0uXLjlqa/N3EEVAz/pGAOrrG1i6xPeAg1i/thaALnVBbV1UxF39JVUlP097N/ru2t6xiJje3okjYoqkEZsRW+omXrAD5098hhPOfBHVBGd+sc2vpFOoqQku/b/72HbIav5y2weYPas/l160Gz/86cNs3FjL2jVd+PZJ+6cdZkWoqQl+fc/zbDtiI3dctTWzn8xO7Q2yUYP7RTvHAjigFAFImgBMAOheU1l9N4cd/Tq//9kOPHzvQD528EJOPf85vv/VMWmHlZpcTpzytYOor9/I/57/KMNHrOCIz77AOd/dl9mz+vNfR89mwonPcOlFu6UdaupyOXHiJ0ZR36eJc66cy/BR63hldo+0wyqdEvXBtdaVJelc4GvAouRt34uIO5Nj3wVOIL/o7rci4p72zt/mKGpE7N/OVpLkllxnYkSMjYixXVVZzZuDxr/Bw/cOAODBewYy6iOrUo6oMqxZ05VnnhrI2D3fYPsPrGD2rPysvin3D+NDH16ScnSVZc3KWp7+Vy923z9DvzuFjqAWVsu7CjiklfJLImJMsjUnt9HAMcCHk89cLqm2vZMXcptIp7VkYTc+svtyAHbeczmvvZKhv8BF6tN3A/X1GwHo2rWJXXZ7k3mv9KFnrwaGDM3/591l7JvMe7V3mmFWhL79G6nvk++77do9x67jVjNvTmX98d5sJUpwETEFWFrgVccDN0TEhoiYC8wB9mjvA36yfeKsC2fy0d2X06dfA1ff9y/++Jvt+NW5H+TrZ8+htkvQsKGGy84dlXaYqem/9XrOOPsJamoC1cCDDwzl8UcH86uLduX7P3yUXIjVq+r45c/dPO0/qIEzL32VmhqoqYEpd/Tlsb9n6/YiFb7g5QBJLZ+xPDEiJhbwuZMlHUf++cxnRMQy8quKP9riPfOTsjaVLcFJup78DIgBkuYD50TEleW63ub6+f+MbrX81KM69y0PzV5+qS+nTDjoPeWPPDSERx5q93es05k7qwcnfTLjfwwLH2RYHBHF/if6LXB+cpXzyY8HfKXIcwCFTdUS+SXLt4+I8yS9H3hfRDze3uci4vObEpCZVTZFeUdRI+LNt64l/R74S/LyNd45yWBoUtamQvrgLgf2BpoT1irgN4UGa2YZVMYlyyUNbvHySODZZH8ycIykbpK2A0YC7Va0Cmmi7hkRu0p6EiAilknquglxm1lWlKgG11pXFrCfpDHJVV4mWYsyImZIuhGYCTQCJ0VEu89oLiTBNSRDsZEENJCqeaaOmZVDqZqobXRltdlXHxE/Bn5c6PkLSXC/Am4FtpH0Y/Kri/xvoRcws4yJokZRU1XIXNRrJU0jv2SSgCMiwk+2N+vMMjBVC4Bk1HQtcEfLsoh4tZyBmVkFy0qCA/7K2w+f6Q5sB8wmP13CzDqhLEy2ByAiPtLydbLKyIltvN3MrGIUPZMhIqZL2rMcwZhZlchKDU7S6S1e1gC7Aq+XLSIzq2xZGkUFWi4P0Ui+T+7m8oRjZlUhCzW45Abf3hFx5haKx8wqnMjAIIOkLhHRKGnfLRmQmVWBak9w5Cex7go8JWkycBOwpvlgRNxS5tjMrBKVeTWRUiqkD647sIT8Mxia74cLwAnOrLPKwCDDNskI6rO8ndiaVUn+NrNyyEINrhboxTsTW7Mq+fHMrCyqJAO0l+AWRMR5WywSM6sOhT8xK3XtJbjSPPjQzDInC03UA7dYFGZWXao9wUVEoc8qNLNOJktTtczM3paRPjgzs/cQ1dNB7wRnZsVzDc7MsioLo6hmZq1zgjOzTMrYgpdmZu/kGpyZZZX74Mwsu5zgihdNTTQtX552GBWrZuPGtEOoeLX9+qYdQkXTytrSnKdECU7SH4BPAQsjYqekrD/wJ2AE8DJwVEQskyTgUuAw8g+j/1JETG/v/DWlCdPMOo0gv+BlIVvHrgIOeVfZ2cB9ETESuC95DXAoMDLZJgC/7ejkTnBmVpTmh84UsnUkIqYA7573Ph6YlOxPAo5oUX515D0K9JM0uL3zO8GZWfGiwA0GSJraYptQwNkHRcSCZP8NYFCyPwSY1+J985OyNlVUH5yZVQdFwZ1wiyNi7KZeJyJC2vQeP9fgzKw4hdbeNn0g4s3mpmfy78Kk/DVgWIv3DU3K2uQEZ2ZFK1UfXBsmA8cn+8cDt7coP055ewErWjRlW+UmqpkVrVRTtSRdD+xHvq9uPnAOcAFwo6QTgFeAo5K330n+FpE55G8T+XJH53eCM7Pileg+uIj4fBuH3vPIhIgI4KRizu8EZ2bFydiT7c3M3skJzsyyqPlG32rgBGdmRVOuOjKcE5yZFcdP1TKzLPOKvmaWXa7BmVlWeZDBzLIpgMIn26fKCc7MiuY+ODPLJN8HZ2bZFeEmqplll2twZpZdTnBmllWuwZlZNgXQVB0ZzgnOzIrmGpyZZZdHUc0sq1yDM7Ns8nJJZpZVAuRBBjPLqiKebJ8qJzgzK46bqGaWXZ6LWtUmPTqDdatryeWgqVGcctiotENK3bd/Ooc99l/K8iV1fPPwXQA49rRX2fvApeQCViyp4xffGcnShV1TjjQ9p/1oNnt8fCnLl9Zx4vixAJz9i1kM2W4tAL16N7J6VRdO+cxuaYZZEp1+FFXSMOBqYBD5Cu3EiLi0XNcrtbM+twMrlzn/N7v3loFMvuZ9nHnhC2+V3XzFtlzzy/cD8J/HLeALJ8/j1z/4QFohpu7vtw7ijmu35YwLZr9VdsEZH3pr/6tnvciaVRn5naqSGlxNGc/dCJwREaOBvYCTJI0u4/WsjJ59oi+rVrzzP+fa1W+/7t6jqWr6Zcrl2Wn9WLWiro2jwccOXsQ/79xmi8ZUFpEfRS1kS1vZ/pxExAJgQbK/StIsYAgws1zXLJkQP7n+RQj46x+35q5rB6QdUcU6/tuvcOCRi1izqpazj90p7XAq1k67rWD5kq68/kqPtEMpjRLlLkkvA6uAJqAxIsZK6g/8CRgBvAwcFRHLNuX85azBvUXSCGAX4LFWjk2QNFXS1AY2bIlwOnT6kTtw8iGj+P5/b89/fmkxO+25Ou2QKtakS4Zz3Lix3D95IJ/+7wVph1OxPn74Ih7IQu0toYiCtgLtHxFjImJs8vps4L6IGAncl7zeJGVPcJJ6ATcDp0XEyncfj4iJETE2IsbW0a3c4RRkyRv5jvIVS+p4+K6+7DhmbcoRVb77Jw9k34OXpB1GRaqpDfY5aDFT7hqYdiil07yqb0fbphkPTEr2JwFHbOqJyprgJNWRT27XRsQt5bxWqXTr0USP+qa39nf7+Cpent095agq07bD1721v/dBS5n/UkaaXyW2y97LmD+3J0verIw/4JstgFyBGwxobqEl24RWzvY3SdNaHBuUdHEBvEF+oHKTlHMUVcCVwKyIuLhc1ym1rQY2cs6VcwGorYX7b+vH1Af6pBxV+r5zyfN8dI8V9NmqkWsenMo1lw5j9/2WMXS7dUROLHy9G5f9YPu0w0zVWRfOyn9H/Rq4+h+P8sdfD+dvtwxm3KGL+Oed2am9iaKan4tbND1b8x8R8ZqkbYB7JT3X8mBEhLTpN6UoyjTcK+k/gAeBf9Ocy+F7EXFnW5/po/6xZ81BZYknC2p69kw7hIqnuozchlEmj6y8nRWNi7Q55+hbv23stePXCnrv36afN62DBPcWSecCq4GvAftFxAJJg4EHImKTbkYt5yjqQ+Tn5ZpZljQ3UTeTpHqgJrnLoh74JHAeMBk4Hrgg+ff2Tb2G/9yZWdFKNNl+EHBrvjeLLsB1EXG3pCeAGyWdALwCHLWpF3CCM7PilSDBRcRLwM6tlC8BDtzsC+AEZ2ZF82R7M8sqP1XLzLLMC16aWXY5wZlZJgWQc4Izs0zyIIOZZZkTnJllUgBNJZjKsAU4wZlZkQLCCc7MsspNVDPLJI+imlmmuQZnZpnlBGdmmRQBTU1pR1EQJzgzK55rcGaWWU5wZpZN4VFUM8uogPCNvmaWWZ6qZWaZFAE5JzgzyyoPMphZVoVrcGaWTV7w0syyypPtzSyrAghP1TKzTAoveGlmGRZuoppZZlVJDU5RQaMhkhYBr6QdRwsDgMVpB1HB/P10rNK+o+ERMXBzTiDpbvI/VyEWR8Qhm3O9zVFRCa7SSJoaEWPTjqNS+fvpmL+jdNWkHYCZWbk4wZlZZjnBtW9i2gFUOH8/HfN3lCL3wZlZZrkGZ2aZ5QRnZpnlBNcKSYdImi1pjqSz046n0kj6g6SFkp5NO5ZKJGmYpPslzZQ0Q9KpacfUWbkP7l0k1QLPA58A5gNPAJ+PiJmpBlZBJI0DVgNXR8ROacdTaSQNBgZHxHRJvYFpwBH+HdryXIN7rz2AORHxUkRsBG4AxqccU0WJiCnA0rTjqFQRsSAipif7q4BZwJB0o+qcnODeawgwr8Xr+fiX0zaRpBHALsBjKYfSKTnBmZWJpF7AzcBpEbEy7Xg6Iye493oNGNbi9dCkzKxgkurIJ7drI+KWtOPprJzg3usJYKSk7SR1BY4BJqcck1URSQKuBGZFxMVpx9OZOcG9S0Q0AicD95DvHL4xImakG1VlkXQ98AgwStJ8SSekHVOF2Rc4FjhA0lPJdljaQXVGvk3EzDLLNTgzyywnODPLLCc4M8ssJzgzyywnODPLLCe4KiKpKbnl4FlJN0nquRnnukrSZ5P9KySNbue9+0naZxOu8bKk9zx9qa3yd71ndZHXOlfSmcXGaNnmBFdd1kXEmGQFj43AN1oelLRJz7mNiK92sNLFfkDRCc4sbU5w1etBYIekdvWgpMnATEm1ki6U9ISkZyR9HfJ310v6dbLO3d+BbZpPJOkBSWOT/UMkTZf0tKT7ksni3wC+ndQePyZpoKSbk2s8IWnf5LNbS/pbsgbaFYA6+iEk3SZpWvKZCe86dklSfp+kgUnZByTdnXzmQUk7luTbtEzyk+2rUFJTOxS4OynaFdgpIuYmSWJFROwuqRvwsKS/kV/RYhQwGhgEzAT+8K7zDgR+D4xLztU/IpZK+h2wOiIuSt53HXBJRDwk6f3kZ318CDgHeCgizpN0OFDIDIevJNfoATwh6eaIWALUA1Mj4tuSfpCc+2TyD3H5RkS8IGlP4HLggE34Gq0TcIKrLj0kPZXsP0h+vuM+wOMRMTcp/yTw0eb+NaAvMBIYB1wfEU3A65L+0cr59wKmNJ8rItpa8+0gYHR+yiUAfZKVM8YBn0k++1dJywr4mb4l6chkf1gS6xIgB/wpKf8jcEtyjX2Am1pcu1sB17BOygmuuqyLiDEtC5L/6GtaFgGnRMQ973pfKedC1gB7RcT6VmIpmKT9yCfLvSNiraQHgO5tvD2S6y5/93dg1hb3wWXPPcA3k+V6kPRBSfXAFODopI9uMLB/K599FBgnabvks/2T8lVA7xbv+xtwSvMLSWOS3SnAF5KyQ4GtOoi1L7AsSW47kq9BNqsBmmuhXyDf9F0JzJX0ueQakrRzB9ewTswJLnuuIN+/Nl35h8L8H/ma+q3AC8mxq8mvBvIOEbEImEC+Ofg0bzcR7wCObB5kAL4FjE0GMWby9mjuD8knyBnkm6qvdhDr3UAXSbOAC8gn2GZrgD2Sn+EA4Lyk/IvACUl8M/By8tYOryZiZpnlGpyZZZYTnJlllhOcmWWWE5yZZZYTnJlllhOcmWWWE5yZZdb/B9co2aFxGhirAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# use display matrix function to heatmap prediction/true cases\n",
    "cm = confusion_matrix(test_y, tuned_preds)\n",
    "disp = ConfusionMatrixDisplay(cm)\n",
    "disp.plot()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "3ab8ebe3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TPR</th>\n",
       "      <th>FNR</th>\n",
       "      <th>FPR</th>\n",
       "      <th>TNR</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.915663</td>\n",
       "      <td>0.084337</td>\n",
       "      <td>0.244681</td>\n",
       "      <td>0.755319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.644068</td>\n",
       "      <td>0.355932</td>\n",
       "      <td>0.103542</td>\n",
       "      <td>0.896458</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.485714</td>\n",
       "      <td>0.514286</td>\n",
       "      <td>0.015345</td>\n",
       "      <td>0.984655</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        TPR       FNR       FPR       TNR\n",
       "1  0.915663  0.084337  0.244681  0.755319\n",
       "2  0.644068  0.355932  0.103542  0.896458\n",
       "3  0.485714  0.514286  0.015345  0.984655"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print the Confusion Matrix\n",
    "\n",
    "conf_mat = pd.DataFrame(get_tpr_fnr_fpr_tnr(cm)).transpose()\n",
    "conf_mat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cf7030c",
   "metadata": {},
   "source": [
    "## Multilayer Perceptron NN Classifier\n",
    "### Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4dd833b",
   "metadata": {},
   "source": [
    "For our neural network model, a multilayer perceptive classifier was chosen for our project. Given the strong imbalances between classes, data was train-test-split in a stratified manner. This offered assurance that classes with small counts were not further reduced during the split.  A train-test-split ratio of 80/20 was chosen to offer our model a wider range   of training cases to learn from. To further accommodate the imbalance between class counts, oversampling was also conducted to increase class counts to match the larger class count, of 1323. This method was chosen instead of under-sampling to better complement   this model type which is characterized to benefit from larger training sets.  Two scalers were tested, and a standard scaler proved to provide slightly higher accuracy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a79b32ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Random Oversampling\n",
    "over_sampler = RandomOverSampler(sampling_strategy = 'auto', random_state = 0)\n",
    "ros_X, ros_y = over_sampler.fit_resample(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7fe5fd68",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0    1323\n",
       "2.0     236\n",
       "3.0     141\n",
       "Name: fetal_health, dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_y.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b32ede4c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0    1323\n",
       "1.0    1323\n",
       "3.0    1323\n",
       "Name: fetal_health, dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check counts/oversampling results\n",
    "ros_y.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "2592b90b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encode classes so they are 0-2 instead of 1-3\n",
    "# Now training y and test_y will be label encoded and oversampled\n",
    "enc = preprocessing.LabelEncoder()\n",
    "ros_y = enc.fit_transform(ros_y)\n",
    "test_y2 = enc.transform(test_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "51d536b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create scaler\n",
    "scaler = StandardScaler()\n",
    "# Create Model Instance for gridsearching parameters\n",
    "mlp = MLPClassifier()\n",
    "# Create pipeline\n",
    "pipe = Pipeline(steps=[('scaler', scaler ), ('mlpc', mlp)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7dd444d",
   "metadata": {},
   "source": [
    "### Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "04159a60",
   "metadata": {},
   "outputs": [],
   "source": [
    "#mlp.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6005dbf5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Define gridspace\n",
    "# param_grid= [{'mlpc__activation': ['relu','tanh'],\n",
    "#              'mlpc__alpha': [0.0025, 0.05, 0.5 ],\n",
    "#              'mlpc__hidden_layer_sizes': [(12,6)],\n",
    "#              'mlpc__max_iter': [2000],\n",
    "#              'mlpc__solver': ['adam','lbfgs'],\n",
    "#              'mlpc__batch_size': [20,30,50],\n",
    "#              'mlpc__learning_rate':['invscaling','adaptive']\n",
    "#              }]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "fc3c1e71",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Run a new gridsearch with new search_space\n",
    "# grid_mlp = GridSearchCV(pipe, param_grid, n_jobs= -1, cv = 2, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "5312400c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Fit grid search\n",
    "# grid_mlp.fit(ros_X,ros_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "a0951078",
   "metadata": {},
   "outputs": [],
   "source": [
    "# grid_mlp.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "011bd930",
   "metadata": {},
   "outputs": [],
   "source": [
    "# print('Best score: ', grid_mlp.best_score_)\n",
    "# print('Test Accuracy: ', grid_mlp.score(test_X, test_y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "9d8ed322",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create MLP model with chosen params\n",
    "best_mlp = MLPClassifier(\n",
    "                        activation= 'tanh', \n",
    "                        alpha = 0.001, \n",
    "                        hidden_layer_sizes = (12,6), \n",
    "                        max_iter = 2000, \n",
    "                        solver = 'lbfgs',\n",
    "                        random_state = 12,\n",
    "                        learning_rate = 'adaptive'\n",
    "                        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "938ae49f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update pipeline\n",
    "pipe = Pipeline(steps=[('scaler', scaler ), ('mlpc', best_mlp)])\n",
    "\n",
    "# Fit to features\n",
    "pipe.fit(ros_X, ros_y)\n",
    "\n",
    "# Get predictions\n",
    "pred_y = pipe.predict(test_X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5db3f628",
   "metadata": {},
   "source": [
    "### Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "fc196a58",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test model accuracy\n",
    "# cross validate\n",
    "mlp_scores = cross_validate(pipe, ros_X, ros_y, return_train_score=True)\n",
    "mlp_scores_df = pd.DataFrame(mlp_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "3bcf13e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       3.068092\n",
       "score_time     0.004839\n",
       "test_score     0.980096\n",
       "train_score    0.995339\n",
       "dtype: float64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlp_scores_df.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "162bef1c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9107981220657277"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get accuracy of predictions\n",
    "accuracy_score(test_y2, pred_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "bbf60cad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create confusion matrix\n",
    "cm = confusion_matrix(test_y2, pred_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "afa6d9f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAEGCAYAAADxD4m3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAAsTAAALEwEAmpwYAAAeUklEQVR4nO3deZgV1bnv8e+vm5ZmHgQVpJ2JBo2iQZwi1zlq4lWTGE1yEm/iOTgmaozRk3MSjc/1ZnI4mhxNEJyNRq+oJBoVp6hJVBAHBCV0nABRZJBZobvf88euxg1C996wd9fe1b/P89TTVatqV729n+ZlVa1aaykiMDPLopq0AzAzKxcnODPLLCc4M8ssJzgzyywnODPLrC5pB5BvQP/a2K6hLu0wKtbMV3qmHULFi5aWtEOoaB+ynFXxkTblHJ8/uEcsWNhc0LHPv/zRQxFx5KZcb1NUVILbrqGO5x5qSDuMinXU0APSDqHitSxfnnYIFe3ZeHSTzzF/YTPPPjSkoGPrBv1zwCZfcBNUVIIzs2oQNEd11JSd4MysKAG0UB0dBJzgzKxoLbgGZ2YZFASrfYtqZlkUQLNvUc0sq/wMzswyKYDmKhmFyD0ZzKxoLQUubZFUL+k5SS9Jmibpp0n59pKeldQo6Q+SNkvKuybbjcn+7dqL0wnOzIoSBM0FLu34CDgkIvYAhgNHStoX+AVwZUTsBCwCTkmOPwVYlJRfmRzXJic4MytKBKwucGn7PBERsSzZrEuWAA4B/n9SfhNwXLJ+bLJNsv9QSW12O3OCM7MiieYCF2CApMl5y+i1ziTVSnoRmAdMBP4JfBARTckhs4Gtk/WtgVkAyf7FwOZtRepGBjMrSgAthbcxzI+IERs8V0QzMFxSX+AeYJdNjS+fa3BmVrQianAFiYgPgMeB/YC+klorX0OAOcn6HKABINnfB1jQ1nmd4MysKLkXfTc9wUkamNTckNQNOBx4lVyi+0py2MnAfcn6hGSbZP9j0c6sWb5FNbOiBLA6SlI3GgTcJKmWXGXrzoj4k6TpwB2S/i/wAjAuOX4ccIukRmAhcFJ7F3CCM7OiBKK5BDd/EfEysOd6yl8HRq6n/EPghGKu4QRnZkVriU0aFLjDOMGZWVFan8FVAyc4MyuSaC7NM7iyc4Izs6LkRvR1gjOzDIoQq6I27TAK4gRnZkVr8TM4M8uiXCODb1HNLJPcyGBmGeVGBjPLtGa/6GtmWRSI1VEdqaM6ojSziuFGBjPLrEC+RTWz7HIjQ4Vb9aE470s7sXpVDc1NcOAXFvOt89/lvusHcM/Ygcx9syt3Tp1Kn82bAbjrmoE8Nr4/AM3NMGtmPX+Y+gq9+zWn+Wt0mHN/1sjIgxfywYI6Tv9CboSbUy54k30OXkTTajH37XquuHAnli/ttH9Sa9R1beHy8Y3UbRbUdgmeur8vt1y2VdphlUwEfk0EQNKRwFVALTA2In5ezusVo65r8Mu7/km3Hi00rYbvHzeUvQ9Zwq57L2efw5fwwy/vtNbxJ5zxPiec8T4Azzzcm/HXDew0yQ1g4viBTLhlK37wq5lryl74a19uuGxbWprFd85/kxNPm831v9ouvSArxOqPxA9P2JEPV9RS2yW44t5GJj3Wi9em9Eg7tJLINTJUR1etsqXhZJTO/waOAoYBX5M0rFzXK5YE3XrkpqZtWi2aVwsJdvrMSrZqWNXmZx+/tx8HHbeoI8KsGK9M6sPSxWv/fzjl6b60NOeexbz2Yi8GbNX299Z5iA9X5BJAl7qgti6okongC9ZMTUFL2soZwUigMSJej4hVwB3k5jWsGM3NcPphO3Pi7rux56il7LLXinY/8+EKMfmJXnzu6MUdEGH1OOIr85j0l35ph1ExamqCaybO4A8vT+OFJ3sy44Vs1N4gV4NricKWtJUzwa2ZwzCRP79hRaithWsfmcFtz09nxovdefO1+nY/88zEPuw6Ynmnuj1tz0mnz6a5STw+YUDaoVSMlhZxxuE7843PDmPn4SvYdueVaYdUUq7BFUjS6NZJYd9fkE7S6NmnmT32X8akx3u1e+xf7uvb6W5P23LYl+Yx8uCF/PK8oVAlI0x0pOVLannpbz3Z++ClaYdSMrl5UWsKWtJWzgjWzGGYyJ/fcI2IGBMRIyJixMDNO+7B5QcLalm2OHe9j1aKKU/2omGnj9r8zPIlNbz8TE/2P3JJR4RY8T574CJO+Lc5/PS0T/PRh9Xx0Lkj9OnfRI/euf+sN6tvYa9Ry5jV2P7dQfUoamb7VJWzFXUSMFTS9uQS20nA18t4vaIsfK+Oy87ehpYW0dICo475gH0PX8K9Ywdw17VbsHBeHacdtgsjD1nCuZfn7rT/+ue+fHbUUuq7t6Qcfce74Mp/sPvIxfTu18QtT03mlqsaOPG0OdRt1sKlN04Dcg0Nv/nJjilHmr7+W67mB1e9TU0N1NTAk3/sw7OP9E47rJLJTRtYHf+hqZ15Uzft5NLRwH+Re03k+oi4tK3jR+xRH8891NDWIZ3aUUMPSDuEiteyfHnaIVS0Z+NRlsTCTapabb1r3zjjzs8VdOx/7nb/8xExYlOutynK+h5cRDwAPFDOa5hZx/OLvmaWSbnx4NJ/vlaI6kjDZlZBciP6FrK0eRapQdLjkqZLmibp7KT8YklzJL2YLEfnfebfJTVKmiHp8+1F6hqcmRUl95pISWpwTcB5ETFFUi/geUkTk31XRsRl+QcnPaFOAnYFBgOPSPpURGzw/TInODMrSqn6okbEXGBusr5U0qu03RngWOCOiPgIeENSI7keU3/f0Ad8i2pmRWuhpqClUJK2A/YEnk2KzpL0sqTrJbX2ASy6d5QTnJkVJTdckgpagAGtPZWSZfS655PUE7gbOCcilgDXAjsCw8nV8C7f2Fh9i2pmRSviGdz8tt6Dk1RHLrndFhHjASLivbz91wF/SjYL6h2VzzU4MytKbjSRTe+LKknAOODViLgir3xQ3mHHA68k6xOAkyR1TXpIDQWea+sarsGZWVFyXbVKUjc6APgmMFXSi0nZj8iNHTk8udSbwKkAETFN0p3AdHItsGe21YIKTnBmVjSVZKSQiHia9Q9Bs8HeT0l3zza7fOZzgjOzolVLTwYnODMrSmsrajVwgjOzolXCYJaFcIIzs6K0zslQDZzgzKwoATS5BmdmWeVbVDPLpgqZErAQTnBmVpRqGvDSCc7MiuYanJllUgkHvCw7JzgzK0ogmlrcyGBmGeVncGaWTeFbVDPLKD+DM7NMc4Izs0wKRLMbGcwsq9zIYGaZFG5kMLMsCyc4M8smd7Y3swxzDW4jzHylJ0cNPSDtMCrXjg3tH9PZvfxa2hFkXgQ0tzjBmVlGuRXVzDIp8C2qmWWWGxnMLMMi0o6gMNXR38LMKkqEClraIqlB0uOSpkuaJunspLy/pImSZiY/+yXlknS1pEZJL0vaq704neDMrCi5VtSagpZ2NAHnRcQwYF/gTEnDgAuBRyNiKPBosg1wFDA0WUYD17Z3ASc4MytaRGFL2+eIuRExJVlfCrwKbA0cC9yUHHYTcFyyfixwc+Q8A/SVNKita/gZnJkVrYhW1AGSJudtj4mIMeseJGk7YE/gWWDLiJib7HoX2DJZ3xqYlfex2UnZXDbACc7MihK0/3wtz/yIGNHWAZJ6AncD50TEEunjc0dESNroJg3foppZ0aLApT2S6sglt9siYnxS/F7rrWfyc15SPgfI784zJCnbICc4MytOQLSooKUtylXVxgGvRsQVebsmACcn6ycD9+WVfytpTd0XWJx3K7tevkU1s6KVqCfDAcA3gamSXkzKfgT8HLhT0inAW8BXk30PAEcDjcAK4NvtXcAJzsyKVooXfSPiadhgp9ZD13N8AGcWc40NJjhJv6aN2+iI+F4xFzKzbMhKX9TJbewzs84qgGpPcBFxU/62pO4RsaL8IZlZpctMX1RJ+0maDryWbO8h6ZqyR2ZmFaqwFtT2WlE7QiGvifwX8HlgAUBEvASMKmNMZlbpSvUiXJkV1IoaEbPy3y4GmssTjplVvMhGI0OrWZL2ByJ56/hscp1izayzqoDaWSEKuUU9jdy7J1sD7wDDKfJdFDPLGhW4pKvdGlxEzAe+0QGxmFm1aEk7gMIU0oq6g6Q/Snpf0jxJ90naoSOCM7MK1PoeXCFLygq5Rf09cCcwCBgM3AXcXs6gzKyylWLAy45QSILrHhG3RERTstwK1Jc7MDOrYNX+moik/snqnyVdCNxBLuQTyfXqN7POqgJuPwvRViPD8+QSWutvcmrevgD+vVxBmVll2/gxdjtWW31Rt+/IQMysSoSgArphFaKgngySdgOGkffsLSJuLldQZlbhqr0G10rSRcBB5BLcA+TmJnwacIIz66yqJMEV0or6FXKja74bEd8G9gD6lDUqM6ts1d6KmmdlRLRIapLUm9wMNw3tfajanPuzRkYevJAPFtRx+hf2BOCb57zNfocupCVg8YI6Lr9gKAvnbZZypOmpqWnh6qsnMn9BNy6+6OMBZU47fQpHHPEGXzr+yylGVzkGDl7F+Ve9Td+BTRDwwK2bc++4gWmHVTpVNOBlITW4yZL6AteRa1mdAvy9vQ9Juj7p+fDKpoXYMSaOH8h/fmfYWmV3jx3MGccM56z/PZxnH+/P18+atYFPdw7HHjeTt2f1Xqts6NCF9Oy5KqWIKlNzkxhzyWBGH7QLZ39xKMf8n/lsM/TDtMMqKUVhS9raTXARcUZEfBARvwUOB05OblXbcyNw5CbG12FemdSHpYvXrtCuWPbxdn235oqocqdlwIAVjNz7HR568ONeejU1LZzyry8xbtweKUZWeRbOq6NxancAVi6vZVZjPQMGrU45qhKr9ltUSXu1tS8iprR14oh4UtJ2mxBbRTj53Lc49Pj3Wb60lgu/uVva4aTm1FNfYNy4PejWvWlN2THHNPLMM4NZtLBbipFVti2HrGLH3Vby2pTuaYdSUpVQOytEW8/gLm9jXwCHlCIASaOB0QD16lGKU5bUTVduy01XbstXT53NMf8yl1uv3ibtkDrcyJHv8MEHXWls7M9nds9NMt6//0oOHDWLH55/cMrRVa767s38eOyb/PYng1mxrDbtcEqrSp7BtfWib4f85UbEGGAMQJ/aARX7/8LjEwZyydjpnTLBDdt1Pvvu+w57j/wjdXUtdO++mt/+7s+sXl3L9TfcD0DXrk2Mu/5+TvnOF1KOtjLUdgl+PPZNHhvfj7/+uW/a4ZRWhdx+FsITP7dh8LYreeet3O3XfoctZPbrnfNW7MYbdufGG3YH4DO7z+PLX35trVZUgPH33O3ktkbw/ctnMWtmPePHZKj1NJ8TXHW54Mp/sPvIxfTu18QtT03mlqsa2PugRQzZfiXRIua905Vf/8TD4Fn7dh25nMNOWMTr0+u5ZuIMAG742SAmPda7nU9WD5VowEtJ1wNfBOZFxG5J2cXAvwHvJ4f9KCIeSPb9O3AKuXlhvhcRD7V5/ijToE2SbifXA2IA8B5wUUSMa+szfWoHxL7dv1iWeDJhx8y9flhyLS+/lnYIFe3ZeJQlsXCTHqB1bWiIIWefW9Cxr59/3vMRMWJD+yWNApYBN6+T4JZFxGXrHDuM3FiUI8mNTfkI8KmI2OAkWIV01RK5Ict3iIhLJG0DbBURz7X1uYj4WnvnNrPqU8p33Ip82+JY4I6I+Ah4Q1IjuWS3wfdyC3nR9xpgP6A1YS0F/rvAgMwsi8o/ZPlZkl5OOgz0S8q2BvLftp+dlG1QIQlun4g4E/gQICIWAZ23v5KZFfOi7wBJk/OW0QWc/VpgR3Iz+M2l7VfW2lRII8NqSbUk4UoaSNXMqWNm5VDELer8tp7BrU9EvLfmOtJ1wJ+SzTms3Q9+SFK2QYXU4K4G7gG2kHQpuaGS/l8xAZtZhkSuFbWQZWNIGpS3eTzQ2p99AnCSpK6StgeGAm22BRQyL+ptkp4nN2SSgOMiwjPbm3VmJWpkyH/bQtJs4CLgIEnDk6u8STJdQkRMk3QnMB1oAs5sqwUVCmtF3QZYAfwxvywi3t6I38fMsqB0rajre9tig6+TRcSlwKWFnr+QZ3D38/HkM/XA9sAMYNdCL2Jm2ZKFzvYARMRn8reTUUbOKFtEZmYlUnRXrYiYImmfcgRjZlUiKzU4Sd/P26wB9gLeKVtEZlbZonR9UcutkBpcr7z1JnLP5O4uTzhmVhWyUINLXvDtFRE/6KB4zKzCiQw0MkjqEhFNkg7oyIDMrApUe4Ij94bwXsCLkiYAdwHLW3dGxPgyx2ZmlahCZswqRCHP4OqBBeTmYGh9Hy4AJzizzioDjQxbJC2or/BxYmtVJfnbzMohCzW4WqAnaye2VlXy65lZWVRJBmgrwc2NiEs6LBIzqw4ZmVWrOiY+NLMOl4Vb1EM7LAozqy7VnuAiYmFHBmJm1SNLXbXMzD6WkWdwZmafIKrnAb0TnJkVzzU4M8uqLLSimpmtnxOcmWVSxga8NDNbm2twZpZVfgZnZtnlBFe8aGmhZfny9g/spDS9Me0QKl6XIVunHUJF07t1pTmPE5yZZVJQNQNe1qQdgJlVl9ZJZwpZ2j2XdL2keZJeySvrL2mipJnJz35JuSRdLalR0svJJPRtcoIzs+JFgUv7bgSOXKfsQuDRiBgKPJpsAxwFDE2W0cC17Z3cCc7MiqaIgpb2RMSTwLojFx0L3JSs3wQcl1d+c+Q8A/SVNKit8zvBmVlxCq295fLbAEmT85bRBVxhy4iYm6y/C2yZrG8NzMo7bnZStkFuZDCzohXRijo/IkZs7HUiIqSNb7N1Dc7MiqaWwpaN9F7rrWfyc15SPgdoyDtuSFK2QU5wZla80jUyrM8E4ORk/WTgvrzybyWtqfsCi/NuZdfLt6hmVpwSzmwv6XbgIHLP6mYDFwE/B+6UdArwFvDV5PAHgKOBRmAF8O32zu8EZ2bFK1GCi4ivbWDXJya9iogAzizm/E5wZlaU1hd9q4ETnJkVTS3VkeGc4MysOJ5Vy8yyzCP6mll2uQZnZlnlRgYzy6YACuhIXwmc4MysaH4GZ2aZ5PfgzCy7InyLambZ5RqcmWWXE5yZZZVrcGaWTQE0V0eGc4Izs6K5Bmdm2eVWVDPLKtfgzCybPFySmWWVALmRwcyyqpBZ6yuBE5yZFce3qNVr4OBVnH/V2/Qd2AQBD9y6OfeOG5h2WBWnR+8mzvnlW2z3qZVEiCvP35ZXp/RMO6zUDNhiJedd/BJ9+68igAfvaWDCH7bngktfYMi2ywDo0bOJ5cu68N1/OTDdYDeZ+6IiqQG4GdiSXL4fExFXlet6pdLcJMZcMpjGqd3p1qOZ3zz4D6Y82Yu3Z9anHVpFOe3iWTz/RB8uPW1HutS10LVblYyfUybNzWLsVZ/mnzP60K17E1fd/DQvPDeAX/zHnmuOOeXsV1mxLBt1imppRS3nzPZNwHkRMQzYFzhT0rAyXq8kFs6ro3FqdwBWLq9lVmM9AwatTjmqytK9VzOfGbmMB+/YHICm1TUsX5KNf7gba9GCev45ow8AK1d0YdYbPdl84Id5RwQHHjaXvzw8OJ0AS611RJH2lpSV7a8yIuYCc5P1pZJeBbYGppfrmqW25ZBV7LjbSl6b0j3tUCrKVg0fsXhhF867/C22//QKGqd259qLG/hoZW3aoVWELQatYIedlzBjWt81ZbvuuYgPFm7GO7N6pBdYqUT1tKKWswa3hqTtgD2BZzvieqVQ372ZH499k9/+ZDArlvkfbr7aLsFOu63gT7cM5Kyjh/HhylpOPOPdtMOqCPXdmviPn0/huiuGsXJ53Zry/3XEO/zloYzU3uDjhob2lpSVPcFJ6gncDZwTEUvWs3+0pMmSJq/mo3KHU5DaLsGPx77JY+P78dc/9007nIozf+5mzJ+7GTNezNVGnnqgLzvttiLlqNJXW9vCj34xhccfGszfnthqTXlNbQv7H/QuTz4yKMXoSksRBS3tnkd6U9JUSS9KmpyU9Zc0UdLM5Ge/jY2zrAlOUh255HZbRIxf3zERMSYiRkTEiDq6ljOcAgXfv3wWs2bWM36MW0/XZ9H7dbw/dzOG7JB7xrTnAUt5e2a3lKNKW3D2j6cy642e3Pv7Hdbas+feC5j9Vk8WzMvQd1TaZ3AHR8TwiBiRbF8IPBoRQ4FHk+2NUs5WVAHjgFcj4opyXafUdh25nMNOWMTr0+u5ZuIMAG742SAmPdY75cgqyzU/aeCHV79BXV0w9+3NuOIH26UdUqqG7bGIQ4+ewxsze/HrW58C4KZrdmby37Zg1BHv8JeHs1N7I4DyNpofCxyUrN8EPAFcsDEnUpSppUPS54CngKl8/HX8KCIe2NBneqt/7KNDyxJPFqhL526pLETtVlumHUJF+9u7t7N41XvalHP06TE49h12akHHPjz54reA+XlFYyJiTOuGpDeAReTS5u8iYoykDyKib7JfwKLW7WKVsxX1aXLd1swsa1oKrsLNz7v1XJ/PRcQcSVsAEyW9lr8zIkLa+LfuOqQV1cwypPUWtZClvVNFzEl+zgPuAUYC70kaBJD8nLexoTrBmVnRStGKKqmHpF6t68ARwCvABODk5LCTgfs2Nk4/1DGz4pXm2f2WwD25x2x0AX4fEQ9KmgTcKekU4C3gqxt7ASc4MytSabphRcTrwB7rKV8AlKS10QnOzIrjWbXMLMs84KWZZZcTnJllUgAtTnBmlkmVMdZbIZzgzKx4TnBmlkkBNFfHEPVOcGZWpIBwgjOzrPItqpllkltRzSzTXIMzs8xygjOzTIqA5ua0oyiIE5yZFc81ODPLLCc4M8umcCuqmWVUQPhFXzPLLHfVMrNMiihm2sBUOcGZWfHcyGBmWRWuwZlZNnnASzPLKne2N7OsCiDcVcvMMik84KWZZVj4FtXMMqtKanCKCmoNkfQ+8FbaceQZAMxPO4gK5u+nfZX2HW0bEQM35QSSHiT3exVifkQcuSnX2xQVleAqjaTJETEi7Tgqlb+f9vk7SldN2gGYmZWLE5yZZZYTXNvGpB1AhfP30z5/RynyMzgzyyzX4Mwss5zgzCyznODWQ9KRkmZIapR0YdrxVBpJ10uaJ+mVtGOpRJIaJD0uabqkaZLOTjumzsrP4NYhqRb4B3A4MBuYBHwtIqanGlgFkTQKWAbcHBG7pR1PpZE0CBgUEVMk9QKeB47z31DHcw3uk0YCjRHxekSsAu4Ajk05pooSEU8CC9OOo1JFxNyImJKsLwVeBbZON6rOyQnuk7YGZuVtz8Z/nLaRJG0H7Ak8m3IonZITnFmZSOoJ3A2cExFL0o6nM3KC+6Q5QEPe9pCkzKxgkurIJbfbImJ82vF0Vk5wnzQJGCppe0mbAScBE1KOyaqIJAHjgFcj4oq04+nMnODWERFNwFnAQ+QeDt8ZEdPSjaqySLod+Duws6TZkk5JO6YKcwDwTeAQSS8my9FpB9UZ+TURM8ss1+DMLLOc4Mwss5zgzCyznODMLLOc4Mwss5zgqoik5uSVg1ck3SWp+yac60ZJX0nWx0oa1saxB0nafyOu8aakT8y+tKHydY5ZVuS1Lpb0g2JjtGxzgqsuKyNieDKCxyrgtPydkjZqntuI+Nd2Rro4CCg6wZmlzQmuej0F7JTUrp6SNAGYLqlW0q8kTZL0sqRTIfd2vaTfJOPcPQJs0XoiSU9IGpGsHylpiqSXJD2adBY/DTg3qT0eKGmgpLuTa0ySdEDy2c0lPZyMgTYWUHu/hKR7JT2ffGb0OvuuTMoflTQwKdtR0oPJZ56StEtJvk3LJM9sX4WSmtpRwINJ0V7AbhHxRpIkFkfE3pK6An+V9DC5ES12BoYBWwLTgevXOe9A4DpgVHKu/hGxUNJvgWURcVly3O+BKyPiaUnbkOv18WngIuDpiLhE0heAQno4fCe5RjdgkqS7I2IB0AOYHBHnSvpJcu6zyE3iclpEzJS0D3ANcMhGfI3WCTjBVZdukl5M1p8i199xf+C5iHgjKT8C2L31+RrQBxgKjAJuj4hm4B1Jj63n/PsCT7aeKyI2NObbYcCwXJdLAHonI2eMAr6UfPZ+SYsK+J2+J+n4ZL0hiXUB0AL8ISm/FRifXGN/4K68a3ct4BrWSTnBVZeVETE8vyD5h748vwj4bkQ8tM5xpewLWQPsGxEfrieWgkk6iFyy3C8iVkh6AqjfwOGRXPeDdb8Dsw3xM7jseQg4PRmuB0mfktQDeBI4MXlGNwg4eD2ffQYYJWn75LP9k/KlQK+84x4Gvtu6IWl4svok8PWk7CigXzux9gEWJcltF3I1yFY1QGst9Ovkbn2XAG9IOiG5hiTt0c41rBNzgsueseSer01RblKY35Grqd8DzEz23UxuNJC1RMT7wGhyt4Mv8fEt4h+B41sbGYDvASOSRozpfNya+1NyCXIauVvVt9uJ9UGgi6RXgZ+TS7CtlgMjk9/hEOCSpPwbwClJfNPwcPLWBo8mYmaZ5RqcmWWWE5yZZZYTnJlllhOcmWWWE5yZZZYTnJlllhOcmWXW/wAqUeIgoYxurAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# use display matrix function to heatmap prediction/true cases\n",
    "disp = ConfusionMatrixDisplay(confusion_matrix=cm)\n",
    "disp.plot()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "7d1fd0be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TPR</th>\n",
       "      <th>FNR</th>\n",
       "      <th>FPR</th>\n",
       "      <th>TNR</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.954819</td>\n",
       "      <td>0.045181</td>\n",
       "      <td>0.159574</td>\n",
       "      <td>0.840426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.745763</td>\n",
       "      <td>0.254237</td>\n",
       "      <td>0.049046</td>\n",
       "      <td>0.950954</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.771429</td>\n",
       "      <td>0.228571</td>\n",
       "      <td>0.012788</td>\n",
       "      <td>0.987212</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        TPR       FNR       FPR       TNR\n",
       "1  0.954819  0.045181  0.159574  0.840426\n",
       "2  0.745763  0.254237  0.049046  0.950954\n",
       "3  0.771429  0.228571  0.012788  0.987212"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rates_df = pd.DataFrame(get_tpr_fnr_fpr_tnr(cm)).transpose()\n",
    "rates_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18075946",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6414de9",
   "metadata": {},
   "source": [
    "In evaluation of the complement naïve Bayes model, we primarily used accuracy but also looked more closely at false positive rates, false negative rates, true positive rates, and true negative rates. The k-fold cross validation of this model has a train score of 0.84 and a test score of 0.84.  This could imply some overfitting is happening within this model. Furthermore, when we look more closely at the true positive rates for each of the categorical outcomes, we do have some serious concerns. It appears that this model is very accurate at predicting healthy outcomes, with a true positive rate of 91%, but much worse at correctly predicting other fetal health outcomes, with true positive rates of less than 65% for each of the other outcomes. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "936de862",
   "metadata": {},
   "source": [
    "For our MLP classifier, a grid search was conducted to find ideal model hyperparameters.  Cross validation however, revealed a lower performance when predicting test data, indicating overfitting. The hyperparameters were tuned manually with cross validation to reach a higher accuracy in predicting outcomes for our test data. The resulting classifier model calculated outcomes with a .91 accuracy score. Outcomes for ‘normal’, ‘suspect’, and ‘pathological’ were returned with True Positive Rates of 95%, 74%, and 77%, respectively. Of particular concern, given the context of this project, were False Negative Rates for non-normal outcomes. Results returned a FNR of 25% for ‘suspect’ and 22% for ‘pathological’ predictions.    While this lends confidence for predicting ‘normal’ outcomes, overall, the results show the model is not dependable for predicting riskier outcomes. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3659d76b",
   "metadata": {},
   "source": [
    "## Conclusion and Ethical Implications"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3da9343c",
   "metadata": {},
   "source": [
    "It is worth noting that both of our models performed much better than the null model with regards to accuracy and in that way, they can be considered successful models. When comparing both model results it appears that the MLP neural network model performed better with an overall accuracy of 91% compared to our Complement Naïve Bayes accuracy at 84%. Both models, however, under-performed when correctly classifying ‘suspect’ and ‘pathological’ outcomes with false negative rates ranging from  22%-51% across both models.   Since   the purpose of these models is to identify unhealthy fetal heart health outcomes, we feel that these models are not accurate enough to be deployed for diagnostic purposes. It is possible that the imbalance between classes and the nuanced differences between class cases are bit ill-suited for training these model types. It would be interesting to see how results compared with a tree-based model.     Although the purpose of this diagnostic tool is to assist physicians in the accurate assessment of fetal heart hearth, if such a tool were to eventually replace physicians, that could result in a lower quality of care for patients.   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbead3c3",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35be9bd9",
   "metadata": {},
   "source": [
    "- Boudet, S., Houzé l’Aulnoit, A., Demailly, R., Delgranche, A., Peyrodie, L., Beuscart, R., & Houzé de l’Aulnoit, D. (2020). A fetal heart rate morphological analysis toolbox for MATLAB. SoftwareX, 11, 100428. https://doi.org/10.1016/j.softx.2020.100428  \n",
    "  \n",
    "  \n",
    "- Fetal Heart Monitoring. (2019, August 14). Johns Hopkins Medicine. https://www.hopkinsmedicine.org/health/treatment-tests-and-therapies/fetal-heart-monitoring \n",
    "  \n",
    "   \n",
    "- Naik, N. (2022, March 14). Legal and Ethical Consideration in Artificial Intelligence in Healthcare: Who Takes Responsibility? Frontiers. https://www.frontiersin.org/articles/10.3389/fsurg.2022.862322/full \n",
    "  \n",
    "  \n",
    "- Simfukwe, M., Kunda, D., &amp; Chembe, C. (2015). Comparing Naive Bayes Method and Artificial Neural Network for Semen Quality Categorization. International Journal of Innovative Science, Engineering &amp; Technology, 2(7) https://ijiset.com/vol2/v2s7/IJISET_V2_I6_90.pdf   \n",
    "  \n",
    "    \n",
    "- Md Abdul Bari (Oct 22, 2020) get_tpr_fnr_fpr_tnr() [Python Function]. https://stackoverflow.com/questions/45053238/how-to-get-all-confusion-matrix-terminologies-tpr-fpr-tnr-fnr-for-a-multi-c."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}