AutoML-based_IDS_Batch_Learning_Dataset_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Case Study: Automated Machine Learning (AutoML) for Autonomous Intrusion Detection System Development \n",
    "This is the code for the paper entitled \"**[Enabling AutoML for Zero-Touch Network Security: Use-Case Driven Analysis](https://ieeexplore.ieee.org/document/10472316)**\" published in *IEEE Transactions on Network and Service Management* (IF:5.3).<br>\n",
    "Authors: Li Yang (liyanghart@gmail.com), Mirna El Rajab, Abdallah Shami, and Sami Muhaidat<br>\n",
    "\n",
    "L. Yang, M. E. Rajab, A. Shami, and S. Muhaidat, \"Enabling AutoML for Zero-Touch Network Security: Use-Case Driven Analysis,\" IEEE Transactions on Network and Service Management, pp. 1-28, 2024, doi: https://doi.org/10.1109/TNSM.2024.3376631."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code Part 1: Automated Offline/Static/Batch Learning\n",
    "Batch learning: Batch learning methods analyze static data in batches and often need access to the entire dataset prior to model training. Traditional ML algorithms can effectively solve batch learning tasks. Although batch learning models often achieve high performance due to their ability to learn diverse data patterns, it is often difficult to update these models once created. Therefore, batch learning faces two significant challenges: model degradation and data unavailability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset 1: CICIDS2017\n",
    "A subset of the network traffic data randomly sampled from the [CICIDS2017 dataset](https://www.unb.ca/cic/datasets/ids-2017.html).  \n",
    "\n",
    "The Canadian Institute for Cybersecurity Intrusion Detection System 2017 (CICIDS2017) dataset has the most updated network threats. The CICIDS2017 dataset is close to real-world network data since it has a large amount of network traffic data, a variety of network features, various types of attacks, and highly imbalanced classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split,cross_val_score\n",
    "import lightgbm as lgb\n",
    "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score, precision_score, recall_score, f1_score\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.feature_selection import RFE\n",
    "from scipy.stats import shapiro\n",
    "from imblearn.over_sampling import SMOTE, ADASYN\n",
    "from hyperopt import fmin, tpe, hp\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read the sampled CICIDS2017 dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"Data/cic_0.01km.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flow Duration</th>\n",
       "      <th>Total Length of Fwd Packets</th>\n",
       "      <th>Fwd Packet Length Max</th>\n",
       "      <th>Fwd Packet Length Mean</th>\n",
       "      <th>Bwd Packet Length Max</th>\n",
       "      <th>Bwd Packet Length Min</th>\n",
       "      <th>Flow IAT Mean</th>\n",
       "      <th>Flow IAT Min</th>\n",
       "      <th>Fwd IAT Min</th>\n",
       "      <th>Fwd Header Length</th>\n",
       "      <th>Bwd Header Length</th>\n",
       "      <th>Fwd Packets/s</th>\n",
       "      <th>Bwd Packets/s</th>\n",
       "      <th>Min Packet Length</th>\n",
       "      <th>URG Flag Count</th>\n",
       "      <th>Down/Up Ratio</th>\n",
       "      <th>Init_Win_bytes_forward</th>\n",
       "      <th>Init_Win_bytes_backward</th>\n",
       "      <th>min_seg_size_forward</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>50833</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.083300e+04</td>\n",
       "      <td>50833</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>19.672260</td>\n",
       "      <td>19.672260</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>319</td>\n",
       "      <td>153</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>49</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4.900000e+01</td>\n",
       "      <td>49</td>\n",
       "      <td>49</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>40816.326530</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>277</td>\n",
       "      <td>-1</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>306</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6.0000</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>3.060000e+02</td>\n",
       "      <td>306</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>3267.973856</td>\n",
       "      <td>3267.973856</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>63041</td>\n",
       "      <td>65</td>\n",
       "      <td>65</td>\n",
       "      <td>65.0000</td>\n",
       "      <td>124</td>\n",
       "      <td>124</td>\n",
       "      <td>6.304100e+04</td>\n",
       "      <td>63041</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>15.862693</td>\n",
       "      <td>15.862693</td>\n",
       "      <td>65</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>47682</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>43.0000</td>\n",
       "      <td>59</td>\n",
       "      <td>59</td>\n",
       "      <td>4.768200e+04</td>\n",
       "      <td>47682</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>20.972275</td>\n",
       "      <td>20.972275</td>\n",
       "      <td>43</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28298</th>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4.500000e+01</td>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>22222.222220</td>\n",
       "      <td>22222.222220</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>349</td>\n",
       "      <td>307</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28299</th>\n",
       "      <td>114309573</td>\n",
       "      <td>511</td>\n",
       "      <td>427</td>\n",
       "      <td>31.9375</td>\n",
       "      <td>746</td>\n",
       "      <td>0</td>\n",
       "      <td>3.941709e+06</td>\n",
       "      <td>94</td>\n",
       "      <td>165</td>\n",
       "      <td>332</td>\n",
       "      <td>424</td>\n",
       "      <td>0.139971</td>\n",
       "      <td>0.122474</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8192</td>\n",
       "      <td>343</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28300</th>\n",
       "      <td>48850</td>\n",
       "      <td>80</td>\n",
       "      <td>40</td>\n",
       "      <td>40.0000</td>\n",
       "      <td>72</td>\n",
       "      <td>72</td>\n",
       "      <td>1.628333e+04</td>\n",
       "      <td>1</td>\n",
       "      <td>48</td>\n",
       "      <td>64</td>\n",
       "      <td>64</td>\n",
       "      <td>40.941658</td>\n",
       "      <td>40.941658</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28301</th>\n",
       "      <td>260</td>\n",
       "      <td>66</td>\n",
       "      <td>33</td>\n",
       "      <td>33.0000</td>\n",
       "      <td>97</td>\n",
       "      <td>97</td>\n",
       "      <td>8.666667e+01</td>\n",
       "      <td>48</td>\n",
       "      <td>48</td>\n",
       "      <td>40</td>\n",
       "      <td>40</td>\n",
       "      <td>7692.307692</td>\n",
       "      <td>7692.307692</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28302</th>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6.0000</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>2.500000e+01</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>40000.000000</td>\n",
       "      <td>40000.000000</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>11256</td>\n",
       "      <td>35000</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>28303 rows × 20 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Flow Duration  Total Length of Fwd Packets  Fwd Packet Length Max  \\\n",
       "0              50833                            0                      0   \n",
       "1                 49                            0                      0   \n",
       "2                306                            6                      6   \n",
       "3              63041                           65                     65   \n",
       "4              47682                           43                     43   \n",
       "...              ...                          ...                    ...   \n",
       "28298             45                            0                      0   \n",
       "28299      114309573                          511                    427   \n",
       "28300          48850                           80                     40   \n",
       "28301            260                           66                     33   \n",
       "28302             25                            6                      6   \n",
       "\n",
       "       Fwd Packet Length Mean  Bwd Packet Length Max  Bwd Packet Length Min  \\\n",
       "0                      0.0000                      0                      0   \n",
       "1                      0.0000                      0                      0   \n",
       "2                      6.0000                      6                      6   \n",
       "3                     65.0000                    124                    124   \n",
       "4                     43.0000                     59                     59   \n",
       "...                       ...                    ...                    ...   \n",
       "28298                  0.0000                      0                      0   \n",
       "28299                 31.9375                    746                      0   \n",
       "28300                 40.0000                     72                     72   \n",
       "28301                 33.0000                     97                     97   \n",
       "28302                  6.0000                      6                      6   \n",
       "\n",
       "       Flow IAT Mean  Flow IAT Min  Fwd IAT Min  Fwd Header Length  \\\n",
       "0       5.083300e+04         50833            0                 32   \n",
       "1       4.900000e+01            49           49                 64   \n",
       "2       3.060000e+02           306            0                 20   \n",
       "3       6.304100e+04         63041            0                 32   \n",
       "4       4.768200e+04         47682            0                 32   \n",
       "...              ...           ...          ...                ...   \n",
       "28298   4.500000e+01            45            0                 32   \n",
       "28299   3.941709e+06            94          165                332   \n",
       "28300   1.628333e+04             1           48                 64   \n",
       "28301   8.666667e+01            48           48                 40   \n",
       "28302   2.500000e+01            25            0                 20   \n",
       "\n",
       "       Bwd Header Length  Fwd Packets/s  Bwd Packets/s  Min Packet Length  \\\n",
       "0                     32      19.672260      19.672260                  0   \n",
       "1                      0   40816.326530       0.000000                  0   \n",
       "2                     20    3267.973856    3267.973856                  6   \n",
       "3                     32      15.862693      15.862693                 65   \n",
       "4                     32      20.972275      20.972275                 43   \n",
       "...                  ...            ...            ...                ...   \n",
       "28298                 32   22222.222220   22222.222220                  0   \n",
       "28299                424       0.139971       0.122474                  0   \n",
       "28300                 64      40.941658      40.941658                 40   \n",
       "28301                 40    7692.307692    7692.307692                 33   \n",
       "28302                 20   40000.000000   40000.000000                  6   \n",
       "\n",
       "       URG Flag Count  Down/Up Ratio  Init_Win_bytes_forward  \\\n",
       "0                   1              1                     319   \n",
       "1                   0              0                     277   \n",
       "2                   0              1                       0   \n",
       "3                   0              1                      -1   \n",
       "4                   0              1                      -1   \n",
       "...               ...            ...                     ...   \n",
       "28298               1              1                     349   \n",
       "28299               0              0                    8192   \n",
       "28300               0              1                      -1   \n",
       "28301               0              1                      -1   \n",
       "28302               1              1                   11256   \n",
       "\n",
       "       Init_Win_bytes_backward  min_seg_size_forward  Label  \n",
       "0                          153                    32      0  \n",
       "1                           -1                    32      0  \n",
       "2                            0                    20      0  \n",
       "3                           -1                    32      0  \n",
       "4                           -1                    32      0  \n",
       "...                        ...                   ...    ...  \n",
       "28298                      307                    32      0  \n",
       "28299                      343                    20      0  \n",
       "28300                       -1                    32      0  \n",
       "28301                       -1                    20      0  \n",
       "28302                    35000                    20      0  \n",
       "\n",
       "[28303 rows x 20 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Automated Data Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated Transformation/Encoding\n",
    "Automatically identify and transform string/text features into numerical features by the label encoding method to make the data more readable by ML models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the automated data encoding function\n",
    "def Auto_Encoding(df):\n",
    "    cat_features=[x for x in df.columns if df[x].dtype==\"object\"] ## Find string/text features\n",
    "    le=LabelEncoder()\n",
    "    for col in cat_features:\n",
    "        if col in df.columns:\n",
    "            i = df.columns.get_loc(col)\n",
    "            # Transform to numerical features\n",
    "            df.iloc[:,i] = df.apply(lambda i:le.fit_transform(i.astype(str)), axis=0, result_type='expand')\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=Auto_Encoding(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated Imputation\n",
    "Detect and impute missing values to improve data quality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the automated data imputation function\n",
    "def Auto_Imputation(df):\n",
    "    if df.isnull().values.any() or np.isinf(df).values.any(): # if there is any empty or infinite values\n",
    "        df.replace([np.inf, -np.inf], np.nan, inplace=True)\n",
    "        df.fillna(df.median(), inplace = True)  # Replace empty values with median values; there are other imputation methods discussed in the paper\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=Auto_Imputation(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated normalization\n",
    "Normalize the range of features to a similar scale to improve data quality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def Auto_Normalization(df):\n",
    "    stat, p = shapiro(df)\n",
    "    print('Statistics=%.3f, p=%.3f' % (stat, p))\n",
    "    # interpret\n",
    "    alpha = 0.05\n",
    "    numeric_features = df.drop(['Label'],axis = 1).dtypes[df.dtypes != 'object'].index\n",
    "    \n",
    "    # check if the data distribution follows a Gaussian/normal distribution\n",
    "    # If so, select the Z-score normalization method; otherwise, select the min-max normalization\n",
    "    # Details are in the paper\n",
    "    if p > alpha:\n",
    "        print('Sample looks Gaussian (fail to reject H0)')\n",
    "        df[numeric_features] = df[numeric_features].apply(\n",
    "            lambda x: (x - x.mean()) / (x.std()))\n",
    "        print('Z-score normalization is automatically chosen and used')\n",
    "    else:\n",
    "        print('Sample does not look Gaussian (reject H0)')\n",
    "        df[numeric_features] = df[numeric_features].apply(\n",
    "            lambda x: (x - x.min()) / (x.max()-x.min()))\n",
    "        print('Min-max normalization is automatically chosen and used')\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Statistics=0.076, p=0.000\n",
      "Sample does not look Gaussian (reject H0)\n",
      "Min-max normalization is automatically chosen and used\n"
     ]
    }
   ],
   "source": [
    "df=Auto_Normalization(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Train-test split\n",
    "Split the dataset into the training and the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop(['Label'],axis=1)\n",
    "y = df['Label']\n",
    "\n",
    "# Here we used the 80%/20% split, it can be changed based on specific tasks\n",
    "#X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2, shuffle=False,random_state = 0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2,random_state = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated data balancing\n",
    "Generate minority class samples to solve class-imbalance and improve data quality.  \n",
    "Adaptive Synthetic (ADASYN) method is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    18126\n",
       "1     4516\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For binary data (can be modified for multi-class data with the same logic)\n",
    "def Auto_Balancing(X_train, y_train):\n",
    "    number0 = pd.Series(y_train).value_counts().iloc[0]\n",
    "    number1 = pd.Series(y_train).value_counts().iloc[1]\n",
    "    \n",
    "    if number0 > number1:\n",
    "        nlarge = number0\n",
    "    else:\n",
    "        nlarge = number1\n",
    "    \n",
    "    # evaluate whether the incoming dataset is imbalanced (the abnormal/normal ratio is smaller than a threshold (e.g., 50%)) \n",
    "    if (number1/number0 > 1.5) or (number0/number1 > 1.5):\n",
    "        balanced=ADASYN(n_jobs=-1,sampling_strategy={0:nlarge, 1:nlarge})\n",
    "\n",
    "        X_train, y_train = balanced.fit_resample(X_train, y_train)\n",
    "        \n",
    "    return X_train, y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = Auto_Balancing(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    18126\n",
       "1    18102\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Model learning (for Comparison)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.753%\n",
      "Precision: 99.029%\n",
      "Recall: 99.733%\n",
      "F1-score: 99.38%\n",
      "Time: 1.23332\n",
      "Wall time: 191 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lg = lgb.LGBMClassifier(verbose = -1)\n",
    "lg.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = lg.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.823%\n",
      "Precision: 99.38%\n",
      "Recall: 99.733%\n",
      "F1-score: 99.556%\n",
      "Time: 7.75258\n",
      "Wall time: 2.66 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = RandomForestClassifier()\n",
    "rf.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 98.34%\n",
      "Precision: 92.63900000000001%\n",
      "Recall: 99.556%\n",
      "F1-score: 95.973%\n",
      "Time: 453.33682\n",
      "Wall time: 2.58 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "knn = KNeighborsClassifier()\n",
    "knn.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = knn.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from keras.layers import Input,Dense,Dropout,BatchNormalization,Activation\n",
    "from keras import Model\n",
    "import keras.backend as K\n",
    "import keras.callbacks as kcallbacks\n",
    "from keras import optimizers\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from keras.wrappers.scikit_learn import KerasClassifier\n",
    "from keras.callbacks import EarlyStopping\n",
    "def ANN(optimizer = 'sgd',neurons=32,batch_size=1024,epochs=80,activation='relu',patience=8,loss='binary_crossentropy'):\n",
    "    K.clear_session()\n",
    "    inputs=Input(shape=(X.shape[1],))\n",
    "    x=Dense(1000)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.3)(x)\n",
    "    x=Dense(256)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.25)(x)\n",
    "    x=Dense(2,activation='softmax')(x)\n",
    "    model=Model(inputs=inputs,outputs=x,name='base_nlp')\n",
    "    model.compile(optimizer='adam',loss='categorical_crossentropy')\n",
    "#     model.compile(optimizer=Adam(lr = 0.01),loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "    early_stopping = EarlyStopping(monitor=\"loss\", patience = patience)# early stop patience\n",
    "    history = model.fit(X, pd.get_dummies(y).values,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              callbacks = [early_stopping],\n",
    "              verbose=0) #verbose set to 1 will show the training process\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 88.801%\n",
      "Precision: 64.199%\n",
      "Recall: 98.667%\n",
      "F1-score: 77.786%\n",
      "Time: 453.33682\n",
      "Wall time: 7.98 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "ann = KerasClassifier(build_fn=ANN, verbose=0)\n",
    "ann.fit(X_train,y_train)\n",
    "predictions = ann.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Automated Feature Engineering\n",
    "Feature selection method 1: **Recursive Feature Elimination (RFE)**, used to remove irrelevant features to improve model efficiency  \n",
    "Feature selection method 2: **Pearson Correlation**, used to remove redundant features to improve model efficiency and accuracy  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Feature_Importance_RFE(data, n_features_to_select=20):\n",
    "    features = data.drop(['Label'], axis=1).values  # \"Label\" should be changed to the target class variable name if different\n",
    "    labels = data['Label'].values\n",
    "\n",
    "    # Extract feature names\n",
    "    feature_names = list(data.drop(['Label'], axis=1).columns)\n",
    "\n",
    "    # Create a base estimator\n",
    "    model = lgb.LGBMRegressor(verbose = -1)\n",
    "\n",
    "    # Create the RFE object and rank each feature\n",
    "    rfe = RFE(estimator=model, n_features_to_select=n_features_to_select, step=1)\n",
    "    rfe.fit(features, labels)\n",
    "\n",
    "    # Get the feature ranking\n",
    "    feature_ranking = rfe.ranking_\n",
    "\n",
    "    # Create a DataFrame for feature importances\n",
    "    feature_importances = pd.DataFrame({'feature': feature_names, 'ranking': feature_ranking})\n",
    "\n",
    "    # Sort features according to their ranking\n",
    "    feature_importances = feature_importances.sort_values('ranking', ascending=True).reset_index(drop=True)\n",
    "\n",
    "    # Get the features to drop\n",
    "    to_drop = list(feature_importances[feature_importances['ranking'] > 1]['feature'])\n",
    "\n",
    "    return to_drop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove redundant features\n",
    "def Feature_Redundancy_Pearson(data):\n",
    "    correlation_threshold=0.90 # Only remove features with the redundancy>90%. It can be changed\n",
    "    features = data.drop(['Label'],axis=1)\n",
    "    corr_matrix = features.corr()\n",
    "\n",
    "    # Extract the upper triangle of the correlation matrix\n",
    "    upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k = 1).astype(np.bool))\n",
    "\n",
    "    # Select the features with correlations above the threshold\n",
    "    # Need to use the absolute value\n",
    "    to_drop = [column for column in upper.columns if any(upper[column].abs() > correlation_threshold)]\n",
    "\n",
    "    # Dataframe to hold correlated pairs\n",
    "    record_collinear = pd.DataFrame(columns = ['drop_feature', 'corr_feature', 'corr_value'])\n",
    "\n",
    "    # Iterate through the columns to drop\n",
    "    for column in to_drop:\n",
    "\n",
    "        # Find the correlated features\n",
    "        corr_features = list(upper.index[upper[column].abs() > correlation_threshold])\n",
    "\n",
    "        # Find the correlated values\n",
    "        corr_values = list(upper[column][upper[column].abs() > correlation_threshold])\n",
    "        drop_features = [column for _ in range(len(corr_features))]    \n",
    "\n",
    "        # Record the information (need a temp df for now)\n",
    "        temp_df = pd.DataFrame.from_dict({'drop_feature': drop_features,\n",
    "                                         'corr_feature': corr_features,\n",
    "                                         'corr_value': corr_values})\n",
    "        record_collinear = record_collinear.append(temp_df, ignore_index = True)\n",
    "#     print(record_collinear)\n",
    "    return to_drop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Auto_Feature_Engineering(df):\n",
    "    drop1 = Feature_Importance_RFE(df)\n",
    "    dfh1 = df.drop(columns = drop1)\n",
    "    \n",
    "    drop2 = Feature_Redundancy_Pearson(dfh1)\n",
    "    dfh2 = dfh1.drop(columns = drop2)\n",
    "    \n",
    "    return dfh2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flow Duration</th>\n",
       "      <th>Total Length of Fwd Packets</th>\n",
       "      <th>Fwd Packet Length Max</th>\n",
       "      <th>Fwd Packet Length Mean</th>\n",
       "      <th>Bwd Packet Length Max</th>\n",
       "      <th>Bwd Packet Length Min</th>\n",
       "      <th>Flow IAT Mean</th>\n",
       "      <th>Flow IAT Min</th>\n",
       "      <th>Fwd IAT Min</th>\n",
       "      <th>Fwd Header Length</th>\n",
       "      <th>Fwd Packets/s</th>\n",
       "      <th>Bwd Packets/s</th>\n",
       "      <th>Min Packet Length</th>\n",
       "      <th>URG Flag Count</th>\n",
       "      <th>Down/Up Ratio</th>\n",
       "      <th>Init_Win_bytes_forward</th>\n",
       "      <th>Init_Win_bytes_backward</th>\n",
       "      <th>min_seg_size_forward</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.236419e-04</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.707129e-04</td>\n",
       "      <td>4.707129e-04</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000150</td>\n",
       "      <td>6.557420e-06</td>\n",
       "      <td>1.311484e-05</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.004883</td>\n",
       "      <td>0.002350</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.416669e-07</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.907407e-07</td>\n",
       "      <td>4.907407e-07</td>\n",
       "      <td>4.083333e-07</td>\n",
       "      <td>0.000299</td>\n",
       "      <td>1.360544e-02</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.004242</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.583334e-06</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>0.000242</td>\n",
       "      <td>0.001556</td>\n",
       "      <td>0.000516</td>\n",
       "      <td>0.004144</td>\n",
       "      <td>2.870370e-06</td>\n",
       "      <td>2.870370e-06</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000094</td>\n",
       "      <td>1.089325e-03</td>\n",
       "      <td>2.178649e-03</td>\n",
       "      <td>0.017804</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.253752e-04</td>\n",
       "      <td>0.000090</td>\n",
       "      <td>0.002619</td>\n",
       "      <td>0.016856</td>\n",
       "      <td>0.010660</td>\n",
       "      <td>0.085635</td>\n",
       "      <td>5.837500e-04</td>\n",
       "      <td>5.837500e-04</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000150</td>\n",
       "      <td>5.287564e-06</td>\n",
       "      <td>1.057513e-05</td>\n",
       "      <td>0.192878</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.973835e-04</td>\n",
       "      <td>0.000060</td>\n",
       "      <td>0.001732</td>\n",
       "      <td>0.011151</td>\n",
       "      <td>0.005072</td>\n",
       "      <td>0.040746</td>\n",
       "      <td>4.415370e-04</td>\n",
       "      <td>4.415370e-04</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000150</td>\n",
       "      <td>6.990758e-06</td>\n",
       "      <td>1.398152e-05</td>\n",
       "      <td>0.127596</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28298</th>\n",
       "      <td>4.083335e-07</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.537037e-07</td>\n",
       "      <td>4.537037e-07</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000150</td>\n",
       "      <td>7.407407e-03</td>\n",
       "      <td>1.481481e-02</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.005341</td>\n",
       "      <td>0.004700</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28299</th>\n",
       "      <td>9.525802e-01</td>\n",
       "      <td>0.000710</td>\n",
       "      <td>0.017204</td>\n",
       "      <td>0.008282</td>\n",
       "      <td>0.064133</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.649735e-02</td>\n",
       "      <td>9.074074e-07</td>\n",
       "      <td>1.375000e-06</td>\n",
       "      <td>0.001554</td>\n",
       "      <td>4.665693e-08</td>\n",
       "      <td>8.164962e-08</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.125015</td>\n",
       "      <td>0.005249</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28300</th>\n",
       "      <td>4.071168e-04</td>\n",
       "      <td>0.000111</td>\n",
       "      <td>0.001612</td>\n",
       "      <td>0.010373</td>\n",
       "      <td>0.006190</td>\n",
       "      <td>0.049724</td>\n",
       "      <td>1.508086e-04</td>\n",
       "      <td>4.629629e-08</td>\n",
       "      <td>4.000000e-07</td>\n",
       "      <td>0.000299</td>\n",
       "      <td>1.364722e-05</td>\n",
       "      <td>2.729444e-05</td>\n",
       "      <td>0.118694</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28301</th>\n",
       "      <td>2.200001e-06</td>\n",
       "      <td>0.000092</td>\n",
       "      <td>0.001330</td>\n",
       "      <td>0.008558</td>\n",
       "      <td>0.008339</td>\n",
       "      <td>0.066989</td>\n",
       "      <td>8.395061e-07</td>\n",
       "      <td>4.814815e-07</td>\n",
       "      <td>4.000000e-07</td>\n",
       "      <td>0.000187</td>\n",
       "      <td>2.564103e-03</td>\n",
       "      <td>5.128205e-03</td>\n",
       "      <td>0.097923</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28302</th>\n",
       "      <td>2.416668e-07</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>0.000242</td>\n",
       "      <td>0.001556</td>\n",
       "      <td>0.000516</td>\n",
       "      <td>0.004144</td>\n",
       "      <td>2.685185e-07</td>\n",
       "      <td>2.685185e-07</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000094</td>\n",
       "      <td>1.333333e-02</td>\n",
       "      <td>2.666667e-02</td>\n",
       "      <td>0.017804</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.171768</td>\n",
       "      <td>0.534073</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>28303 rows × 19 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Flow Duration  Total Length of Fwd Packets  Fwd Packet Length Max  \\\n",
       "0       4.236419e-04                     0.000000               0.000000   \n",
       "1       4.416669e-07                     0.000000               0.000000   \n",
       "2       2.583334e-06                     0.000008               0.000242   \n",
       "3       5.253752e-04                     0.000090               0.002619   \n",
       "4       3.973835e-04                     0.000060               0.001732   \n",
       "...              ...                          ...                    ...   \n",
       "28298   4.083335e-07                     0.000000               0.000000   \n",
       "28299   9.525802e-01                     0.000710               0.017204   \n",
       "28300   4.071168e-04                     0.000111               0.001612   \n",
       "28301   2.200001e-06                     0.000092               0.001330   \n",
       "28302   2.416668e-07                     0.000008               0.000242   \n",
       "\n",
       "       Fwd Packet Length Mean  Bwd Packet Length Max  Bwd Packet Length Min  \\\n",
       "0                    0.000000               0.000000               0.000000   \n",
       "1                    0.000000               0.000000               0.000000   \n",
       "2                    0.001556               0.000516               0.004144   \n",
       "3                    0.016856               0.010660               0.085635   \n",
       "4                    0.011151               0.005072               0.040746   \n",
       "...                       ...                    ...                    ...   \n",
       "28298                0.000000               0.000000               0.000000   \n",
       "28299                0.008282               0.064133               0.000000   \n",
       "28300                0.010373               0.006190               0.049724   \n",
       "28301                0.008558               0.008339               0.066989   \n",
       "28302                0.001556               0.000516               0.004144   \n",
       "\n",
       "       Flow IAT Mean  Flow IAT Min   Fwd IAT Min  Fwd Header Length  \\\n",
       "0       4.707129e-04  4.707129e-04  0.000000e+00           0.000150   \n",
       "1       4.907407e-07  4.907407e-07  4.083333e-07           0.000299   \n",
       "2       2.870370e-06  2.870370e-06  0.000000e+00           0.000094   \n",
       "3       5.837500e-04  5.837500e-04  0.000000e+00           0.000150   \n",
       "4       4.415370e-04  4.415370e-04  0.000000e+00           0.000150   \n",
       "...              ...           ...           ...                ...   \n",
       "28298   4.537037e-07  4.537037e-07  0.000000e+00           0.000150   \n",
       "28299   3.649735e-02  9.074074e-07  1.375000e-06           0.001554   \n",
       "28300   1.508086e-04  4.629629e-08  4.000000e-07           0.000299   \n",
       "28301   8.395061e-07  4.814815e-07  4.000000e-07           0.000187   \n",
       "28302   2.685185e-07  2.685185e-07  0.000000e+00           0.000094   \n",
       "\n",
       "       Fwd Packets/s  Bwd Packets/s  Min Packet Length  URG Flag Count  \\\n",
       "0       6.557420e-06   1.311484e-05           0.000000             1.0   \n",
       "1       1.360544e-02   0.000000e+00           0.000000             0.0   \n",
       "2       1.089325e-03   2.178649e-03           0.017804             0.0   \n",
       "3       5.287564e-06   1.057513e-05           0.192878             0.0   \n",
       "4       6.990758e-06   1.398152e-05           0.127596             0.0   \n",
       "...              ...            ...                ...             ...   \n",
       "28298   7.407407e-03   1.481481e-02           0.000000             1.0   \n",
       "28299   4.665693e-08   8.164962e-08           0.000000             0.0   \n",
       "28300   1.364722e-05   2.729444e-05           0.118694             0.0   \n",
       "28301   2.564103e-03   5.128205e-03           0.097923             0.0   \n",
       "28302   1.333333e-02   2.666667e-02           0.017804             1.0   \n",
       "\n",
       "       Down/Up Ratio  Init_Win_bytes_forward  Init_Win_bytes_backward  \\\n",
       "0           0.142857                0.004883                 0.002350   \n",
       "1           0.000000                0.004242                 0.000000   \n",
       "2           0.142857                0.000015                 0.000015   \n",
       "3           0.142857                0.000000                 0.000000   \n",
       "4           0.142857                0.000000                 0.000000   \n",
       "...              ...                     ...                      ...   \n",
       "28298       0.142857                0.005341                 0.004700   \n",
       "28299       0.000000                0.125015                 0.005249   \n",
       "28300       0.142857                0.000000                 0.000000   \n",
       "28301       0.142857                0.000000                 0.000000   \n",
       "28302       0.142857                0.171768                 0.534073   \n",
       "\n",
       "       min_seg_size_forward  Label  \n",
       "0                  0.571429      0  \n",
       "1                  0.571429      0  \n",
       "2                  0.357143      0  \n",
       "3                  0.571429      0  \n",
       "4                  0.571429      0  \n",
       "...                     ...    ...  \n",
       "28298              0.571429      0  \n",
       "28299              0.357143      0  \n",
       "28300              0.571429      0  \n",
       "28301              0.357143      0  \n",
       "28302              0.357143      0  \n",
       "\n",
       "[28303 rows x 19 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfh2 = Auto_Feature_Engineering(df)\n",
    "dfh2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Split & Balancing (After Feature Engineering)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = dfh2.drop(['Label'],axis=1)\n",
    "y = dfh2['Label']\n",
    "\n",
    "#X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2, shuffle=False,random_state = 0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2,random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = Auto_Balancing(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Automated Model Selection\n",
    "Select the best-performing model among five common machine learning models (Naive Bayes, KNN, random forest, LightGBM, and ANN/MLP) by evaluating their learning performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grid Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a pipeline\n",
    "pipe = Pipeline([('classifier', KNeighborsClassifier())])\n",
    "\n",
    "# Create space of candidate learning algorithms and their hyperparameters\n",
    "search_space = [\n",
    "                {'classifier': [KNeighborsClassifier()]},\n",
    "                {'classifier': [RandomForestClassifier()]},\n",
    "                {'classifier': [lgb.LGBMClassifier(verbose = -1)]},\n",
    "                {'classifier': [KerasClassifier(build_fn=ANN, verbose=0)]},\n",
    "                 ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = GridSearchCV(pipe, search_space, cv=3, verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3,\n",
       "             estimator=Pipeline(steps=[('classifier', KNeighborsClassifier())]),\n",
       "             param_grid=[{'classifier': [KNeighborsClassifier()]},\n",
       "                         {'classifier': [RandomForestClassifier()]},\n",
       "                         {'classifier': [LGBMClassifier(verbose=-1)]},\n",
       "                         {'classifier': [<keras.wrappers.scikit_learn.KerasClassifier object at 0x000001D50A392988>]}])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Model:{'classifier': LGBMClassifier(verbose=-1)}\n",
      "Accuracy:0.952513976271599\n"
     ]
    }
   ],
   "source": [
    "print(\"Best Model:\"+ str(clf.best_params_))\n",
    "print(\"Accuracy:\"+ str(clf.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([3.98953756e-03, 1.04863024e+00, 1.10047976e-01, 7.77408489e+00]),\n",
       " 'std_fit_time': array([0.00081391, 0.05885326, 0.00497592, 0.3468082 ]),\n",
       " 'mean_score_time': array([2.47092915, 0.05418356, 0.00930921, 0.2336181 ]),\n",
       " 'std_score_time': array([0.03166898, 0.00168619, 0.00047042, 0.03251903]),\n",
       " 'param_classifier': masked_array(data=[KNeighborsClassifier(), RandomForestClassifier(),\n",
       "                    LGBMClassifier(verbose=-1),\n",
       "                    <keras.wrappers.scikit_learn.KerasClassifier object at 0x000001D50A392988>],\n",
       "              mask=[False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'classifier': KNeighborsClassifier()},\n",
       "  {'classifier': RandomForestClassifier()},\n",
       "  {'classifier': LGBMClassifier(verbose=-1)},\n",
       "  {'classifier': <keras.wrappers.scikit_learn.KerasClassifier at 0x1d50a392988>}],\n",
       " 'split0_test_score': array([0.881823  , 0.90768415, 0.94944356,        nan]),\n",
       " 'split1_test_score': array([0.91912232, 0.92993428, 0.91763833,        nan]),\n",
       " 'split2_test_score': array([0.96979012, 0.98367607, 0.99046004,        nan]),\n",
       " 'mean_test_score': array([0.92357848, 0.9404315 , 0.95251398,        nan]),\n",
       " 'std_test_score': array([0.0360504 , 0.03189918, 0.02980851,        nan]),\n",
       " 'rank_test_score': array([3, 2, 1, 4])}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.cv_results_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LightGBM model is the best performing machine learning model, and the best cross-validation accuracy is 95.251%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model learning (for Comparison)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.717%\n",
      "Precision: 98.855%\n",
      "Recall: 99.733%\n",
      "F1-score: 99.292%\n",
      "Time: 0.8809\n",
      "Wall time: 184 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lg = lgb.LGBMClassifier(verbose = -1)\n",
    "lg.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = lg.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.84100000000001%\n",
      "Precision: 99.468%\n",
      "Recall: 99.733%\n",
      "F1-score: 99.601%\n",
      "Time: 7.75292\n",
      "Wall time: 2.72 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = RandomForestClassifier()\n",
    "rf.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 98.21600000000001%\n",
      "Precision: 92.036%\n",
      "Recall: 99.644%\n",
      "F1-score: 95.68900000000001%\n",
      "Time: 468.84165\n",
      "Wall time: 2.66 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "knn = KNeighborsClassifier()\n",
    "knn.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = knn.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from keras.layers import Input,Dense,Dropout,BatchNormalization,Activation\n",
    "from keras import Model\n",
    "import keras.backend as K\n",
    "import keras.callbacks as kcallbacks\n",
    "from keras import optimizers\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from keras.wrappers.scikit_learn import KerasClassifier\n",
    "from keras.callbacks import EarlyStopping\n",
    "def ANN(optimizer = 'sgd',neurons=32,batch_size=1024,epochs=80,activation='relu',patience=8,loss='binary_crossentropy'):\n",
    "    K.clear_session()\n",
    "    inputs=Input(shape=(X_train.shape[1],))\n",
    "    x=Dense(1000)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.3)(x)\n",
    "    x=Dense(256)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.25)(x)\n",
    "    x=Dense(2,activation='softmax')(x)\n",
    "    model=Model(inputs=inputs,outputs=x,name='base_nlp')\n",
    "    model.compile(optimizer='adam',loss='categorical_crossentropy')\n",
    "#     model.compile(optimizer=Adam(lr = 0.01),loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "    early_stopping = EarlyStopping(monitor=\"loss\", patience = patience)# early stop patience\n",
    "    history = model.fit(X_train, pd.get_dummies(y_train).values,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              callbacks = [early_stopping],\n",
    "              verbose=0) #verbose set to 1 will show the training process\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 88.39399999999999%\n",
      "Precision: 63.31099999999999%\n",
      "Recall: 98.933%\n",
      "F1-score: 77.211%\n",
      "Time: 468.84165\n",
      "Wall time: 10.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "ann = KerasClassifier(build_fn=ANN, verbose=0)\n",
    "ann.fit(X_train,y_train)\n",
    "predictions = ann.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 4. Hyperparameter Optimization\n",
    "Optimize the best performing machine learning model (lightGBM) by tuning its hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hold-out validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': 419.580078125, 'max_depth': 43.4521484375, 'learning_rate': 0.2587890625, 'num_leaves': 1163.18359375, 'min_child_samples': 38.7109375}\n",
      "Accuracy:0.9980568804098215\n"
     ]
    }
   ],
   "source": [
    "#Particle Swarm Optimization\n",
    "import optunity\n",
    "import optunity.metrics\n",
    "\n",
    "# Define the hyperparameter configuration space\n",
    "search = {\n",
    "    'n_estimators': [50, 500],\n",
    "    'max_depth': [5, 50],\n",
    "    'learning_rate': (0, 1),\n",
    "    \"num_leaves\":[100, 2000],\n",
    "    \"min_child_samples\":[10, 50],\n",
    "         }\n",
    "# Define the objective function\n",
    "def performance(n_estimators=None, max_depth=None,learning_rate=None,num_leaves=None,min_child_samples=None):\n",
    "    clf = lgb.LGBMClassifier(n_estimators=int(n_estimators),\n",
    "                                   max_depth=int(max_depth),\n",
    "                                   learning_rate=float(learning_rate),\n",
    "                                   num_leaves=int(num_leaves),\n",
    "                                   min_child_samples=int(min_child_samples),\n",
    "                                  )\n",
    "    clf.fit(X_train,y_train)\n",
    "    prediction = clf.predict(X_test)\n",
    "    score = accuracy_score(y_test,prediction)\n",
    "    return score\n",
    "\n",
    "# Detect the optimal hyperparameter values\n",
    "optimal_configuration, info, _ = optunity.maximize(performance,\n",
    "                                                  solver_name='particle swarm',\n",
    "                                                  num_evals=20,\n",
    "                                                   **search\n",
    "                                                  )\n",
    "print(optimal_configuration)\n",
    "print(\"Accuracy:\"+ str(info.optimum))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.806%\n",
      "Precision: 99.37899999999999%\n",
      "Recall: 99.644%\n",
      "F1-score: 99.512%\n",
      "Wall time: 1.66 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.LGBMClassifier(max_depth=43, learning_rate=  0.2587890625, n_estimators = 419, \n",
    "                         num_leaves = 1163, min_child_samples = 38)\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "After hyperparameter optimization, the hold-out accuracy has been improved from 99.717% to 99.806%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': 113.5009765625, 'max_depth': 40.96923828125, 'min_samples_split': 6.03857421875, 'min_samples_leaf': 1.8935546875, 'criterion_index': 0.30224609375}\n",
      "Accuracy:0.9982335276452924\n"
     ]
    }
   ],
   "source": [
    "import optunity\n",
    "import optunity.metrics\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# Define the hyperparameter configuration space for RandomForestClassifier\n",
    "search = {\n",
    "    'n_estimators': [50, 500],  # Number of trees in the forest\n",
    "    'max_depth': [5, 50],  # Maximum depth of the tree\n",
    "    'min_samples_split': [2, 11],  # Minimum number of samples required to split an internal node\n",
    "    'min_samples_leaf': [1, 11],  # Minimum number of samples required to be at a leaf node\n",
    "    'criterion_index': [0, 1],  # Index to select criterion, 0 for \"gini\", 1 for \"entropy\"\n",
    "}\n",
    "\n",
    "# Define the objective function for RandomForestClassifier\n",
    "def performance(n_estimators=None, max_depth=None, min_samples_split=None, min_samples_leaf=None, criterion_index=None):\n",
    "    # Convert criterion_index to actual criterion string\n",
    "    criterion = [\"gini\", \"entropy\"][int(criterion_index)]\n",
    "    \n",
    "    # Define and fit the model\n",
    "    clf = RandomForestClassifier(\n",
    "        n_estimators=int(n_estimators),\n",
    "        max_depth=int(max_depth),\n",
    "        min_samples_split=int(min_samples_split),\n",
    "        min_samples_leaf=int(min_samples_leaf),\n",
    "        criterion=criterion\n",
    "    )\n",
    "    clf.fit(X_train, y_train)\n",
    "    prediction = clf.predict(X_test)\n",
    "    score = accuracy_score(y_test, prediction)\n",
    "    return score\n",
    "\n",
    "# Detect the optimal hyperparameter values using PSO\n",
    "optimal_configuration, info, _ = optunity.maximize(performance,\n",
    "                                                   solver_name='particle swarm',\n",
    "                                                   num_evals=20,\n",
    "                                                   **search\n",
    "                                                  )\n",
    "\n",
    "print(optimal_configuration)\n",
    "print(\"Accuracy:\" + str(info.optimum))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.823%\n",
      "Precision: 99.38%\n",
      "Recall: 99.733%\n",
      "F1-score: 99.556%\n",
      "Wall time: 3.14 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = RandomForestClassifier(max_depth=40, n_estimators = 113, min_samples_split = 6,\n",
    "                         min_samples_leaf = 1, criterion = 'gini')\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Combined Algorithm Selection and Hyperparameter tuning (CASH)\n",
    "CASH is the process of combining the two AutoML procedures: model selection and hyperparameter optimization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Method: Particle Swarm Optimization (PSO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'algorithm': 'random-forest', 'epochs': None, 'neurons': None, 'patience': None, 'n_neighbors': None, 'learning_rate': None, 'max_depth': 31.3232421875, 'min_child_samples': None, 'n_estimators': 191.064453125, 'num_leaves': None, 'max_features': 5.4443359375, 'min_samples_leaf': 1.185546875, 'min_samples_split': 8.1787109375}\n",
      "0.9982335276452924\n"
     ]
    }
   ],
   "source": [
    "import optunity\n",
    "import optunity.metrics\n",
    "\n",
    "search = {'algorithm': {'k-nn': {'n_neighbors': [3, 10]},\n",
    "                        'random-forest': {\n",
    "                                'n_estimators': [50, 500],\n",
    "                                'max_features': [5, 12],\n",
    "                                'max_depth': [5,50],\n",
    "                                \"min_samples_split\":[2,11],\n",
    "                                \"min_samples_leaf\":[1,11]},\n",
    "                        'lightgbm': {\n",
    "                                'n_estimators': [50, 500],\n",
    "                                'max_depth': [5, 50],\n",
    "                                'learning_rate': (0, 1),\n",
    "                                \"num_leaves\":[100, 2000],\n",
    "                                \"min_child_samples\":[10, 50],\n",
    "                                    },\n",
    "                        'ann': {\n",
    "                                'neurons': [10, 100],\n",
    "                                'epochs': [20, 50],\n",
    "                                'patience': [3, 20],\n",
    "                                }\n",
    "                        }\n",
    "          \n",
    "         }\n",
    "def performance(\n",
    "                algorithm, n_neighbors=None, \n",
    "    n_estimators=None, max_features=None,max_depth=None,min_samples_split=None,min_samples_leaf=None,\n",
    "    learning_rate=None,num_leaves=None,min_child_samples=None,\n",
    "    neurons=None,epochs=None,patience=None\n",
    "):\n",
    "    # fit the model\n",
    "    if algorithm == 'k-nn':\n",
    "        model = KNeighborsClassifier(n_neighbors=int(n_neighbors))\n",
    "    elif algorithm == 'random-forest':\n",
    "        model = RandomForestClassifier(n_estimators=int(n_estimators),\n",
    "                                       max_features=int(max_features),\n",
    "                                       max_depth=int(max_depth),\n",
    "                                       min_samples_split=int(min_samples_split),\n",
    "                                       min_samples_leaf=int(min_samples_leaf))\n",
    "    elif algorithm == 'lightgbm':\n",
    "        model = lgb.LGBMClassifier(n_estimators=int(n_estimators),\n",
    "                                   max_depth=int(max_depth),\n",
    "                                   learning_rate=float(learning_rate),\n",
    "                                   num_leaves=int(num_leaves),\n",
    "                                   min_child_samples=int(min_child_samples),\n",
    "                                  )\n",
    "    elif algorithm == 'ann':\n",
    "        model = KerasClassifier(build_fn=ANN, verbose=0,\n",
    "                               neurons=int(neurons),\n",
    "                                epochs=int(epochs),\n",
    "                                patience=int(patience)\n",
    "                               )\n",
    "    else:\n",
    "        raise ArgumentError('Unknown algorithm: %s' % algorithm)\n",
    "# predict the test set\n",
    "    model.fit(X_train,y_train)\n",
    "    prediction = model.predict(X_test)\n",
    "    score = accuracy_score(y_test,prediction)\n",
    "    return score\n",
    "\n",
    "# Run the CASH process\n",
    "optimal_configuration, info, _ = optunity.maximize_structured(performance, \n",
    "                                                              search_space=search, \n",
    "                                                              num_evals=10)\n",
    "print(optimal_configuration)\n",
    "print(info.optimum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.823%\n",
      "Precision: 99.468%\n",
      "Recall: 99.644%\n",
      "F1-score: 99.556%\n",
      "Wall time: 6.76 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = RandomForestClassifier(max_depth=31, n_estimators = 191, max_features=5, min_samples_split = 8,\n",
    "                         min_samples_leaf = 1, criterion = 'gini')\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random Forest with the above hyperparameter values is identified as the optimal model"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}