AML_Attack_and_Defense_Dataset_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Case Study: Automated Machine Learning (AutoML) for Autonomous Intrusion Detection System Development \n",
    "This is the code for the paper entitled \"**[Enabling AutoML for Zero-Touch Network Security: Use-Case Driven Analysis](https://ieeexplore.ieee.org/document/10472316)**\" published in *IEEE Transactions on Network and Service Management* (IF:5.3).<br>\n",
    "Authors: Li Yang (liyanghart@gmail.com), Mirna El Rajab, Abdallah Shami, and Sami Muhaidat<br>\n",
    "\n",
    "L. Yang, M. E. Rajab, A. Shami, and S. Muhaidat, \"Enabling AutoML for Zero-Touch Network Security: Use-Case Driven Analysis,\" IEEE Transactions on Network and Service Management, pp. 1-28, 2024, doi: https://doi.org/10.1109/TNSM.2024.3376631."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code Part 3: Adversarial Machine Learning (AML) Attack and Defense\n",
    "As many network services and functionalities rely on AI/ML models, they are becoming increasingly vulnerable to AML attacks. AML attacks exploit the weaknesses and vulnerabilities of ML models by generating adversarial inputs that can deceive or manipulate the models into making incorrect predictions. In networks, AML attacks pose a significant threat to overall network security and reliability.  \n",
    "This case study aims to demonstrate the detrimental impact that AML attacks can have on ML models in networks and presents basic defense strategies to mitigate these attacks, thereby ensuring the accuracy of the ML-based IDS. In this case study, three common types of adversarial attacks ((i.e., DTA, FGSM, and BIM)) are used to generate adversarial samples to probe the vulnerability of the IDS.  \n",
    "Subsequently, basic defense mechanisms, including adversarial sample detection and filtering, are devised to safeguard the AutoML-based IDS against AML attacks.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset 1: CICIDS2017\n",
    "A subset of the network traffic data randomly sampled from the [CICIDS2017 dataset](https://www.unb.ca/cic/datasets/ids-2017.html).  \n",
    "\n",
    "The Canadian Institute for Cybersecurity Intrusion Detection System 2017 (CICIDS2017) dataset has the most updated network threats. The CICIDS2017 dataset is close to real-world network data since it has a large amount of network traffic data, a variety of network features, various types of attacks, and highly imbalanced classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split,cross_val_score\n",
    "import lightgbm as lgb\n",
    "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score, precision_score, recall_score, f1_score\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from scipy.stats import shapiro\n",
    "from imblearn.over_sampling import SMOTE\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read the sampled CICIDS2017 dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"Data/cic_0.01km.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flow Duration</th>\n",
       "      <th>Total Length of Fwd Packets</th>\n",
       "      <th>Fwd Packet Length Max</th>\n",
       "      <th>Fwd Packet Length Mean</th>\n",
       "      <th>Bwd Packet Length Max</th>\n",
       "      <th>Bwd Packet Length Min</th>\n",
       "      <th>Flow IAT Mean</th>\n",
       "      <th>Flow IAT Min</th>\n",
       "      <th>Fwd IAT Min</th>\n",
       "      <th>Fwd Header Length</th>\n",
       "      <th>Bwd Header Length</th>\n",
       "      <th>Fwd Packets/s</th>\n",
       "      <th>Bwd Packets/s</th>\n",
       "      <th>Min Packet Length</th>\n",
       "      <th>URG Flag Count</th>\n",
       "      <th>Down/Up Ratio</th>\n",
       "      <th>Init_Win_bytes_forward</th>\n",
       "      <th>Init_Win_bytes_backward</th>\n",
       "      <th>min_seg_size_forward</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>50833</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.083300e+04</td>\n",
       "      <td>50833</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>19.672260</td>\n",
       "      <td>19.672260</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>319</td>\n",
       "      <td>153</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>49</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4.900000e+01</td>\n",
       "      <td>49</td>\n",
       "      <td>49</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>40816.326530</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>277</td>\n",
       "      <td>-1</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>306</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6.0000</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>3.060000e+02</td>\n",
       "      <td>306</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>3267.973856</td>\n",
       "      <td>3267.973856</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>63041</td>\n",
       "      <td>65</td>\n",
       "      <td>65</td>\n",
       "      <td>65.0000</td>\n",
       "      <td>124</td>\n",
       "      <td>124</td>\n",
       "      <td>6.304100e+04</td>\n",
       "      <td>63041</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>15.862693</td>\n",
       "      <td>15.862693</td>\n",
       "      <td>65</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>47682</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>43.0000</td>\n",
       "      <td>59</td>\n",
       "      <td>59</td>\n",
       "      <td>4.768200e+04</td>\n",
       "      <td>47682</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>20.972275</td>\n",
       "      <td>20.972275</td>\n",
       "      <td>43</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28298</th>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4.500000e+01</td>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>22222.222220</td>\n",
       "      <td>22222.222220</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>349</td>\n",
       "      <td>307</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28299</th>\n",
       "      <td>114309573</td>\n",
       "      <td>511</td>\n",
       "      <td>427</td>\n",
       "      <td>31.9375</td>\n",
       "      <td>746</td>\n",
       "      <td>0</td>\n",
       "      <td>3.941709e+06</td>\n",
       "      <td>94</td>\n",
       "      <td>165</td>\n",
       "      <td>332</td>\n",
       "      <td>424</td>\n",
       "      <td>0.139971</td>\n",
       "      <td>0.122474</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8192</td>\n",
       "      <td>343</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28300</th>\n",
       "      <td>48850</td>\n",
       "      <td>80</td>\n",
       "      <td>40</td>\n",
       "      <td>40.0000</td>\n",
       "      <td>72</td>\n",
       "      <td>72</td>\n",
       "      <td>1.628333e+04</td>\n",
       "      <td>1</td>\n",
       "      <td>48</td>\n",
       "      <td>64</td>\n",
       "      <td>64</td>\n",
       "      <td>40.941658</td>\n",
       "      <td>40.941658</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28301</th>\n",
       "      <td>260</td>\n",
       "      <td>66</td>\n",
       "      <td>33</td>\n",
       "      <td>33.0000</td>\n",
       "      <td>97</td>\n",
       "      <td>97</td>\n",
       "      <td>8.666667e+01</td>\n",
       "      <td>48</td>\n",
       "      <td>48</td>\n",
       "      <td>40</td>\n",
       "      <td>40</td>\n",
       "      <td>7692.307692</td>\n",
       "      <td>7692.307692</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28302</th>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6.0000</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>2.500000e+01</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>40000.000000</td>\n",
       "      <td>40000.000000</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>11256</td>\n",
       "      <td>35000</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>28303 rows × 20 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Flow Duration  Total Length of Fwd Packets  Fwd Packet Length Max  \\\n",
       "0              50833                            0                      0   \n",
       "1                 49                            0                      0   \n",
       "2                306                            6                      6   \n",
       "3              63041                           65                     65   \n",
       "4              47682                           43                     43   \n",
       "...              ...                          ...                    ...   \n",
       "28298             45                            0                      0   \n",
       "28299      114309573                          511                    427   \n",
       "28300          48850                           80                     40   \n",
       "28301            260                           66                     33   \n",
       "28302             25                            6                      6   \n",
       "\n",
       "       Fwd Packet Length Mean  Bwd Packet Length Max  Bwd Packet Length Min  \\\n",
       "0                      0.0000                      0                      0   \n",
       "1                      0.0000                      0                      0   \n",
       "2                      6.0000                      6                      6   \n",
       "3                     65.0000                    124                    124   \n",
       "4                     43.0000                     59                     59   \n",
       "...                       ...                    ...                    ...   \n",
       "28298                  0.0000                      0                      0   \n",
       "28299                 31.9375                    746                      0   \n",
       "28300                 40.0000                     72                     72   \n",
       "28301                 33.0000                     97                     97   \n",
       "28302                  6.0000                      6                      6   \n",
       "\n",
       "       Flow IAT Mean  Flow IAT Min  Fwd IAT Min  Fwd Header Length  \\\n",
       "0       5.083300e+04         50833            0                 32   \n",
       "1       4.900000e+01            49           49                 64   \n",
       "2       3.060000e+02           306            0                 20   \n",
       "3       6.304100e+04         63041            0                 32   \n",
       "4       4.768200e+04         47682            0                 32   \n",
       "...              ...           ...          ...                ...   \n",
       "28298   4.500000e+01            45            0                 32   \n",
       "28299   3.941709e+06            94          165                332   \n",
       "28300   1.628333e+04             1           48                 64   \n",
       "28301   8.666667e+01            48           48                 40   \n",
       "28302   2.500000e+01            25            0                 20   \n",
       "\n",
       "       Bwd Header Length  Fwd Packets/s  Bwd Packets/s  Min Packet Length  \\\n",
       "0                     32      19.672260      19.672260                  0   \n",
       "1                      0   40816.326530       0.000000                  0   \n",
       "2                     20    3267.973856    3267.973856                  6   \n",
       "3                     32      15.862693      15.862693                 65   \n",
       "4                     32      20.972275      20.972275                 43   \n",
       "...                  ...            ...            ...                ...   \n",
       "28298                 32   22222.222220   22222.222220                  0   \n",
       "28299                424       0.139971       0.122474                  0   \n",
       "28300                 64      40.941658      40.941658                 40   \n",
       "28301                 40    7692.307692    7692.307692                 33   \n",
       "28302                 20   40000.000000   40000.000000                  6   \n",
       "\n",
       "       URG Flag Count  Down/Up Ratio  Init_Win_bytes_forward  \\\n",
       "0                   1              1                     319   \n",
       "1                   0              0                     277   \n",
       "2                   0              1                       0   \n",
       "3                   0              1                      -1   \n",
       "4                   0              1                      -1   \n",
       "...               ...            ...                     ...   \n",
       "28298               1              1                     349   \n",
       "28299               0              0                    8192   \n",
       "28300               0              1                      -1   \n",
       "28301               0              1                      -1   \n",
       "28302               1              1                   11256   \n",
       "\n",
       "       Init_Win_bytes_backward  min_seg_size_forward  Label  \n",
       "0                          153                    32      0  \n",
       "1                           -1                    32      0  \n",
       "2                            0                    20      0  \n",
       "3                           -1                    32      0  \n",
       "4                           -1                    32      0  \n",
       "...                        ...                   ...    ...  \n",
       "28298                      307                    32      0  \n",
       "28299                      343                    20      0  \n",
       "28300                       -1                    32      0  \n",
       "28301                       -1                    20      0  \n",
       "28302                    35000                    20      0  \n",
       "\n",
       "[28303 rows x 20 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Automated Data Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated Transformation/Encoding\n",
    "Automatically identify and transform string/text features into numerical features to make the data more readable by ML models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the automated data encoding function\n",
    "def Auto_Encoding(df):\n",
    "    cat_features=[x for x in df.columns if df[x].dtype==\"object\"] ## Find string/text features\n",
    "    le=LabelEncoder()\n",
    "    for col in cat_features:\n",
    "        if col in df.columns:\n",
    "            i = df.columns.get_loc(col)\n",
    "            # Transform to numerical features\n",
    "            df.iloc[:,i] = df.apply(lambda i:le.fit_transform(i.astype(str)), axis=0, result_type='expand')\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=Auto_Encoding(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated Imputation\n",
    "Detect and impute missing values to improve data quality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the automated data imputation function\n",
    "def Auto_Imputation(df):\n",
    "    if df.isnull().values.any() or np.isinf(df).values.any(): # if there is any empty or infinite values\n",
    "        df.replace([np.inf, -np.inf], np.nan, inplace=True)\n",
    "        df.fillna(0, inplace = True)  # Replace empty values with zeros; there are other imputation methods discussed in the paper\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=Auto_Imputation(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated normalization\n",
    "Normalize the range of features to a similar scale to improve data quality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def Auto_Normalization(df):\n",
    "    stat, p = shapiro(df)\n",
    "    print('Statistics=%.3f, p=%.3f' % (stat, p))\n",
    "    # interpret\n",
    "    alpha = 0.05\n",
    "    numeric_features = df.drop(['Label'],axis = 1).dtypes[df.dtypes != 'object'].index\n",
    "    \n",
    "    # check if the data distribution follows a Gaussian/normal distribution\n",
    "    # If so, select the Z-score normalization method; otherwise, select the min-max normalization\n",
    "    # Details are in the paper\n",
    "    if p > alpha:\n",
    "        print('Sample looks Gaussian (fail to reject H0)')\n",
    "        df[numeric_features] = df[numeric_features].apply(\n",
    "            lambda x: (x - x.mean()) / (x.std()))\n",
    "        print('Z-score normalization is automatically chosen and used')\n",
    "    else:\n",
    "        print('Sample does not look Gaussian (reject H0)')\n",
    "        df[numeric_features] = df[numeric_features].apply(\n",
    "            lambda x: (x - x.min()) / (x.max()-x.min()))\n",
    "        print('Min-max normalization is automatically chosen and used')\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Statistics=0.076, p=0.000\n",
      "Sample does not look Gaussian (reject H0)\n",
      "Min-max normalization is automatically chosen and used\n"
     ]
    }
   ],
   "source": [
    "df=Auto_Normalization(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=Auto_Imputation(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Train-test split\n",
    "Split the dataset into the training and the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop(['Label'],axis=1)\n",
    "y = df['Label']\n",
    "\n",
    "# Here we used the 80%/20% split, it can be changed based on specific tasks\n",
    "#X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2, shuffle=False,random_state = 0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2,random_state = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated data balancing\n",
    "Generate minority class samples to solve class-imbalance and improve data quality.  \n",
    "Synthetic Minority Over-sampling Technique (SMOTE) method is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    18126\n",
       "1     4516\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For binary data (can be modified for multi-class data with the same logic)\n",
    "def Auto_Balancing(X_train, y_train):\n",
    "    number0 = pd.Series(y_train).value_counts().iloc[0]\n",
    "    number1 = pd.Series(y_train).value_counts().iloc[1]\n",
    "    \n",
    "    if number0 > number1:\n",
    "        nlarge = number0\n",
    "    else:\n",
    "        nlarge = number1\n",
    "    \n",
    "    # evaluate whether the incoming dataset is imbalanced (the abnormal/normal ratio is smaller than a threshold (e.g., 50%)) \n",
    "    if (number1/number0 > 1.5) or (number0/number1 > 1.5):\n",
    "        smote=SMOTE(n_jobs=-1,sampling_strategy={0:nlarge, 1:nlarge})\n",
    "        X_train, y_train = smote.fit_sample(X_train, y_train)\n",
    "        \n",
    "    return X_train, y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = Auto_Balancing(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    18126\n",
       "0    18126\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Automated Feature Engineering\n",
    "Feature selection method 1: **Information Gain (IG)**, used to remove irrelevant features to improve model efficiency  \n",
    "Feature selection method 2: **Pearson Correlation**, used to remove redundant features to improve model efficiency and accuracy  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove irrelevant features and select important features\n",
    "def Feature_Importance_IG(data):\n",
    "    features = data.drop(['Label'],axis=1).values  # \"Label\" should be changed to the target class variable name if different\n",
    "    labels = data['Label'].values\n",
    "    \n",
    "    # Extract feature names\n",
    "    feature_names = list(data.drop(['Label'],axis=1).columns)\n",
    "\n",
    "    # Empty array for feature importances\n",
    "    feature_importance_values = np.zeros(len(feature_names))\n",
    "    model = lgb.LGBMRegressor(verbose = -1)\n",
    "    model.fit(features, labels)\n",
    "    feature_importances = pd.DataFrame({'feature': feature_names, 'importance': model.feature_importances_})\n",
    "\n",
    "    # Sort features according to importance\n",
    "    feature_importances = feature_importances.sort_values('importance', ascending = False).reset_index(drop = True)\n",
    "\n",
    "    # Normalize the feature importances to add up to one\n",
    "    feature_importances['normalized_importance'] = feature_importances['importance'] / feature_importances['importance'].sum()\n",
    "    feature_importances['cumulative_importance'] = np.cumsum(feature_importances['normalized_importance'])\n",
    "    \n",
    "    cumulative_importance=0.90 # Only keep the important features with cumulative importance scores>=90%. It can be changed.\n",
    "\n",
    "    # Make sure most important features are on top\n",
    "    feature_importances = feature_importances.sort_values('cumulative_importance')\n",
    "\n",
    "    # Identify the features not needed to reach the cumulative_importance\n",
    "    record_low_importance = feature_importances[feature_importances['cumulative_importance'] > cumulative_importance]\n",
    "\n",
    "    to_drop = list(record_low_importance['feature'])\n",
    "#     print(feature_importances.drop(['importance'],axis=1))\n",
    "    return to_drop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove redundant features\n",
    "def Feature_Redundancy_Pearson(data):\n",
    "    correlation_threshold=0.90 # Only remove features with the redundancy>90%. It can be changed\n",
    "    features = data.drop(['Label'],axis=1)\n",
    "    corr_matrix = features.corr()\n",
    "\n",
    "    # Extract the upper triangle of the correlation matrix\n",
    "    upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k = 1).astype(np.bool))\n",
    "\n",
    "    # Select the features with correlations above the threshold\n",
    "    # Need to use the absolute value\n",
    "    to_drop = [column for column in upper.columns if any(upper[column].abs() > correlation_threshold)]\n",
    "\n",
    "    # Dataframe to hold correlated pairs\n",
    "    record_collinear = pd.DataFrame(columns = ['drop_feature', 'corr_feature', 'corr_value'])\n",
    "\n",
    "    # Iterate through the columns to drop\n",
    "    for column in to_drop:\n",
    "\n",
    "        # Find the correlated features\n",
    "        corr_features = list(upper.index[upper[column].abs() > correlation_threshold])\n",
    "\n",
    "        # Find the correlated values\n",
    "        corr_values = list(upper[column][upper[column].abs() > correlation_threshold])\n",
    "        drop_features = [column for _ in range(len(corr_features))]    \n",
    "\n",
    "        # Record the information (need a temp df for now)\n",
    "        temp_df = pd.DataFrame.from_dict({'drop_feature': drop_features,\n",
    "                                         'corr_feature': corr_features,\n",
    "                                         'corr_value': corr_values})\n",
    "        record_collinear = record_collinear.append(temp_df, ignore_index = True)\n",
    "#     print(record_collinear)\n",
    "    return to_drop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Auto_Feature_Engineering(df):\n",
    "    drop1 = Feature_Importance_IG(df)\n",
    "    dfh1 = df.drop(columns = drop1)\n",
    "    \n",
    "    drop2 = Feature_Redundancy_Pearson(dfh1)\n",
    "    dfh2 = dfh1.drop(columns = drop2)\n",
    "    \n",
    "    return dfh2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flow Duration</th>\n",
       "      <th>Total Length of Fwd Packets</th>\n",
       "      <th>Fwd Packet Length Max</th>\n",
       "      <th>Fwd Packet Length Mean</th>\n",
       "      <th>Bwd Packet Length Max</th>\n",
       "      <th>Flow IAT Mean</th>\n",
       "      <th>Flow IAT Min</th>\n",
       "      <th>Fwd IAT Min</th>\n",
       "      <th>Fwd Header Length</th>\n",
       "      <th>Bwd Packets/s</th>\n",
       "      <th>Init_Win_bytes_forward</th>\n",
       "      <th>Init_Win_bytes_backward</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.236419e-04</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.707129e-04</td>\n",
       "      <td>4.707129e-04</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000150</td>\n",
       "      <td>1.311484e-05</td>\n",
       "      <td>0.004883</td>\n",
       "      <td>0.002350</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.416669e-07</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.907407e-07</td>\n",
       "      <td>4.907407e-07</td>\n",
       "      <td>4.083333e-07</td>\n",
       "      <td>0.000299</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.004242</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.583334e-06</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>0.000242</td>\n",
       "      <td>0.001556</td>\n",
       "      <td>0.000516</td>\n",
       "      <td>2.870370e-06</td>\n",
       "      <td>2.870370e-06</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000094</td>\n",
       "      <td>2.178649e-03</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.253752e-04</td>\n",
       "      <td>0.000090</td>\n",
       "      <td>0.002619</td>\n",
       "      <td>0.016856</td>\n",
       "      <td>0.010660</td>\n",
       "      <td>5.837500e-04</td>\n",
       "      <td>5.837500e-04</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000150</td>\n",
       "      <td>1.057513e-05</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.973835e-04</td>\n",
       "      <td>0.000060</td>\n",
       "      <td>0.001732</td>\n",
       "      <td>0.011151</td>\n",
       "      <td>0.005072</td>\n",
       "      <td>4.415370e-04</td>\n",
       "      <td>4.415370e-04</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000150</td>\n",
       "      <td>1.398152e-05</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28298</th>\n",
       "      <td>4.083335e-07</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.537037e-07</td>\n",
       "      <td>4.537037e-07</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000150</td>\n",
       "      <td>1.481481e-02</td>\n",
       "      <td>0.005341</td>\n",
       "      <td>0.004700</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28299</th>\n",
       "      <td>9.525802e-01</td>\n",
       "      <td>0.000710</td>\n",
       "      <td>0.017204</td>\n",
       "      <td>0.008282</td>\n",
       "      <td>0.064133</td>\n",
       "      <td>3.649735e-02</td>\n",
       "      <td>9.074074e-07</td>\n",
       "      <td>1.375000e-06</td>\n",
       "      <td>0.001554</td>\n",
       "      <td>8.164962e-08</td>\n",
       "      <td>0.125015</td>\n",
       "      <td>0.005249</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28300</th>\n",
       "      <td>4.071168e-04</td>\n",
       "      <td>0.000111</td>\n",
       "      <td>0.001612</td>\n",
       "      <td>0.010373</td>\n",
       "      <td>0.006190</td>\n",
       "      <td>1.508086e-04</td>\n",
       "      <td>4.629629e-08</td>\n",
       "      <td>4.000000e-07</td>\n",
       "      <td>0.000299</td>\n",
       "      <td>2.729444e-05</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28301</th>\n",
       "      <td>2.200001e-06</td>\n",
       "      <td>0.000092</td>\n",
       "      <td>0.001330</td>\n",
       "      <td>0.008558</td>\n",
       "      <td>0.008339</td>\n",
       "      <td>8.395061e-07</td>\n",
       "      <td>4.814815e-07</td>\n",
       "      <td>4.000000e-07</td>\n",
       "      <td>0.000187</td>\n",
       "      <td>5.128205e-03</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28302</th>\n",
       "      <td>2.416668e-07</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>0.000242</td>\n",
       "      <td>0.001556</td>\n",
       "      <td>0.000516</td>\n",
       "      <td>2.685185e-07</td>\n",
       "      <td>2.685185e-07</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000094</td>\n",
       "      <td>2.666667e-02</td>\n",
       "      <td>0.171768</td>\n",
       "      <td>0.534073</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>28303 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Flow Duration  Total Length of Fwd Packets  Fwd Packet Length Max  \\\n",
       "0       4.236419e-04                     0.000000               0.000000   \n",
       "1       4.416669e-07                     0.000000               0.000000   \n",
       "2       2.583334e-06                     0.000008               0.000242   \n",
       "3       5.253752e-04                     0.000090               0.002619   \n",
       "4       3.973835e-04                     0.000060               0.001732   \n",
       "...              ...                          ...                    ...   \n",
       "28298   4.083335e-07                     0.000000               0.000000   \n",
       "28299   9.525802e-01                     0.000710               0.017204   \n",
       "28300   4.071168e-04                     0.000111               0.001612   \n",
       "28301   2.200001e-06                     0.000092               0.001330   \n",
       "28302   2.416668e-07                     0.000008               0.000242   \n",
       "\n",
       "       Fwd Packet Length Mean  Bwd Packet Length Max  Flow IAT Mean  \\\n",
       "0                    0.000000               0.000000   4.707129e-04   \n",
       "1                    0.000000               0.000000   4.907407e-07   \n",
       "2                    0.001556               0.000516   2.870370e-06   \n",
       "3                    0.016856               0.010660   5.837500e-04   \n",
       "4                    0.011151               0.005072   4.415370e-04   \n",
       "...                       ...                    ...            ...   \n",
       "28298                0.000000               0.000000   4.537037e-07   \n",
       "28299                0.008282               0.064133   3.649735e-02   \n",
       "28300                0.010373               0.006190   1.508086e-04   \n",
       "28301                0.008558               0.008339   8.395061e-07   \n",
       "28302                0.001556               0.000516   2.685185e-07   \n",
       "\n",
       "       Flow IAT Min   Fwd IAT Min  Fwd Header Length  Bwd Packets/s  \\\n",
       "0      4.707129e-04  0.000000e+00           0.000150   1.311484e-05   \n",
       "1      4.907407e-07  4.083333e-07           0.000299   0.000000e+00   \n",
       "2      2.870370e-06  0.000000e+00           0.000094   2.178649e-03   \n",
       "3      5.837500e-04  0.000000e+00           0.000150   1.057513e-05   \n",
       "4      4.415370e-04  0.000000e+00           0.000150   1.398152e-05   \n",
       "...             ...           ...                ...            ...   \n",
       "28298  4.537037e-07  0.000000e+00           0.000150   1.481481e-02   \n",
       "28299  9.074074e-07  1.375000e-06           0.001554   8.164962e-08   \n",
       "28300  4.629629e-08  4.000000e-07           0.000299   2.729444e-05   \n",
       "28301  4.814815e-07  4.000000e-07           0.000187   5.128205e-03   \n",
       "28302  2.685185e-07  0.000000e+00           0.000094   2.666667e-02   \n",
       "\n",
       "       Init_Win_bytes_forward  Init_Win_bytes_backward  Label  \n",
       "0                    0.004883                 0.002350      0  \n",
       "1                    0.004242                 0.000000      0  \n",
       "2                    0.000015                 0.000015      0  \n",
       "3                    0.000000                 0.000000      0  \n",
       "4                    0.000000                 0.000000      0  \n",
       "...                       ...                      ...    ...  \n",
       "28298                0.005341                 0.004700      0  \n",
       "28299                0.125015                 0.005249      0  \n",
       "28300                0.000000                 0.000000      0  \n",
       "28301                0.000000                 0.000000      0  \n",
       "28302                0.171768                 0.534073      0  \n",
       "\n",
       "[28303 rows x 13 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfh2 = Auto_Feature_Engineering(df)\n",
    "dfh2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Split & Balancing (After Feature Engineering)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = dfh2.drop(['Label'],axis=1)\n",
    "y = dfh2['Label']\n",
    "\n",
    "#X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2, shuffle=False,random_state = 0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2,random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = Auto_Balancing(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Automated Model Selection\n",
    "Select the best-performing model among five common machine learning models (Naive Bayes, KNN, random forest, LightGBM, and ANN/MLP) by evaluating their learning performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model learning (for Comparison)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.77000000000001%\n",
      "Precision: 99.466%\n",
      "Recall: 99.378%\n",
      "F1-score: 99.422%\n",
      "Time: 2.29035\n",
      "Wall time: 315 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lg = lgb.LGBMClassifier(verbose = -1)\n",
    "lg.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = lg.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.735%\n",
      "Precision: 99.465%\n",
      "Recall: 99.2%\n",
      "F1-score: 99.332%\n",
      "Time: 9.51379\n",
      "Wall time: 2.82 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = RandomForestClassifier()\n",
    "rf.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 4. Hyperparameter Optimization\n",
    "Optimize the best performing machine learning model (lightGBM) by tuning its hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hold-out validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 50/50 [01:22<00:00,  1.65s/trial, best loss: -0.9982335276452924]\n",
      "LightGBM: Hyperopt estimated optimum {'learning_rate': 0.06157312210088248, 'max_depth': 22.0, 'min_child_samples': 20.0, 'n_estimators': 360.0, 'num_leaves': 2000.0}\n"
     ]
    }
   ],
   "source": [
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "\n",
    "# Define the objective function\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_estimators': int(params['n_estimators']), \n",
    "        'max_depth': int(params['max_depth']),\n",
    "        'learning_rate': abs(float(params['learning_rate'])),\n",
    "        \"num_leaves\": int(params['num_leaves']),\n",
    "        \"min_child_samples\": int(params['min_child_samples']),\n",
    "    }\n",
    "    clf = lgb.LGBMClassifier( **params)\n",
    "    clf.fit(X_train,y_train)\n",
    "    predictions = clf.predict(X_test)\n",
    "    score = accuracy_score(y_test,predictions)\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "\n",
    "# Define the hyperparameter configuration space\n",
    "space = {\n",
    "    'n_estimators': hp.quniform('n_estimators', 50, 500, 20),\n",
    "    'max_depth': hp.quniform('max_depth', 5, 50, 1),\n",
    "    \"learning_rate\":hp.uniform('learning_rate', 0, 1),\n",
    "    \"num_leaves\":hp.quniform('num_leaves',100,2000,100),\n",
    "    \"min_child_samples\":hp.quniform('min_child_samples',10,50,5),\n",
    "}\n",
    "\n",
    "# Detect the optimal hyperparameter values\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=50)\n",
    "print(\"LightGBM: Hyperopt estimated optimum {}\".format(best))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.823%\n",
      "Precision: 99.468%\n",
      "Recall: 99.644%\n",
      "F1-score: 99.556%\n",
      "Wall time: 7.11 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.LGBMClassifier(max_depth=22, learning_rate= 0.06157312210088248, n_estimators = 360, \n",
    "                         num_leaves = 2000, min_child_samples = 20)\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "After hyperparameter optimization, the hold-out accuracy has been improved from 99.806% to 99.841%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 50/50 [07:12<00:00,  8.65s/trial, best loss: -0.9973502914679385]\n",
      "RandomForest: Hyperopt estimated optimum {'criterion': 0, 'max_depth': 19.0, 'min_samples_leaf': 1.0, 'min_samples_split': 3.0, 'n_estimators': 380.0}\n"
     ]
    }
   ],
   "source": [
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Define the objective function\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_estimators': int(params['n_estimators']), \n",
    "        'max_depth': int(params['max_depth']),\n",
    "        'min_samples_split': int(params['min_samples_split']),\n",
    "        'min_samples_leaf': int(params['min_samples_leaf']),\n",
    "        'criterion': str(params['criterion'])\n",
    "    }\n",
    "    clf = RandomForestClassifier(**params)\n",
    "    clf.fit(X_train,y_train)\n",
    "    predictions = clf.predict(X_test)\n",
    "    score = accuracy_score(y_test,predictions)\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "\n",
    "# Define the hyperparameter configuration space\n",
    "space = {\n",
    "    'n_estimators': hp.quniform('n_estimators', 20, 500, 20),\n",
    "    'max_depth': hp.quniform('max_depth', 5, 50, 1),\n",
    "    'min_samples_split': hp.quniform('min_samples_split', 2, 11, 1),\n",
    "    'min_samples_leaf': hp.quniform('min_samples_leaf', 1, 11, 1),\n",
    "    'criterion': hp.choice('criterion', ['gini', 'entropy'])\n",
    "}\n",
    "\n",
    "# Detect the optimal hyperparameter values\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=50)\n",
    "print(\"RandomForest: Hyperopt estimated optimum {}\".format(best))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.735%\n",
      "Precision: 99.465%\n",
      "Recall: 99.2%\n",
      "F1-score: 99.332%\n",
      "Wall time: 10.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = RandomForestClassifier(max_depth=19, n_estimators = 380, min_samples_split = 3,\n",
    "                         min_samples_leaf = 1, criterion = 'gini')\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AML"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experiment 1\n",
    "* 1. Original ML model for IDS\n",
    "* 2. generate adversarial samples using DecisionTreeAttack or other attacks\n",
    "* 3. test ML model under attack\n",
    "* 4. develop adversarial sample detection model\n",
    "* 5. remove adversarial samples from the training set\n",
    "* 6. re-train IDS model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DecisionTreeAttack"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Original ML model for IDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.823%\n",
      "Precision: 99.468%\n",
      "Recall: 99.644%\n",
      "F1-score: 99.556%\n",
      "Wall time: 6.91 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.LGBMClassifier(max_depth=22, learning_rate= 0.06157312210088248, n_estimators = 360, \n",
    "                         num_leaves = 2000, min_child_samples = 20)\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: generate adversarial samples using DecisionTreeAttack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Decision tree attack: 100%|██████████| 36252/36252 [00:18<00:00, 1999.66it/s]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.datasets import load_digits\n",
    "from matplotlib import pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from art.attacks.evasion import DecisionTreeAttack\n",
    "from art.estimators.classification import SklearnClassifier\n",
    "\n",
    "clf = DecisionTreeClassifier()\n",
    "clf.fit(X_train,y_train)\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "y_train1 = y_train.values.reshape(-1, 1)\n",
    "y_train1_AML = OneHotEncoder().fit_transform(y_train1).toarray()\n",
    "\n",
    "clf_art = SklearnClassifier(clf)\n",
    "attack = DecisionTreeAttack(clf_art)\n",
    "\n",
    "# Generate adversarial examples\n",
    "x_adv = attack.generate(X_train.values,y_train1_AML)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: test ML model under attack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 78.608%\n",
      "Precision: 15.873000000000001%\n",
      "Recall: 1.778%\n",
      "F1-score: 3.197%\n",
      "Time: 6.69349\n",
      "Wall time: 1.31 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#DTA\n",
    "rf = lgb.LGBMClassifier(max_depth=22, learning_rate= 0.06157312210088248, n_estimators = 360, \n",
    "                         num_leaves = 2000, min_child_samples = 20)\n",
    "rf.fit(x_adv,y_train)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: develop adversarial sample detection model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = np.concatenate([x_adv,X_train])\n",
    "y1 = pd.Series(np.ones(y_train.shape[0]))\n",
    "y2 = pd.Series(np.zeros(y_train.shape[0]))\n",
    "y_new = np.concatenate([y1,y2])\n",
    "\n",
    "\n",
    "X_train_d, X_test_d, y_train_d, y_test_d = train_test_split(X_new,y_new, train_size = 0.1, test_size = 0.9,random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.937%\n",
      "Precision: 99.932%\n",
      "Recall: 99.942%\n",
      "F1-score: 99.937%\n",
      "Time: 4.44764\n",
      "Wall time: 1.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = lgb.LGBMClassifier(max_depth=22, learning_rate= 0.06157312210088248, n_estimators = 360, \n",
    "                         num_leaves = 2000, min_child_samples = 20)\n",
    "rf.fit(X_train_d,y_train_d)\n",
    "t1=time.time()\n",
    "predictions2 = rf.predict(X_test_d)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test_d)*1000000,5)))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: remove adversarial samples from the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "detection_results = rf.predict(X_new)\n",
    "X_new1 = X_new\n",
    "y_new1 = np.concatenate([y_train,y_train])\n",
    "\n",
    "indices_to_remove = [i for i in range(len(detection_results)) if detection_results[i] == 1]\n",
    "for i in reversed(indices_to_remove):\n",
    "    X_new1 = np.delete(X_new1, i, axis=0)\n",
    "    y_new1 = np.delete(y_new1, i)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: re-train IDS model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.806%\n",
      "Precision: 99.37899999999999%\n",
      "Recall: 99.644%\n",
      "F1-score: 99.512%\n",
      "Time: 10.21826\n",
      "Wall time: 5.66 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#DTA\n",
    "rf = lgb.LGBMClassifier(max_depth=22, learning_rate= 0.06157312210088248, n_estimators = 360, \n",
    "                         num_leaves = 2000, min_child_samples = 20)\n",
    "rf.fit(X_new1,y_new1)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FastGradientMethod"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Original ML model for IDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.823%\n",
      "Precision: 99.468%\n",
      "Recall: 99.644%\n",
      "F1-score: 99.556%\n",
      "Wall time: 6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.LGBMClassifier(max_depth=22, learning_rate= 0.06157312210088248, n_estimators = 360, \n",
    "                         num_leaves = 2000, min_child_samples = 20)\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: generate adversarial samples using DecisionTreeAttack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.layers import Input,Dense,Dropout,BatchNormalization,Activation\n",
    "from art.estimators.classification import KerasClassifier\n",
    "from keras import Model\n",
    "import keras.backend as K\n",
    "import keras.callbacks as kcallbacks\n",
    "from keras import optimizers\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "from art.attacks import evasion\n",
    "from art.attacks.evasion import FastGradientMethod\n",
    "from art.estimators.classification import SklearnClassifier\n",
    "\n",
    "def ANN(optimizer = 'sgd',neurons=16,batch_size=1024,epochs=80,activation='relu',patience=8,loss='binary_crossentropy'):\n",
    "    inputs=Input(shape=(X.shape[1],))\n",
    "    x=Dense(1000)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.3)(x)\n",
    "    x=Dense(256)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.25)(x)\n",
    "    x=Dense(2,activation='softmax')(x)\n",
    "    model=Model(inputs=inputs,outputs=x,name='base_nlp')\n",
    "    model.compile(optimizer='adam',loss='categorical_crossentropy')\n",
    "#     model.compile(optimizer=Adam(lr = 0.01),loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "    early_stopping = EarlyStopping(monitor=\"loss\", patience = patience)# early stop patience\n",
    "    history = model.fit(X, pd.get_dummies(y).values,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              callbacks = [early_stopping],\n",
    "              verbose=0) #verbose set to 1 will show the training process\n",
    "    return model\n",
    "\n",
    "# Create a KerasClassifier\n",
    "classifier = KerasClassifier(model=ANN(), clip_values=(0, 1))\n",
    "\n",
    "# Create the FastGradientMethod attack\n",
    "attack = FastGradientMethod(estimator=classifier, eps=0.1)\n",
    "\n",
    "# Generate adversarial examples\n",
    "x_adv = attack.generate(X_train.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: test ML model under attack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 86.31%\n",
      "Precision: 96.296%\n",
      "Recall: 32.356%\n",
      "F1-score: 48.436%\n",
      "Time: 15.67928\n",
      "Wall time: 8.78 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#FGM\n",
    "rf = lgb.LGBMClassifier(max_depth=22, learning_rate= 0.06157312210088248, n_estimators = 360, \n",
    "                         num_leaves = 2000, min_child_samples = 20)\n",
    "rf.fit(x_adv,y_train)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: develop adversarial sample detection model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = np.concatenate([x_adv,X_train])\n",
    "y1 = pd.Series(np.ones(y_train.shape[0]))\n",
    "y2 = pd.Series(np.zeros(y_train.shape[0]))\n",
    "y_new = np.concatenate([y1,y2])\n",
    "\n",
    "\n",
    "X_train_d, X_test_d, y_train_d, y_test_d = train_test_split(X_new,y_new, train_size = 0.1, test_size = 0.9,random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.407%\n",
      "Precision: 99.667%\n",
      "Recall: 99.144%\n",
      "F1-score: 99.405%\n",
      "Time: 8.21566\n",
      "Wall time: 2.44 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = lgb.LGBMClassifier(max_depth=22, learning_rate= 0.06157312210088248, n_estimators = 360, \n",
    "                         num_leaves = 2000, min_child_samples = 20)\n",
    "rf.fit(X_train_d,y_train_d)\n",
    "t1=time.time()\n",
    "predictions2 = rf.predict(X_test_d)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test_d)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: remove adversarial samples from the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "detection_results = rf.predict(X_new)\n",
    "X_new1 = X_new\n",
    "y_new1 = np.concatenate([y_train,y_train])\n",
    "\n",
    "indices_to_remove = [i for i in range(len(detection_results)) if detection_results[i] == 1]\n",
    "for i in reversed(indices_to_remove):\n",
    "    X_new1 = np.delete(X_new1, i, axis=0)\n",
    "    y_new1 = np.delete(y_new1, i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: re-train IDS model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.77000000000001%\n",
      "Precision: 99.291%\n",
      "Recall: 99.556%\n",
      "F1-score: 99.423%\n",
      "Time: 13.91828\n",
      "Wall time: 6.78 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#FGM\n",
    "rf = lgb.LGBMClassifier(max_depth=22, learning_rate= 0.06157312210088248, n_estimators = 360, \n",
    "                         num_leaves = 2000, min_child_samples = 20)\n",
    "rf.fit(X_new1,y_new1)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BasicIterativeMethod"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Original ML model for IDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.823%\n",
      "Precision: 99.468%\n",
      "Recall: 99.644%\n",
      "F1-score: 99.556%\n",
      "Wall time: 8.45 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.LGBMClassifier(max_depth=22, learning_rate= 0.06157312210088248, n_estimators = 360, \n",
    "                         num_leaves = 2000, min_child_samples = 20)\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: generate adversarial samples using FastGradientMethod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Input,Dense,Dropout,BatchNormalization,Activation\n",
    "from art.estimators.classification import KerasClassifier\n",
    "from keras import Model\n",
    "import keras.backend as K\n",
    "import keras.callbacks as kcallbacks\n",
    "from keras import optimizers\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "from art.attacks import evasion\n",
    "from art.attacks.evasion import FastGradientMethod\n",
    "from art.estimators.classification import SklearnClassifier\n",
    "\n",
    "def ANN(optimizer = 'sgd',neurons=16,batch_size=1024,epochs=80,activation='relu',patience=8,loss='binary_crossentropy'):\n",
    "    inputs=Input(shape=(X.shape[1],))\n",
    "    x=Dense(1000)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.3)(x)\n",
    "    x=Dense(256)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.25)(x)\n",
    "    x=Dense(2,activation='softmax')(x)\n",
    "    model=Model(inputs=inputs,outputs=x,name='base_nlp')\n",
    "    model.compile(optimizer='adam',loss='categorical_crossentropy')\n",
    "#     model.compile(optimizer=Adam(lr = 0.01),loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "    early_stopping = EarlyStopping(monitor=\"loss\", patience = patience)# early stop patience\n",
    "    history = model.fit(X, pd.get_dummies(y).values,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              callbacks = [early_stopping],\n",
    "              verbose=0) #verbose set to 1 will show the training process\n",
    "    return model\n",
    "\n",
    "# Create a KerasClassifier\n",
    "classifier = KerasClassifier(model=ANN(), clip_values=(0, 1))\n",
    "\n",
    "# Create the FastGradientMethod attack\n",
    "attack = evasion.BasicIterativeMethod(estimator=classifier, eps=0.1, eps_step=0.1, max_iter=200, batch_size=32, verbose = False)\n",
    "\n",
    "# Generate adversarial examples\n",
    "x_adv = attack.generate(X_train.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: test ML model under attack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 88.324%\n",
      "Precision: 96.586%\n",
      "Recall: 42.756%\n",
      "F1-score: 59.272999999999996%\n",
      "Time: 16.47628\n",
      "Wall time: 16.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#BIM\n",
    "rf = lgb.LGBMClassifier(max_depth=22, learning_rate= 0.06157312210088248, n_estimators = 360, \n",
    "                         num_leaves = 2000, min_child_samples = 20)\n",
    "rf.fit(x_adv,y_train)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: develop adversarial sample detection model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = np.concatenate([x_adv,X_train])\n",
    "y1 = pd.Series(np.ones(y_train.shape[0]))\n",
    "y2 = pd.Series(np.zeros(y_train.shape[0]))\n",
    "y_new = np.concatenate([y1,y2])\n",
    "\n",
    "\n",
    "X_train_d, X_test_d, y_train_d, y_test_d = train_test_split(X_new,y_new, train_size = 0.1, test_size = 0.9,random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.453%\n",
      "Precision: 99.673%\n",
      "Recall: 99.22999999999999%\n",
      "F1-score: 99.451%\n",
      "Time: 9.85973\n",
      "Wall time: 2.88 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = lgb.LGBMClassifier(max_depth=22, learning_rate= 0.06157312210088248, n_estimators = 360, \n",
    "                         num_leaves = 2000, min_child_samples = 20)\n",
    "rf.fit(X_train_d,y_train_d)\n",
    "t1=time.time()\n",
    "predictions2 = rf.predict(X_test_d)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test_d)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: remove adversarial samples from the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "detection_results = rf.predict(X_new)\n",
    "X_new1 = X_new\n",
    "y_new1 = np.concatenate([y_train,y_train])\n",
    "\n",
    "indices_to_remove = [i for i in range(len(detection_results)) if detection_results[i] == 1]\n",
    "for i in reversed(indices_to_remove):\n",
    "    X_new1 = np.delete(X_new1, i, axis=0)\n",
    "    y_new1 = np.delete(y_new1, i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: re-train IDS model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.806%\n",
      "Precision: 99.37899999999999%\n",
      "Recall: 99.644%\n",
      "F1-score: 99.512%\n",
      "Time: 16.12124\n",
      "Wall time: 8.05 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#BIM\n",
    "rf = lgb.LGBMClassifier(max_depth=22, learning_rate= 0.06157312210088248, n_estimators = 360, \n",
    "                         num_leaves = 2000, min_child_samples = 20)\n",
    "rf.fit(X_new1,y_new1)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}