Clustering_Consulting_Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering Consulting Project \n",
    "\n",
    "A large technology firm needs your help, they've been hacked! Luckily their forensic engineers have grabbed valuable data about the hacks, including information like session time,locations, wpm typing speed, etc. The forensic engineer relates to you what she has been able to figure out so far, she has been able to grab meta data of each session that the hackers used to connect to their servers. These are the features of the data:\n",
    "\n",
    "* 'Session_Connection_Time': How long the session lasted in minutes\n",
    "* 'Bytes Transferred': Number of MB transferred during session\n",
    "* 'Kali_Trace_Used': Indicates if the hacker was using Kali Linux\n",
    "* 'Servers_Corrupted': Number of server corrupted during the attack\n",
    "* 'Pages_Corrupted': Number of pages illegally accessed\n",
    "* 'Location': Location attack came from (Probably useless because the hackers used VPNs)\n",
    "* 'WPM_Typing_Speed': Their estimated typing speed based on session logs.\n",
    "\n",
    "\n",
    "The technology firm has 3 potential hackers that perpetrated the attack. Their certain of the first two hackers but they aren't very sure if the third hacker was involved or not. They have requested your help! Can you help figure out whether or not the third suspect had anything to do with the attacks, or was it just two hackers? It's probably not possible to know for sure, but maybe what you've just learned about Clustering can help!\n",
    "\n",
    "**One last key fact, the forensic engineer knows that the hackers trade off attacks. Meaning they should each have roughly the same amount of attacks. For example if there were 100 total attacks, then in a 2 hacker situation each should have about 50 hacks, in a three hacker situation each would have about 33 hacks. The engineer believes this is the key element to solving this, but doesn't know how to distinguish this unlabeled data into groups of hackers.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "from pyspark.ml.clustering import KMeans\n",
    "from pyspark.ml.feature import VectorAssembler, StandardScaler, StringIndexer\n",
    "from pyspark.ml.evaluation import ClusteringEvaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <div>\n",
       "                <p><b>SparkSession - in-memory</b></p>\n",
       "                \n",
       "        <div>\n",
       "            <p><b>SparkContext</b></p>\n",
       "\n",
       "            <p><a href=\"http://DESKTOP-BJS1Q67:4043\">Spark UI</a></p>\n",
       "\n",
       "            <dl>\n",
       "              <dt>Version</dt>\n",
       "                <dd><code>v3.5.0</code></dd>\n",
       "              <dt>Master</dt>\n",
       "                <dd><code>local[*]</code></dd>\n",
       "              <dt>AppName</dt>\n",
       "                <dd><code>cluster_consult</code></dd>\n",
       "            </dl>\n",
       "        </div>\n",
       "        \n",
       "            </div>\n",
       "        "
      ],
      "text/plain": [
       "<pyspark.sql.session.SparkSession at 0x289cdc925d0>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spark = SparkSession.builder.appName('cluster_consult').getOrCreate()\n",
    "spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Session_Connection_Time: double (nullable = true)\n",
      " |-- Bytes Transferred: double (nullable = true)\n",
      " |-- Kali_Trace_Used: integer (nullable = true)\n",
      " |-- Servers_Corrupted: double (nullable = true)\n",
      " |-- Pages_Corrupted: double (nullable = true)\n",
      " |-- Location: string (nullable = true)\n",
      " |-- WPM_Typing_Speed: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df = spark.read.csv('hack_data.csv',inferSchema=True,header=True)\n",
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "334"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------------+-----------------+---------------+-----------------+---------------+--------------------+----------------+\n",
      "|Session_Connection_Time|Bytes Transferred|Kali_Trace_Used|Servers_Corrupted|Pages_Corrupted|            Location|WPM_Typing_Speed|\n",
      "+-----------------------+-----------------+---------------+-----------------+---------------+--------------------+----------------+\n",
      "|                    8.0|           391.09|              1|             2.96|            7.0|            Slovenia|           72.37|\n",
      "|                   20.0|           720.99|              0|             3.04|            9.0|British Virgin Is...|           69.08|\n",
      "|                   31.0|           356.32|              1|             3.71|            8.0|             Tokelau|           70.58|\n",
      "|                    2.0|           228.08|              1|             2.48|            8.0|             Bolivia|            70.8|\n",
      "|                   20.0|            408.5|              0|             3.57|            8.0|                Iraq|           71.28|\n",
      "+-----------------------+-----------------+---------------+-----------------+---------------+--------------------+----------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-----------------------+------------------+------------------+-----------------+------------------+-----------+------------------+\n",
      "|summary|Session_Connection_Time| Bytes Transferred|   Kali_Trace_Used|Servers_Corrupted|   Pages_Corrupted|   Location|  WPM_Typing_Speed|\n",
      "+-------+-----------------------+------------------+------------------+-----------------+------------------+-----------+------------------+\n",
      "|  count|                    334|               334|               334|              334|               334|        334|               334|\n",
      "|   mean|     30.008982035928145| 607.2452694610777|0.5119760479041916|5.258502994011977|10.838323353293413|       NULL|57.342395209580864|\n",
      "| stddev|     14.088200614636158|286.33593163576757|0.5006065264451406| 2.30190693339697|  3.06352633036022|       NULL| 13.41106336843464|\n",
      "|    min|                    1.0|              10.0|                 0|              1.0|               6.0|Afghanistan|              40.0|\n",
      "|    max|                   60.0|            1330.5|                 1|             10.0|              15.0|   Zimbabwe|              75.0|\n",
      "+-------+-----------------------+------------------+------------------+-----------------+------------------+-----------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Session_Connection_Time',\n",
       " 'Bytes Transferred',\n",
       " 'Kali_Trace_Used',\n",
       " 'Servers_Corrupted',\n",
       " 'Pages_Corrupted',\n",
       " 'Location',\n",
       " 'WPM_Typing_Speed']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# indexer = StringIndexer(inputCol='Location',outputCol='LocationIndex')\n",
    "# indexer_model = indexer.fit(df)\n",
    "# indexer_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# output = indexer_model.transform(df)\n",
    "# output.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "VectorAssembler_44374b740baa"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assembler = VectorAssembler(inputCols=['Session_Connection_Time',\n",
    " 'Bytes Transferred',\n",
    " 'Kali_Trace_Used',\n",
    " 'Servers_Corrupted',\n",
    " 'Pages_Corrupted',\n",
    " 'WPM_Typing_Speed'],outputCol='features')\n",
    "assembler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------------+-----------------+---------------+-----------------+---------------+--------------------+----------------+--------------------+\n",
      "|Session_Connection_Time|Bytes Transferred|Kali_Trace_Used|Servers_Corrupted|Pages_Corrupted|            Location|WPM_Typing_Speed|            features|\n",
      "+-----------------------+-----------------+---------------+-----------------+---------------+--------------------+----------------+--------------------+\n",
      "|                    8.0|           391.09|              1|             2.96|            7.0|            Slovenia|           72.37|[8.0,391.09,1.0,2...|\n",
      "|                   20.0|           720.99|              0|             3.04|            9.0|British Virgin Is...|           69.08|[20.0,720.99,0.0,...|\n",
      "|                   31.0|           356.32|              1|             3.71|            8.0|             Tokelau|           70.58|[31.0,356.32,1.0,...|\n",
      "|                    2.0|           228.08|              1|             2.48|            8.0|             Bolivia|            70.8|[2.0,228.08,1.0,2...|\n",
      "|                   20.0|            408.5|              0|             3.57|            8.0|                Iraq|           71.28|[20.0,408.5,0.0,3...|\n",
      "|                    1.0|           390.69|              1|             2.79|            9.0|    Marshall Islands|           71.57|[1.0,390.69,1.0,2...|\n",
      "|                   18.0|           342.97|              1|              5.1|            7.0|             Georgia|           72.32|[18.0,342.97,1.0,...|\n",
      "|                   22.0|           101.61|              1|             3.03|            7.0|         Timor-Leste|           72.03|[22.0,101.61,1.0,...|\n",
      "|                   15.0|           275.53|              1|             3.53|            8.0|Palestinian Terri...|           70.17|[15.0,275.53,1.0,...|\n",
      "|                   12.0|           424.83|              1|             2.53|            8.0|          Bangladesh|           69.99|[12.0,424.83,1.0,...|\n",
      "|                   15.0|           249.09|              1|             3.39|            9.0|Northern Mariana ...|           70.77|[15.0,249.09,1.0,...|\n",
      "|                   32.0|           242.48|              0|             4.24|            8.0|            Zimbabwe|           67.93|[32.0,242.48,0.0,...|\n",
      "|                   23.0|           514.54|              0|             3.18|            8.0|         Isle of Man|           68.56|[23.0,514.54,0.0,...|\n",
      "|                    9.0|           284.77|              0|             3.12|            9.0|Sao Tome and Prin...|           70.82|[9.0,284.77,0.0,3...|\n",
      "|                   27.0|           779.25|              1|             2.37|            8.0|              Greece|           72.73|[27.0,779.25,1.0,...|\n",
      "|                   12.0|           307.31|              1|             3.22|            7.0|     Solomon Islands|           67.95|[12.0,307.31,1.0,...|\n",
      "|                   21.0|           355.94|              1|              2.0|            7.0|       Guinea-Bissau|            72.0|[21.0,355.94,1.0,...|\n",
      "|                   10.0|           372.65|              0|             3.33|            7.0|        Burkina Faso|           69.19|[10.0,372.65,0.0,...|\n",
      "|                   20.0|           347.23|              1|             2.33|            7.0|            Mongolia|           70.41|[20.0,347.23,1.0,...|\n",
      "|                   22.0|           456.57|              0|             1.52|            8.0|             Nigeria|           69.35|[22.0,456.57,0.0,...|\n",
      "+-----------------------+-----------------+---------------+-----------------+---------------+--------------------+----------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "final_data = assembler.transform(df)\n",
    "final_data.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Session_Connection_Time: double (nullable = true)\n",
      " |-- Bytes Transferred: double (nullable = true)\n",
      " |-- Kali_Trace_Used: integer (nullable = true)\n",
      " |-- Servers_Corrupted: double (nullable = true)\n",
      " |-- Pages_Corrupted: double (nullable = true)\n",
      " |-- Location: string (nullable = true)\n",
      " |-- WPM_Typing_Speed: double (nullable = true)\n",
      " |-- features: vector (nullable = true)\n",
      " |-- scaled_features: vector (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "scaler = StandardScaler(inputCol='features',outputCol='scaled_features')\n",
    "scaler_model = scaler.fit(final_data)\n",
    "final_data = scaler_model.transform(final_data)\n",
    "final_data.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first check for a KMeans model with 3 clusters to see if there were 3 hackers involved in the attack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KMeansModel: uid=KMeans_5cc963663eda, k=3, distanceMeasure=euclidean, numFeatures=6"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kmeans = KMeans().setFeaturesCol('scaled_features').setK(3).setSeed(48)\n",
    "kmeans_model = kmeans.fit(final_data)\n",
    "kmeans_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([1.21780112, 1.37901802, 1.99757683, 1.37198977, 2.55237797,\n",
       "        5.29152222]),\n",
       " array([2.99991988, 2.92319035, 1.05261534, 3.20390443, 4.51321315,\n",
       "        3.28474   ]),\n",
       " array([1.30217042, 1.25830099, 0.        , 1.35793211, 2.57251009,\n",
       "        5.24230473])]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster_centers = kmeans_model.clusterCenters()\n",
    "cluster_centers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "434.1492898715821"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sse = kmeans_model.summary.trainingCost\n",
    "sse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------------+-----------------+---------------+-----------------+---------------+--------------------+----------------+--------------------+--------------------+----------+\n",
      "|Session_Connection_Time|Bytes Transferred|Kali_Trace_Used|Servers_Corrupted|Pages_Corrupted|            Location|WPM_Typing_Speed|            features|     scaled_features|prediction|\n",
      "+-----------------------+-----------------+---------------+-----------------+---------------+--------------------+----------------+--------------------+--------------------+----------+\n",
      "|                    8.0|           391.09|              1|             2.96|            7.0|            Slovenia|           72.37|[8.0,391.09,1.0,2...|[0.56785108466505...|         0|\n",
      "|                   20.0|           720.99|              0|             3.04|            9.0|British Virgin Is...|           69.08|[20.0,720.99,0.0,...|[1.41962771166263...|         2|\n",
      "|                   31.0|           356.32|              1|             3.71|            8.0|             Tokelau|           70.58|[31.0,356.32,1.0,...|[2.20042295307707...|         0|\n",
      "|                    2.0|           228.08|              1|             2.48|            8.0|             Bolivia|            70.8|[2.0,228.08,1.0,2...|[0.14196277116626...|         0|\n",
      "|                   20.0|            408.5|              0|             3.57|            8.0|                Iraq|           71.28|[20.0,408.5,0.0,3...|[1.41962771166263...|         2|\n",
      "|                    1.0|           390.69|              1|             2.79|            9.0|    Marshall Islands|           71.57|[1.0,390.69,1.0,2...|[0.07098138558313...|         0|\n",
      "|                   18.0|           342.97|              1|              5.1|            7.0|             Georgia|           72.32|[18.0,342.97,1.0,...|[1.27766494049636...|         0|\n",
      "|                   22.0|           101.61|              1|             3.03|            7.0|         Timor-Leste|           72.03|[22.0,101.61,1.0,...|[1.56159048282889...|         0|\n",
      "|                   15.0|           275.53|              1|             3.53|            8.0|Palestinian Terri...|           70.17|[15.0,275.53,1.0,...|[1.06472078374697...|         0|\n",
      "|                   12.0|           424.83|              1|             2.53|            8.0|          Bangladesh|           69.99|[12.0,424.83,1.0,...|[0.85177662699757...|         0|\n",
      "|                   15.0|           249.09|              1|             3.39|            9.0|Northern Mariana ...|           70.77|[15.0,249.09,1.0,...|[1.06472078374697...|         0|\n",
      "|                   32.0|           242.48|              0|             4.24|            8.0|            Zimbabwe|           67.93|[32.0,242.48,0.0,...|[2.27140433866020...|         2|\n",
      "|                   23.0|           514.54|              0|             3.18|            8.0|         Isle of Man|           68.56|[23.0,514.54,0.0,...|[1.63257186841202...|         2|\n",
      "|                    9.0|           284.77|              0|             3.12|            9.0|Sao Tome and Prin...|           70.82|[9.0,284.77,0.0,3...|[0.63883247024818...|         2|\n",
      "|                   27.0|           779.25|              1|             2.37|            8.0|              Greece|           72.73|[27.0,779.25,1.0,...|[1.91649741074455...|         0|\n",
      "|                   12.0|           307.31|              1|             3.22|            7.0|     Solomon Islands|           67.95|[12.0,307.31,1.0,...|[0.85177662699757...|         0|\n",
      "|                   21.0|           355.94|              1|              2.0|            7.0|       Guinea-Bissau|            72.0|[21.0,355.94,1.0,...|[1.49060909724576...|         0|\n",
      "|                   10.0|           372.65|              0|             3.33|            7.0|        Burkina Faso|           69.19|[10.0,372.65,0.0,...|[0.70981385583131...|         2|\n",
      "|                   20.0|           347.23|              1|             2.33|            7.0|            Mongolia|           70.41|[20.0,347.23,1.0,...|[1.41962771166263...|         0|\n",
      "|                   22.0|           456.57|              0|             1.52|            8.0|             Nigeria|           69.35|[22.0,456.57,0.0,...|[1.56159048282889...|         2|\n",
      "+-----------------------+-----------------+---------------+-----------------+---------------+--------------------+----------------+--------------------+--------------------+----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kmeans_model.summary.predictions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Session_Connection_Time: double (nullable = true)\n",
      " |-- Bytes Transferred: double (nullable = true)\n",
      " |-- Kali_Trace_Used: integer (nullable = true)\n",
      " |-- Servers_Corrupted: double (nullable = true)\n",
      " |-- Pages_Corrupted: double (nullable = true)\n",
      " |-- Location: string (nullable = true)\n",
      " |-- WPM_Typing_Speed: double (nullable = true)\n",
      " |-- features: vector (nullable = true)\n",
      " |-- scaled_features: vector (nullable = true)\n",
      " |-- prediction: integer (nullable = false)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "results = kmeans_model.transform(final_data)\n",
    "results.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+\n",
      "|prediction|\n",
      "+----------+\n",
      "|         0|\n",
      "|         2|\n",
      "|         0|\n",
      "|         0|\n",
      "|         2|\n",
      "+----------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "results.select('prediction').show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(prediction=1), Row(prediction=2), Row(prediction=0)]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.select('prediction').distinct().collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+-----+\n",
      "|prediction|count|\n",
      "+----------+-----+\n",
      "|         1|  167|\n",
      "|         2|   84|\n",
      "|         0|   83|\n",
      "+----------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "results.groupBy('prediction').count().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Silhouette Score of KMeans: 0.755239186655932\n"
     ]
    }
   ],
   "source": [
    "clusteval = ClusteringEvaluator(featuresCol='scaled_features',predictionCol='prediction')\n",
    "silhouette_score = clusteval.evaluate(results)\n",
    "print(\"Silhouette Score of KMeans:\",silhouette_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Session_Connection_Time: double (nullable = true)\n",
      " |-- Bytes Transferred: double (nullable = true)\n",
      " |-- Kali_Trace_Used: integer (nullable = true)\n",
      " |-- Servers_Corrupted: double (nullable = true)\n",
      " |-- Pages_Corrupted: double (nullable = true)\n",
      " |-- Location: string (nullable = true)\n",
      " |-- WPM_Typing_Speed: double (nullable = true)\n",
      " |-- features: vector (nullable = true)\n",
      " |-- scaled_features: vector (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "final_data = final_data.drop('prediction')\n",
    "final_data.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's evaluate a K Means model with 2 clusters to determine if there were only 2 hackers involved in the attack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KMeansModel: uid=KMeans_076f16744221, k=2, distanceMeasure=euclidean, numFeatures=6"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kmeans = KMeans(k=2,seed=48,featuresCol='scaled_features')\n",
    "kmeans_model = kmeans.fit(final_data)\n",
    "kmeans_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([2.99991988, 2.92319035, 1.05261534, 3.20390443, 4.51321315,\n",
       "        3.28474   ]),\n",
       " array([1.26023837, 1.31829808, 0.99280765, 1.36491885, 2.5625043 ,\n",
       "        5.26676612])]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster_centers = kmeans_model.clusterCenters()\n",
    "cluster_centers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "601.7707512676691"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sse = kmeans_model.summary.trainingCost\n",
    "sse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+----------+\n",
      "|     scaled_features|prediction|\n",
      "+--------------------+----------+\n",
      "|[0.56785108466505...|         1|\n",
      "|[1.41962771166263...|         1|\n",
      "|[2.20042295307707...|         1|\n",
      "|[0.14196277116626...|         1|\n",
      "|[1.41962771166263...|         1|\n",
      "|[0.07098138558313...|         1|\n",
      "|[1.27766494049636...|         1|\n",
      "|[1.56159048282889...|         1|\n",
      "|[1.06472078374697...|         1|\n",
      "|[0.85177662699757...|         1|\n",
      "|[1.06472078374697...|         1|\n",
      "|[2.27140433866020...|         1|\n",
      "|[1.63257186841202...|         1|\n",
      "|[0.63883247024818...|         1|\n",
      "|[1.91649741074455...|         1|\n",
      "|[0.85177662699757...|         1|\n",
      "|[1.49060909724576...|         1|\n",
      "|[0.70981385583131...|         1|\n",
      "|[1.41962771166263...|         1|\n",
      "|[1.56159048282889...|         1|\n",
      "+--------------------+----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kmeans_model.transform(final_data).select('scaled_features','prediction').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+\n",
      "|prediction|\n",
      "+----------+\n",
      "|         1|\n",
      "|         1|\n",
      "|         1|\n",
      "|         1|\n",
      "|         1|\n",
      "|         1|\n",
      "|         1|\n",
      "|         1|\n",
      "|         1|\n",
      "|         1|\n",
      "|         1|\n",
      "|         1|\n",
      "|         1|\n",
      "|         1|\n",
      "|         1|\n",
      "+----------+\n",
      "only showing top 15 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "results = kmeans_model.transform(final_data)\n",
    "results.sample(withReplacement=False,fraction=0.2,seed=48).select('prediction').show(15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(prediction=1), Row(prediction=0)]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.select('prediction').distinct().collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+-----+\n",
      "|prediction|count|\n",
      "+----------+-----+\n",
      "|         1|  167|\n",
      "|         0|  167|\n",
      "+----------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "results.groupBy('prediction').count().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Silhouette Score for a K Means model with 2 clusters: 0.817646009401248\n"
     ]
    }
   ],
   "source": [
    "print(\"Silhouette Score for a K Means model with 2 clusters:\", clusteval.evaluate(kmeans_model.transform(final_data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the results, we can conclude that there were only 2 hackers involved in the attack. There were 334 attacks in total out of which each of those hackers traded 167 attacks. This is proved by the higher value of Silhouette coefficient in the case of the K Means model with 2 clusters as compared to the one with 3 clusters."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}