L11_pipelines_and_gridsearch.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kX-ucI4cF-KI"
      },
      "source": [
        "# Intro Model Selection\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sfNMI8zNItX7"
      },
      "source": [
        "## Scikit-learn pipelines\n",
        "- Scikit-learn pipelines are an extremely convenient and powerful concept -- one of the things that sets scikit-learn apart from other machine learning libraries.\n",
        "- Pipelines basically let us define a series of perprocessing steps together with fitting an estimator.\n",
        "- Pipelines will automatically take care of pitfalls like estimating feature scaling parameters from the training set and applying those to scale new data (which we discussed earlier in the context of z-score standardization).\n",
        "- Below is an visualization of how pipelines work.\n",
        "\n",
        "<img src=\"https://github.com/rasbt/stat451-machine-learning-fs20/raw/ee813e1c30a5610a2e6475a77c67c1174a63b75c/L05/code/images/sklearn-pipeline.png\" width=\"400\">\n",
        "\n",
        "Below is an example pipeline that combines the feature scaling step, PCA and the kNN classifier."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eK85714z7pDB"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "from sklearn import datasets\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.model_selection import train_test_split"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0_L1E_lI8kqS"
      },
      "source": [
        "iris = datasets.load_iris()\n",
        "\n",
        "X = iris.data\n",
        "y = iris.target"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ILzUen7FAlua"
      },
      "source": [
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=True, random_state=1, stratify=y)"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_hboroBDGKCM"
      },
      "source": [
        "pipe = Pipeline([\n",
        "        ('z-score', StandardScaler()),\n",
        "        ('reduce_dim', PCA(n_components=3)),\n",
        "        ('classify', KNeighborsClassifier(n_neighbors=1))])"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RsH66Pb1GMJ_",
        "outputId": "8f37b2dd-ac7f-4fc2-bcba-62392389e0ba",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 284
        }
      },
      "source": [
        "pipe.fit(X_train, y_train)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pipeline(steps=[('z-score', StandardScaler()),\n",
              "                ('reduce_dim', PCA(n_components=3)),\n",
              "                ('classify', KNeighborsClassifier(n_neighbors=1))])"
            ],
            "text/html": [
              "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;z-score&#x27;, StandardScaler()),\n",
              "                (&#x27;reduce_dim&#x27;, PCA(n_components=3)),\n",
              "                (&#x27;classify&#x27;, KNeighborsClassifier(n_neighbors=1))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;z-score&#x27;, StandardScaler()),\n",
              "                (&#x27;reduce_dim&#x27;, PCA(n_components=3)),\n",
              "                (&#x27;classify&#x27;, KNeighborsClassifier(n_neighbors=1))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">PCA</label><div class=\"sk-toggleable__content\"><pre>PCA(n_components=3)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KNeighborsClassifier</label><div class=\"sk-toggleable__content\"><pre>KNeighborsClassifier(n_neighbors=1)</pre></div></div></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t5uVQBY_XjGr",
        "outputId": "961bddb5-5eeb-498e-b4e1-eb10a6fc8223",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from sklearn.metrics import accuracy_score\n",
        "\n",
        "y_train_pred = pipe.predict(X_train)\n",
        "accuracy_score(y_train, y_train_pred)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1.0"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(y_train)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OCOc6dZgDzm9",
        "outputId": "0c16a42a-474d-4472-9ca8-3a5ea75c0b54"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[0 0 2 1 1 1 1 2 2 1 2 0 1 1 2 0 2 1 1 0 0 2 2 1 2 2 0 0 2 0 0 0 0 2 0 1 2\n",
            " 1 2 2 1 1 2 2 1 1 2 1 1 0 2 0 2 0 0 1 0 1 2 0 2 0 0 2 2 0 1 0 0 1 2 1 1 0\n",
            " 0 1 2 1 1 2 1 2 0 1 2 1 1 0 0 2 1 2 1 0 1 2 2 0 0 2 2 0 2 1 1 1 2 0 2 1 2\n",
            " 0 2 0 1 0 1 0 0 0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(y_train_pred)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IaVK2QecD2zH",
        "outputId": "188d2941-03fa-4337-ce45-0e437d3e0b27"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[0 0 2 1 1 1 1 2 2 1 2 0 1 1 2 0 2 1 1 0 0 2 2 1 2 2 0 0 2 0 0 0 0 2 0 1 2\n",
            " 1 2 2 1 1 2 2 1 1 2 1 1 0 2 0 2 0 0 1 0 1 2 0 2 0 0 2 2 0 1 0 0 1 2 1 1 0\n",
            " 0 1 2 1 1 2 1 2 0 1 2 1 1 0 0 2 1 2 1 0 1 2 2 0 0 2 2 0 2 1 1 1 2 0 2 1 2\n",
            " 0 2 0 1 0 1 0 0 0]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8IqvCg8cGQvH",
        "outputId": "d1011974-4994-4b64-b7c3-dfa3f561159a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from sklearn.metrics import accuracy_score\n",
        "\n",
        "y_test_pred = pipe.predict(X_test)\n",
        "accuracy_score(y_test, y_test_pred)"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9333333333333332"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(y_test)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4IUYtNBpFIxT",
        "outputId": "4ed94b1a-ae76-4103-de3d-fc6b6030157c"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[2 0 1 0 0 0 2 2 2 1 0 1 2 1 2 0 2 1 1 2 1 1 0 0 2 2 0 0 1 1]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(y_test_pred)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kZ9jUnMbFKSs",
        "outputId": "6693541f-99c1-4859-b905-c19383839200"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[2 0 1 0 0 0 2 2 2 1 0 1 2 1 2 0 2 1 1 2 1 1 0 0 2 1 0 0 1 2]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jCjd3_-CH3gT"
      },
      "source": [
        "As you can see above, the Pipeline itself follows the scikit-learn estimator API."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bUyQQAo1FqUI"
      },
      "source": [
        "## Scikit-learn grid-search\n",
        "\n",
        "- In machine learning practice, we often need to experiment with an machine learning algorithm's hyperparameters to find a good setting.\n",
        "- The process of tuning hyperparameters and comparing and selecting the resulting models is also called \"model selection\" (in contrast to \"algorithm selection\").\n",
        "- We will cover topics such as \"model selection\" and \"algorithm selection\" in more detail later in this course.\n",
        "- For now, we are introducing the simplest way of performing model selection: using the \"holdout method.\"\n",
        "- In the holdout method, we split a dataset into 3 subsets: a training, a validation, and a test datatset.\n",
        "- To avoid biasing the estimate of the generalization performance, we only want to use the test dataset once, which is why we use the validation dataset for hyperparameter tuning (model selection).\n",
        "- Here, the validation dataset serves as an estimate of the generalization performance, too, but it becomes more biased than the final estimate on the test data because of its repeated re-use during model selection (think of \"multiple hypothesis testing\").\n",
        "\n",
        "<img src=\"https://github.com/rasbt/stat451-machine-learning-fs20/raw/ee813e1c30a5610a2e6475a77c67c1174a63b75c/L05/code/images/holdout-tuning.png\" width=\"400\">"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KDXrsUgy8jhA"
      },
      "source": [
        "param_grid = {\n",
        "    'reduce_dim__n_components': [1, 2, 3, 4],\n",
        "    'classify__n_neighbors': [2, 3, 4, 5]\n",
        "}\n",
        "\n",
        "grid = GridSearchCV(pipe, cv=2, n_jobs=1, param_grid=param_grid, scoring='accuracy')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9hZnMqMGAO6A",
        "outputId": "64899b74-ea6b-4e9a-c2ce-9e0512d1323b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "grid.fit(X_train, y_train)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(cv=2, error_score=nan,\n",
              "             estimator=Pipeline(memory=None,\n",
              "                                steps=[('z-score',\n",
              "                                        StandardScaler(copy=True,\n",
              "                                                       with_mean=True,\n",
              "                                                       with_std=True)),\n",
              "                                       ('reduce_dim',\n",
              "                                        PCA(copy=True, iterated_power='auto',\n",
              "                                            n_components=3, random_state=None,\n",
              "                                            svd_solver='auto', tol=0.0,\n",
              "                                            whiten=False)),\n",
              "                                       ('classify',\n",
              "                                        KNeighborsClassifier(algorithm='auto',\n",
              "                                                             leaf_size=30,\n",
              "                                                             metric='minkowski',\n",
              "                                                             metric_params=None,\n",
              "                                                             n_jobs=None,\n",
              "                                                             n_neighbors=1, p=2,\n",
              "                                                             weights='uniform'))],\n",
              "                                verbose=False),\n",
              "             iid='deprecated', n_jobs=1,\n",
              "             param_grid={'classify__n_neighbors': [2, 3, 4, 5],\n",
              "                         'reduce_dim__n_components': [1, 2, 3, 4]},\n",
              "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
              "             scoring='accuracy', verbose=0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 205
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hQqyJKNS-7RL",
        "outputId": "ea9f62b6-6159-478e-befb-d55af57d81a4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(grid.cv_results_)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'mean_fit_time': array([0.00215685, 0.00161123, 0.00155306, 0.00156486, 0.00198913,\n",
            "       0.00156486, 0.00145936, 0.00168526, 0.00168669, 0.00142121,\n",
            "       0.00150418, 0.00141299, 0.00137055, 0.00140047, 0.0014323 ,\n",
            "       0.00141287]), 'std_fit_time': array([5.41329384e-04, 6.43730164e-05, 2.38418579e-06, 9.26256180e-05,\n",
            "       4.93526459e-04, 1.56044960e-04, 5.36441803e-05, 2.14219093e-04,\n",
            "       2.21371651e-04, 2.02655792e-05, 2.62260437e-05, 2.41994858e-05,\n",
            "       2.70605087e-05, 1.47819519e-05, 5.37633896e-05, 4.17232513e-05]), 'mean_score_time': array([0.00443637, 0.0036118 , 0.00266862, 0.00273967, 0.00266659,\n",
            "       0.0026015 , 0.00256932, 0.0033294 , 0.00295794, 0.00275695,\n",
            "       0.00273621, 0.00272202, 0.00253725, 0.00274074, 0.00270486,\n",
            "       0.00268829]), 'std_score_time': array([7.32779503e-04, 9.33170319e-04, 4.33921814e-05, 1.34229660e-04,\n",
            "       3.27825546e-05, 1.37090683e-05, 2.02655792e-06, 4.45008278e-04,\n",
            "       3.64661217e-04, 1.69634819e-04, 1.84774399e-05, 9.77516174e-06,\n",
            "       5.43594360e-05, 7.90357590e-05, 5.69820404e-05, 1.19209290e-07]), 'param_classify__n_neighbors': masked_array(data=[2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5],\n",
            "             mask=[False, False, False, False, False, False, False, False,\n",
            "                   False, False, False, False, False, False, False, False],\n",
            "       fill_value='?',\n",
            "            dtype=object), 'param_reduce_dim__n_components': masked_array(data=[1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4],\n",
            "             mask=[False, False, False, False, False, False, False, False,\n",
            "                   False, False, False, False, False, False, False, False],\n",
            "       fill_value='?',\n",
            "            dtype=object), 'params': [{'classify__n_neighbors': 2, 'reduce_dim__n_components': 1}, {'classify__n_neighbors': 2, 'reduce_dim__n_components': 2}, {'classify__n_neighbors': 2, 'reduce_dim__n_components': 3}, {'classify__n_neighbors': 2, 'reduce_dim__n_components': 4}, {'classify__n_neighbors': 3, 'reduce_dim__n_components': 1}, {'classify__n_neighbors': 3, 'reduce_dim__n_components': 2}, {'classify__n_neighbors': 3, 'reduce_dim__n_components': 3}, {'classify__n_neighbors': 3, 'reduce_dim__n_components': 4}, {'classify__n_neighbors': 4, 'reduce_dim__n_components': 1}, {'classify__n_neighbors': 4, 'reduce_dim__n_components': 2}, {'classify__n_neighbors': 4, 'reduce_dim__n_components': 3}, {'classify__n_neighbors': 4, 'reduce_dim__n_components': 4}, {'classify__n_neighbors': 5, 'reduce_dim__n_components': 1}, {'classify__n_neighbors': 5, 'reduce_dim__n_components': 2}, {'classify__n_neighbors': 5, 'reduce_dim__n_components': 3}, {'classify__n_neighbors': 5, 'reduce_dim__n_components': 4}], 'split0_test_score': array([0.83333333, 0.91666667, 0.91666667, 0.91666667, 0.93333333,\n",
            "       0.93333333, 0.95      , 0.93333333, 0.9       , 0.93333333,\n",
            "       0.93333333, 0.93333333, 0.93333333, 0.93333333, 0.93333333,\n",
            "       0.93333333]), 'split1_test_score': array([0.91666667, 0.91666667, 0.91666667, 0.9       , 0.88333333,\n",
            "       0.93333333, 0.95      , 0.95      , 0.9       , 0.91666667,\n",
            "       0.95      , 0.95      , 0.88333333, 0.91666667, 0.96666667,\n",
            "       0.93333333]), 'mean_test_score': array([0.875     , 0.91666667, 0.91666667, 0.90833333, 0.90833333,\n",
            "       0.93333333, 0.95      , 0.94166667, 0.9       , 0.925     ,\n",
            "       0.94166667, 0.94166667, 0.90833333, 0.925     , 0.95      ,\n",
            "       0.93333333]), 'std_test_score': array([0.04166667, 0.        , 0.        , 0.00833333, 0.025     ,\n",
            "       0.        , 0.        , 0.00833333, 0.        , 0.00833333,\n",
            "       0.00833333, 0.00833333, 0.025     , 0.00833333, 0.01666667,\n",
            "       0.        ]), 'rank_test_score': array([16, 10, 10, 12, 12,  6,  1,  3, 15,  8,  3,  3, 12,  8,  1,  6],\n",
            "      dtype=int32)}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hMf0AnegNmjK",
        "outputId": "a817e91c-b921-438d-8b6b-c53e0f4c2ab9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "grid.cv_results_['mean_test_score']"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.875     , 0.91666667, 0.91666667, 0.90833333, 0.90833333,\n",
              "       0.93333333, 0.95      , 0.94166667, 0.9       , 0.925     ,\n",
              "       0.94166667, 0.94166667, 0.90833333, 0.925     , 0.95      ,\n",
              "       0.93333333])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 207
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VeB5Dem9_QCi",
        "outputId": "fce02cbe-9119-4fc1-84b1-ba4b401b6407",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(grid.best_score_)\n",
        "print(grid.best_params_)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.95\n",
            "{'classify__n_neighbors': 3, 'reduce_dim__n_components': 3}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w7g_5QwUEJ6q"
      },
      "source": [
        "clf = grid.best_estimator_"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gFnX1XlhC8KM",
        "outputId": "2b34c8eb-b1fa-4e84-f415-3f5bab15b26a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "y_test_pred = clf.predict(X_test)\n",
        "accuracy_score(y_test, y_test_pred)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9666666666666667"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 209
        }
      ]
    }
  ]
}