8389_Text_Classification__models_final.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "8389_Text_Classification__models_final.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## Second Model - Naive Bayes classifier : Supervised \n"
      ],
      "metadata": {
        "id": "MruO8P_YGg3Z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**What is Naive Bayes?**\n",
        "\n",
        "In this portion of the project, we are building a predictive model.  Naive Bayes is a supervised machine learning model based off Bayes theorem. Bayes theorem is using the knowledge of other events to calculate the probability of a future event happening. Many people used Naive Bayes classifier with text data because it calculates the features individually. \n",
        "\n",
        "**How does the Naive Bayes classifier work?**\n",
        "\n",
        "The algorithm uses Bayes theorem to calculate the posterior probability. A posterior probability, in Bayesian statistics, is the revised or updated probability of an event occurring after taking into consideration new information. [4] We have certain amount of feature that we need to calculate the probability for.  The algorithm applies Bayes theorem to each feature and the chain rule is applied to calculate the individual probability. Then, the individual probability is combine to calculate the posterior probability.The main objective function in naive Bayes classifier is to maximize the posterior probability given the training data for each class. [1]"
      ],
      "metadata": {
        "id": "9_OvZcAWHNRI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 1. Import necessary packages "
      ],
      "metadata": {
        "id": "-oHgiRUzHP6n"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "yOtMEvpVW7O2"
      },
      "outputs": [],
      "source": [
        "import pandas as pd \n",
        "import matplotlib.pyplot as plt\n",
        "from string import punctuation\n",
        "import string\n",
        "import regex\n",
        "import io\n",
        "import nltk\n",
        "from nltk.corpus import stopwords, wordnet\n",
        "from nltk import pos_tag\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "from nltk.tokenize import word_tokenize,sent_tokenize\n",
        "from wordcloud import WordCloud\n",
        "from sklearn.preprocessing import OneHotEncoder\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.linear_model import LogisticRegression,SGDClassifier, LinearRegression\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from sklearn.svm import SVC\n",
        "from keras.layers import Dense, Conv1D, MaxPool1D, Flatten, Dropout\n",
        "from keras.models import Sequential\n",
        "from tensorflow.keras.utils import to_categorical"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        " ### 2. Import Data \n",
        " \n",
        "The dataframe used for the next to models was built in our Juptyer notebook with the code from the first half of our project. The dataset has been sorted in articles that contain licensing agreement language and news articles that do not. A label column has been created and number 0 for nonlicensing and 1 for licensing. We have taken a sample of 5000 articles instead of training on all 40,000. The remaining articles can be used for further model testing. To elimenate the imbalance of categorical data. We selected only 3000 of the non-licensing articles and all of the licensing articles which totaled to a little over 2500. "
      ],
      "metadata": {
        "id": "YSsUrLwCrzGA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        " \n",
        " \n",
        "uploaded = files.upload()\n"
      ],
      "metadata": {
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 74
        },
        "id": "GsGCZFvaXCSn",
        "outputId": "78bab254-a733-4a3a-faa5-c266c242044b"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-06fa15bf-c1d2-4f36-91df-d90b3c17158f\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-06fa15bf-c1d2-4f36-91df-d90b3c17158f\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving df_equal_labels.csv to df_equal_labels.csv\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.read_csv(io.BytesIO(uploaded['df_equal_labels.csv']))"
      ],
      "metadata": {
        "id": "8kjBynint_56"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3. Clean data for models\n",
        "\n",
        "In this section, we are cleaning the data specifically for the machine learning models. We are following similar logic and procedure as the first half of the project. Making text lowercase, removing numbers, spliting contractions, and removing punctuation\n"
      ],
      "metadata": {
        "id": "DramX9PpAasi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def clean_text(text):\n",
        "        \n",
        "    '''Make text lowercase, ,remove punctuation, split contractions, remove text in square brackets, remove punctuation and remove words containing numbers.'''\n",
        "    text = re.sub('\\[.*?\\]', '', text)\n",
        "    text = re.sub('\\w*\\d\\w*', '', text)\n",
        "    text = re.sub('[‘’“”…]', '', text)\n",
        "    text = re.sub('\\n', '', text)\n",
        "    text = str(text)\n",
        "    text = text.lower()\n",
        "    text = text.replace('\\n','')\n",
        "    text = re.sub('[%s]' % re.escape(string.punctuation), '', text)\n",
        "    return text\n"
      ],
      "metadata": {
        "id": "IV3-Ri98rYU1"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 3.0.1 Remove stop words"
      ],
      "metadata": {
        "id": "woAVvgpSBY8-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nltk.download('stopwords')\n",
        "stop = set(nltk.corpus.stopwords.words('english'))\n",
        "stop.update(punctuation)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8UKVHYYIrXuK",
        "outputId": "58693450-b1a1-47cc-8ab7-50fce792059d"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.0.2 Add positional tagging \n",
        "\n",
        "This section is somewhat similar to tagging the nouns during topic modeling. The positional tagging is done with wordnet. WordNet is a lexical database of semantic relations between words in more than 200 languages. It was a hand coded by linguist. "
      ],
      "metadata": {
        "id": "Fa941NDFBhZ4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_simple_pos(tag):\n",
        "    if tag.startswith('J'):\n",
        "        return wordnet.ADJ\n",
        "    elif tag.startswith('V'):\n",
        "        return wordnet.VERB\n",
        "    elif tag.startswith('N'):\n",
        "        return wordnet.NOUN\n",
        "    elif tag.startswith('R'):\n",
        "        return wordnet.ADV\n",
        "    else:\n",
        "        return wordnet.NOUN"
      ],
      "metadata": {
        "id": "KPAMZ_FFss38"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.0.3 Word Lemmatizing \n",
        "\n",
        "This is the process of getting the text down to its base or lemma. It save on processing time. The NLTK has a package that we will use to assist us. Sometimes, the same word can have multiple different ‘lemma’s. So, based on the context it’s used, you should identify the ‘part-of-speech’ (POS) tag for the word in that specific context and extract the appropriate lemma.[3]"
      ],
      "metadata": {
        "id": "6eUL3uqh4Mo3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "lemmatizer = WordNetLemmatizer()\n",
        "def clean_text_2(text):\n",
        "    clean_text = []\n",
        "    for w in word_tokenize(text):\n",
        "        if w.lower() not in stop:\n",
        "            pos = pos_tag([w])\n",
        "            new_w = lemmatizer.lemmatize(w, pos=get_simple_pos(pos[0][1]))\n",
        "            clean_text.append(new_w)\n",
        "    return clean_text\n",
        "\n",
        "def join_text(text):\n",
        "    return \" \".join(text)"
      ],
      "metadata": {
        "id": "oWpJ52G1swSS"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.0.4 Apply user define preprocessing functions to text\n",
        "\n",
        "This section will apply all the functions defined above to the dataframes text. "
      ],
      "metadata": {
        "id": "yuzykZjr505R"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nltk.download('punkt')\n",
        "nltk.download('averaged_perceptron_tagger')\n",
        "nltk.download('wordnet')\n",
        "nltk.download('omw-1.4')\n",
        "df['Content'] = df['Content'].apply(clean_text_2)\n",
        "df['Content'] = df['Content'].apply(join_text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0FkDL0gKs16N",
        "outputId": "c50cde89-c17e-41c7-e619-a82b7d453928"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data] Downloading package omw-1.4 to /root/nltk_data...\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4. Train and Test Split \n",
        "\n",
        "The data has been cleaned and preprocessed. Now, it needs to split so we can train the model and then test it. The standard split for train and test data is 80 (train) and 20 (test). When training the model, it can over or under fit the data. So, we need to test it with fresh data to get a better and more true accuracy score. "
      ],
      "metadata": {
        "id": "4WLOaLip6Nif"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# splitting data.\n",
        "x_train,x_test,y_train,y_test = train_test_split(df.Content,df.label,test_size = 0.2 , random_state = 0)"
      ],
      "metadata": {
        "id": "oaM04-Qtt9J3"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "x_train.shape, y_train.shape, x_test.shape, y_test.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BYc9dkijYibu",
        "outputId": "cf2c7865-50c4-43dd-8aac-8c3c846dce11"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((4419,), (4419,), (1105,), (1105,))"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 4.0.1 Identifying Labels"
      ],
      "metadata": {
        "id": "8FxjBbMh691a"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "non_licensing = x_train[y_train[y_train==0].index]\n",
        "licensing = x_train[y_train[y_train==1].index]\n"
      ],
      "metadata": {
        "id": "cTmamtNyYs9q"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 5. Vectorize the Text Data\n",
        "\n",
        "Now, we need to formulate the data in a way our model can understand it. This is a process that we also did during the topic modelling and was happening behind the scene during some of the keyword extractions methods used as well. It is used to transform a given text into a vector on the basis of the frequency (count) of each word that occurs in the entire text."
      ],
      "metadata": {
        "id": "TRGXY8NS7TgI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# creating a variable for count vectorizer which gives us features using the whole text of data.\n",
        "count_vec = CountVectorizer(max_features=4000, ngram_range=(1,2), max_df=0.9, min_df=0)\n",
        "# max_df insures to remove most frequent words as we discussed earlier.\n",
        "# ngram_range is used to select words at a time like 1 or 2 like if a sentence have 'not happy' in text then it can mean two things if we pick the word 'happy' and pick the words 'not happy' both.\n"
      ],
      "metadata": {
        "id": "GUPj8pWYY79R"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "x_train_features = count_vec.fit_transform(x_train).todense()\n",
        "x_test_features = count_vec.transform(x_test).todense()\n",
        "x_train_features.shape, x_test_features.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CHlPu6d3Y9mM",
        "outputId": "48da6ee8-93f1-4c68-a6fe-74f50c06aa82"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((4419, 4000), (1105, 4000))"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 6. Create and fit the model\n",
        "\n",
        "Now, the model is ready to be created and fitted. Once we train the model, we will assess its accuracy with the test data. "
      ],
      "metadata": {
        "id": "pRXLSspk8enY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nb_clf = MultinomialNB()\n",
        "nb_clf.fit(x_train_features, y_train)\n",
        "y_pred = nb_clf.predict(x_test_features)\n",
        "print(accuracy_score(y_test,y_pred)*100)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kpUJl_l9ZDVX",
        "outputId": "df68e50e-ec98-436b-8a5f-e1e7415eb770"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py:598: FutureWarning: np.matrix usage is deprecated in 1.0 and will raise a TypeError in 1.2. Please convert to a numpy array with np.asarray. For more information see: https://numpy.org/doc/stable/reference/generated/numpy.matrix.html\n",
            "  FutureWarning,\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "69.32126696832579\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py:598: FutureWarning: np.matrix usage is deprecated in 1.0 and will raise a TypeError in 1.2. Please convert to a numpy array with np.asarray. For more information see: https://numpy.org/doc/stable/reference/generated/numpy.matrix.html\n",
            "  FutureWarning,\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 7. Examine the Results \n",
        "\n",
        "To get a deeper understanding of our models performance, we will print out the classification report. A classification report is similar to a confusion matrix. It tells us the precision and recall based on the true and false negatives and positives. If something is 'true', it was classified correctly. If it is 'false' , it has been labeled incorrectly. Precision is the number of true positives divided by the total number of positives(true and false) classified. Recall is the number of true positives divided by the total number of elements that actually belong to the positive class (i.e. the sum of true positives and false negatives, which are items which were not labelled as belonging to the positive class but should have been). This is done for each category. From our classification report, we can identify that the model had a more difficult time labelling the 1 or licensing articles correctly. Overall, our model did OK. This is not something we could provide to a company to execute a sorting task or any high confidence predictive measure. However, we can fine tune the parameters or adjust the preprocessing technique to try to improve upon our model. \n"
      ],
      "metadata": {
        "id": "cVk4akNF9X5o"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 7.0.1 Classification report"
      ],
      "metadata": {
        "id": "m8T_ZnBHfhRl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(classification_report(y_test, y_pred))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IWxP6cyUZHR0",
        "outputId": "b9efa157-b00a-4266-d3ad-d528513f80e1"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.98      0.45      0.62       606\n",
            "           1       0.60      0.99      0.74       499\n",
            "\n",
            "    accuracy                           0.69      1105\n",
            "   macro avg       0.79      0.72      0.68      1105\n",
            "weighted avg       0.81      0.69      0.67      1105\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Sources:\n",
        "1. [Naive Bayes Classifier (NB)](https://medium.com/@akshayc123/naive-bayes-classifier-nb-7429a1bdb2c0#:~:text=Multinomial%20NB%20is%20used%20for%20multinomial%20distribution%20that,It%20is%20used%20when%20data%20has%20Gaussian%20distribution.)\n",
        "2. [Code Source](https://www.kaggle.com/code/amananandrai/nlp-using-ml-algorithms-news-articles/notebook)\n",
        "3. [Word Lemmatization](https://www.machinelearningplus.com/nlp/lemmatization-examples-python/#:~:text=Lemmatization%20is%20the%20process%20of%20converting%20a%20word,outputs%20from%20these%20packages.%20Skip%20to%20content%20Blogs)\n",
        "4. [Posterior Probability](https://www.investopedia.com/terms/p/posterior-probability.asp)"
      ],
      "metadata": {
        "id": "tc1jm2BAkAmM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Third Model - Gradient Boost : Supervised"
      ],
      "metadata": {
        "id": "Bfpz3k6GlR4i"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**What is Gradient Boosting?**\n",
        "\n",
        "Our third and last model of the project will be using Gradient Boosting Classifier. This is a very popular machine learning algorithm and can be used for both regression and classification. This classifier is a boosting type of ensemble learning method. Boosting works by taking poor performing predictors and turning them into strong performing predictors. \n",
        "\n",
        "**How does Gradient Boosting work?**\n",
        "\n",
        "In order to make initial predictions on the data, the algorithm will get the log of the odds of the target features. Once it has done this, it has built a mini decision tree. Next, it will calculate a residuals for each target feature. Unlike the regression Gradient Boost, all the predictions cannot simply be added together without some sort of transformation because the predictions are in terms of the log of the odds. The transformation formula is the numerator is the sum of all the residuals and the denominator is sum of the previous predicted probability multiplied by 1 minus the previous predicted probability. This transformer is applied to every leaf on the tree.[2] Now, it’s ready to update the prediction by the initial leaf with a new tree. The new tree is scaled by a learning rate. So, we take the the log(odds) prediction or the residual and add that to the learning rate multiplied by the updated prediction. This process is continued and repeated for more new trees until a certain predefined threshold is reached, or the residuals are negligible.[1]\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "Y1dj9djqXJkq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.ensemble import GradientBoostingClassifier\n",
        "from pprint import pprint\n",
        "from sklearn.model_selection import RandomizedSearchCV\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "from sklearn.metrics import classification_report, confusion_matrix, accuracy_score\n",
        "from sklearn.model_selection import ShuffleSplit\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "import pickle\n",
        "import pandas as pd\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.feature_selection import chi2\n",
        "import numpy as np"
      ],
      "metadata": {
        "id": "BmPtBBY4lY5-"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2. Define Parameters\n",
        "We have to define the different parameters:\n",
        "\n",
        "* ngram_range: This is were ngrams is decided. \n",
        "We want to consider both unigrams and bigrams.\n",
        "\n",
        "* max_df: When building the vocabulary ignore terms that have a document frequency strictly higher than the given threshold\n",
        "\n",
        "* min_df: When building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold.\n",
        "\n",
        "* max_features: If not None, build a vocabulary that only consider the top max_features ordered by term frequency across the corpus.[3]"
      ],
      "metadata": {
        "id": "bpa4gmlPpDdY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Parameter election\n",
        "ngram_range = (1,2)\n",
        "min_df = 10\n",
        "max_df = 1.\n",
        "max_features = 300\n"
      ],
      "metadata": {
        "id": "qw3sPw6VpGxU"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3. Vectorize the Features\n",
        "\n",
        "This is the same process done for our previous model but a different formula. Tf stands for term frequency and idf, inverse document frequency. This method multiple tf by idf. Some data scienctist prefer TF-IDF because it not only focuses on the frequency of words present in the corpus but also provides the importance of the words.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "Please note that we have fitted and then transformed the training set, but we have only transformed the test set."
      ],
      "metadata": {
        "id": "bDmVP5alqwQG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tfidf = TfidfVectorizer(encoding='utf-8',\n",
        "                        ngram_range=ngram_range,\n",
        "                        stop_words=None,\n",
        "                        lowercase=False,\n",
        "                        max_df=max_df,\n",
        "                        min_df=min_df,\n",
        "                        max_features=max_features,\n",
        "                        norm='l2',\n",
        "                        sublinear_tf=True)\n",
        "                        \n",
        "features_train = tfidf.fit_transform(x_train).toarray()\n",
        "labels_train = y_train\n",
        "print(features_train.shape)\n",
        "\n",
        "features_test = tfidf.transform(x_test).toarray()\n",
        "labels_test = y_test\n",
        "print(features_test.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "U7I4KEDOpIaA",
        "outputId": "cdd7a4ee-7c2b-4d42-d5c5-c31751a14ab7"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(4419, 300)\n",
            "(1105, 300)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3. Hyperparameter tuning\n",
        "\n",
        "**What is hyperparameter tuning?**\n",
        "\n",
        "Hyperparameter are the varaibles that govern the training processes.  parameters are the variables that your chosen machine learning technique uses to adjust to your data. These are tuned so we can fine the best fit for our model and data to produce the highest accuracy.\n",
        "\n",
        "Let's check the current model parameters. Then, we will use cross vaildation to tune the hyperparameters. "
      ],
      "metadata": {
        "id": "CVk9T4yWtABE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "gb_0 = GradientBoostingClassifier(random_state = 8)\n",
        "\n",
        "print('Parameters currently in use:\\n')\n",
        "## pretty print\n",
        "pprint(gb_0.get_params())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k0PBuXp-lXMZ",
        "outputId": "6974d88f-041f-4f16-f2c5-4af06df49305"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Parameters currently in use:\n",
            "\n",
            "{'ccp_alpha': 0.0,\n",
            " 'criterion': 'friedman_mse',\n",
            " 'init': None,\n",
            " 'learning_rate': 0.1,\n",
            " 'loss': 'deviance',\n",
            " 'max_depth': 3,\n",
            " 'max_features': None,\n",
            " 'max_leaf_nodes': None,\n",
            " 'min_impurity_decrease': 0.0,\n",
            " 'min_samples_leaf': 1,\n",
            " 'min_samples_split': 2,\n",
            " 'min_weight_fraction_leaf': 0.0,\n",
            " 'n_estimators': 100,\n",
            " 'n_iter_no_change': None,\n",
            " 'random_state': 8,\n",
            " 'subsample': 1.0,\n",
            " 'tol': 0.0001,\n",
            " 'validation_fraction': 0.1,\n",
            " 'verbose': 0,\n",
            " 'warm_start': False}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Tree-related hyperparameters**:\n",
        "\n",
        "* n_estimators = number of trees in the forest.\n",
        "* max_features = max number of features considered for splitting a node\n",
        "* max_depth = max number of levels in each decision tree\n",
        "* min_samples_split = min number of data points placed in a node before the node is split\n",
        "* min_samples_leaf = min number of data points allowed in a leaf node\n",
        "\n",
        "\n",
        "**Boosting-related hyperparameters**:\n",
        "\n",
        "* learning_rate= learning rate shrinks the contribution of each tree by learning_rate.\n",
        "* subsample= the fraction of samples to be used for fitting the individual base learners.\n"
      ],
      "metadata": {
        "id": "XxrJnyTO8oxO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 3.0.1 Random Search Cross Validation\n",
        "\n",
        "When performing cross validation, either random search or grid search are used to try out different hyperparameters in hopes to fine the best fitting for our model and data. We will use random search. \n",
        "\n",
        "**What is random search?**\n",
        "\n",
        "A randomized search on hyper parameters. In contrast to GridSearchCV, not all parameter values are tried out, but rather a fixed number of parameter settings is sampled from the specified distributions [3] Because grid search is an exhaustive approach it is very time consuming, but random search is a very effective and time-efficient. The logic behind random search is that by randomly picking different hyper-parameter combination, we will mostly likely pick one similar to the exhaustive solution of grid search in less time. \n",
        "\n",
        "**What is Cross Validation? (CV)**\n",
        "\n",
        "Cross validation is very commonly used when building a machine learning model. It can help when there is not a robust amount of data to train and test on. It, also, helps to test hyper-parameter with overfitting. Cross validation takes a samples of the data to train and validate with. These are called folds and the user can designate how many folds are performed when cross validating. All folds are used to train except one and the one is used for validation. \n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "SOMhKzFh072h"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 3.0.2 Define Search Parameters"
      ],
      "metadata": {
        "id": "NnWEUNnh9lRC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# n_estimators\n",
        "n_estimators = [200, 800]\n",
        "\n",
        "# max_features\n",
        "max_features = ['auto', 'sqrt']\n",
        "\n",
        "# max_depth\n",
        "max_depth = [10, 40]\n",
        "max_depth.append(None)\n",
        "\n",
        "# min_samples_split\n",
        "min_samples_split = [10, 30, 50]\n",
        "\n",
        "# min_samples_leaf\n",
        "min_samples_leaf = [1, 2, 4]\n",
        "\n",
        "# learning rate\n",
        "learning_rate = [.1, .5]\n",
        "\n",
        "# subsample\n",
        "subsample = [.5, 1.]\n",
        "\n",
        "# Create the random grid\n",
        "random_grid = {'n_estimators': n_estimators,\n",
        "               'max_features': max_features,\n",
        "               'max_depth': max_depth,\n",
        "               'min_samples_split': min_samples_split,\n",
        "               'min_samples_leaf': min_samples_leaf,\n",
        "               'learning_rate': learning_rate,\n",
        "               'subsample': subsample}\n",
        "\n",
        "pprint(random_grid)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dgrQLiaJluCD",
        "outputId": "93a160fa-52aa-4247-df92-4ce31efb8ab5"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'learning_rate': [0.1, 0.5],\n",
            " 'max_depth': [10, 40, None],\n",
            " 'max_features': ['auto', 'sqrt'],\n",
            " 'min_samples_leaf': [1, 2, 4],\n",
            " 'min_samples_split': [10, 30, 50],\n",
            " 'n_estimators': [200, 800],\n",
            " 'subsample': [0.5, 1.0]}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.0.3 Create Base Gradient Booting Model"
      ],
      "metadata": {
        "id": "YlF2LcUK941N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# First create the base model to tune\n",
        "gbc = GradientBoostingClassifier(random_state=8)\n",
        "\n",
        "# Definition of the random search\n",
        "random_search = RandomizedSearchCV(estimator=gbc,\n",
        "                                   param_distributions=random_grid,\n",
        "                                   n_iter=10,\n",
        "                                   scoring='accuracy',\n",
        "                                   cv=3, \n",
        "                                   verbose=1, \n",
        "                                   random_state=8)\n",
        "\n",
        "# Fit the random search model\n",
        "random_search.fit(features_train, labels_train)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZY7JFZ8FlxF4",
        "outputId": "06b4ee10-5065-4442-be7a-8612166d92a7"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 10 candidates, totalling 30 fits\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RandomizedSearchCV(cv=3, estimator=GradientBoostingClassifier(random_state=8),\n",
              "                   param_distributions={'learning_rate': [0.1, 0.5],\n",
              "                                        'max_depth': [10, 40, None],\n",
              "                                        'max_features': ['auto', 'sqrt'],\n",
              "                                        'min_samples_leaf': [1, 2, 4],\n",
              "                                        'min_samples_split': [10, 30, 50],\n",
              "                                        'n_estimators': [200, 800],\n",
              "                                        'subsample': [0.5, 1.0]},\n",
              "                   random_state=8, scoring='accuracy', verbose=1)"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 3.0.4 Print Out Hyperparameters and Accuracy Score"
      ],
      "metadata": {
        "id": "9Y_588AY-Bth"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"The best hyperparameters from Random Search are:\")\n",
        "print(random_search.best_params_)\n",
        "print(\"\")\n",
        "print(\"The mean accuracy of a model with these hyperparameters is:\")\n",
        "print(random_search.best_score_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R8ClEsEil4gX",
        "outputId": "c2dd6911-83bd-4fda-8b80-3fd99c63e308"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The best hyperparameters from Random Search are:\n",
            "{'subsample': 0.5, 'n_estimators': 800, 'min_samples_split': 30, 'min_samples_leaf': 4, 'max_features': 'sqrt', 'max_depth': 10, 'learning_rate': 0.1}\n",
            "\n",
            "The mean accuracy of a model with these hyperparameters is:\n",
            "0.949988685222901\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 3.0.5 Grid Search CV \n",
        "\n",
        "**What is Grid Search?**\n",
        "\n",
        "\n",
        "Similar to random search, Grid search is an exhaustive search on all the hyperparmeters. This will help us build a high preforming model. Grid search is great for spot-checking combinations that are known to perform well generally. Random search is great for discovery and getting hyperparameter combinations that you would not have guessed intuitively, although it often requires more time to execute. We will use our discovery found in random search and perfect it in grid search.\n"
      ],
      "metadata": {
        "id": "-4MVls0r-WsZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create the parameter grid based on the results of random search \n",
        "max_depth = [5, 10, 15]\n",
        "max_features = ['sqrt']\n",
        "min_samples_leaf = [2]\n",
        "min_samples_split = [50, 100]\n",
        "n_estimators = [800]\n",
        "learning_rate = [.1, .5]\n",
        "subsample = [1.]\n",
        "\n",
        "param_grid = {\n",
        "    'max_depth': max_depth,\n",
        "    'max_features': max_features,\n",
        "    'min_samples_leaf': min_samples_leaf,\n",
        "    'min_samples_split': min_samples_split,\n",
        "    'n_estimators': n_estimators,\n",
        "    'learning_rate': learning_rate,\n",
        "    'subsample': subsample\n",
        "\n",
        "}\n",
        "\n",
        "# Create a base model\n",
        "gbc = GradientBoostingClassifier(random_state=8)\n",
        "\n",
        "# Manually create the splits in CV in order to be able to fix a random_state (GridSearchCV doesn't have that argument)\n",
        "cv_sets = ShuffleSplit(n_splits = 3, test_size = .33, random_state = 8)\n",
        "\n",
        "# Instantiate the grid search model\n",
        "grid_search = GridSearchCV(estimator=gbc, \n",
        "                           param_grid=param_grid,\n",
        "                           scoring='accuracy',\n",
        "                           cv=cv_sets,\n",
        "                           verbose=1)\n",
        "\n",
        "# Fit the grid search to the data\n",
        "grid_search.fit(features_train, labels_train)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4XlDbFTtmVND",
        "outputId": "65069592-b49d-44eb-eb7b-1becf01826c5"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 12 candidates, totalling 36 fits\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(cv=ShuffleSplit(n_splits=3, random_state=8, test_size=0.33, train_size=None),\n",
              "             estimator=GradientBoostingClassifier(random_state=8),\n",
              "             param_grid={'learning_rate': [0.1, 0.5], 'max_depth': [5, 10, 15],\n",
              "                         'max_features': ['sqrt'], 'min_samples_leaf': [2],\n",
              "                         'min_samples_split': [50, 100], 'n_estimators': [800],\n",
              "                         'subsample': [1.0]},\n",
              "             scoring='accuracy', verbose=1)"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 3.0.7 Print Out Best Hyperparameters from Grid Search"
      ],
      "metadata": {
        "id": "v0cv6A_n_NMY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"The best hyperparameters from Grid Search are:\")\n",
        "print(grid_search.best_params_)\n",
        "print(\"\")\n",
        "print(\"The mean accuracy of a model with these hyperparameters is:\")\n",
        "print(grid_search.best_score_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XqLN1_a6mWb-",
        "outputId": "b8b7a197-7aa3-4fc5-ac93-27fb8847bc52"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The best hyperparameters from Grid Search are:\n",
            "{'learning_rate': 0.1, 'max_depth': 15, 'max_features': 'sqrt', 'min_samples_leaf': 2, 'min_samples_split': 100, 'n_estimators': 800, 'subsample': 1.0}\n",
            "\n",
            "The mean accuracy of a model with these hyperparameters is:\n",
            "0.9529358007767877\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "best_gbc = grid_search.best_estimator_"
      ],
      "metadata": {
        "id": "9apHJ53SmLjG"
      },
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4. Create and Fit Gradient Boost Model\n",
        "\n",
        "In this section, we are ready to create and train our model. We will use the best parameters from from grid search. Grid search preformed 1 percent better then random search. "
      ],
      "metadata": {
        "id": "53YlFcu7_YJr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "best_gbc.fit(features_train, labels_train)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O4azXvzImE7J",
        "outputId": "2bda4b94-7141-48b8-a990-a675efe60428"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GradientBoostingClassifier(max_depth=15, max_features='sqrt',\n",
              "                           min_samples_leaf=2, min_samples_split=100,\n",
              "                           n_estimators=800, random_state=8)"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 5. Evaluate the Model \n",
        "\n",
        "We will now see how well the model will perform with are unseen test data. "
      ],
      "metadata": {
        "id": "NBvEXOY2_z35"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "gbc_pred = best_gbc.predict(features_test)"
      ],
      "metadata": {
        "id": "1RCTjIrVmHyv"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Training accuracy\n",
        "print(\"The training accuracy is: \")\n",
        "print(accuracy_score(labels_train, best_gbc.predict(features_train)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aSXHkIKcmqT_",
        "outputId": "40fa67b0-2cfb-42c5-98f3-cbb81d8ffccd"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The training accuracy is: \n",
            "1.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Test accuracy\n",
        "print(\"The test accuracy is: \")\n",
        "print(accuracy_score(labels_test, gbc_pred))\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f-cfPaSJmtbI",
        "outputId": "d4234c8f-4ad0-42fb-d8f1-a52655fea34c"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The test accuracy is: \n",
            "0.9638009049773756\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 6. Examine Results \n",
        "\n",
        "Similar to our previous model, we will print out a classification report to identify strengths and weaknesses. For the overall accuracy on the test data, our model got a **96 percent**. This is very good and has sucessfully achieved our task. This is something we could use if continuing this project and offer to a business as a sucessful predictive measure. \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "IOhdGFmt_8Mn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Classification report\n",
        "print(\"Classification report\")\n",
        "print(classification_report(labels_test,gbc_pred))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PLlFBfyymwe4",
        "outputId": "4a6d45db-8eeb-4cf4-ca0a-4e12e9aaa1ad"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Classification report\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.96      0.98      0.97       606\n",
            "           1       0.97      0.95      0.96       499\n",
            "\n",
            "    accuracy                           0.96      1105\n",
            "   macro avg       0.96      0.96      0.96      1105\n",
            "weighted avg       0.96      0.96      0.96      1105\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 6.0.1 Classification Report\n",
        "Now, lets break down our classification report: \n",
        "\n",
        "**Precision**: Out of all the articles that the model predicted were licensing articles, only 97% actually are.\n",
        "\n",
        "**Recall**: Out of all the articles that actually were licensing, the model predicted this outcome correctly for 95% of those articles\n",
        "\n",
        "**F1 Score**: This value is calculated as:\n",
        "\n",
        "F1 Score: 2 * (Precision * Recall) / (Precision + Recall)\n",
        "F1 Score: 2 * (.97 * .95) / (.97 + .95)\n",
        "F1 Score: 0.96\n",
        "\n",
        "\n",
        "Since this value is very close to 1, it tells us that the model does a good job of predicting whether or not the article is a licensing agreement article or not."
      ],
      "metadata": {
        "id": "5xeR1jngI2Fj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Sources: \n",
        "\n",
        "1. [Gradient Boosting Classification ](https://towardsdatascience.com/gradient-boosting-classification-explained-through-python-60cc980eeb3d#:~:text=Boosting%20is%20a%20special%20type,attention%20to%20its%20predecessor's%20mistakes.)\n",
        "2. [Gradient Boost Classification Video](https://www.youtube.com/watch?v=jxuNLH5dXCs&t=346s)\n",
        "3. [Code source Github](https://github.com/miguelfzafra/Latest-News-Classifier/blob/master/0.%20Latest%20News%20Classifier/03.%20Feature%20Engineering/03.%20Feature%20Engineering.ipynb)\n",
        "4. [Random Search CV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html)\n",
        "\n"
      ],
      "metadata": {
        "id": "yb34inedeOJu"
      }
    }
  ]
}