Amazon_Fine_Food_Reviews_Analysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Amazon Fine Food Reviews Analysis.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/arunsingh22/CaseStudies/blob/master/Amazon_Fine_Food_Reviews_Analysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X-cp_kAxCAY2"
      },
      "source": [
        "# Amazon Fine Food Reviews Analysis\n",
        "\n",
        "\n",
        "Data Source: https://www.kaggle.com/snap/amazon-fine-food-reviews <br>\n",
        "\n",
        "EDA: https://nycdatascience.com/blog/student-works/amazon-fine-foods-visualization/\n",
        "\n",
        "\n",
        "The Amazon Fine Food Reviews dataset consists of reviews of fine foods from Amazon.<br>\n",
        "\n",
        "Number of reviews: 568,454<br>\n",
        "Number of users: 256,059<br>\n",
        "Number of products: 74,258<br>\n",
        "Timespan: Oct 1999 - Oct 2012<br>\n",
        "Number of Attributes/Columns in data: 10 \n",
        "\n",
        "Attribute Information:\n",
        "\n",
        "1. Id\n",
        "2. ProductId - unique identifier for the product\n",
        "3. UserId - unqiue identifier for the user\n",
        "4. ProfileName\n",
        "5. HelpfulnessNumerator - number of users who found the review helpful\n",
        "6. HelpfulnessDenominator - number of users who indicated whether they found the review helpful or not\n",
        "7. Score - rating between 1 and 5\n",
        "8. Time - timestamp for the review\n",
        "9. Summary - brief summary of the review\n",
        "10. Text - text of the review\n",
        "\n",
        "\n",
        "#### Objective:\n",
        "Given a review, determine whether the review is positive (Rating of 4 or 5) or negative (rating of 1 or 2).\n",
        "\n",
        "<br>\n",
        "[Q] How to determine if a review is positive or negative?<br>\n",
        "<br> \n",
        "[Ans] We could use the Score/Rating. A rating of 4 or 5 could be cosnidered a positive review. A review of 1 or 2 could be considered negative. A review of 3 is nuetral and ignored. This is an approximate and proxy way of determining the polarity (positivity/negativity) of a review.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHC_UQTuCAY4"
      },
      "source": [
        "## Loading the data\n",
        "\n",
        "The dataset is available in two forms\n",
        "1. .csv file\n",
        "2. SQLite Database\n",
        "\n",
        "In order to load the data, We have used the SQLITE dataset as it easier to query the data and visualise the data efficiently.\n",
        "<br> \n",
        "\n",
        "Here as we only want to get the global sentiment of the recommendations (positive or negative), we will purposefully ignore all Scores equal to 3. If the score id above 3, then the recommendation wil be set to \"positive\". Otherwise, it will be set to \"negative\"."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PvDdRsHgCAY5",
        "outputId": "65b237f2-2ead-4721-f95a-15ed0bb06d04"
      },
      "source": [
        "%matplotlib inline\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "\n",
        "\n",
        "\n",
        "import sqlite3\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import nltk\n",
        "import string\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from sklearn.feature_extraction.text import TfidfTransformer\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.metrics import confusion_matrix\n",
        "from sklearn import metrics\n",
        "from sklearn.metrics import roc_curve, auc\n",
        "from nltk.stem.porter import PorterStemmer\n",
        "\n",
        "import re\n",
        "# Tutorial about Python regular expressions: https://pymotw.com/2/re/\n",
        "import string\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.stem import PorterStemmer\n",
        "from nltk.stem.wordnet import WordNetLemmatizer\n",
        "\n",
        "from gensim.models import Word2Vec\n",
        "from gensim.models import KeyedVectors\n",
        "import pickle\n",
        "\n",
        "from tqdm import tqdm\n",
        "import os"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "D:\\installed\\Anaconda3\\lib\\site-packages\\gensim\\utils.py:1197: UserWarning: detected Windows; aliasing chunkize to chunkize_serial\n",
            "  warnings.warn(\"detected Windows; aliasing chunkize to chunkize_serial\")\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QOI7X2YgCAY_"
      },
      "source": [
        "# [1]. Reading Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3iYH2p1ECAZA",
        "outputId": "3feca330-8e21-4173-ad7c-88eb4d81668e"
      },
      "source": [
        "\n",
        "# using the SQLite Table to read data.\n",
        "con = sqlite3.connect('database.sqlite') \n",
        "#filtering only positive and negative reviews i.e. \n",
        "# not taking into consideration those reviews with Score=3\n",
        "# SELECT * FROM Reviews WHERE Score != 3 LIMIT 500000, will give top 500000 data points\n",
        "# you can change the number to any other number based on your computing power\n",
        "\n",
        "# filtered_data = pd.read_sql_query(\"\"\" SELECT * FROM Reviews WHERE Score != 3 LIMIT 500000\"\"\", con) \n",
        "# for tsne assignment you can take 5k data points\n",
        "\n",
        "filtered_data = pd.read_sql_query(\"\"\" SELECT * FROM Reviews WHERE Score != 3 LIMIT 5000\"\"\", con) \n",
        "\n",
        "# Give reviews with Score>3 a positive rating, and reviews with a score<3 a negative rating.\n",
        "def partition(x):\n",
        "    if x < 3:\n",
        "        return 0\n",
        "    return 1\n",
        "\n",
        "#changing reviews with score less than 3 to be positive and vice-versa\n",
        "actualScore = filtered_data['Score']\n",
        "positiveNegative = actualScore.map(partition) \n",
        "filtered_data['Score'] = positiveNegative\n",
        "print(\"Number of data points in our data\", filtered_data.shape)\n",
        "filtered_data.head(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of data points in our data (5000, 10)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Id</th>\n",
              "      <th>ProductId</th>\n",
              "      <th>UserId</th>\n",
              "      <th>ProfileName</th>\n",
              "      <th>HelpfulnessNumerator</th>\n",
              "      <th>HelpfulnessDenominator</th>\n",
              "      <th>Score</th>\n",
              "      <th>Time</th>\n",
              "      <th>Summary</th>\n",
              "      <th>Text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>B001E4KFG0</td>\n",
              "      <td>A3SGXH7AUHU8GW</td>\n",
              "      <td>delmartian</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1303862400</td>\n",
              "      <td>Good Quality Dog Food</td>\n",
              "      <td>I have bought several of the Vitality canned d...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>B00813GRG4</td>\n",
              "      <td>A1D87F6ZCVE5NK</td>\n",
              "      <td>dll pa</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1346976000</td>\n",
              "      <td>Not as Advertised</td>\n",
              "      <td>Product arrived labeled as Jumbo Salted Peanut...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>B000LQOCH0</td>\n",
              "      <td>ABXLMWJIXXAIN</td>\n",
              "      <td>Natalia Corres \"Natalia Corres\"</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1219017600</td>\n",
              "      <td>\"Delight\" says it all</td>\n",
              "      <td>This is a confection that has been around a fe...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Id   ProductId          UserId                      ProfileName  \\\n",
              "0   1  B001E4KFG0  A3SGXH7AUHU8GW                       delmartian   \n",
              "1   2  B00813GRG4  A1D87F6ZCVE5NK                           dll pa   \n",
              "2   3  B000LQOCH0   ABXLMWJIXXAIN  Natalia Corres \"Natalia Corres\"   \n",
              "\n",
              "   HelpfulnessNumerator  HelpfulnessDenominator  Score        Time  \\\n",
              "0                     1                       1      1  1303862400   \n",
              "1                     0                       0      0  1346976000   \n",
              "2                     1                       1      1  1219017600   \n",
              "\n",
              "                 Summary                                               Text  \n",
              "0  Good Quality Dog Food  I have bought several of the Vitality canned d...  \n",
              "1      Not as Advertised  Product arrived labeled as Jumbo Salted Peanut...  \n",
              "2  \"Delight\" says it all  This is a confection that has been around a fe...  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I1jf03kECAZF"
      },
      "source": [
        "display = pd.read_sql_query(\"\"\"\n",
        "SELECT UserId, ProductId, ProfileName, Time, Score, Text, COUNT(*)\n",
        "FROM Reviews\n",
        "GROUP BY UserId\n",
        "HAVING COUNT(*)>1\n",
        "\"\"\", con)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ja8xSefOCAZH",
        "outputId": "2cbd0705-618b-47cf-ce18-f3d0d4063cbf"
      },
      "source": [
        "print(display.shape)\n",
        "display.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(80668, 7)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>UserId</th>\n",
              "      <th>ProductId</th>\n",
              "      <th>ProfileName</th>\n",
              "      <th>Time</th>\n",
              "      <th>Score</th>\n",
              "      <th>Text</th>\n",
              "      <th>COUNT(*)</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>#oc-R115TNMSPFT9I7</td>\n",
              "      <td>B007Y59HVM</td>\n",
              "      <td>Breyton</td>\n",
              "      <td>1331510400</td>\n",
              "      <td>2</td>\n",
              "      <td>Overall its just OK when considering the price...</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>#oc-R11D9D7SHXIJB9</td>\n",
              "      <td>B005HG9ET0</td>\n",
              "      <td>Louis E. Emory \"hoppy\"</td>\n",
              "      <td>1342396800</td>\n",
              "      <td>5</td>\n",
              "      <td>My wife has recurring extreme muscle spasms, u...</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>#oc-R11DNU2NBKQ23Z</td>\n",
              "      <td>B007Y59HVM</td>\n",
              "      <td>Kim Cieszykowski</td>\n",
              "      <td>1348531200</td>\n",
              "      <td>1</td>\n",
              "      <td>This coffee is horrible and unfortunately not ...</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>#oc-R11O5J5ZVQE25C</td>\n",
              "      <td>B005HG9ET0</td>\n",
              "      <td>Penguin Chick</td>\n",
              "      <td>1346889600</td>\n",
              "      <td>5</td>\n",
              "      <td>This will be the bottle that you grab from the...</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>#oc-R12KPBODL2B5ZD</td>\n",
              "      <td>B007OSBE1U</td>\n",
              "      <td>Christopher P. Presta</td>\n",
              "      <td>1348617600</td>\n",
              "      <td>1</td>\n",
              "      <td>I didnt like this coffee. Instead of telling y...</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               UserId   ProductId             ProfileName        Time  Score  \\\n",
              "0  #oc-R115TNMSPFT9I7  B007Y59HVM                 Breyton  1331510400      2   \n",
              "1  #oc-R11D9D7SHXIJB9  B005HG9ET0  Louis E. Emory \"hoppy\"  1342396800      5   \n",
              "2  #oc-R11DNU2NBKQ23Z  B007Y59HVM        Kim Cieszykowski  1348531200      1   \n",
              "3  #oc-R11O5J5ZVQE25C  B005HG9ET0           Penguin Chick  1346889600      5   \n",
              "4  #oc-R12KPBODL2B5ZD  B007OSBE1U   Christopher P. Presta  1348617600      1   \n",
              "\n",
              "                                                Text  COUNT(*)  \n",
              "0  Overall its just OK when considering the price...         2  \n",
              "1  My wife has recurring extreme muscle spasms, u...         3  \n",
              "2  This coffee is horrible and unfortunately not ...         2  \n",
              "3  This will be the bottle that you grab from the...         3  \n",
              "4  I didnt like this coffee. Instead of telling y...         2  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u5ctUTW-CAZK",
        "outputId": "41439570-0ff9-44bb-9c05-e87bcb3dd6b7"
      },
      "source": [
        "display[display['UserId']=='AZY10LLTJ71NX']"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>UserId</th>\n",
              "      <th>ProductId</th>\n",
              "      <th>ProfileName</th>\n",
              "      <th>Time</th>\n",
              "      <th>Score</th>\n",
              "      <th>Text</th>\n",
              "      <th>COUNT(*)</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>80638</th>\n",
              "      <td>AZY10LLTJ71NX</td>\n",
              "      <td>B006P7E5ZI</td>\n",
              "      <td>undertheshrine \"undertheshrine\"</td>\n",
              "      <td>1334707200</td>\n",
              "      <td>5</td>\n",
              "      <td>I was recommended to try green tea extract to ...</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              UserId   ProductId                      ProfileName        Time  \\\n",
              "80638  AZY10LLTJ71NX  B006P7E5ZI  undertheshrine \"undertheshrine\"  1334707200   \n",
              "\n",
              "       Score                                               Text  COUNT(*)  \n",
              "80638      5  I was recommended to try green tea extract to ...         5  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tiaXnkZxCAZO",
        "outputId": "0bbd60e6-e66b-4f20-9060-bf0cfe143b1f"
      },
      "source": [
        "display['COUNT(*)'].sum()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "393063"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TX5b3dc5CAZS"
      },
      "source": [
        "#  Exploratory Data Analysis\n",
        "\n",
        "## [2] Data Cleaning: Deduplication\n",
        "\n",
        "It is observed (as shown in the table below) that the reviews data had many duplicate entries. Hence it was necessary to remove duplicates in order to get unbiased results for the analysis of the data.  Following is an example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": false,
        "id": "4jW_0WxCCAZT",
        "outputId": "4e526ab2-98f3-46ae-93dc-3e061cb06d59"
      },
      "source": [
        "display= pd.read_sql_query(\"\"\"\n",
        "SELECT *\n",
        "FROM Reviews\n",
        "WHERE Score != 3 AND UserId=\"AR5J8UI46CURR\"\n",
        "ORDER BY ProductID\n",
        "\"\"\", con)\n",
        "display.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Id</th>\n",
              "      <th>ProductId</th>\n",
              "      <th>UserId</th>\n",
              "      <th>ProfileName</th>\n",
              "      <th>HelpfulnessNumerator</th>\n",
              "      <th>HelpfulnessDenominator</th>\n",
              "      <th>Score</th>\n",
              "      <th>Time</th>\n",
              "      <th>Summary</th>\n",
              "      <th>Text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>78445</td>\n",
              "      <td>B000HDL1RQ</td>\n",
              "      <td>AR5J8UI46CURR</td>\n",
              "      <td>Geetha Krishnan</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>1199577600</td>\n",
              "      <td>LOACKER QUADRATINI VANILLA WAFERS</td>\n",
              "      <td>DELICIOUS WAFERS. I FIND THAT EUROPEAN WAFERS ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>138317</td>\n",
              "      <td>B000HDOPYC</td>\n",
              "      <td>AR5J8UI46CURR</td>\n",
              "      <td>Geetha Krishnan</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>1199577600</td>\n",
              "      <td>LOACKER QUADRATINI VANILLA WAFERS</td>\n",
              "      <td>DELICIOUS WAFERS. I FIND THAT EUROPEAN WAFERS ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>138277</td>\n",
              "      <td>B000HDOPYM</td>\n",
              "      <td>AR5J8UI46CURR</td>\n",
              "      <td>Geetha Krishnan</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>1199577600</td>\n",
              "      <td>LOACKER QUADRATINI VANILLA WAFERS</td>\n",
              "      <td>DELICIOUS WAFERS. I FIND THAT EUROPEAN WAFERS ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>73791</td>\n",
              "      <td>B000HDOPZG</td>\n",
              "      <td>AR5J8UI46CURR</td>\n",
              "      <td>Geetha Krishnan</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>1199577600</td>\n",
              "      <td>LOACKER QUADRATINI VANILLA WAFERS</td>\n",
              "      <td>DELICIOUS WAFERS. I FIND THAT EUROPEAN WAFERS ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>155049</td>\n",
              "      <td>B000PAQ75C</td>\n",
              "      <td>AR5J8UI46CURR</td>\n",
              "      <td>Geetha Krishnan</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>1199577600</td>\n",
              "      <td>LOACKER QUADRATINI VANILLA WAFERS</td>\n",
              "      <td>DELICIOUS WAFERS. I FIND THAT EUROPEAN WAFERS ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       Id   ProductId         UserId      ProfileName  HelpfulnessNumerator  \\\n",
              "0   78445  B000HDL1RQ  AR5J8UI46CURR  Geetha Krishnan                     2   \n",
              "1  138317  B000HDOPYC  AR5J8UI46CURR  Geetha Krishnan                     2   \n",
              "2  138277  B000HDOPYM  AR5J8UI46CURR  Geetha Krishnan                     2   \n",
              "3   73791  B000HDOPZG  AR5J8UI46CURR  Geetha Krishnan                     2   \n",
              "4  155049  B000PAQ75C  AR5J8UI46CURR  Geetha Krishnan                     2   \n",
              "\n",
              "   HelpfulnessDenominator  Score        Time  \\\n",
              "0                       2      5  1199577600   \n",
              "1                       2      5  1199577600   \n",
              "2                       2      5  1199577600   \n",
              "3                       2      5  1199577600   \n",
              "4                       2      5  1199577600   \n",
              "\n",
              "                             Summary  \\\n",
              "0  LOACKER QUADRATINI VANILLA WAFERS   \n",
              "1  LOACKER QUADRATINI VANILLA WAFERS   \n",
              "2  LOACKER QUADRATINI VANILLA WAFERS   \n",
              "3  LOACKER QUADRATINI VANILLA WAFERS   \n",
              "4  LOACKER QUADRATINI VANILLA WAFERS   \n",
              "\n",
              "                                                Text  \n",
              "0  DELICIOUS WAFERS. I FIND THAT EUROPEAN WAFERS ...  \n",
              "1  DELICIOUS WAFERS. I FIND THAT EUROPEAN WAFERS ...  \n",
              "2  DELICIOUS WAFERS. I FIND THAT EUROPEAN WAFERS ...  \n",
              "3  DELICIOUS WAFERS. I FIND THAT EUROPEAN WAFERS ...  \n",
              "4  DELICIOUS WAFERS. I FIND THAT EUROPEAN WAFERS ...  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BzVTlHiMCAZV"
      },
      "source": [
        "As can be seen above the same user has multiple reviews of the with the same values for HelpfulnessNumerator, HelpfulnessDenominator, Score, Time, Summary and Text  and on doing analysis it was found that <br>\n",
        "<br> \n",
        "ProductId=B000HDOPZG was Loacker Quadratini Vanilla Wafer Cookies, 8.82-Ounce Packages (Pack of 8)<br>\n",
        "<br> \n",
        "ProductId=B000HDL1RQ was Loacker Quadratini Lemon Wafer Cookies, 8.82-Ounce Packages (Pack of 8) and so on<br>\n",
        "\n",
        "It was inferred after analysis that reviews with same parameters other than ProductId belonged to the same product just having different flavour or quantity. Hence in order to reduce redundancy it was decided to eliminate the rows having same parameters.<br>\n",
        "\n",
        "The method used for the same was that we first sort the data according to ProductId and then just keep the first similar product review and delelte the others. for eg. in the above just the review for ProductId=B000HDL1RQ remains. This method ensures that there is only one representative for each product and deduplication without sorting would lead to possibility of different representatives still existing for the same product."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DGy2NM7BCAZX"
      },
      "source": [
        "#Sorting data according to ProductId in ascending order\n",
        "sorted_data=filtered_data.sort_values('ProductId', axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8b-O13XVCAZZ",
        "outputId": "12b254d4-826d-4cfc-bb6d-455f4d5b2a4f"
      },
      "source": [
        "#Deduplication of entries\n",
        "final=sorted_data.drop_duplicates(subset={\"UserId\",\"ProfileName\",\"Time\",\"Text\"}, keep='first', inplace=False)\n",
        "final.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(4986, 10)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S72Yh-rHCAZc",
        "outputId": "146c067f-081c-4e26-f527-71442ff83b8d"
      },
      "source": [
        "#Checking to see how much % of data still remains\n",
        "(final['Id'].size*1.0)/(filtered_data['Id'].size*1.0)*100"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "99.72"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LO5nLKeFCAZf"
      },
      "source": [
        "<b>Observation:-</b> It was also seen that in two rows given below the value of HelpfulnessNumerator is greater than HelpfulnessDenominator which is not practically possible hence these two rows too are removed from calcualtions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L9z5lrsiCAZh",
        "outputId": "1a0dfa7c-62f4-4469-a97b-38f77fa3b2ab"
      },
      "source": [
        "display= pd.read_sql_query(\"\"\"\n",
        "SELECT *\n",
        "FROM Reviews\n",
        "WHERE Score != 3 AND Id=44737 OR Id=64422\n",
        "ORDER BY ProductID\n",
        "\"\"\", con)\n",
        "\n",
        "display.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Id</th>\n",
              "      <th>ProductId</th>\n",
              "      <th>UserId</th>\n",
              "      <th>ProfileName</th>\n",
              "      <th>HelpfulnessNumerator</th>\n",
              "      <th>HelpfulnessDenominator</th>\n",
              "      <th>Score</th>\n",
              "      <th>Time</th>\n",
              "      <th>Summary</th>\n",
              "      <th>Text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>64422</td>\n",
              "      <td>B000MIDROQ</td>\n",
              "      <td>A161DK06JJMCYF</td>\n",
              "      <td>J. E. Stephens \"Jeanne\"</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>1224892800</td>\n",
              "      <td>Bought This for My Son at College</td>\n",
              "      <td>My son loves spaghetti so I didn't hesitate or...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>44737</td>\n",
              "      <td>B001EQ55RW</td>\n",
              "      <td>A2V0I904FH7ABY</td>\n",
              "      <td>Ram</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>4</td>\n",
              "      <td>1212883200</td>\n",
              "      <td>Pure cocoa taste with crunchy almonds inside</td>\n",
              "      <td>It was almost a 'love at first bite' - the per...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      Id   ProductId          UserId              ProfileName  \\\n",
              "0  64422  B000MIDROQ  A161DK06JJMCYF  J. E. Stephens \"Jeanne\"   \n",
              "1  44737  B001EQ55RW  A2V0I904FH7ABY                      Ram   \n",
              "\n",
              "   HelpfulnessNumerator  HelpfulnessDenominator  Score        Time  \\\n",
              "0                     3                       1      5  1224892800   \n",
              "1                     3                       2      4  1212883200   \n",
              "\n",
              "                                        Summary  \\\n",
              "0             Bought This for My Son at College   \n",
              "1  Pure cocoa taste with crunchy almonds inside   \n",
              "\n",
              "                                                Text  \n",
              "0  My son loves spaghetti so I didn't hesitate or...  \n",
              "1  It was almost a 'love at first bite' - the per...  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Tphk1piCAZk"
      },
      "source": [
        "final=final[final.HelpfulnessNumerator<=final.HelpfulnessDenominator]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8TpsGDdJCAZm",
        "outputId": "287c6c08-d222-4848-b677-c9a5e769d0b1"
      },
      "source": [
        "#Before starting the next phase of preprocessing lets see the number of entries left\n",
        "print(final.shape)\n",
        "\n",
        "#How many positive and negative reviews are present in our dataset?\n",
        "final['Score'].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(4986, 10)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1    4178\n",
              "0     808\n",
              "Name: Score, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RxBsuBylCAZr"
      },
      "source": [
        "# [3].  Text Preprocessing.\n",
        "\n",
        "Now that we have finished deduplication our data requires some preprocessing before we go on further with analysis and making the prediction model.\n",
        "\n",
        "Hence in the Preprocessing phase we do the following in the order below:-\n",
        "\n",
        "1. Begin by removing the html tags\n",
        "2. Remove any punctuations or limited set of special characters like , or . or # etc.\n",
        "3. Check if the word is made up of english letters and is not alpha-numeric\n",
        "4. Check to see if the length of the word is greater than 2 (as it was researched that there is no adjective in 2-letters)\n",
        "5. Convert the word to lowercase\n",
        "6. Remove Stopwords\n",
        "7. Finally Snowball Stemming the word (it was obsereved to be better than Porter Stemming)<br>\n",
        "\n",
        "After which we collect the words used to describe positive and negative reviews"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "laOGMFB7CAZt",
        "outputId": "49d82795-cea4-4695-9834-45978266f6f4"
      },
      "source": [
        "# printing some random reviews\n",
        "sent_0 = final['Text'].values[0]\n",
        "print(sent_0)\n",
        "print(\"=\"*50)\n",
        "\n",
        "sent_1000 = final['Text'].values[1000]\n",
        "print(sent_1000)\n",
        "print(\"=\"*50)\n",
        "\n",
        "sent_1500 = final['Text'].values[1500]\n",
        "print(sent_1500)\n",
        "print(\"=\"*50)\n",
        "\n",
        "sent_4900 = final['Text'].values[4900]\n",
        "print(sent_4900)\n",
        "print(\"=\"*50)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Why is this $[...] when the same product is available for $[...] here?<br />http://www.amazon.com/VICTOR-FLY-MAGNET-BAIT-REFILL/dp/B00004RBDY<br /><br />The Victor M380 and M502 traps are unreal, of course -- total fly genocide. Pretty stinky, but only right nearby.\n",
            "==================================================\n",
            "I recently tried this flavor/brand and was surprised at how delicious these chips are.  The best thing was that there were a lot of \"brown\" chips in the bsg (my favorite), so I bought some more through amazon and shared with family and friends.  I am a little disappointed that there are not, so far, very many brown chips in these bags, but the flavor is still very good.  I like them better than the yogurt and green onion flavor because they do not seem to be as salty, and the onion flavor is better.  If you haven't eaten Kettle chips before, I recommend that you try a bag before buying bulk.  They are thicker and crunchier than Lays but just as fresh out of the bag.\n",
            "==================================================\n",
            "Wow.  So far, two two-star reviews.  One obviously had no idea what they were ordering; the other wants crispy cookies.  Hey, I'm sorry; but these reviews do nobody any good beyond reminding us to look  before ordering.<br /><br />These are chocolate-oatmeal cookies.  If you don't like that combination, don't order this type of cookie.  I find the combo quite nice, really.  The oatmeal sort of \"calms\" the rich chocolate flavor and gives the cookie sort of a coconut-type consistency.  Now let's also remember that tastes differ; so, I've given my opinion.<br /><br />Then, these are soft, chewy cookies -- as advertised.  They are not \"crispy\" cookies, or the blurb would say \"crispy,\" rather than \"chewy.\"  I happen to like raw cookie dough; however, I don't see where these taste like raw cookie dough.  Both are soft, however, so is this the confusion?  And, yes, they stick together.  Soft cookies tend to do that.  They aren't individually wrapped, which would add to the cost.  Oh yeah, chocolate chip cookies tend to be somewhat sweet.<br /><br />So, if you want something hard and crisp, I suggest Nabiso's Ginger Snaps.  If you want a cookie that's soft, chewy and tastes like a combination of chocolate and oatmeal, give these a try.  I'm here to place my second order.\n",
            "==================================================\n",
            "love to order my coffee on amazon.  easy and shows up quickly.<br />This k cup is great coffee.  dcaf is very good as well\n",
            "==================================================\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MvD0JaQ9CAZx",
        "outputId": "dcc683d4-6014-4430-fb91-f9526bca4475"
      },
      "source": [
        "# remove urls from text python: https://stackoverflow.com/a/40823105/4084039\n",
        "sent_0 = re.sub(r\"http\\S+\", \"\", sent_0)\n",
        "sent_1000 = re.sub(r\"http\\S+\", \"\", sent_1000)\n",
        "sent_150 = re.sub(r\"http\\S+\", \"\", sent_1500)\n",
        "sent_4900 = re.sub(r\"http\\S+\", \"\", sent_4900)\n",
        "\n",
        "print(sent_0)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Why is this $[...] when the same product is available for $[...] here?<br /> /><br />The Victor M380 and M502 traps are unreal, of course -- total fly genocide. Pretty stinky, but only right nearby.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Liu2zNFLCAZ0",
        "outputId": "f66770ef-17d3-4a99-df7f-75242858701d"
      },
      "source": [
        "# https://stackoverflow.com/questions/16206380/python-beautifulsoup-how-to-remove-all-tags-from-an-element\n",
        "from bs4 import BeautifulSoup\n",
        "\n",
        "soup = BeautifulSoup(sent_0, 'lxml')\n",
        "text = soup.get_text()\n",
        "print(text)\n",
        "print(\"=\"*50)\n",
        "\n",
        "soup = BeautifulSoup(sent_1000, 'lxml')\n",
        "text = soup.get_text()\n",
        "print(text)\n",
        "print(\"=\"*50)\n",
        "\n",
        "soup = BeautifulSoup(sent_1500, 'lxml')\n",
        "text = soup.get_text()\n",
        "print(text)\n",
        "print(\"=\"*50)\n",
        "\n",
        "soup = BeautifulSoup(sent_4900, 'lxml')\n",
        "text = soup.get_text()\n",
        "print(text)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Why is this $[...] when the same product is available for $[...] here? />The Victor M380 and M502 traps are unreal, of course -- total fly genocide. Pretty stinky, but only right nearby.\n",
            "==================================================\n",
            "I recently tried this flavor/brand and was surprised at how delicious these chips are.  The best thing was that there were a lot of \"brown\" chips in the bsg (my favorite), so I bought some more through amazon and shared with family and friends.  I am a little disappointed that there are not, so far, very many brown chips in these bags, but the flavor is still very good.  I like them better than the yogurt and green onion flavor because they do not seem to be as salty, and the onion flavor is better.  If you haven't eaten Kettle chips before, I recommend that you try a bag before buying bulk.  They are thicker and crunchier than Lays but just as fresh out of the bag.\n",
            "==================================================\n",
            "Wow.  So far, two two-star reviews.  One obviously had no idea what they were ordering; the other wants crispy cookies.  Hey, I'm sorry; but these reviews do nobody any good beyond reminding us to look  before ordering.These are chocolate-oatmeal cookies.  If you don't like that combination, don't order this type of cookie.  I find the combo quite nice, really.  The oatmeal sort of \"calms\" the rich chocolate flavor and gives the cookie sort of a coconut-type consistency.  Now let's also remember that tastes differ; so, I've given my opinion.Then, these are soft, chewy cookies -- as advertised.  They are not \"crispy\" cookies, or the blurb would say \"crispy,\" rather than \"chewy.\"  I happen to like raw cookie dough; however, I don't see where these taste like raw cookie dough.  Both are soft, however, so is this the confusion?  And, yes, they stick together.  Soft cookies tend to do that.  They aren't individually wrapped, which would add to the cost.  Oh yeah, chocolate chip cookies tend to be somewhat sweet.So, if you want something hard and crisp, I suggest Nabiso's Ginger Snaps.  If you want a cookie that's soft, chewy and tastes like a combination of chocolate and oatmeal, give these a try.  I'm here to place my second order.\n",
            "==================================================\n",
            "love to order my coffee on amazon.  easy and shows up quickly.This k cup is great coffee.  dcaf is very good as well\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lNHWbzBaCAZ3"
      },
      "source": [
        "# https://stackoverflow.com/a/47091490/4084039\n",
        "import re\n",
        "\n",
        "def decontracted(phrase):\n",
        "    # specific\n",
        "    phrase = re.sub(r\"won't\", \"will not\", phrase)\n",
        "    phrase = re.sub(r\"can\\'t\", \"can not\", phrase)\n",
        "\n",
        "    # general\n",
        "    phrase = re.sub(r\"n\\'t\", \" not\", phrase)\n",
        "    phrase = re.sub(r\"\\'re\", \" are\", phrase)\n",
        "    phrase = re.sub(r\"\\'s\", \" is\", phrase)\n",
        "    phrase = re.sub(r\"\\'d\", \" would\", phrase)\n",
        "    phrase = re.sub(r\"\\'ll\", \" will\", phrase)\n",
        "    phrase = re.sub(r\"\\'t\", \" not\", phrase)\n",
        "    phrase = re.sub(r\"\\'ve\", \" have\", phrase)\n",
        "    phrase = re.sub(r\"\\'m\", \" am\", phrase)\n",
        "    return phrase"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y7v1fmngCAZ5",
        "outputId": "22e3fcb6-c2e6-4c92-be48-f65543b1140f"
      },
      "source": [
        "sent_1500 = decontracted(sent_1500)\n",
        "print(sent_1500)\n",
        "print(\"=\"*50)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Wow.  So far, two two-star reviews.  One obviously had no idea what they were ordering; the other wants crispy cookies.  Hey, I am sorry; but these reviews do nobody any good beyond reminding us to look  before ordering.<br /><br />These are chocolate-oatmeal cookies.  If you do not like that combination, do not order this type of cookie.  I find the combo quite nice, really.  The oatmeal sort of \"calms\" the rich chocolate flavor and gives the cookie sort of a coconut-type consistency.  Now let is also remember that tastes differ; so, I have given my opinion.<br /><br />Then, these are soft, chewy cookies -- as advertised.  They are not \"crispy\" cookies, or the blurb would say \"crispy,\" rather than \"chewy.\"  I happen to like raw cookie dough; however, I do not see where these taste like raw cookie dough.  Both are soft, however, so is this the confusion?  And, yes, they stick together.  Soft cookies tend to do that.  They are not individually wrapped, which would add to the cost.  Oh yeah, chocolate chip cookies tend to be somewhat sweet.<br /><br />So, if you want something hard and crisp, I suggest Nabiso is Ginger Snaps.  If you want a cookie that is soft, chewy and tastes like a combination of chocolate and oatmeal, give these a try.  I am here to place my second order.\n",
            "==================================================\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jKIBLrAjCAZ7",
        "outputId": "376f1e55-511a-4352-c345-a39f1d2a2224"
      },
      "source": [
        "#remove words with numbers python: https://stackoverflow.com/a/18082370/4084039\n",
        "sent_0 = re.sub(\"\\S*\\d\\S*\", \"\", sent_0).strip()\n",
        "print(sent_0)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Why is this $[...] when the same product is available for $[...] here?<br /> /><br />The Victor  and  traps are unreal, of course -- total fly genocide. Pretty stinky, but only right nearby.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sU3eY6geCAZ9",
        "outputId": "90d91c93-0d77-47c0-e496-77d1611a86fb"
      },
      "source": [
        "#remove spacial character: https://stackoverflow.com/a/5843547/4084039\n",
        "sent_1500 = re.sub('[^A-Za-z0-9]+', ' ', sent_1500)\n",
        "print(sent_1500)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Wow So far two two star reviews One obviously had no idea what they were ordering the other wants crispy cookies Hey I am sorry but these reviews do nobody any good beyond reminding us to look before ordering br br These are chocolate oatmeal cookies If you do not like that combination do not order this type of cookie I find the combo quite nice really The oatmeal sort of calms the rich chocolate flavor and gives the cookie sort of a coconut type consistency Now let is also remember that tastes differ so I have given my opinion br br Then these are soft chewy cookies as advertised They are not crispy cookies or the blurb would say crispy rather than chewy I happen to like raw cookie dough however I do not see where these taste like raw cookie dough Both are soft however so is this the confusion And yes they stick together Soft cookies tend to do that They are not individually wrapped which would add to the cost Oh yeah chocolate chip cookies tend to be somewhat sweet br br So if you want something hard and crisp I suggest Nabiso is Ginger Snaps If you want a cookie that is soft chewy and tastes like a combination of chocolate and oatmeal give these a try I am here to place my second order \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xsr4xHkJCAaA"
      },
      "source": [
        "# https://gist.github.com/sebleier/554280\n",
        "# we are removing the words from the stop words list: 'no', 'nor', 'not'\n",
        "# <br /><br /> ==> after the above steps, we are getting \"br br\"\n",
        "# we are including them into stop words list\n",
        "# instead of <br /> if we have <br/> these tags would have revmoved in the 1st step\n",
        "\n",
        "stopwords= set(['br', 'the', 'i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\",\\\n",
        "            \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', \\\n",
        "            'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their',\\\n",
        "            'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', \\\n",
        "            'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', \\\n",
        "            'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', \\\n",
        "            'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after',\\\n",
        "            'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further',\\\n",
        "            'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more',\\\n",
        "            'most', 'other', 'some', 'such', 'only', 'own', 'same', 'so', 'than', 'too', 'very', \\\n",
        "            's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', \\\n",
        "            've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn',\\\n",
        "            \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn',\\\n",
        "            \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", \\\n",
        "            'won', \"won't\", 'wouldn', \"wouldn't\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MgsJ09NpCAaB",
        "outputId": "1d7ef7e7-d03e-4bff-850a-a2aef70af7d9"
      },
      "source": [
        "# Combining all the above stundents \n",
        "from tqdm import tqdm\n",
        "preprocessed_reviews = []\n",
        "# tqdm is for printing the status bar\n",
        "for sentance in tqdm(final['Text'].values):\n",
        "    sentance = re.sub(r\"http\\S+\", \"\", sentance)\n",
        "    sentance = BeautifulSoup(sentance, 'lxml').get_text()\n",
        "    sentance = decontracted(sentance)\n",
        "    sentance = re.sub(\"\\S*\\d\\S*\", \"\", sentance).strip()\n",
        "    sentance = re.sub('[^A-Za-z]+', ' ', sentance)\n",
        "    # https://gist.github.com/sebleier/554280\n",
        "    sentance = ' '.join(e.lower() for e in sentance.split() if e.lower() not in stopwords)\n",
        "    preprocessed_reviews.append(sentance.strip())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|████████████████████████████████████████████████████████████████████████████| 4986/4986 [00:01<00:00, 3137.37it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f_zfwXXWCAaE",
        "outputId": "a8ba4cbf-3a4c-4447-f06a-2b639e1a5f88"
      },
      "source": [
        "preprocessed_reviews[1500]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'wow far two two star reviews one obviously no idea ordering wants crispy cookies hey sorry reviews nobody good beyond reminding us look ordering chocolate oatmeal cookies not like combination not order type cookie find combo quite nice really oatmeal sort calms rich chocolate flavor gives cookie sort coconut type consistency let also remember tastes differ given opinion soft chewy cookies advertised not crispy cookies blurb would say crispy rather chewy happen like raw cookie dough however not see taste like raw cookie dough soft however confusion yes stick together soft cookies tend not individually wrapped would add cost oh yeah chocolate chip cookies tend somewhat sweet want something hard crisp suggest nabiso ginger snaps want cookie soft chewy tastes like combination chocolate oatmeal give try place second order'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tmeU-LoVCAaI"
      },
      "source": [
        "<h2><font color='red'>[3.2] Preprocess Summary</font></h2>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wtJ2hcy9CAaI"
      },
      "source": [
        "## Similartly you can do preprocessing for review summary also."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ytBZVEs2CAaL"
      },
      "source": [
        "# [4] Featurization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9amw6tMZCAaL"
      },
      "source": [
        "## [4.1] BAG OF WORDS"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3hasoJ-_CAaN",
        "outputId": "12aa3f3f-083c-4de4-a6e7-3dbde6b5a905"
      },
      "source": [
        "#BoW\n",
        "count_vect = CountVectorizer() #in scikit-learn\n",
        "count_vect.fit(preprocessed_reviews)\n",
        "print(\"some feature names \", count_vect.get_feature_names()[:10])\n",
        "print('='*50)\n",
        "\n",
        "final_counts = count_vect.transform(preprocessed_reviews)\n",
        "print(\"the type of count vectorizer \",type(final_counts))\n",
        "print(\"the shape of out text BOW vectorizer \",final_counts.get_shape())\n",
        "print(\"the number of unique words \", final_counts.get_shape()[1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "some feature names  ['aa', 'aahhhs', 'aback', 'abandon', 'abates', 'abbott', 'abby', 'abdominal', 'abiding', 'ability']\n",
            "==================================================\n",
            "the type of count vectorizer  <class 'scipy.sparse.csr.csr_matrix'>\n",
            "the shape of out text BOW vectorizer  (4986, 12997)\n",
            "the number of unique words  12997\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1dhfJZ7ZCAaR"
      },
      "source": [
        "## [4.2] Bi-Grams and n-Grams."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u1Ma2LBfCAaR",
        "outputId": "a5208a4c-c0ad-4360-f021-b2e8715610a4"
      },
      "source": [
        "#bi-gram, tri-gram and n-gram\n",
        "\n",
        "#removing stop words like \"not\" should be avoided before building n-grams\n",
        "# count_vect = CountVectorizer(ngram_range=(1,2))\n",
        "# please do read the CountVectorizer documentation http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html\n",
        "# you can choose these numebrs min_df=10, max_features=5000, of your choice\n",
        "count_vect = CountVectorizer(ngram_range=(1,2), min_df=10, max_features=5000)\n",
        "final_bigram_counts = count_vect.fit_transform(preprocessed_reviews)\n",
        "print(\"the type of count vectorizer \",type(final_bigram_counts))\n",
        "print(\"the shape of out text BOW vectorizer \",final_bigram_counts.get_shape())\n",
        "print(\"the number of unique words including both unigrams and bigrams \", final_bigram_counts.get_shape()[1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "the type of count vectorizer  <class 'scipy.sparse.csr.csr_matrix'>\n",
            "the shape of out text BOW vectorizer  (4986, 3144)\n",
            "the number of unique words including both unigrams and bigrams  3144\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F--Xk5fhCAaV"
      },
      "source": [
        "## [4.3] TF-IDF"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6heiZFZ-CAaW",
        "outputId": "08103e90-4bd8-410e-b3dc-84a02e01aa33"
      },
      "source": [
        "tf_idf_vect = TfidfVectorizer(ngram_range=(1,2), min_df=10)\n",
        "tf_idf_vect.fit(preprocessed_reviews)\n",
        "print(\"some sample features(unique words in the corpus)\",tf_idf_vect.get_feature_names()[0:10])\n",
        "print('='*50)\n",
        "\n",
        "final_tf_idf = tf_idf_vect.transform(preprocessed_reviews)\n",
        "print(\"the type of count vectorizer \",type(final_tf_idf))\n",
        "print(\"the shape of out text TFIDF vectorizer \",final_tf_idf.get_shape())\n",
        "print(\"the number of unique words including both unigrams and bigrams \", final_tf_idf.get_shape()[1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "some sample features(unique words in the corpus) ['ability', 'able', 'able find', 'able get', 'absolute', 'absolutely', 'absolutely delicious', 'absolutely love', 'absolutely no', 'according']\n",
            "==================================================\n",
            "the type of count vectorizer  <class 'scipy.sparse.csr.csr_matrix'>\n",
            "the shape of out text TFIDF vectorizer  (4986, 3144)\n",
            "the number of unique words including both unigrams and bigrams  3144\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XnzP-eZdCAaa"
      },
      "source": [
        "## [4.4] Word2Vec"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e-IuUZsTCAaa"
      },
      "source": [
        "# Train your own Word2Vec model using your own text corpus\n",
        "i=0\n",
        "list_of_sentance=[]\n",
        "for sentance in preprocessed_reviews:\n",
        "    list_of_sentance.append(sentance.split())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "GIbKBSkRCAac",
        "outputId": "d72c6206-2c3f-4143-8c21-3f5b674310df"
      },
      "source": [
        "# Using Google News Word2Vectors\n",
        "\n",
        "# in this project we are using a pretrained model by google\n",
        "# its 3.3G file, once you load this into your memory \n",
        "# it occupies ~9Gb, so please do this step only if you have >12G of ram\n",
        "# we will provide a pickle file wich contains a dict , \n",
        "# and it contains all our courpus words as keys and  model[word] as values\n",
        "# To use this code-snippet, download \"GoogleNews-vectors-negative300.bin\" \n",
        "# from https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit\n",
        "# it's 1.9GB in size.\n",
        "\n",
        "\n",
        "# http://kavita-ganesan.com/gensim-word2vec-tutorial-starter-code/#.W17SRFAzZPY\n",
        "# you can comment this whole cell\n",
        "# or change these varible according to your need\n",
        "\n",
        "is_your_ram_gt_16g=False\n",
        "want_to_use_google_w2v = False\n",
        "want_to_train_w2v = True\n",
        "\n",
        "if want_to_train_w2v:\n",
        "    # min_count = 5 considers only words that occured atleast 5 times\n",
        "    w2v_model=Word2Vec(list_of_sentance,min_count=5,size=50, workers=4)\n",
        "    print(w2v_model.wv.most_similar('great'))\n",
        "    print('='*50)\n",
        "    print(w2v_model.wv.most_similar('worst'))\n",
        "    \n",
        "elif want_to_use_google_w2v and is_your_ram_gt_16g:\n",
        "    if os.path.isfile('GoogleNews-vectors-negative300.bin'):\n",
        "        w2v_model=KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)\n",
        "        print(w2v_model.wv.most_similar('great'))\n",
        "        print(w2v_model.wv.most_similar('worst'))\n",
        "    else:\n",
        "        print(\"you don't have gogole's word2vec file, keep want_to_train_w2v = True, to train your own w2v \")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('snack', 0.9951335191726685), ('calorie', 0.9946465492248535), ('wonderful', 0.9946032166481018), ('excellent', 0.9944332838058472), ('especially', 0.9941144585609436), ('baked', 0.9940600395202637), ('salted', 0.994047224521637), ('alternative', 0.9937226176261902), ('tasty', 0.9936816692352295), ('healthy', 0.9936649799346924)]\n",
            "==================================================\n",
            "[('varieties', 0.9994194507598877), ('become', 0.9992934465408325), ('popcorn', 0.9992750883102417), ('de', 0.9992610216140747), ('miss', 0.9992451071739197), ('melitta', 0.999218761920929), ('choice', 0.9992102384567261), ('american', 0.9991837739944458), ('beef', 0.9991780519485474), ('finish', 0.9991567134857178)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JEJGArtUCAae",
        "outputId": "943e0fc6-83f8-455b-ba53-8dd05428fc92"
      },
      "source": [
        "w2v_words = list(w2v_model.wv.vocab)\n",
        "print(\"number of words that occured minimum 5 times \",len(w2v_words))\n",
        "print(\"sample words \", w2v_words[0:50])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "number of words that occured minimum 5 times  3817\n",
            "sample words  ['product', 'available', 'course', 'total', 'pretty', 'stinky', 'right', 'nearby', 'used', 'ca', 'not', 'beat', 'great', 'received', 'shipment', 'could', 'hardly', 'wait', 'try', 'love', 'call', 'instead', 'removed', 'easily', 'daughter', 'designed', 'printed', 'use', 'car', 'windows', 'beautifully', 'shop', 'program', 'going', 'lot', 'fun', 'everywhere', 'like', 'tv', 'computer', 'really', 'good', 'idea', 'final', 'outstanding', 'window', 'everybody', 'asks', 'bought', 'made']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EPjGCg7UCAag"
      },
      "source": [
        "## [4.4.1] Converting text into vectors using wAvg W2V, TFIDF-W2V"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oPxfYXhMCAag"
      },
      "source": [
        "#### [4.4.1.1] Avg W2v"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sB4Y18rQCAag",
        "outputId": "c9f64dac-cc89-43e3-9820-fbc18c39a69e"
      },
      "source": [
        "# average Word2Vec\n",
        "# compute average word2vec for each review.\n",
        "sent_vectors = []; # the avg-w2v for each sentence/review is stored in this list\n",
        "for sent in tqdm(list_of_sentance): # for each review/sentence\n",
        "    sent_vec = np.zeros(50) # as word vectors are of zero length 50, you might need to change this to 300 if you use google's w2v\n",
        "    cnt_words =0; # num of words with a valid vector in the sentence/review\n",
        "    for word in sent: # for each word in a review/sentence\n",
        "        if word in w2v_words:\n",
        "            vec = w2v_model.wv[word]\n",
        "            sent_vec += vec\n",
        "            cnt_words += 1\n",
        "    if cnt_words != 0:\n",
        "        sent_vec /= cnt_words\n",
        "    sent_vectors.append(sent_vec)\n",
        "print(len(sent_vectors))\n",
        "print(len(sent_vectors[0]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|████████████████████████████████████████████████████████████████████████████| 4986/4986 [00:03<00:00, 1330.47it/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "4986\n",
            "50\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sKAEsZZLCAam"
      },
      "source": [
        "#### [4.4.1.2] TFIDF weighted W2v"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pAVTG3brCAao"
      },
      "source": [
        "# S = [\"abc def pqr\", \"def def def abc\", \"pqr pqr def\"]\n",
        "model = TfidfVectorizer()\n",
        "model.fit(preprocessed_reviews)\n",
        "# we are converting a dictionary with word as a key, and the idf as a value\n",
        "dictionary = dict(zip(model.get_feature_names(), list(model.idf_)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "Tyxhz3XQCAap",
        "outputId": "e72f3ca0-7d29-4657-a107-c5d678514cf3"
      },
      "source": [
        "# TF-IDF weighted Word2Vec\n",
        "tfidf_feat = model.get_feature_names() # tfidf words/col-names\n",
        "# final_tf_idf is the sparse matrix with row= sentence, col=word and cell_val = tfidf\n",
        "\n",
        "tfidf_sent_vectors = []; # the tfidf-w2v for each sentence/review is stored in this list\n",
        "row=0;\n",
        "for sent in tqdm(list_of_sentance): # for each review/sentence \n",
        "    sent_vec = np.zeros(50) # as word vectors are of zero length\n",
        "    weight_sum =0; # num of words with a valid vector in the sentence/review\n",
        "    for word in sent: # for each word in a review/sentence\n",
        "        if word in w2v_words and word in tfidf_feat:\n",
        "            vec = w2v_model.wv[word]\n",
        "#             tf_idf = tf_idf_matrix[row, tfidf_feat.index(word)]\n",
        "            # to reduce the computation we are \n",
        "            # dictionary[word] = idf value of word in whole courpus\n",
        "            # sent.count(word) = tf valeus of word in this review\n",
        "            tf_idf = dictionary[word]*(sent.count(word)/len(sent))\n",
        "            sent_vec += (vec * tf_idf)\n",
        "            weight_sum += tf_idf\n",
        "    if weight_sum != 0:\n",
        "        sent_vec /= weight_sum\n",
        "    tfidf_sent_vectors.append(sent_vec)\n",
        "    row += 1"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|█████████████████████████████████████████████████████████████████████████████| 4986/4986 [00:20<00:00, 245.63it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    }
  ]
}