Sitemap_Canonical_Information.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Sitemap Canonical Information",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/anirudh-tatavarthi/Twittorials/blob/master/Sitemap_Canonical_Information.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nAtF3YV2Ddgf",
        "colab_type": "text"
      },
      "source": [
        "# Hello there! This script will allow you to verify the quality of your sitemap regarding the canonical tags and the status codes of the URLs included in the sitemap"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZEqer8ezyg3O",
        "colab_type": "text"
      },
      "source": [
        "**Note:** All of the URLs in a sitemap should contain canonical tags and return a 200 status code. With this script, you can pinpoint URLs that break this guidelines."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MmQvIteWA-t8",
        "colab_type": "text"
      },
      "source": [
        "[1] This first block installs the necessary libraries for the code to run"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gqrlp8T-s7PS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "!pip install --upgrade -q gspread\n",
        "!pip install requests-html"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JIuB3EG1BGup",
        "colab_type": "text"
      },
      "source": [
        "[2] The block below imports everything needed"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8l5ffiD9sXEp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import gspread\n",
        "from google.colab import auth\n",
        "from bs4 import BeautifulSoup\n",
        "import requests\n",
        "auth.authenticate_user()\n",
        "from requests_html import HTMLSession\n",
        "from oauth2client.client import GoogleCredentials\n",
        "gc = gspread.authorize(GoogleCredentials.get_application_default())"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uEnMXTGUBmAL",
        "colab_type": "text"
      },
      "source": [
        "[3] When you run the block above, you will have to provide an **authorization code** from the link provided"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z8SsWBoGBw0q",
        "colab_type": "text"
      },
      "source": [
        "[4] Next, you will have to **enter the name of the spreadsheet** which you would like the results to be written on. Feel free to make a copy of this template: https://docs.google.com/spreadsheets/d/1KH1Jxsx77xwv-1wbrq3K5_Ga6-jUGibYWSIPBQw46IA/edit?usp=sharing\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-dQUHv-B83H",
        "colab_type": "text"
      },
      "source": [
        "**Note:** You can also specify which columns you would like each piece of information to be displayed in. (You can keep the values already listed)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h9fVFnGasb0S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Replace with your spreadsheet name\n",
        "spreadsheetName = \"Sitemap Canonical Information [Template]\" #@param {type:\"string\"}\n",
        "\n",
        "sheet = gc.open(spreadsheetName).sheet1\n",
        "\n",
        "#Enter the column you'd like the URLs to be displayed:\n",
        "URLcol =   1#@param {type:\"number\"}\n",
        "\n",
        "#Enter the column you'd like the canonical URLs to be displayed:\n",
        "Canonicalcol =   2#@param {type:\"number\"}\n",
        "\n",
        "#Enter the column you'd like the self-referential canonical information to be displayed:\n",
        "ContainsSelfRefCanonicalCol =   3#@param {type:\"number\"}\n",
        "\n",
        "#Enter the column you'd like the status code to be displayed:\n",
        "StatusCodecol =   4#@param {type:\"number\"}"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G6bljLdWC31p",
        "colab_type": "text"
      },
      "source": [
        "[5] First, you will be prompted to **enter the URL of the sitemap** which you would like to use"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tA0Ip1gjDDId",
        "colab_type": "text"
      },
      "source": [
        "[6] The code will go through the specified sitemap and output the URLs contained in the sitemap, the canonical URL (if applicable), whether or not the canonical is self-referential, and the status code."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ymowbGRWswsz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        },
        "outputId": "4d62b6b7-6fd2-4571-9201-3965e00702bf"
      },
      "source": [
        "session = HTMLSession()\n",
        "\n",
        "#Prompts user to input a sitemap URL to analyze\n",
        "sitemap_url = input(\"Please enter the sitemap URL: \")\n",
        "\n",
        "#Finds any rel=canonical and grabs the href\n",
        "canonical_xpath = \"//link[@rel='canonical']/@href\"\n",
        "\n",
        "#Stores the webpage from the URL in a response object (r)\n",
        "r = requests.get(sitemap_url)\n",
        "xml=r.text\n",
        "\n",
        "#We use beautiful soup to fetch the data we need from the HTML of the sitemap\n",
        "soup = BeautifulSoup(xml, \"html.parser\")\n",
        "#find the \"loc\" tags which include the URL\n",
        "URLs = soup.find_all('loc')\n",
        "\n",
        "print(\"The number of URLs found in the sitemap are {0}\".format(len(URLs)))\n",
        "num_of_URLs = len(URLs)\n",
        "\n",
        "#If you would like your output to start on a different row, modify this variable\n",
        "start_row =   2#@param {type:\"number\"}"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Please enter the sitemap URL: https://www.wawak.com/products-sitemap.xml\n",
            "The number of URLs found in the sitemap are 1894\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0kWkLm6wCtDE",
        "colab_type": "text"
      },
      "source": [
        "[7] This next block of code gathers all of the information and updates the google sheet. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gtjD-nFYAn89",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for each in URLs:\n",
        "  #Grab the text within the loc tag \n",
        "  url = str(each.get_text())\n",
        "\n",
        "  \"\"\"\n",
        "  Since we already parsed the HTML with beautiful soup, \n",
        "  requests obtains the information from the HTML snippets\n",
        "  \"\"\"\n",
        "  with session.get(str(each.get_text())) as r:\n",
        "\n",
        "    #canonical informaiton stored in the variable\n",
        "    canonical=r.html.xpath(canonical_xpath, first=True)\n",
        "    #update URL column and canonical column\n",
        "    sheet.update_cell(start_row, URLcol, url)\n",
        "    sheet.update_cell(start_row, Canonicalcol, canonical)\n",
        "\n",
        "    #Checks for self-referential canonical or no canonical\n",
        "    if canonical==url:\n",
        "      sheet.update_cell(start_row, ContainsSelfRefCanonicalCol, \"True\")\n",
        "    elif canonical == None:\n",
        "      sheet.update_cell(start_row, ContainsSelfRefCanonicalCol, \"N/A\")\n",
        "    else:\n",
        "      sheet.update_cell(start_row, ContainsSelfRefCanonicalCol, \"False\")\n",
        "\n",
        "    #URL information stored in the response object\n",
        "    resp = requests.get(url)\n",
        "    #grabs the status code of the URL\n",
        "    status_code = resp.status_code\n",
        "    #update the status code column\n",
        "    sheet.update_cell(start_row, StatusCodecol, status_code)\n",
        "\n",
        "    start_row+=1\n",
        "\n",
        "print(\"Sitemap crawl has finished!\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hmr99pxAZ2zo",
        "colab_type": "text"
      },
      "source": [
        "**Note:** Be sure to resize your sheet according to the number of URLs present in your sitemap"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "voZs54UeDxZe",
        "colab_type": "text"
      },
      "source": [
        "[8] Thats it! View your google sheet to see the magic happen! "
      ]
    }
  ]
}