robots_txt_Sitemap_Testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/ranksense/Twittorials/blob/master/robots_txt_Sitemap_Testing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_Hl2WtsFaFMh"
   },
   "source": [
    "# 🐍🔥\n",
    "#  Is your `robots.txt` file blocking any of the URLs in your sitemap?  \n",
    "\n",
    "We know this generally shouldn't happen, but we also know that it might!  \n",
    "\n",
    "Is it correctly blocking certain URLs for certain User-agents, or is there an error somewhere?  \n",
    "This is quick testing tool that allows you to do just that. \n",
    "\n",
    "The plan: \n",
    "\n",
    "1. Specify a `robots.txt` file URL\n",
    "2. Extract and select one of the available sitemaps\n",
    "3. Download the sitemap\n",
    "4. Extract the `User-agent`s\n",
    "5. For each (User-agent, URL) combination, check whether the user-agent can fetch the URL\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6FXZPTz7idfR"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install advertools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iuC7wVawijqQ"
   },
   "source": [
    "Import the required packages, and display their version numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "p2WE1c3XVwdH",
    "outputId": "96f58b80-15b5-4623-bb53-2faed2fae7e4"
   },
   "outputs": [],
   "source": [
    "import advertools as adv\n",
    "import pandas as pd\n",
    "pd.options.display.max_columns = None\n",
    "for p in [adv, pd]:\n",
    "    print(f'{p.__name__:<13}', 'v' + p.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "RvSV2dtdWM9d"
   },
   "outputs": [],
   "source": [
    "#@title Please enter a robots.txt URL e.g. https://www.nytimes.com/robots.txt\n",
    "robotstxt_url = \"\" #@param {type:\"string\"}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1_0Hvp2eWUR5"
   },
   "outputs": [],
   "source": [
    "robotstxt_df = adv.robotstxt_to_df(robotstxt_url)\n",
    "robotstxt_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4qithSoXczeb"
   },
   "source": [
    "#### Extract sitemaps from `robots.txt`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GTtbN6EVWils"
   },
   "outputs": [],
   "source": [
    "sitemaps = (robotstxt_df                             # take the robotstxt_df DataFrame\n",
    "            [robotstxt_df['directive']               # select its \"directive\" column\n",
    "             .str.contains('^sitemap$', case=False)] # filter values that contain \"sitemap\" case insensitive\n",
    "            ['content']                              # now select the \"content\" column\n",
    "            .tolist())                               # convert it to a list\n",
    "\n",
    "sitemaps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "jJVT7Ir_XQyd"
   },
   "outputs": [],
   "source": [
    "#@title Enter one of the sitemaps extracted (if none exist, try to get one from the website). If you provide a sitemap index, you will get all the sub-sitemaps (might take long with large websites). Try with a regular sitemap for faster results.\n",
    "sitemap = \"\" #@param {type:\"string\"}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hkf1WvdtcvTa"
   },
   "source": [
    "#### Download selected sitemap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YKZOhcJDYbuY"
   },
   "outputs": [],
   "source": [
    "sitemap_df = adv.sitemap_to_df(sitemap)\n",
    "print('Sitemap rows:', sitemap_df.shape[0])\n",
    "print('Sitemap columns:', sitemap_df.shape[1])\n",
    "sitemap_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i_a6D-hXcoPL"
   },
   "source": [
    "#### Extract `User-agent`s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "eZuEyr8PY0uo"
   },
   "outputs": [],
   "source": [
    "user_agents = (robotstxt_df                              # take the robotstxt_df DataFrame\n",
    "               [robotstxt_df['directive']                # select its \"directive\" column\n",
    "                .str.contains('user-agent',case=False)]  # filter values that contain \"user-agent\" case insensitive\n",
    "               ['content']                               # now select the \"content\" column\n",
    "               .tolist())                                # convert it to a list\n",
    "user_agents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "K4M9O4RDetwx"
   },
   "source": [
    "#### Run the report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xvLPonVKYp5t"
   },
   "outputs": [],
   "source": [
    "robots_test_report = adv.robotstxt_test(robotstxt_url, user_agents, sitemap_df['loc'])\n",
    "robots_test_report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Homework\n",
    "\n",
    "Now that you have parsed the XML sitemap, and got all its URLs, how can you further analyze those URLs?\n",
    "\n",
    "Hint: `adv.url_to_df`"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "robots_txt_sitemap_test.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}