Analyzing News Articles With Python (Part 1) - Gathering News Sources.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing News Articles With Python (Part 1)\n",
    "\n",
    "*By Andre Sealy (Updated 6/21/2020)*\n",
    "\n",
    "This is the first part of what I believe to be a multi-part series in analyzing news articles on Python.\n",
    "\n",
    "Never before has America become more polarized than we are today. As such, the news we consume is a function of our polarization. Right-leaning people tend to get their information from Fox News, The Wall Street Journal, National Review, etc. People who lean left tend to get their information from MSNBC, The New York Times, The Huffington Post, etc. There are a handful of neutral publications (Reuters, Politico, Associated Press, etc.). Still, even their alignment falls into question, based on how they report the news.\n",
    "\n",
    "This project will attempt to analyze how polarizing mainstream articles are for stories typical in the U.S. news cycle, as well as measuring the sentiment for each item. To help guide us for this project, we will be using the website All Sides.\n",
    "\n",
    "### All Sides News Aggregator\n",
    "\n",
    "All Sides is a bipartisan organization that looks at a more balanced approach to news coverage by collecting the top headlines of the day and showcasing the reporting of the news outlet on the left, right, and center. The platform also allows readers the rate the lean of the publication for further analysis.\n",
    "\n",
    "#### Importing the Modules\n",
    "\n",
    "Let's begin to load the necessary modules for the project, which include the following:\n",
    "\n",
    "* **requests**\n",
    "* **BeautifulSoup**\n",
    "* **csv**\n",
    "* **re**\n",
    "* **socket**\n",
    "\n",
    "The `requests` module allows you to send HTTP requests using Python, which will enable you to inspect the structure of many different websites. From there, you will be able to scrape the relevant information of the website with the `BeautifulSoup` module. Regular Expressions or the `re` module allows Python to look for specific text that follows a particular pattern. (doing this technique several times, you'll soon realize that certain outlets follow a pattern when publishing articles) \n",
    "\n",
    "The `csv` module will just allow us to create and save our information to a csv for further analysis later on. There is nothing reall important about the `socket` module; it's used just in case we run into an error during the extraction process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import csv\n",
    "import re\n",
    "import socket"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first mode of attack is figuring out how we can go about extracting these stories for our analysis. We can extract all stories from all publications, but what if we want a more targeted focus? One interesting feature of the All Sides website is that we can look for articles based on the topic.\n",
    "\n",
    "![](https://kidquant.com/post/images/Analyzing-News/topics-allsides.PNG)\n",
    "\n",
    "We can get articles from pretty much any topic: **Criminal Justice**, **Education**, and the **Economy**. All Sides even collects the news perspectives on the most important topic in the world right now, the Coronavirus. (Yes, our society is so divided right now, we've even managed to politize a deadly virus)\n",
    "\n",
    "I'm going to use immigration as our topic; I feel it's pretty easy to understand where both sides (the political left and right) stand on this issue. \n",
    "\n",
    "The following code chunk will extract the most important news stories involving immigration within the last two years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://www.allsides.com/story/admin?tid=&field_story_topic_tid=Immigration&page=0',\n",
       " 'https://www.allsides.com/story/admin?tid=&field_story_topic_tid=Immigration&page=1']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# creates a empty list to store the story pages from AllSides.com\n",
    "pages = []\n",
    "\n",
    "# We only want to extract stories about immigration\n",
    "story = 'Immigration'\n",
    "\n",
    "\n",
    "def get_seed(n):\n",
    "    \"\"\"\n",
    "    n defines the number of pages back to pull\n",
    "    n=1 steps back to April 2018 (as of April 2020)\n",
    "    \"\"\"\n",
    "    for i in range(0, n+1):\n",
    "        url = 'https://www.allsides.com/story/admin?tid=&field_story_topic_tid=' + \\\n",
    "            str(story) + '&page=' + str(i)\n",
    "        pages.append(url)\n",
    "\n",
    "get_seed(1)\n",
    "\n",
    "pages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output should give you a list of page links, which provides a link of stories on the All Sides website.\n",
    "\n",
    "![](https://kidquant.com/post/images/Analyzing-News/topics-allsides2.PNG)\n",
    "\n",
    "We want to create functions that will allow us to parse through the HTML format of the [AllSides](https://www.allsides.com/unbiased-balanced-news) website so we can extract the URLs of these stories. After the extraction process is done, we're going to store all of these links into a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://www.allsides.com/story/trump-restricts-travel-six-new-countries',\n",
       " 'https://www.allsides.com/story/officials-uncertain-2021-goal-border-wall-completion',\n",
       " 'https://www.allsides.com/story/ny-nj-enact-laws-let-undocumented-immigrants-apply-drivers-licenses-lices',\n",
       " 'https://www.allsides.com/story/pentagon-watchdog-review-400m-border-contract',\n",
       " 'https://www.allsides.com/story/trump-gets-jeers-after-colorado-wall-comments',\n",
       " 'https://www.allsides.com/story/trump-administration-require-visa-applicants-be-able-afford-health-care',\n",
       " 'https://www.allsides.com/story/honduras-will-accept-more-asylum-seekers',\n",
       " 'https://www.allsides.com/story/judge-reinstates-injunction-against-asylum-ban',\n",
       " 'https://www.allsides.com/story/incoming-palestinian-harvard-student-deemed-inadmissible-cbp',\n",
       " 'https://www.allsides.com/story/statue-liberty-poem-revised']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# set up BeautifulSoup to run over All Sides Media\n",
    "link_harvest = []\n",
    "\n",
    "# helper function to harvest and parse pages\n",
    "def soup_basics(item):\n",
    "    page = requests.get(item)\n",
    "    soup = BeautifulSoup(page.text, 'html.parser')\n",
    "    return soup\n",
    "\n",
    "def harvest_links(pages):\n",
    "    '''\n",
    "    runs the parser over submitted pages\n",
    "    identifies headline link content in the extracted page\n",
    "    appends relevant links to a list\n",
    "    '''\n",
    "    for item in pages:\n",
    "        soup = soup_basics(item)\n",
    "\n",
    "        # Pull all headlines from the featured stories under class 'view-content'\n",
    "        story_headline_list = soup.find(class_='view-content')\n",
    "        # Pull headline/link text from all instances of <a> tag\n",
    "        story_list_items = story_headline_list.find_all('a')\n",
    "\n",
    "        # harvest the headline and link information\n",
    "        for story_headline in story_list_items:\n",
    "            #headline = story_headline.contents[0]\n",
    "            #headline = headline.encode(\"utf8\").strip()\n",
    "            link = 'https://www.allsides.com'+story_headline.get('href')\n",
    "            if '/story/' in link:\n",
    "                link_harvest.append(link)\n",
    "\n",
    "harvest_links(pages)\n",
    "\n",
    "link_harvest[5:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following output shows an example of 10 links to AllSide stories on immigration. Now we want to extract the URLs to the articles from the following stories. The links can be found as follows:\n",
    "\n",
    "![](https://kidquant.com/post/images/Analyzing-News/topics-allsides3.PNG)\n",
    "\n",
    "We want to be able to locate the href tag that contains the link to the article. We can do this using the `BeautifulSoup` module I mentioned earlier.\n",
    "\n",
    "![](https://kidquant.com/post/images/Analyzing-News/topics-allsides4.PNG)\n",
    "\n",
    "If we expect the elements of the AllSides webpage, we review the HTML and CSS structure of the webpage. The link to the Reuters article is wrapped around an href container. We use the `soup.find_all` method to locate the title and the URL containers.\n",
    "\n",
    "Once we have located the containers, we can extract the information inside; namely the Title and the URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://www.washingtonexaminer.com/opinion/just-imagine-if-trump-had-stopped-immigration-in-early-february',\n",
       " 'https://www.washingtonpost.com/immigration/coronavirus-trump-immigration/2020/04/21/a2a465aa-837a-11ea-9728-c74380d9d410_story.html',\n",
       " 'https://www.reuters.com/article/us-health-coronavirus-usa/u-s-coronavirus-response-deepens-divide-as-trump-suspends-immigration-idUSKCN2231TU',\n",
       " 'https://www.foxnews.com/politics/trump-suspend-immigration-executive-order-coronavirus',\n",
       " 'https://www.foxnews.com/politics/court-hands-trump-win-in-sanctuary-city-grant']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get all news article links\n",
    "all_articles = []\n",
    "\n",
    "\n",
    "def extract_articles(link_harvest):\n",
    "    for link_content in link_harvest:\n",
    "        soup = soup_basics(link_content)\n",
    "\n",
    "        # locate relevant information within the extracted page\n",
    "        substory_list = soup.find_all(class_='news-title')\n",
    "\n",
    "        # loop through the different news sources within each major news story\n",
    "        for i in range(0, len(substory_list)):\n",
    "            substory_items = substory_list[i].find_all('a')\n",
    "            for substory_headline in substory_items:\n",
    "                link = substory_headline.get('href')\n",
    "                all_articles.append(link)\n",
    "\n",
    "extract_articles(link_harvest)\n",
    "\n",
    "all_articles[5:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've extracted all of the necessary articles, we want to be able to save these articles for research later on. The following chunk will allow us to keep all of the links into a single text file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save news article links to file\n",
    "link_file = open('link_file.txt', 'w')\n",
    "\n",
    "for item in all_articles:\n",
    "    link_file.write(\"%s\\n\" % item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we want to create a function that allows us to extract the important information for the articles we're planning on extracting. This information includes the following: **Date**, **Headline**, **Story Description**, **Source/Publication**, **Bias/Lean**, and **Link**.\n",
    "\n",
    "We're going to be using the same technique to extract this information, as we did with the article URLs. However, the only difference is that this information is located in HTML class attributes. As long as we are aware of the exact attributes, we can extract the relevant information.\n",
    "\n",
    "Afterward, we will review the results of the function by using `pandas` to open the CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>AllMedia_Headline</th>\n",
       "      <th>Description</th>\n",
       "      <th>Source_Name</th>\n",
       "      <th>Source_Bias</th>\n",
       "      <th>Source_Headline</th>\n",
       "      <th>Source_Link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>June 22nd, 2020</td>\n",
       "      <td>b'Trump Admin Suspends Certain Visas Through E...</td>\n",
       "      <td>The Trump administration announced Monday that...</td>\n",
       "      <td>BuzzFeed News</td>\n",
       "      <td>Left</td>\n",
       "      <td>b'Trump Is Suspending Certain Visas For Foreig...</td>\n",
       "      <td>https://www.buzzfeednews.com/article/adolfoflo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>June 22nd, 2020</td>\n",
       "      <td>b'Trump Admin Suspends Certain Visas Through E...</td>\n",
       "      <td>The Trump administration announced Monday that...</td>\n",
       "      <td>Reuters</td>\n",
       "      <td>Center</td>\n",
       "      <td>b'Trump to suspend entry of certain foreign wo...</td>\n",
       "      <td>https://www.reuters.com/article/us-usa-immigra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>June 22nd, 2020</td>\n",
       "      <td>b'Trump Admin Suspends Certain Visas Through E...</td>\n",
       "      <td>The Trump administration announced Monday that...</td>\n",
       "      <td>The Daily Caller</td>\n",
       "      <td>Right</td>\n",
       "      <td>b'Trump To Suspend Visas Through End Of The Ye...</td>\n",
       "      <td>https://dailycaller.com/2020/06/22/exclusive-t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>April 24th, 2020</td>\n",
       "      <td>b'Perspectives Trump s Immigration Executive O...</td>\n",
       "      <td>President Donald Trump's executive order suspe...</td>\n",
       "      <td>Reuters</td>\n",
       "      <td>Center</td>\n",
       "      <td>b\"Inside Trump's proposal to suspend some lega...</td>\n",
       "      <td>https://af.reuters.com/article/worldNews/idAFK...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>April 24th, 2020</td>\n",
       "      <td>b'Perspectives Trump s Immigration Executive O...</td>\n",
       "      <td>President Donald Trump's executive order suspe...</td>\n",
       "      <td>CNN - Editorial</td>\n",
       "      <td>Left</td>\n",
       "      <td>b\"Trump's moves on immigration reveal his true...</td>\n",
       "      <td>https://www.cnn.com/2020/04/22/opinions/trump-...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Date                                  AllMedia_Headline  \\\n",
       "0   June 22nd, 2020  b'Trump Admin Suspends Certain Visas Through E...   \n",
       "1   June 22nd, 2020  b'Trump Admin Suspends Certain Visas Through E...   \n",
       "2   June 22nd, 2020  b'Trump Admin Suspends Certain Visas Through E...   \n",
       "3  April 24th, 2020  b'Perspectives Trump s Immigration Executive O...   \n",
       "4  April 24th, 2020  b'Perspectives Trump s Immigration Executive O...   \n",
       "\n",
       "                                         Description       Source_Name  \\\n",
       "0  The Trump administration announced Monday that...     BuzzFeed News   \n",
       "1  The Trump administration announced Monday that...           Reuters   \n",
       "2  The Trump administration announced Monday that...  The Daily Caller   \n",
       "3  President Donald Trump's executive order suspe...           Reuters   \n",
       "4  President Donald Trump's executive order suspe...   CNN - Editorial   \n",
       "\n",
       "  Source_Bias                                    Source_Headline  \\\n",
       "0       Left   b'Trump Is Suspending Certain Visas For Foreig...   \n",
       "1     Center   b'Trump to suspend entry of certain foreign wo...   \n",
       "2      Right   b'Trump To Suspend Visas Through End Of The Ye...   \n",
       "3     Center   b\"Inside Trump's proposal to suspend some lega...   \n",
       "4       Left   b\"Trump's moves on immigration reveal his true...   \n",
       "\n",
       "                                         Source_Link  \n",
       "0  https://www.buzzfeednews.com/article/adolfoflo...  \n",
       "1  https://www.reuters.com/article/us-usa-immigra...  \n",
       "2  https://dailycaller.com/2020/06/22/exclusive-t...  \n",
       "3  https://af.reuters.com/article/worldNews/idAFK...  \n",
       "4  https://www.cnn.com/2020/04/22/opinions/trump-...  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "socket.socket\n",
    "\n",
    "def csv_encoder(text_string):\n",
    "    coded = text_string.encode(\"utf-8\").strip()\n",
    "    return coded\n",
    "\n",
    "# extract all content\n",
    "def extract_content(link_harvest):\n",
    "    '''\n",
    "    for each story, pulls the shared news headline, date, and summary description\n",
    "    for each news source, identifies the source bias (liberal, conservative, center) & outgoing link\n",
    "    uses re and .contents to clean harvested text\n",
    "    writes collected, cleaned data to csv\n",
    "    '''\n",
    "\n",
    "    # open csv file to store info\n",
    "    file = open('allsides-content.csv', 'w', newline=\"\", encoding='utf-8')\n",
    "    fileWriter = csv.writer(file)\n",
    "    fileWriter.writerow(['Date', 'AllMedia_Headline', 'Description',\n",
    "                         'Source_Name', 'Source_Bias', 'Source_Headline', 'Source_Link'])\n",
    "\n",
    "    try:\n",
    "        for link_content in link_harvest:\n",
    "            soup = soup_basics(link_content)\n",
    "\n",
    "            # locate relevant information within the extracted page\n",
    "            story_headline = soup.find(class_='taxonomy-heading')\n",
    "            story_date = soup.find(property='dc:date')\n",
    "            story_description = soup.find(class_='story-id-page-description')\n",
    "            substory_source = soup.find_all(class_='news-source')\n",
    "            substory_bias = soup.find_all(class_='global-bias')\n",
    "            substory_list = soup.find_all(class_='news-title')\n",
    "\n",
    "            # loop through the different news sources within each major news story\n",
    "            n = 0\n",
    "            for i in range(0, len(substory_list)):\n",
    "                substory_items = substory_list[i].find_all('a')\n",
    "                for substory_headline in substory_items:\n",
    "\n",
    "                    # extracting the date\n",
    "                    clean_date = story_date.contents[0]\n",
    "\n",
    "                    # extracting the headline\n",
    "                    clean_headline = re.sub(\n",
    "                        '\\W+', ' ', story_headline.contents[0])[1:][:-1]\n",
    "                    clean_headline = csv_encoder(clean_headline)\n",
    "\n",
    "                    # extracting the description\n",
    "                    try:\n",
    "                        clean_description = str(\n",
    "                            story_description.contents[1].text)\n",
    "                    except IndexError:\n",
    "                        clean_description = 'Null'\n",
    "                        \n",
    "                    # clean_authors = 'None'\n",
    "\n",
    "                    # extracting the story source\n",
    "                    try:\n",
    "                        clean_source = substory_source[n].contents[1]\n",
    "                    except (AttributeError, IndexError):\n",
    "                        clean_source = 'Unknown'\n",
    "\n",
    "                    # extracting the lean and the bias\n",
    "                    clean_bias = substory_bias[n]\n",
    "                    clean_bias = re.sub(\n",
    "                        '\\W+', ' ', clean_bias.contents[0])[10:]\n",
    "                    n = n+1\n",
    "\n",
    "                    # extracting the headline\n",
    "                    headline = substory_headline.contents[0]\n",
    "                    headline = csv_encoder(headline)\n",
    "\n",
    "                    # extracting the link\n",
    "                    link = substory_headline.get('href')\n",
    "\n",
    "                    fileWriter.writerow(\n",
    "                        [clean_date, clean_headline, clean_description, clean_source, clean_bias, headline, link])\n",
    "\n",
    "    except socket.error as err:\n",
    "        print('Socket connection error... Waiting 10 seconds to retry.')\n",
    "        del self.sock\n",
    "        time.sleep(10)\n",
    "        try_count += 1\n",
    "\n",
    "    file.close()\n",
    "\n",
    "# running the function\n",
    "extract_content(link_harvest)\n",
    "\n",
    "# showing the results of the csv\n",
    "pd.read_csv('allsides-content.csv').head(5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're showing the first five rows of the CSV we created. So far, everything looks satisfactory.\n",
    "\n",
    "Keep in mind; this DataFrame lacks specific information for analyzing news articles. Such as the actual body or text of the article (and perhaps the authors of the article). For Part 2, we will be using the same CSV to extract the necessary information."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}