twitter_network_analysis/twitter_network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b50a0dba",
   "metadata": {},
   "source": [
    "# Exploring a Twitter network in Memgraph\n",
    "\n",
    "Through this short tutorial, you will learn how to install Memgraph, connect to it from a Jupyter Notebook and perform data analysis using graph algorithms.\n",
    "\n",
    "## 1. Prerequisites\n",
    "\n",
    "For this tutorial, you will need to install:\n",
    "\n",
    "- [Jupyter](https://jupyter.org/install)\n",
    "- [Docker](https://docs.docker.com/get-docker/)\n",
    "- [GQLAlchemy](https://pypi.org/project/gqlalchemy/)\n",
    "- [Pandas](https://pypi.org/project/pandas/)\n",
    "\n",
    "Docker is used because Memgraph is a native Linux application and cannot be installed on Windows and macOS.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d81bb720",
   "metadata": {},
   "source": [
    "## 2. Installation using Docker\n",
    "\n",
    "After you install Docker, you can set up Memgraph by running:\n",
    "\n",
    "```\n",
    "docker run -it -p 7687:7687 -p 3000:3000 memgraph/memgraph-platform\n",
    "```\n",
    "\n",
    "This command will start the download and after it finishes, run the Memgraph container.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c53ab73",
   "metadata": {},
   "source": [
    "## 3. Connecting to Memgraph with GQLAlchemy\n",
    "\n",
    "We will be using the **GQLAlchemy** object graph mapper (OGM) to connect to Memgraph and execute **Cypher** queries easily. GQLAlchemy also serves as a Python driver/client for Memgraph. You can install it using:\n",
    "\n",
    "```\n",
    "pip install gqlalchemy\n",
    "```\n",
    "\n",
    "> **Hint**: You may need to install [CMake](https://cmake.org/download/) before installing GQLAlchemy.\n",
    "\n",
    "Maybe you got confused when I mentioned Cypher. You can think of Cypher as SQL for graph databases. It contains many of the same language constructs like `CREATE`, `UPDATE`, `DELETE`... and it's used to query the database.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d6e35b90",
   "metadata": {},
   "outputs": [],
   "source": [
    "from gqlalchemy import Memgraph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "17b8101a",
   "metadata": {},
   "outputs": [],
   "source": [
    "memgraph = Memgraph(\"127.0.0.1\", 7687)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "743d7b6b",
   "metadata": {},
   "source": [
    "Let's make sure that Memgraph is empty before we start with anything else.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5746a3bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "memgraph.drop_database()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "244c7031",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'number_of_nodes': 0}\n"
     ]
    }
   ],
   "source": [
    "results = memgraph.execute_and_fetch(\n",
    "    \"\"\"\n",
    "    MATCH (n) RETURN count(n) AS number_of_nodes ;\n",
    "    \"\"\"\n",
    ")\n",
    "print(next(results))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a575bda4",
   "metadata": {},
   "source": [
    "## 4. Define a graph schema\n",
    "\n",
    "We are going to create Python classes that will represent our graph schema. This way, all the objects that are returned from Memgraph will be of the correct type if the class definition can be found.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e2dd6420",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Optional\n",
    "from gqlalchemy import Field, Node, Relationship\n",
    "\n",
    "\n",
    "class User(Node):\n",
    "    username: str = Field(index=True, unique=True, db=memgraph)\n",
    "\n",
    "\n",
    "class Retweeted(Relationship, type=\"RETWEETED\"):\n",
    "    pass\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe38e88b",
   "metadata": {},
   "source": [
    "## 5. Creating and returning nodes\n",
    "\n",
    "We are going to create `User` nodes, save them to the database and return them to our program:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8d077675",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<User id=1874 labels={'User'} properties={'username': 'ivan_g_despot'}>\n",
      "<User id=1875 labels={'User'} properties={'username': 'supe_katarina'}>\n"
     ]
    }
   ],
   "source": [
    "user1 = User(username=\"ivan_g_despot\")\n",
    "user2 = User(username=\"supe_katarina\")\n",
    "\n",
    "user1.save(memgraph)\n",
    "user2.save(memgraph)\n",
    "\n",
    "print(user1)\n",
    "print(user2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "788a73f1",
   "metadata": {},
   "source": [
    "Now, let's try to create a node using the Cypher query language. We are going to create a node with an existing username just to check if the existance constraint on the property `username` is set correctly.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8a3ef2a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error: A user with the username supe_katarina is already in the database.\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    results = memgraph.execute(\n",
    "        \"\"\"\n",
    "        CREATE (:User {username: \"supe_katarina\"});\n",
    "        \"\"\"\n",
    "    )\n",
    "except Exception:\n",
    "    print(\"Error: A user with the username supe_katarina is already in the database.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "947c2325",
   "metadata": {},
   "source": [
    "## 6. Creating and returning relationships\n",
    "\n",
    "We are going to create a `Retweeted` relationship, save it to the database and return it to our program:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "08161204",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Retweeted id=1670 start_node_id=1874 end_node_id=1875 nodes=(1874, 1875) type=RETWEETED properties={}>\n"
     ]
    }
   ],
   "source": [
    "retweeted = Retweeted(_start_node_id=user1._id, _end_node_id=user2._id)\n",
    "\n",
    "retweeted.save(memgraph)\n",
    "\n",
    "print(retweeted)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d9da1f4",
   "metadata": {},
   "source": [
    "## 7. Importing data from CSV files\n",
    "\n",
    "You will need to download [this file]() which contains a simple dataset of scraped tweets. To import it into Memgraph, we will first need to copy it to the Docker container where Memgraph is running.\n",
    "Find the `CONTAINER_ID` by running:\n",
    "\n",
    "```\n",
    "docker ps\n",
    "```\n",
    "\n",
    "Copy the file with the following command (don't forget to replace `CONTAINER_ID`):\n",
    "\n",
    "```\n",
    "docker cp scraped_tweets.csv CONTAINER_ID:scraped_tweets.csv\n",
    "```\n",
    "\n",
    "We are going to see what our CSV file looks like with the help of the pandas library. To install it, run: \n",
    "```\n",
    "pip install pandas\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c6a3c78a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_username</th>\n",
       "      <th>target_username</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CapeCodGiftShop</td>\n",
       "      <td>RetroCEO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CodeAttBot</td>\n",
       "      <td>LeeHillerLondon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BattlegroundHs</td>\n",
       "      <td>getwhalinvest</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>botpokemongofr1</td>\n",
       "      <td>TrevorAllenPKMN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AnyaSha13331181</td>\n",
       "      <td>WORLDMUSICAWARD</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   source_username  target_username\n",
       "0  CapeCodGiftShop         RetroCEO\n",
       "1       CodeAttBot  LeeHillerLondon\n",
       "2   BattlegroundHs    getwhalinvest\n",
       "3  botpokemongofr1  TrevorAllenPKMN\n",
       "4  AnyaSha13331181  WORLDMUSICAWARD"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv(\"scraped_tweets.csv\")\n",
    "data.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12cbdcc3",
   "metadata": {},
   "source": [
    "Now, we can execute the Cypher command `LOAD CSV`, which is used for loading data from CSV files:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b34cac6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "memgraph.execute(\n",
    "    \"\"\"\n",
    "    LOAD CSV FROM \"/scraped_tweets.csv\" WITH HEADER AS row\n",
    "    MERGE (u1:User {username: row.source_username}) \n",
    "    MERGE (u2:User {username: row.target_username}) \n",
    "    MERGE (u1)-[:RETWEETED]->(u2);\n",
    "    \"\"\"\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "445c7990",
   "metadata": {},
   "source": [
    "You can think of the `LOAD CSV` clause as a loop that will go over every row in the CSV file and execute the specified Cypher commands.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d46b4c05",
   "metadata": {},
   "source": [
    "## 8. Querying the database and retrieving results\n",
    "\n",
    "Let's make sure that our data was imported correctly by retrieving it:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7da3d5a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = memgraph.execute_and_fetch(\n",
    "    \"\"\"\n",
    "    MATCH (u:User)\n",
    "    RETURN u\n",
    "    ORDER BY u.username DESC\n",
    "    LIMIT 10;\n",
    "    \"\"\"\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "bd3bd070",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = list(results)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3cfb2c13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<User id=3692 labels={'User'} properties={'username': 'zziru67'}>\n",
      "<User id=3240 labels={'User'} properties={'username': 'zippydjh'}>\n",
      "<User id=3725 labels={'User'} properties={'username': 'zee_row_ex'}>\n",
      "<User id=3591 labels={'User'} properties={'username': 'yvonneqqm'}>\n",
      "<User id=3212 labels={'User'} properties={'username': 'yujulia999'}>\n",
      "<User id=2378 labels={'User'} properties={'username': 'yudhapati88'}>\n",
      "<User id=2655 labels={'User'} properties={'username': 'yu100_kun'}>\n",
      "<User id=2302 labels={'User'} properties={'username': 'youth_tree'}>\n",
      "<User id=2432 labels={'User'} properties={'username': 'yourkpopsoul'}>\n",
      "<User id=2132 labels={'User'} properties={'username': 'your_harrogate'}>\n"
     ]
    }
   ],
   "source": [
    "for result in results:\n",
    "    print(result[\"u\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "093d5745",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "User:  zziru67\n",
      "Type:  <class '__main__.User'>\n"
     ]
    }
   ],
   "source": [
    "u = results[0][\"u\"]\n",
    "\n",
    "print(\"User: \", u.username)\n",
    "print(\"Type: \", type(u))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48bf0ce0",
   "metadata": {},
   "source": [
    "Let's try to execute the same query with the GQLAlchemy query builder:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "54027fea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<User id=3692 labels={'User'} properties={'username': 'zziru67'}>\n",
      "<User id=3240 labels={'User'} properties={'username': 'zippydjh'}>\n",
      "<User id=3725 labels={'User'} properties={'username': 'zee_row_ex'}>\n",
      "<User id=3591 labels={'User'} properties={'username': 'yvonneqqm'}>\n",
      "<User id=3212 labels={'User'} properties={'username': 'yujulia999'}>\n",
      "<User id=2378 labels={'User'} properties={'username': 'yudhapati88'}>\n",
      "<User id=2655 labels={'User'} properties={'username': 'yu100_kun'}>\n",
      "<User id=2302 labels={'User'} properties={'username': 'youth_tree'}>\n",
      "<User id=2432 labels={'User'} properties={'username': 'yourkpopsoul'}>\n",
      "<User id=2132 labels={'User'} properties={'username': 'your_harrogate'}>\n"
     ]
    }
   ],
   "source": [
    "from gqlalchemy import match\n",
    "\n",
    "results_from_qb = (\n",
    "    match()\n",
    "    .node(labels=\"User\", variable=\"u\")\n",
    "    .return_()\n",
    "    .order_by(\"u.username DESC\")\n",
    "    .limit(10)\n",
    "    .execute()\n",
    ")\n",
    "results_from_qb = list(results_from_qb)\n",
    "\n",
    "for result in results_from_qb:\n",
    "    print(result[\"u\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e87b224",
   "metadata": {},
   "source": [
    "## 9. Calculating PageRank\n",
    "\n",
    "Now, let's do something clever with our graph.\n",
    "For example, calculating PageRank for each node and then adding a `rank` property that stores the PageRank value to each node:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "1a868444",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = memgraph.execute_and_fetch(\n",
    "    \"\"\"\n",
    "    CALL pagerank.get()\n",
    "    YIELD node, rank\n",
    "    SET node.rank = rank\n",
    "    RETURN node, rank\n",
    "    ORDER BY rank DESC\n",
    "    LIMIT 10;\n",
    "    \"\"\"\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "bcc07704",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The PageRank of node  WORLDMUSICAWARD :  0.13278838151391434\n",
      "The PageRank of node  Kidzcoolit :  0.018924764871246207\n",
      "The PageRank of node  HuobiGlobal :  0.011314994833838172\n",
      "The PageRank of node  ChloeLe39602964 :  0.010011755296388128\n",
      "The PageRank of node  getwhalinvest :  0.007228675936490175\n",
      "The PageRank of node  Cooper_Lechat :  0.005577971882231625\n",
      "The PageRank of node  Phemex_official :  0.005413803151353543\n",
      "The PageRank of node  HamleysOfficial :  0.005325936307836382\n",
      "The PageRank of node  bmstores :  0.00524546649693655\n",
      "The PageRank of node  TheStourbridge :  0.004422198431576731\n"
     ]
    }
   ],
   "source": [
    "for result in results:\n",
    "    print(\"The PageRank of node \", result[\"node\"].username, \": \", result[\"rank\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef5de90b",
   "metadata": {},
   "source": [
    "Visit the [Memgraph MAGE](https://github.com/memgraph/mage) graph library (and throw us a star ⭐) and take a look at all of the graph algorithms that have been implemented. You can also implement and submit your own algorithms and utility procedures.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d4d4319",
   "metadata": {},
   "source": [
    "## 10. Visualizing the graph in Memgraph Lab\n",
    "\n",
    "Open Memgraph Lab in your browser on the address [localhost:3000](localhost:3000).\n",
    "Execute the following Cypher query:\n",
    "\n",
    "```cypher\n",
    "MATCH (n)-[r]-(m)\n",
    "RETURN n, r, m\n",
    "LIMIT 100;\n",
    "```\n",
    "\n",
    "Apply the following graph style:\n",
    "\n",
    "```\n",
    "@NodeStyle {\n",
    "  size: Sqrt(Mul(Div(Property(node, \"rank\"), 1), 200000))\n",
    "  border-width: 1\n",
    "  border-color: #000000\n",
    "  shadow-color: #1D9BF0\n",
    "  shadow-size: 10\n",
    "  image-url: \"https://i.imgur.com/UV7Nl0i.png\"\n",
    "}\n",
    "\n",
    "@NodeStyle Greater?(Size(Labels(node)), 0) {\n",
    "  label: Format(\":{}\", Join(Labels(node), \" :\"))\n",
    "}\n",
    "\n",
    "@NodeStyle HasLabel?(node, \"User\") {\n",
    "  color: #1D9BF0\n",
    "  color-hover: Darker(#dd2222)\n",
    "  color-selected: #dd2222\n",
    "}\n",
    "\n",
    "@NodeStyle HasProperty?(node, \"username\") {\n",
    "  label: AsText(Property(node, \"username\"))\n",
    "}\n",
    "\n",
    "@EdgeStyle {\n",
    "  width: 1\n",
    "}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbbd74c8",
   "metadata": {},
   "source": [
    "## What's next?\n",
    "\n",
    "Now it's time for you to use Memgraph on a graph problem!\n",
    "\n",
    "You can always check out [Memgraph Playground](https://playground.memgraph.com/) for some cool use cases and examples.\n",
    "If you have any questions or you want to share your work with the rest of the community, join our **[Discord Server](https://discord.gg/memgraph)**.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}