docs/tutorials/quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quickstart tutorial\n",
    "\n",
    "This tutorial will go through the basic ways to use `floweaver` to process and transform data into many different Sankey diagrams.\n",
    "\n",
    ">  If you are reading the static documentation, you can also try an interactive version of this tutorial online [using MyBinder](https://mybinder.org/v2/gh/ricklupton/floweaver/master?filepath=docs%2Ftutorials%2Fquickstart.ipynb)\n",
    "\n",
    "Let's start by making a really simple dataset. Imagine we have some farms, which grow apples and bananas to sell to a few different customers. We can describe the *flow* of fruit from the farms (the *source* of the flow) to the customers (the *target* of the flow):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "flows = pd.read_csv('simple_fruit_sales.csv')\n",
    "flows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drawn directly as a Sankey diagram, this data would look something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipysankeywidget import SankeyWidget\n",
    "SankeyWidget(links=flows.to_dict('records'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But you don't always want a direct correspondence between the flows in your data and the links that you see in the Sankey diagram. For example:\n",
    "\n",
    "* Farms 4, 5 and 6 are all pretty small, and to make the diagram clearer we might want to group them in an \"other\" category.\n",
    "* The flows of apples are mixed in with the flows of bananas -- we might want to group the kinds of fruit together to make them easier to compare\n",
    "* We might want to group farms or customers based on some other attributes -- to see difference between genders, locations, or organic/non-organic farms, say.\n",
    "\n",
    "This introduction shows how to use `floweaver` to do some of these for this simple example, in the simplest possible way. Later tutorials will show how to use it on real data, and more efficient ways to do the same things.\n",
    "\n",
    "## Basic diagram\n",
    "\n",
    "Let's start with the first example: grouping farms 4, 5 and 6 into an \"other\" category. `floweaver` works by setting up a \"Sankey diagram definition\" which describes the structure of the diagram we want to see. In this case, we need to set up some groups:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from floweaver import *\n",
    "\n",
    "# Set the default size to fit the documentation better.\n",
    "size = dict(width=570, height=300)\n",
    "\n",
    "nodes = {\n",
    "    'farms': ProcessGroup(['farm1', 'farm2', 'farm3', \n",
    "                           'farm4', 'farm5', 'farm6']),\n",
    "    'customers': ProcessGroup(['James', 'Mary', 'Fred', 'Susan']),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to describe roughly how these groups should be placed in the final diagram by defining an \"ordering\" -- a list of vertical slices, each containing a list of node ids:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ordering = [\n",
    "    ['farms'],       # put \"farms\" on the left...\n",
    "    ['customers'],   # ... and \"customers\" on the right.\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we also need to say which connections should appear in the diagram (sometimes you don't want to actually see all the connections). This is called a \"bundle\" because it bundles up multiple flows -- in this case all of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bundles = [\n",
    "    Bundle('farms', 'customers'),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Putting that together into a Sankey diagram definition (SDD) and applying it to the data gives this result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sdd = SankeyDefinition(nodes, bundles, ordering)\n",
    "weave(sdd, flows).to_widget(**size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's not very useful. What's happened? Every farm and every customer has been lumped together into one group. To get the picture we want -- like the first one, but with an \"other\" group containing farms 4, 5 and 6 -- we need to *partition* the groups:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The first argument is the dimension name -- for now we're using\n",
    "# \"process\" to group by process ids. The second argument is a list\n",
    "# of groups.\n",
    "farms_with_other = Partition.Simple('process', [\n",
    "    'farm1',  # the groups within the partition can be a single id...\n",
    "    'farm2',\n",
    "    'farm3',\n",
    "    ('other', ['farm4', 'farm5', 'farm6']),   # ... or a group\n",
    "])\n",
    "\n",
    "# This is another partition.\n",
    "customers_by_name = Partition.Simple('process', [\n",
    "    'James', 'Mary', 'Fred', 'Susan'\n",
    "])\n",
    "\n",
    "# Update the ProcessGroup nodes to use the partitions\n",
    "nodes['farms'].partition = farms_with_other\n",
    "nodes['customers'].partition = customers_by_name\n",
    "\n",
    "# New Sankey!\n",
    "weave(sdd, flows).to_widget(**size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's better: now the farms are split up appropriately with an \"other\" category, and the customers are shown separately as well. We don't have to stop there -- what about showing sales to men and women?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "customers_by_gender = Partition.Simple('process', [\n",
    "    ('Men', ['Fred', 'James']),\n",
    "    ('Women', ['Susan', 'Mary']),\n",
    "])\n",
    "\n",
    "nodes['customers'].partition = customers_by_gender\n",
    "\n",
    "weave(sdd, flows).to_widget(**size).auto_save_png('quickstart_example1.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> There is a better way of specifying this type of partition that doesn't involve hard-coding who's a man and who's a woman into the code -- see the later tutorial on using *dimension tables*.\n",
    "\n",
    "## Distinguishing flow types\n",
    "\n",
    "These diagrams have lost sight of the kind of fruit that is actually being sold -- are the men buying apples, bananas or both from farm1? To show this we need to split up the flows in the diagram based on their *type*. Just like we split up the `ProcessGroups` by defining a partition of processes, we split up flows by defining a partition of flows.\n",
    "\n",
    "(While we're at it let's choose some colours that look vaguely like apples and bananas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Another partition -- but this time the dimension is the \"type\"\n",
    "# column of the flows table\n",
    "fruits_by_type = Partition.Simple('type', ['apples', 'bananas'])\n",
    "\n",
    "# Set the colours for the labels in the partition.\n",
    "palette = {'apples': 'yellowgreen', 'bananas': 'gold'}\n",
    "\n",
    "# New SDD with the flow_partition set\n",
    "sdd = SankeyDefinition(nodes, bundles, ordering, \n",
    "                       flow_partition=fruits_by_type)\n",
    "\n",
    "weave(sdd, flows, palette=palette).to_widget(**size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a last step, it would be nice to label which flows are apples and which are bananas. One way to do this would be to use a legend next to the diagram, or to put labels on every flow. Here, we'll add a new layer in the middle of the diagram which temporarily groups together the different fruit types on their way from the farms to the customers. This temporary/additional grouping point is called a *waypoint*.\n",
    "\n",
    "To add a waypoint, we need to do three things:\n",
    "\n",
    "1. Define it as a node\n",
    "2. Position it in the ordering (between `farms` and `customers`)\n",
    "3. Add it to the bundle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Define a new waypoint node\n",
    "nodes['waypoint'] = Waypoint()\n",
    "\n",
    "# 2. Update the ordering to show where the waypoint goes: in the middle\n",
    "ordering = [\n",
    "    ['farms'],\n",
    "    ['waypoint'],\n",
    "    ['customers'],\n",
    "]\n",
    "\n",
    "# 3. Update the bundle definition to send the flows via the waypoint\n",
    "bundles = [\n",
    "    Bundle('farms', 'customers', waypoints=['waypoint']),\n",
    "]\n",
    "\n",
    "# Update the SDD with the new nodes, ordering & bundles.\n",
    "sdd = SankeyDefinition(nodes, bundles, ordering, \n",
    "                       flow_partition=fruits_by_type)\n",
    "\n",
    "weave(sdd, flows, palette=palette).to_widget(**size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's not yet very useful. Just like above, the default for Waypoints is to group everything togeter. We need to set a partition on the waypoint to split apart apples and bananas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Redefine the waypoint with a partition (same one defined above)\n",
    "nodes['waypoint'] = Waypoint(fruits_by_type)\n",
    "\n",
    "weave(sdd, flows, palette=palette).to_widget(**size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "This has demonstrated the basic usage of `floweaver`: defining `ProcessGroup`s, `Waypoint`s, `Partition`s, and `Bundle`s. If you are reading the interactive version, why not go back and try out some different ways to present the data? Here are some suggestions:\n",
    "\n",
    "1. Farms 1, 3 and 5 are organic. Can you change the farm Partition to show two groups, organic and non-organic?\n",
    "2. What happens if you remove `\"farm1\"` from the original definition of the `farms` `ProcessGroup`? (Hint: those apples that James and Mary are eating have to come from somewhere -- so they are shown as coming from \"elsewhere\". See [later tutorial on moving the system boundary](/tutorials/system-boundary.ipynb))\n",
    "\n",
    "If you are reading the static documentation, you can easily experiment with editing and rerunning this tutorial online [using MyBinder](https://mybinder.org/v2/gh/ricklupton/floweaver/master?filepath=docs%2Ftutorials%2Fquickstart.ipynb), or download it to run on your computer from [GitHub](https://github.com/ricklupton/floweaver/archive/master.zip)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}