Advanced_Simulation_Project.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Advanced_Simulation_Project.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MGbq_t1fDo_z",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "# Advanced Simulation\n",
        "## Final Project\n",
        "\n",
        "![logo_sgh](http://administracja.sgh.waw.pl/pl/dpir/obowiazki/PublishingImages/ksiega2019/SGHlogoCMYK.png)\n",
        "\n",
        "June 2019, Warsaw School of Economics\n",
        " * Roberto Sannazzaro 79240\n",
        " \n",
        "  \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Ou5LeP-UDJd",
        "colab_type": "text"
      },
      "source": [
        "## Introduction \n",
        "\n",
        "The use of robotics is constantly expanding in every business sector, automation takes repetitive tasks and aims to automatize them, in order to optimize processes and cut costs.\n",
        "In 2012 Amazon purchased Kiva Systems, a company which developed warehouse robots and related technologies, and which was acquired for $775 million. Moreover many other companies implement robots in their warehouses, even robots that can work in 3 dimensions.\n",
        "\n",
        "<br/><br/>\n",
        "\n",
        "![Autonomous warehouse robot](https://media.giphy.com/media/s0urqX40zokIo/giphy.gif)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EywE3y-rbc1l",
        "colab_type": "text"
      },
      "source": [
        "## Scope \n",
        "\n",
        "The goal of this study is to simulate the actions an autonomous warehouse robot needs to do in order to collect products for deliveries in an optimal way. The warehouse used in this simulation consists in different 12 locations:\n",
        "\n",
        "<img src=\"https://i.ibb.co/LrTfrgc/warehouse.png\">\n",
        "\n",
        "The system needs to rank in real time the priorities for collecting the products in those 12 locations, for example, at a specific time `t` the system will return the following ranking:\n",
        "\n",
        "\n",
        "Rank | Location |\n",
        ":---:|:---:|\n",
        "1|G|\n",
        "2|K|\n",
        "3|L|\n",
        "4|J|\n",
        "5|A|\n",
        "6|I\n",
        "7|H\n",
        "8|C|\n",
        "9|B|\n",
        "10|D|\n",
        "11|F|\n",
        "12|E|\n",
        "\n",
        "\n",
        "In this example the location G has the highest priority, and the robot must move to this location with the shortest route calculatd by the system, moreover, location K and L are in the top 3 priorities. hence the system will calculate the shortest route by passing to some intermediary locations before reaching its final top priority location.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h3_TSG9zddny",
        "colab_type": "text"
      },
      "source": [
        "## Definition of the environment \n",
        "\n",
        "In order to implement this simulation it is necessary to create the environment by defining the 3 following elements:\n",
        "\n",
        "* Defining the states\n",
        "* Defining the actions\n",
        "* Defining the rewards\n",
        "\n",
        "The states are the locations where the robot can be at each time `t`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d_Nj4DuPcil6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "location_to_state = {'A': 0,\n",
        "                     'B': 1,\n",
        "                     'C': 2,\n",
        "                     'D': 3,\n",
        "                     'E': 4,\n",
        "                     'F': 5,\n",
        "                     'G': 6,\n",
        "                     'H': 7,\n",
        "                     'I': 8,\n",
        "                     'J': 9,\n",
        "                     'K': 10,\n",
        "                     'L': 11}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r_LOJ5yImA0r",
        "colab_type": "text"
      },
      "source": [
        "The actions are the possible moves the robot can do while moving from one location to another"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QEISuD43mRPv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "actions = [0,1,2,3,4,5,6,7,8,9,10,11]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h8vIAxxUmWsr",
        "colab_type": "text"
      },
      "source": [
        "Of course, when the robot is at a specific location there are some actions it cannot perform, this is specified in the simulation by a matrix of rewards and by attributing a reward of 0 for actions it cannot perform.\n",
        "The matrix of rewards consists in a matrix of states and actions, containing a 0 for actions the robot cannot perform in that state and a 1 for actions it can perform.\n",
        "Starting from location A, according to the warehouse map, the robot can only go to location B, while being in location B gives the possibility to move to location A, C or F\n",
        "\n",
        "||A | B | C | D | E | F | G | H | I | J | K | L |\n",
        ":---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n",
        "A|0|1|0|0|0|0|0|0|0|0|0|0|0\n",
        "B|1|0|1|0|0|1|0|0|0|0|0|0|0\n",
        "C\n",
        "D\n",
        "E\n",
        "F\n",
        "G\n",
        "H\n",
        "I\n",
        "J\n",
        "K\n",
        "L\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SzZ9ZcB9scw8",
        "colab_type": "text"
      },
      "source": [
        "The system takes the matrix of rewards and assign a high reward to the top priority location, returning the optmial path for the location, this system is based on the **Markov decision process**, which consists in a tuple:\n",
        "\n",
        "$(S, A, T, R)$  \n",
        "\n",
        "Where:\n",
        "* _S_ is the set of the states\n",
        "* _A_ the set of actions that can be played \n",
        "* _T_ the transition rule defining the probability that action  $a$ in state $s$ at time $t$ will lead to state  $s'$ at time $ t+1$\n",
        "* _R_  the reward function received after transitioning from state  $s$ to state $s'$, due to action $a$\n",
        "\n",
        "The system contains a policy function, which given a state $s_t$ returns the action $a_t$.\n",
        "Denoting with $Π$ the set of all the possible policy actions create the optimization problem where the optimal policy $π^*$ that maximizes the accumulated reward.\n",
        "\n",
        "To each couple of action $(s, a)$ a numeric value is associated, noted as $Q-value$; \n",
        "at $t = 0$ all the $Q-values$ are initialised to 0, when at a time $t$ and state $s_t$ a random action $a_t$ is being played, it brings to a state $s_{t+1}$ and a reward $R(s_t, a_t)$\n",
        "\n",
        "This whole algorithm, named Q-Learning can be summarised as follows:\n",
        "\n",
        "For all couples of states s and actions $a$, the Q-Values are initialized to 0.\n",
        "The initial state is $s_0$. A random possible action is played and the first state $s_1$ is reached.\n",
        "For each $t ≥ 1$, a certain number of times (1000 times in this case study) the following steps are repeated:\n",
        "\n",
        "While the \n",
        "*A random state $s_t$ is selected from the possible 12 states\n",
        "* A random action $a_t$, that leads to the next possible state is played\n",
        "* The next state $s_t +1$ is reached, and a reward $R(s_t, a_t)$ is generated\n",
        "*The temporal difference $TD_t(s_t; a_t):$ is computed as follows:\n",
        " $TD_t(s_t, a_t) = R(s_t, a_t) +  γmax_a(Q(s_{t+1}, a)) - Q(s_t, a_t)$\n",
        " \n",
        "* The $Q-Value$ is updated by applying the Bellman equation:\n",
        " $Q_t(s_t, a_t) = Q_{t-1}(s_t, a_t) + α TD_t(s_t, a_t)$\n",
        " \n",
        "##  Implementation \n",
        " \n",
        " In this part, the Q-learning process is being implemented for creating the reward matrix for the location G.\n",
        " While the actions and location to state were defined previously it is now necessary to define the matrix of rewards and the parameters $γ$ and $α$:\n",
        " \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yPokptZX7rYx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gamma = 0.75\n",
        "alpha = 0.9"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tgz56ADj73lO",
        "colab_type": "text"
      },
      "source": [
        "[link text](https://)Since the location G has the top priority it is possible to define the matrix of rewards as follows, giving a high reward for the location G:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qkgMi5QD-P2O",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "R = np.array([[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
        "              [1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0],\n",
        "              [0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0],\n",
        "              [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0],\n",
        "              [0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0],\n",
        "              [0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0],\n",
        "              [0, 0, 1, 0, 0, 0, 1000, 1, 0, 0, 0, 0],\n",
        "              [0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1],\n",
        "              [0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0],\n",
        "              [0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0],\n",
        "              [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1],\n",
        "              [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OC1194qZ-k6u",
        "colab_type": "text"
      },
      "source": [
        "$Q-values$ are being initialised by a matrix of zeros"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x1zN0TQb-syw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "Q = np.array(np.zeros([12,12]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IqabvMz7-vOg",
        "colab_type": "text"
      },
      "source": [
        "Then the $Q-Learning$ process is implemented, with a for loop over 1000 iterations, repeating 1000 times the steps of the algorithm"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QGJwVl-r8Jqk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for i in range(1000):\n",
        "  current_state = np.random.randint(0,12)\n",
        "  playable_actions = []\n",
        "  for j in range(12):\n",
        "    if R[current_state, j] > 0:\n",
        "      playable_actions.append(j)\n",
        "  next_state = np.random.choice(playable_actions)\n",
        "  TD = R[current_state, next_state] + gamma*Q[next_state, np.argmax(Q[next_state,])] \n",
        "  - Q[current_state, next_state]\n",
        "  Q[current_state, next_state] = Q[current_state, next_state] + alpha*TD"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fc11aE5g_csb",
        "colab_type": "text"
      },
      "source": [
        "**The** Q values for the location G are calculated by the algorithm and it is possible to visualise them:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SqAe8Pys8Lxc",
        "colab_type": "code",
        "outputId": "f8bdcb14-2b6e-46b8-9a42-15ca84e348ee",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 312
        }
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "q_values = pd.DataFrame(Q, columns=[location for location in location_to_state])\n",
        "s = q_values.round().style.background_gradient(cmap='GnBu')\n",
        "s"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<style  type=\"text/css\" >\n",
              "    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col0 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col1 {\n",
              "            background-color:  #085ea1;\n",
              "            color:  #f1f1f1;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col2 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col3 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col4 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col5 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col6 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col7 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col8 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col9 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col10 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col11 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col0 {\n",
              "            background-color:  #084081;\n",
              "            color:  #f1f1f1;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col1 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col2 {\n",
              "            background-color:  #f1faeb;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col3 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col4 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col5 {\n",
              "            background-color:  #084081;\n",
              "            color:  #f1f1f1;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col6 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col7 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col8 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col9 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col10 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col11 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col0 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col1 {\n",
              "            background-color:  #d9f0d3;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col2 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col3 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col4 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col5 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col6 {\n",
              "            background-color:  #8dd3be;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col7 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col8 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col9 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col10 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col11 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col0 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col1 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col2 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col3 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col4 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col5 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col6 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col7 {\n",
              "            background-color:  #084081;\n",
              "            color:  #f1f1f1;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col8 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col9 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col10 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col11 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col0 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col1 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col2 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col3 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col4 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col5 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col6 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col7 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col8 {\n",
              "            background-color:  #3597c4;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col9 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col10 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col11 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col0 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col1 {\n",
              "            background-color:  #084081;\n",
              "            color:  #f1f1f1;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col2 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col3 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col4 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col5 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col6 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col7 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col8 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col9 {\n",
              "            background-color:  #ebf7e5;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col10 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col11 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col0 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col1 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col2 {\n",
              "            background-color:  #084081;\n",
              "            color:  #f1f1f1;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col3 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col4 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col5 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col6 {\n",
              "            background-color:  #dbf1d5;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col7 {\n",
              "            background-color:  #99d7ba;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col8 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col9 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col10 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col11 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col0 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col1 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col2 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col3 {\n",
              "            background-color:  #084081;\n",
              "            color:  #f1f1f1;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col4 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col5 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col6 {\n",
              "            background-color:  #084081;\n",
              "            color:  #f1f1f1;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col7 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col8 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col9 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col10 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col11 {\n",
              "            background-color:  #dff2da;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col0 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col1 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col2 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col3 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col4 {\n",
              "            background-color:  #084081;\n",
              "            color:  #f1f1f1;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col5 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col6 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col7 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col8 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col9 {\n",
              "            background-color:  #e5f5e0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col10 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col11 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col0 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col1 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col2 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col3 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col4 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col5 {\n",
              "            background-color:  #ccebc5;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col6 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col7 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col8 {\n",
              "            background-color:  #084081;\n",
              "            color:  #f1f1f1;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col9 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col10 {\n",
              "            background-color:  #c0e6c0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col11 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col0 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col1 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col2 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col3 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col4 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col5 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col6 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col7 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col8 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col9 {\n",
              "            background-color:  #084081;\n",
              "            color:  #f1f1f1;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col10 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col11 {\n",
              "            background-color:  #084081;\n",
              "            color:  #f1f1f1;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col0 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col1 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col2 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col3 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col4 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col5 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col6 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col7 {\n",
              "            background-color:  #7dcdc3;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col8 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col9 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col10 {\n",
              "            background-color:  #084081;\n",
              "            color:  #f1f1f1;\n",
              "        }    #T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col11 {\n",
              "            background-color:  #f7fcf0;\n",
              "            color:  #000000;\n",
              "        }</style><table id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >A</th>        <th class=\"col_heading level0 col1\" >B</th>        <th class=\"col_heading level0 col2\" >C</th>        <th class=\"col_heading level0 col3\" >D</th>        <th class=\"col_heading level0 col4\" >E</th>        <th class=\"col_heading level0 col5\" >F</th>        <th class=\"col_heading level0 col6\" >G</th>        <th class=\"col_heading level0 col7\" >H</th>        <th class=\"col_heading level0 col8\" >I</th>        <th class=\"col_heading level0 col9\" >J</th>        <th class=\"col_heading level0 col10\" >K</th>        <th class=\"col_heading level0 col11\" >L</th>    </tr></thead><tbody>\n",
              "                <tr>\n",
              "                        <th id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col0\" class=\"data row0 col0\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col1\" class=\"data row0 col1\" >1.76404e+19</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col2\" class=\"data row0 col2\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col3\" class=\"data row0 col3\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col4\" class=\"data row0 col4\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col5\" class=\"data row0 col5\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col6\" class=\"data row0 col6\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col7\" class=\"data row0 col7\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col8\" class=\"data row0 col8\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col9\" class=\"data row0 col9\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col10\" class=\"data row0 col10\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row0_col11\" class=\"data row0 col11\" >0</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col0\" class=\"data row1 col0\" >9.44967e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col1\" class=\"data row1 col1\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col2\" class=\"data row1 col2\" >7.58909e+16</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col3\" class=\"data row1 col3\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col4\" class=\"data row1 col4\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col5\" class=\"data row1 col5\" >6.49436e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col6\" class=\"data row1 col6\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col7\" class=\"data row1 col7\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col8\" class=\"data row1 col8\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col9\" class=\"data row1 col9\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col10\" class=\"data row1 col10\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row1_col11\" class=\"data row1 col11\" >0</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col0\" class=\"data row2 col0\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col1\" class=\"data row2 col1\" >3.33521e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col2\" class=\"data row2 col2\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col3\" class=\"data row2 col3\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col4\" class=\"data row2 col4\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col5\" class=\"data row2 col5\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col6\" class=\"data row2 col6\" >1.55824e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col7\" class=\"data row2 col7\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col8\" class=\"data row2 col8\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col9\" class=\"data row2 col9\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col10\" class=\"data row2 col10\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row2_col11\" class=\"data row2 col11\" >0</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col0\" class=\"data row3 col0\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col1\" class=\"data row3 col1\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col2\" class=\"data row3 col2\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col3\" class=\"data row3 col3\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col4\" class=\"data row3 col4\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col5\" class=\"data row3 col5\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col6\" class=\"data row3 col6\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col7\" class=\"data row3 col7\" >8.96045e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col8\" class=\"data row3 col8\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col9\" class=\"data row3 col9\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col10\" class=\"data row3 col10\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row3_col11\" class=\"data row3 col11\" >0</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col0\" class=\"data row4 col0\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col1\" class=\"data row4 col1\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col2\" class=\"data row4 col2\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col3\" class=\"data row4 col3\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col4\" class=\"data row4 col4\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col5\" class=\"data row4 col5\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col6\" class=\"data row4 col6\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col7\" class=\"data row4 col7\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col8\" class=\"data row4 col8\" >8.14949e+19</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col9\" class=\"data row4 col9\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col10\" class=\"data row4 col10\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row4_col11\" class=\"data row4 col11\" >0</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col0\" class=\"data row5 col0\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col1\" class=\"data row5 col1\" >1.95406e+19</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col2\" class=\"data row5 col2\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col3\" class=\"data row5 col3\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col4\" class=\"data row5 col4\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col5\" class=\"data row5 col5\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col6\" class=\"data row5 col6\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col7\" class=\"data row5 col7\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col8\" class=\"data row5 col8\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col9\" class=\"data row5 col9\" >5.03319e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col10\" class=\"data row5 col10\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row5_col11\" class=\"data row5 col11\" >0</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col0\" class=\"data row6 col0\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col1\" class=\"data row6 col1\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col2\" class=\"data row6 col2\" >2.27096e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col3\" class=\"data row6 col3\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col4\" class=\"data row6 col4\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col5\" class=\"data row6 col5\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col6\" class=\"data row6 col6\" >5.42501e+17</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col7\" class=\"data row6 col7\" >3.71871e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col8\" class=\"data row6 col8\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col9\" class=\"data row6 col9\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col10\" class=\"data row6 col10\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row6_col11\" class=\"data row6 col11\" >0</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col0\" class=\"data row7 col0\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col1\" class=\"data row7 col1\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col2\" class=\"data row7 col2\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col3\" class=\"data row7 col3\" >1.53848e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col4\" class=\"data row7 col4\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col5\" class=\"data row7 col5\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col6\" class=\"data row7 col6\" >3.45036e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col7\" class=\"data row7 col7\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col8\" class=\"data row7 col8\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col9\" class=\"data row7 col9\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col10\" class=\"data row7 col10\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row7_col11\" class=\"data row7 col11\" >1.77536e+18</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col0\" class=\"data row8 col0\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col1\" class=\"data row8 col1\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col2\" class=\"data row8 col2\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col3\" class=\"data row8 col3\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col4\" class=\"data row8 col4\" >8.15794e+19</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col5\" class=\"data row8 col5\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col6\" class=\"data row8 col6\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col7\" class=\"data row8 col7\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col8\" class=\"data row8 col8\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col9\" class=\"data row8 col9\" >7.81187e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col10\" class=\"data row8 col10\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row8_col11\" class=\"data row8 col11\" >0</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col0\" class=\"data row9 col0\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col1\" class=\"data row9 col1\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col2\" class=\"data row9 col2\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col3\" class=\"data row9 col3\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col4\" class=\"data row9 col4\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col5\" class=\"data row9 col5\" >1.64841e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col6\" class=\"data row9 col6\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col7\" class=\"data row9 col7\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col8\" class=\"data row9 col8\" >1.14199e+20</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col9\" class=\"data row9 col9\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col10\" class=\"data row9 col10\" >1.61021e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row9_col11\" class=\"data row9 col11\" >0</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col0\" class=\"data row10 col0\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col1\" class=\"data row10 col1\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col2\" class=\"data row10 col2\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col3\" class=\"data row10 col3\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col4\" class=\"data row10 col4\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col5\" class=\"data row10 col5\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col6\" class=\"data row10 col6\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col7\" class=\"data row10 col7\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col8\" class=\"data row10 col8\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col9\" class=\"data row10 col9\" >7.98802e+19</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col10\" class=\"data row10 col10\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row10_col11\" class=\"data row10 col11\" >1.32241e+19</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col0\" class=\"data row11 col0\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col1\" class=\"data row11 col1\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col2\" class=\"data row11 col2\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col3\" class=\"data row11 col3\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col4\" class=\"data row11 col4\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col5\" class=\"data row11 col5\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col6\" class=\"data row11 col6\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col7\" class=\"data row11 col7\" >4.42958e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col8\" class=\"data row11 col8\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col9\" class=\"data row11 col9\" >0</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col10\" class=\"data row11 col10\" >5.52426e+18</td>\n",
              "                        <td id=\"T_3177d75e_12b4_11ea_9dcf_0242ac1c0002row11_col11\" class=\"data row11 col11\" >0</td>\n",
              "            </tr>\n",
              "    </tbody></table>"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7f6d546d2b70>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e20ipdEmIClg",
        "colab_type": "text"
      },
      "source": [
        "From the visualisation it is possible to see that the location G got the highest $Q-value$, while locations far from G got a lower $Q-value$."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d3rZa9HZI1_x",
        "colab_type": "text"
      },
      "source": [
        "It is now possible to compute a function that returns the optimal route for any arbitrary location:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Czn8TP9nJZ3o",
        "colab_type": "code",
        "outputId": "763931fd-cea7-48d4-f2b1-d3e9060b60a2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "# Making a mapping from the states to the locations\n",
        "\n",
        "state_to_location = {state: location for location, \n",
        "                     state in location_to_state.items()}\n",
        "state_to_location            "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{0: 'A',\n",
              " 1: 'B',\n",
              " 2: 'C',\n",
              " 3: 'D',\n",
              " 4: 'E',\n",
              " 5: 'F',\n",
              " 6: 'G',\n",
              " 7: 'H',\n",
              " 8: 'I',\n",
              " 9: 'J',\n",
              " 10: 'K',\n",
              " 11: 'L'}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dYxmnptdTn0m",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "```\n",
        "# This is formatted as code\n",
        "```\n",
        "\n",
        "To compute a function that returns the optimal path for any arbitrary location it is necessary to redefine the `R` matrix:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_OX4N4WBTw3r",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "R = np.array([[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
        "              [1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0],\n",
        "              [0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0],\n",
        "              [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0],\n",
        "              [0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0],\n",
        "              [0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0],\n",
        "              [0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0],\n",
        "              [0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1],\n",
        "              [0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0],\n",
        "              [0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0],\n",
        "              [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1],\n",
        "              [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SdYSDdMeJGHV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def route(starting_location, ending_location):\n",
        "    R_new = np.copy(R)\n",
        "    ending_state = location_to_state[ending_location]\n",
        "    R_new[ending_state, ending_state] = 1000\n",
        "    Q = np.array(np.zeros([12,12]))\n",
        "    for i in range(1000):\n",
        "        current_state = np.random.randint(0,12)\n",
        "        playable_actions = []\n",
        "        for j in range(12):\n",
        "            if R_new[current_state, j] > 0:\n",
        "                playable_actions.append(j)\n",
        "        next_state = np.random.choice(playable_actions)\n",
        "        TD = R_new[current_state, next_state] + gamma * Q[next_state, np.argmax(Q[next_state,])] - Q[current_state, next_state]\n",
        "        Q[current_state, next_state] = Q[current_state, next_state] + alpha * TD\n",
        "    route = [starting_location]\n",
        "    next_location = starting_location\n",
        "    while (next_location != ending_location):\n",
        "        starting_state = location_to_state[starting_location]\n",
        "        next_state = np.argmax(Q[starting_state,])\n",
        "        next_location = state_to_location[next_state]\n",
        "        route.append(next_location)\n",
        "        starting_location = next_location\n",
        "    return route"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SjH1bhzkKd-o",
        "colab_type": "code",
        "outputId": "b499d750-97a1-402d-9546-603ebd87db1d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "route('E', 'G')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['E', 'I', 'J', 'F', 'B', 'C', 'G']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o8BsIZbzeGeq",
        "colab_type": "text"
      },
      "source": [
        "<img src=\"https://i.ibb.co/VJ1KKcR/warehouse-1.png\">"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fQI4ourzWzZP",
        "colab_type": "text"
      },
      "source": [
        "It is now possible to make an additional function `best_route()` that takes as input the starting, the intermediary and the ending location, that will call the `route()` function twice, the first time between the starting and the intermediary location, and the second time between the intermediary to the ending location:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4KqzfWIOKgJG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def best_route(starting_location, intermediary_location, ending_location):\n",
        "    return route(starting_location, intermediary_location) + route(intermediary_location, ending_location)[1:]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-aPuiFqOXLUb",
        "colab_type": "code",
        "outputId": "d955d94f-e30a-4941-e2f5-4f3adf9de33c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "best_route('E', 'K', 'G')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['E', 'I', 'J', 'K', 'L', 'H', 'G']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DqOdt4emXUij",
        "colab_type": "text"
      },
      "source": [
        "<img src=\"https://i.ibb.co/k2HBnyy/warehouse-2.png\">"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bnsswkrpdoVp",
        "colab_type": "text"
      },
      "source": [
        "## Demonstration"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BNEVlr0lbFys",
        "colab_type": "code",
        "cellView": "form",
        "outputId": "07153cc3-391f-4065-a7fc-5b7d8525923b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 263
        }
      },
      "source": [
        "#@markdown Please choose an initial, intermediary and final location:\n",
        "\n",
        "initial = \"A\" #@param [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\", \"K\", \"L\"] {allow-input: true}\n",
        "intermediary = \"G\" #@param [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\", \"K\", \"L\"] {allow-input: true}\n",
        "final = \"K\" #@param [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\", \"K\", \"L\"] {allow-input: true}\n",
        "#@markdown ---\n",
        "\n",
        "best = best_route(initial, intermediary, final)\n",
        "print('The best route is: ')\n",
        "print(*best, sep=', ')\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-f15ea9354362>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;31m#@markdown ---\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0mbest\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbest_route\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minitial\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mintermediary\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfinal\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      8\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'The best route is: '\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mbest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m', '\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'best_route' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "agxsJJQQZLwW",
        "colab_type": "text"
      },
      "source": [
        "## References \n",
        "\n",
        "* [An introduction to Q-Learning](https://www.freecodecamp.org/news/an-introduction-to-q-learning-reinforcement-learning-14ac0b4493cc/)\n",
        "* [Reinforcement learning](https://medium.com/machine-learning-for-humans/reinforcement-learning-6eacf258b265)\n",
        "*[Math of Q-Learning](https://medium.com/datadriveninvestor/math-of-q-learning-python-code-5dcbdc49b6f6)"
      ]
    }
  ]
}