05a_foundations.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to opening comments and overview of lesson 10](https://course.fast.ai/videos/?lesson=10&t=108)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Callbacks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Callbacks as GUI events"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=432)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipywidgets as widgets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(o): print('hi')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the [ipywidget docs](https://ipywidgets.readthedocs.io/en/stable/examples/Widget%20Events.html):\n",
    "\n",
    "- *the button widget is used to handle mouse clicks. The on_click method of the Button can be used to register function to be called when the button is clicked*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = widgets.Button(description='Click me')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "15fc98bc106246bf959514cfc19b1f49",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Button(description='Click me', style=ButtonStyle())"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hi\n",
      "hi\n"
     ]
    }
   ],
   "source": [
    "w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w.on_click(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*NB: When callbacks are used in this way they are often called \"events\".*\n",
    "\n",
    "Did you know what you can create interactive apps in Jupyter with these widgets? Here's an example from [plotly](https://plot.ly/python/widget-app/):\n",
    "\n",
    "![](https://cloud.githubusercontent.com/assets/12302455/16637308/4e476280-43ac-11e6-9fd3-ada2c9506ee1.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating your own callback"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=680)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import sleep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def slow_calculation():\n",
    "    res = 0\n",
    "    for i in range(5):\n",
    "        res += i*i\n",
    "        sleep(1)\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slow_calculation()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def slow_calculation(cb=None):\n",
    "    res = 0\n",
    "    for i in range(5):\n",
    "        res += i*i\n",
    "        sleep(1)\n",
    "        if cb: cb(i)\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_progress(epoch):\n",
    "    print(f\"Awesome! We've finished epoch {epoch}!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Awesome! We've finished epoch 0!\n",
      "Awesome! We've finished epoch 1!\n",
      "Awesome! We've finished epoch 2!\n",
      "Awesome! We've finished epoch 3!\n",
      "Awesome! We've finished epoch 4!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slow_calculation(show_progress)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lambdas and partials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=811)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Awesome! We've finished epoch 0!\n",
      "Awesome! We've finished epoch 1!\n",
      "Awesome! We've finished epoch 2!\n",
      "Awesome! We've finished epoch 3!\n",
      "Awesome! We've finished epoch 4!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slow_calculation(lambda o: print(f\"Awesome! We've finished epoch {o}!\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_progress(exclamation, epoch):\n",
    "    print(f\"{exclamation}! We've finished epoch {epoch}!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OK I guess! We've finished epoch 0!\n",
      "OK I guess! We've finished epoch 1!\n",
      "OK I guess! We've finished epoch 2!\n",
      "OK I guess! We've finished epoch 3!\n",
      "OK I guess! We've finished epoch 4!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slow_calculation(lambda o: show_progress(\"OK I guess\", o))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_show_progress(exclamation):\n",
    "    _inner = lambda epoch: print(f\"{exclamation}! We've finished epoch {epoch}!\")\n",
    "    return _inner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nice!! We've finished epoch 0!\n",
      "Nice!! We've finished epoch 1!\n",
      "Nice!! We've finished epoch 2!\n",
      "Nice!! We've finished epoch 3!\n",
      "Nice!! We've finished epoch 4!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slow_calculation(make_show_progress(\"Nice!\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_show_progress(exclamation):\n",
    "    # Leading \"_\" is generally understood to be \"private\"\n",
    "    def _inner(epoch): print(f\"{exclamation}! We've finished epoch {epoch}!\")\n",
    "    return _inner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nice!! We've finished epoch 0!\n",
      "Nice!! We've finished epoch 1!\n",
      "Nice!! We've finished epoch 2!\n",
      "Nice!! We've finished epoch 3!\n",
      "Nice!! We've finished epoch 4!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slow_calculation(make_show_progress(\"Nice!\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f2 = make_show_progress(\"Terrific\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Terrific! We've finished epoch 0!\n",
      "Terrific! We've finished epoch 1!\n",
      "Terrific! We've finished epoch 2!\n",
      "Terrific! We've finished epoch 3!\n",
      "Terrific! We've finished epoch 4!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slow_calculation(f2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Amazing! We've finished epoch 0!\n",
      "Amazing! We've finished epoch 1!\n",
      "Amazing! We've finished epoch 2!\n",
      "Amazing! We've finished epoch 3!\n",
      "Amazing! We've finished epoch 4!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slow_calculation(make_show_progress(\"Amazing\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import partial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OK I guess! We've finished epoch 0!\n",
      "OK I guess! We've finished epoch 1!\n",
      "OK I guess! We've finished epoch 2!\n",
      "OK I guess! We've finished epoch 3!\n",
      "OK I guess! We've finished epoch 4!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slow_calculation(partial(show_progress, \"OK I guess\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f2 = partial(show_progress, \"OK I guess\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Callbacks as callable classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=1122)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ProgressShowingCallback():\n",
    "    def __init__(self, exclamation=\"Awesome\"): self.exclamation = exclamation\n",
    "    def __call__(self, epoch): print(f\"{self.exclamation}! We've finished epoch {epoch}!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cb = ProgressShowingCallback(\"Just super\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Just super! We've finished epoch 0!\n",
      "Just super! We've finished epoch 1!\n",
      "Just super! We've finished epoch 2!\n",
      "Just super! We've finished epoch 3!\n",
      "Just super! We've finished epoch 4!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slow_calculation(cb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiple callback funcs; `*args` and `**kwargs`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=1194)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(*args, **kwargs): print(f\"args: {args}; kwargs: {kwargs}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "args: (3, 'a'); kwargs: {'thing1': 'hello'}\n"
     ]
    }
   ],
   "source": [
    "f(3, 'a', thing1=\"hello\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NB: We've been guilty of over-using kwargs in fastai - it's very convenient for the developer, but is annoying for the end-user unless care is taken to ensure docs show all kwargs too. kwargs can also hide bugs (because it might not tell you about a typo in a param name). In [R](https://www.r-project.org/) there's a very similar issue (R uses `...` for the same thing), and matplotlib uses kwargs a lot too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def slow_calculation(cb=None):\n",
    "    res = 0\n",
    "    for i in range(5):\n",
    "        if cb: cb.before_calc(i)\n",
    "        res += i*i\n",
    "        sleep(1)\n",
    "        if cb: cb.after_calc(i, val=res)\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PrintStepCallback():\n",
    "    def __init__(self): pass\n",
    "    def before_calc(self, *args, **kwargs): print(f\"About to start\")\n",
    "    def after_calc (self, *args, **kwargs): print(f\"Done step\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "About to start\n",
      "Done step\n",
      "About to start\n",
      "Done step\n",
      "About to start\n",
      "Done step\n",
      "About to start\n",
      "Done step\n",
      "About to start\n",
      "Done step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slow_calculation(PrintStepCallback())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PrintStatusCallback():\n",
    "    def __init__(self): pass\n",
    "    def before_calc(self, epoch, **kwargs): print(f\"About to start: {epoch}\")\n",
    "    def after_calc (self, epoch, val, **kwargs): print(f\"After {epoch}: {val}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "About to start: 0\n",
      "After 0: 0\n",
      "About to start: 1\n",
      "After 1: 1\n",
      "About to start: 2\n",
      "After 2: 5\n",
      "About to start: 3\n",
      "After 3: 14\n",
      "About to start: 4\n",
      "After 4: 30\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slow_calculation(PrintStatusCallback())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modifying behavior"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=1454)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def slow_calculation(cb=None):\n",
    "    res = 0\n",
    "    for i in range(5):\n",
    "        if cb and hasattr(cb,'before_calc'): cb.before_calc(i)\n",
    "        res += i*i\n",
    "        sleep(1)\n",
    "        if cb and hasattr(cb,'after_calc'):\n",
    "            if cb.after_calc(i, res):\n",
    "                print(\"stopping early\")\n",
    "                break\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PrintAfterCallback():\n",
    "    def after_calc (self, epoch, val):\n",
    "        print(f\"After {epoch}: {val}\")\n",
    "        if val>10: return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After 0: 0\n",
      "After 1: 1\n",
      "After 2: 5\n",
      "After 3: 14\n",
      "stopping early\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "14"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "slow_calculation(PrintAfterCallback())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SlowCalculator():\n",
    "    def __init__(self, cb=None): self.cb,self.res = cb,0\n",
    "    \n",
    "    def callback(self, cb_name, *args):\n",
    "        if not self.cb: return\n",
    "        cb = getattr(self.cb,cb_name, None)\n",
    "        if cb: return cb(self, *args)\n",
    "\n",
    "    def calc(self):\n",
    "        for i in range(5):\n",
    "            self.callback('before_calc', i)\n",
    "            self.res += i*i\n",
    "            sleep(1)\n",
    "            if self.callback('after_calc', i):\n",
    "                print(\"stopping early\")\n",
    "                break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ModifyingCallback():\n",
    "    def after_calc (self, calc, epoch):\n",
    "        print(f\"After {epoch}: {calc.res}\")\n",
    "        if calc.res>10: return True\n",
    "        if calc.res<3: calc.res = calc.res*2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calculator = SlowCalculator(ModifyingCallback())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After 0: 0\n",
      "After 1: 1\n",
      "After 2: 6\n",
      "After 3: 15\n",
      "stopping early\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "15"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calculator.calc()\n",
    "calculator.res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `__dunder__` thingies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Anything that looks like `__this__` is, in some way, *special*. Python, or some library, can define some functions that they will call at certain documented times. For instance, when your class is setting up a new object, python will call `__init__`. These are defined as part of the python [data model](https://docs.python.org/3/reference/datamodel.html#object.__init__).\n",
    "\n",
    "For instance, if python sees `+`, then it will call the special method `__add__`. If you try to display an object in Jupyter (or lots of other places in Python) it will call `__repr__`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=1647)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SloppyAdder():\n",
    "    def __init__(self,o): self.o=o\n",
    "    def __add__(self,b): return SloppyAdder(self.o + b.o + 0.01)\n",
    "    def __repr__(self): return str(self.o)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.01"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = SloppyAdder(1)\n",
    "b = SloppyAdder(2)\n",
    "a+b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Special methods you should probably know about (see data model link above) are:\n",
    "\n",
    "- `__getitem__`\n",
    "- `__getattr__`\n",
    "- `__setattr__`\n",
    "- `__del__`\n",
    "- `__init__`\n",
    "- `__new__`\n",
    "- `__enter__`\n",
    "- `__exit__`\n",
    "- `__len__`\n",
    "- `__repr__`\n",
    "- `__str__`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Variance and stuff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Variance is the average of how far away each data point is from the mean. E.g.:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=2133)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t = torch.tensor([1.,2.,4.,18])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(6.2500)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = t.mean(); m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(t-m).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops. We can't do that. Because by definition the positives and negatives cancel out. So we can fix that in one of (at least) two ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(47.1875)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(t-m).pow(2).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(5.8750)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(t-m).abs().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the first of these is now a totally different scale, since we squared. So let's undo that at the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(6.8693)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(t-m).pow(2).mean().sqrt()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They're still different. Why?\n",
    "\n",
    "Note that we have one outlier (`18`). In the version where we square everything, it makes that much bigger than everything else.\n",
    "\n",
    "`(t-m).pow(2).mean()` is refered to as **variance**. It's a measure of how spread out the data is, and is particularly sensitive to outliers.\n",
    "\n",
    "When we take the sqrt of the variance, we get the **standard deviation**. Since it's on the same kind of scale as the original data, it's generally more interpretable. However, since `sqrt(1)==1`, it doesn't much matter which we use when talking about *unit variance* for initializing neural nets.\n",
    "\n",
    "`(t-m).abs().mean()` is referred to as the **mean absolute deviation**. It isn't used nearly as much as it deserves to be, because mathematicians don't like how awkward it is to work with. But that shouldn't stop us, because we have computers and stuff.\n",
    "\n",
    "Here's a useful thing to note about variance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(47.1875), tensor(47.1875))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(t-m).pow(2).mean(), (t*t).mean() - (m*m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see why these are equal if you want to work thru the algebra. Or not.\n",
    "\n",
    "But, what's important here is that the latter is generally much easier to work with. In particular, you only have to track two things: the sum of the data, and the sum of squares of the data. Whereas in the first form you actually have to go thru all the data twice (once to calculate the mean, once to calculate the differences).\n",
    "\n",
    "Let's go steal the LaTeX from [Wikipedia](https://en.wikipedia.org/wiki/Variance):\n",
    "\n",
    "$$\\operatorname{E}\\left[X^2 \\right] - \\operatorname{E}[X]^2$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Covariance and correlation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how Wikipedia defines covariance:\n",
    "\n",
    "$$\\operatorname{cov}(X,Y) = \\operatorname{E}{\\big[(X - \\operatorname{E}[X])(Y - \\operatorname{E}[Y])\\big]}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=2414)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 1.,  2.,  4., 18.])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see that in code. So now we need two vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAERJJREFUeJzt3X+s3Xddx/Hni7ZIBbQbu2C3MYtkqaAJLbk20ymZ40fHJFCMGojBRpYUEkggYmWDBMFoAlaY0RhMYZNqkB9C2QgBSzNHCIkM77Zu3Sy1Yw5dW9eLUDZig1t5+8f5llyu5/aee3vOPbef+3wkJ+d7Pt/P2Xnle89e93u/5/vtSVUhSTr/PWncASRJw2GhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhqxeilf7KKLLqoNGzYs5UtK0nnvzjvv/FZVTcw3b0kLfcOGDUxNTS3lS0rSeS/JNweZ5yEXSWrEwIWeZFWSu5N8rnv8nCR3JDmS5BNJnjy6mJKk+SxkD/0twKEZj98H3FhVlwPfAa4bZjBJ0sIMVOhJLgV+Dfhw9zjA1cCnuil7gG2jCChJGsyge+h/DvwB8IPu8TOAk1X1RPf4YeCSIWeTJC3AvGe5JHkFcKKq7kxy1ZnhPlP7flNGkh3ADoDLLrtskTEl6fxzy91H2bXvMMdOnuLidWvZuXUj2zaPbt93kNMWrwRemeRa4CnAT9DbY1+XZHW3l34pcKzfk6tqN7AbYHJy0q9HkrQi3HL3UW7Ye5BTj58G4OjJU9yw9yDAyEp93kMuVXVDVV1aVRuA1wD/VFW/DdwO/EY3bTtw60gSStJ5aNe+wz8s8zNOPX6aXfsOj+w1z+U89LcDv5fkAXrH1G8aTiRJOv8dO3lqQePDsKArRavqS8CXuuUHgS3DjyRJ57+L163laJ/yvnjd2pG9pleKStII7Ny6kbVrVv3I2No1q9i5dePIXnNJ/y0XSVopznzwudzOcpEkLcK2zZeMtMBn85CLJDXCQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0SWqEhS5JjZi30JM8JcnXktyT5P4k7+nGP5Lk35Mc6G6bRh9XkjSXQb7g4vvA1VX1vSRrgK8k+UK3bmdVfWp08SRJg5q30KuqgO91D9d0txplKEnSwg10DD3JqiQHgBPA/qq6o1v1J0nuTXJjkh+b47k7kkwlmZqenh5SbEnSbAMVelWdrqpNwKXAliQ/D9wA/CzwC8CFwNvneO7uqpqsqsmJiYkhxZYkzbags1yq6iTwJeCaqjpePd8H/gbYMoJ8kqQBDXKWy0SSdd3yWuAlwNeTrO/GAmwD7htlUEnS2Q1ylst6YE+SVfR+AXyyqj6X5J+STAABDgBvHGFOSdI8BjnL5V5gc5/xq0eSSJK0KF4pKkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0SWqEhS5JjbDQJakRFrokNcJCl6RGWOiS1AgLXZIaYaFLUiMsdElqhIUuSY0Y5DtFn5Lka0nuSXJ/kvd0489JckeSI0k+keTJo48rSZrLIHvo3weurqoXAJuAa5JcAbwPuLGqLge+A1w3upiSpPnMW+jV873u4ZruVsDVwKe68T3AtpEklCQNZKBj6ElWJTkAnAD2A98ATlbVE92Uh4FLRhNRkjSIgQq9qk5X1SbgUmAL8Lx+0/o9N8mOJFNJpqanpxefVJJ0Vgs6y6WqTgJfAq4A1iVZ3a26FDg2x3N2V9VkVU1OTEycS1ZJ0lkMcpbLRJJ13fJa4CXAIeB24De6aduBW0cVUpI0v9XzT2E9sCfJKnq/AD5ZVZ9L8q/Ax5P8MXA3cNMIc0qS5jFvoVfVvcDmPuMP0jueLklaBrxSVJIaYaFLUiMsdElqhIUuSY2w0CWpERa6JDXCQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIwb5kuhnJ7k9yaEk9yd5Szf+7iRHkxzobteOPq4kaS6DfEn0E8DbququJE8H7kyyv1t3Y1X92ejiSZIGNciXRB8HjnfLjyU5BFwy6mCSpIVZ0DH0JBuAzcAd3dCbk9yb5OYkF8zxnB1JppJMTU9Pn1NYSdLcBi70JE8DPg28taoeBT4IPBfYRG8P/v39nldVu6tqsqomJyYmhhBZktTPQIWeZA29Mv9oVe0FqKpHqup0Vf0A+BCwZXQxJUnzGeQslwA3AYeq6gMzxtfPmPZq4L7hx5MkDWqQs1yuBF4HHExyoBt7B/DaJJuAAh4C3jCShJKkgQxylstXgPRZ9fnhx5EkLZZXikpSIyx0SWqEhS5JjbDQJakRFrokNcJCl6RGWOiS1AgLXZIaYaFLUiMsdElqhIUuSY2w0CWpERa6JDXCQpekRljoktQIC12SGmGhS1IjBvlO0WcnuT3JoST3J3lLN35hkv1JjnT3F4w+riRpLoPsoT8BvK2qngdcAbwpyfOB64Hbqupy4LbusSRpTOYt9Ko6XlV3dcuPAYeAS4BXAXu6aXuAbaMKKUma34KOoSfZAGwG7gCeVVXHoVf6wDOHHU6SNLiBCz3J04BPA2+tqkcX8LwdSaaSTE1PTy8moyRpAAMVepI19Mr8o1W1txt+JMn6bv164ES/51bV7qqarKrJiYmJYWSWJPUxyFkuAW4CDlXVB2as+iywvVveDtw6/HiSpEGtHmDOlcDrgINJDnRj7wDeC3wyyXXAfwC/OZqIkqRBzFvoVfUVIHOsfvFw40iSFssrRSWpERa6JDXCQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0SWqEhS5JjbDQJakRg3yn6M1JTiS5b8bYu5McTXKgu1072piSpPkMsof+EeCaPuM3VtWm7vb54caSJC3UvIVeVV8Gvr0EWSRJ5+BcjqG/Ocm93SGZC4aWSJK0KIst9A8CzwU2AceB9881McmOJFNJpqanpxf5cpKk+Syq0Kvqkao6XVU/AD4EbDnL3N1VNVlVkxMTE4vNKUmax6IKPcn6GQ9fDdw311xJ0tJYPd+EJB8DrgIuSvIw8IfAVUk2AQU8BLxhhBklSQOYt9Cr6rV9hm8aQRZJ0jnwSlFJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0SWqEhS5JjbDQJakRFrokNcJCl6RGWOiS1AgLXZIaYaFLUiMsdElqhIUuSY2Yt9CT3JzkRJL7ZoxdmGR/kiPd/QWjjSlJms8ge+gfAa6ZNXY9cFtVXQ7c1j2WJI3RvIVeVV8Gvj1r+FXAnm55D7BtyLkkSQu02GPoz6qq4wDd/TOHF0mStBgj/1A0yY4kU0mmpqenR/1ykrRiLbbQH0myHqC7PzHXxKraXVWTVTU5MTGxyJeTJM1nsYX+WWB7t7wduHU4cSRJizXIaYsfA/4Z2Jjk4STXAe8FXprkCPDS7rEkaYxWzzehql47x6oXDzmLJOkceKWoJDVi3j10jcYtdx9l177DHDt5iovXrWXn1o1s23zJuGNJOo9Z6GNwy91HuWHvQU49fhqAoydPccPegwCWuqRF85DLGOzad/iHZX7GqcdPs2vf4TElktQCC30Mjp08taBxSRqEhT4GF69bu6BxSRqEhT4GO7duZO2aVT8ytnbNKnZu3TimRJJa4IeiY3Dmg0/PcpE0TBb6mGzbfIkFLmmoPOQiSY2w0CWpERa6JDXCQpekRljoktQIC12SGmGhS1IjLHRJasQ5XViU5CHgMeA08ERVTQ4jlCRp4YZxpeivVtW3hvDfkSSdAw+5SFIjzrXQC/hikjuT7BhGIEnS4pzrIZcrq+pYkmcC+5N8vaq+PHNCV/Q7AC677LJzfDlJ0lzOaQ+9qo519yeAzwBb+szZXVWTVTU5MTFxLi8nSTqLRe+hJ3kq8KSqeqxbfhnwR0NLNmS33H3Uf39cUtPO5ZDLs4DPJDnz3/n7qvrHoaQaslvuPsoNew/+8IuZj548xQ17DwJY6pKasehCr6oHgRcMMcvI7Np3+Idlfsapx0+za99hC11SM1bEaYvHTp5a0LgknY9WRKFfvG7tgsYl6Xy0Igp959aNrF2z6kfG1q5Zxc6tG8eUSJKGb0V8SfSZ4+Se5SKpZSui0KFX6ha4pJatiEMukrQSWOiS1AgLXZIaseyPoXvJviQNZlkXupfsS9LglvUhl7Ndsi9J+lHLutC9ZF+SBresC91L9iVpcMu60L1kX5IGt6w/FPWSfUka3LIudPCSfUka1LI+5CJJGpyFLkmNsNAlqREWuiQ1wkKXpEakqpbuxZJp4JtL9oKLdxHwrXGHWCAzj975lhfMvBSWIu9PV9XEfJOWtNDPF0mmqmpy3DkWwsyjd77lBTMvheWU10MuktQIC12SGmGh97d73AEWwcyjd77lBTMvhWWT12PoktQI99AlqRErttCTPDvJ7UkOJbk/yVv6zLkqyXeTHOhu7xpH1lmZHkpysMsz1Wd9kvxFkgeS3JvkhePI2WXZOGPbHUjyaJK3zpoz9m2c5OYkJ5LcN2PswiT7kxzp7i+Y47nbuzlHkmwfc+ZdSb7e/dw/k2TdHM8963toiTO/O8nRGT//a+d47jVJDnfv6+vHmPcTM7I+lOTAHM8dyzamqlbkDVgPvLBbfjrwb8DzZ825CvjcuLPOyvQQcNFZ1l8LfAEIcAVwx7gzd7lWAf9F73zaZbWNgRcBLwTumzH2p8D13fL1wPv6PO9C4MHu/oJu+YIxZn4ZsLpbfl+/zIO8h5Y487uB3x/gvfMN4GeAJwP3zP5/danyzlr/fuBdy2kbr9g99Ko6XlV3dcuPAYeAFv6d3lcBf1s9XwXWJVk/7lDAi4FvVNWyu7Csqr4MfHvW8KuAPd3yHmBbn6duBfZX1ber6jvAfuCakQWdoV/mqvpiVT3RPfwqcOlSZBnUHNt5EFuAB6rqwar6X+Dj9H4+I3W2vEkC/BbwsVHnWIgVW+gzJdkAbAbu6LP6F5Pck+QLSX5uSYP1V8AXk9yZZEef9ZcA/znj8cMsj19Ur2HuN/9y28YAz6qq49D75Q88s8+c5bqtAV5P7y+1fuZ7Dy21N3eHiW6e49DWctzOvwI8UlVH5lg/lm284gs9ydOATwNvrapHZ62+i94hghcAfwncstT5+riyql4IvBx4U5IXzVqfPs8Z66lMSZ4MvBL4hz6rl+M2HtSy29YASd4JPAF8dI4p872HltIHgecCm4Dj9A5jzLYct/NrOfve+Vi28You9CRr6JX5R6tq7+z1VfVoVX2vW/48sCbJRUscc3amY939CeAz9P4cnelh4NkzHl8KHFuadHN6OXBXVT0ye8Vy3MadR84cquruT/SZs+y2dffB7CuA367uYO5sA7yHlkxVPVJVp6vqB8CH5siyrLZzktXArwOfmGvOuLbxii307hjYTcChqvrAHHN+qptHki30ttd/L13K/5fnqUmefmaZ3odg982a9lngd7qzXa4Avnvm0MEYzbk3s9y28QyfBc6ctbIduLXPnH3Ay5Jc0B0qeFk3NhZJrgHeDryyqv5njjmDvIeWzKzPd149R5Z/AS5P8pzur73X0Pv5jMtLgK9X1cP9Vo51Gy/1p7DL5Qb8Mr0/2+4FDnS3a4E3Am/s5rwZuJ/ep+pfBX5pzJl/pstyT5frnd34zMwB/oreWQEHgckxZ/5xegX9kzPGltU2pvfL5jjwOL29weuAZwC3AUe6+wu7uZPAh2c89/XAA93td8ec+QF6x5rPvJ//upt7MfD5s72Hxpj577r36b30Snr97Mzd42vpnYn2jaXK3C9vN/6RM+/fGXOXxTb2SlFJasSKPeQiSa2x0CWpERa6JDXCQpekRljoktQIC12SGmGhS1IjLHRJasT/AQya0c286UAAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# `u` is twice `t`, plus a bit of randomness\n",
    "u = t*2\n",
    "u *= torch.randn_like(t)/10+0.95\n",
    "\n",
    "plt.scatter(t, u);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 59.8856,  39.8543,  11.6089, 304.8394])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prod = (t-t.mean())*(u-u.mean()); prod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(104.0471)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prod.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAEJVJREFUeJzt3W2MXFd9x/Hvr45Dty3CARsSOwkObWQ1iILTVQSlRZF4cBKhxFCoElUiPMmiImp5UYtESIB4A9RqK/EgkIGIgBBJS41xW6PlWalUBWUTJ3FCcONEoNibJibBSRHb4ph/X+w12mxm7bVnPLO75/uRRnPvuWfm/H12/Ju7Z+7YqSokSW35rVEXIEkaPsNfkhpk+EtSgwx/SWqQ4S9JDTL8JalBhr8kNcjwl6QGGf6S1KAzRl3AfFavXl3r168fdRmStKTccccdP6uqNSfqt2jDf/369UxOTo66DElaUpL8dCH9XPaRpAYZ/pLUIMNfkhpk+EtSgwx/SWqQ4S9JDVq0l3qO2s49B9k2sY+pw9OsXTXG1k0b2Lxx3ajLkqSBMPx72LnnIDfs2Mv0kaMAHDw8zQ079gL4BiBpWXDZp4dtE/t+E/zHTB85yraJfSOqSJIGy/DvYerw9Em1S9JSY/j3sHbV2Em1S9JSY/j3sHXTBsZWrnhG29jKFWzdtGFEFUnSYPmBbw/HPtT1ah9Jy5XhP4/NG9cZ9pKWLZd9JKlBhr8kNcjwl6QGGf6S1CDDX5IaZPhLUoMMf0lqkOEvSQ0y/CWpQQMJ/yQ3Jnksyb3zHL80yZNJ7upuHxzEuJKkUzOof97hi8CngC8dp89/VNUbBzSeJKkPAznzr6pbgScG8VySpNNvmGv+r0pyd5JvJnlprw5JtiSZTDJ56NChIZYmSW0ZVvjfCby4ql4OfBLY2atTVW2vqvGqGl+zZs2QSpOk9gwl/Kvqqar6Rbe9G1iZZPUwxpYkPdtQwj/J2UnSbV/Sjfv4MMaWJD3bQK72SfJV4FJgdZIDwIeAlQBV9VngLcBfJXkamAaurqoaxNiSpJM3kPCvqmtOcPxTzFwKKklaBPyGryQ1yPCXpAYZ/pLUIMNfkhpk+EtSgwx/SWqQ4S9JDTL8JalBhr8kNcjwl6QGGf6S1CDDX5IaZPhLUoMMf0lqkOEvSQ0y/CWpQYa/JDVoIP+TlySpfzv3HGTbxD6mDk+zdtUYWzdtYPPGdadlLMNfkhaBnXsOcsOOvUwfOQrAwcPT3LBjL8BpeQNw2UeSFoFtE/t+E/zHTB85yraJfadlPMNfkhaBqcPTJ9XeL8NfkhaBtavGTqq9XwMJ/yQ3Jnksyb3zHE+STyTZn+SeJBcPYlxJWi62btrA2MoVz2gbW7mCrZs2nJbxBnXm/0XgsuMcvxy4sLttAT4zoHElaVnYvHEdH33zy1i3aowA61aN8dE3v2xxX+1TVbcmWX+cLlcBX6qqAm5LsirJOVX1yCDGl6TlYPPGdact7Oca1pr/OuDhWfsHujZJ0ggMK/zTo62e1SnZkmQyyeShQ4eGUJYktWlY4X8AOG/W/rnA1NxOVbW9qsaranzNmjVDKk2S2jOs8N8FvK276ueVwJOu90vS6AzkA98kXwUuBVYnOQB8CFgJUFWfBXYDVwD7gV8C7xjEuJKkUzOoq32uOcHxAt47iLEkSf3zG76S1CDDX5IaZPhLUoMMf0lqkOEvSQ0y/CWpQYa/JDXI8JekBhn+ktQgw1+SGmT4S1KDDH9JapDhL0kNMvwlqUGGvyQ1yPCXpAYZ/pLUIMNfkhpk+EtSgwx/SWqQ4S9JDTL8JalBhr8kNWgg4Z/ksiT7kuxPcn2P429PcijJXd3t3YMYV5J0as7o9wmSrAA+DbweOADcnmRXVf1oTtdbquq6fseTJPVvEGf+lwD7q+qhqvoVcDNw1QCeV5J0mgwi/NcBD8/aP9C1zfXnSe5J8rUk5w1gXEnSKRpE+KdHW83Z/1dgfVX9EfAd4KaeT5RsSTKZZPLQoUMDKE2S1Msgwv8AMPtM/lxganaHqnq8qv6v2/0c8Me9nqiqtlfVeFWNr1mzZgClSZJ6GUT43w5cmOSCJGcCVwO7ZndIcs6s3SuB+wcwriTpFPV9tU9VPZ3kOmACWAHcWFX3JfkIMFlVu4C/TnIl8DTwBPD2fseVJJ26VM1dnl8cxsfHa3JyctRlSNKSkuSOqho/UT+/4StJDTL8JalBhr8kNcjwl6QGGf6S1CDDX5IaZPhLUoMMf0lqkOEvSQ0y/CWpQYa/JDXI8JekBhn+ktQgw1+SGmT4S1KDDH9JapDhL0kNMvwlqUGGvyQ1yPCXpAYZ/pLUIMNfkhpk+EtSgwYS/kkuS7Ivyf4k1/c4/pwkt3THf5hk/SDGlSSdmr7DP8kK4NPA5cBFwDVJLprT7V3Az6vqD4B/BD7e77iSpFM3iDP/S4D9VfVQVf0KuBm4ak6fq4Cbuu2vAa9NkgGMLUk6BYMI/3XAw7P2D3RtPftU1dPAk8AL5j5Rki1JJpNMHjp0aAClSZJ6GUT49zqDr1PoQ1Vtr6rxqhpfs2bNAEqTJPUyiPA/AJw3a/9cYGq+PknOAJ4HPDGAsSVJp2AQ4X87cGGSC5KcCVwN7JrTZxdwbbf9FuB7VfWsM39J0nCc0e8TVNXTSa4DJoAVwI1VdV+SjwCTVbUL+ALw5ST7mTnjv7rfcSVJp67v8Aeoqt3A7jltH5y1/b/AWwcxliSpf37DV5IaZPhLUoMMf0lqkOEvSQ0y/CWpQYa/JDXI8JekBhn+ktQgw1+SGmT4S1KDDH9JapDhL0kNMvwlqUGGvyQ1yPCXpAYZ/pLUIMNfkhpk+EtSgwx/SWqQ4S9JDTL8JalBhr8kNaiv8E/y/CTfTvJAd3/WPP2OJrmru+3qZ0xJUv/6PfO/HvhuVV0IfLfb72W6ql7R3a7sc0xJUp/O6PPxVwGXdts3AT8A3t/nc2qOnXsOsm1iH1OHp1m7aoytmzaweeO6UZclaQnr98z/RVX1CEB3/8J5+v12kskktyXZ3OeYTdm55yA37NjLwcPTFHDw8DQ37NjLzj0HR12apCXshGf+Sb4DnN3j0AdOYpzzq2oqyUuA7yXZW1UP9hhrC7AF4Pzzzz+Jp1++tk3sY/rI0We0TR85yraJfZ79SzplJwz/qnrdfMeSPJrknKp6JMk5wGPzPMdUd/9Qkh8AG4FnhX9VbQe2A4yPj9eC/gTL3NTh6ZNql6SF6HfZZxdwbbd9LfCNuR2SnJXkOd32auDVwI/6HLcZa1eNnVS7JC1Ev+H/MeD1SR4AXt/tk2Q8yee7Pn8ITCa5G/g+8LGqMvwXaOumDYytXPGMtrGVK9i6acOIKpK0HPR1tU9VPQ68tkf7JPDubvs/gZf1M07Ljq3re7WPpEHq91JPDcHmjesMe0kD5T/vIEkNMvwlqUGGvyQ1yPCXpAYZ/pLUIMNfkhpk+EtSgwx/SWqQ4S9JDTL8JalBhr8kNcjwl6QGGf6S1CDDX5IaZPhLUoMMf0lqkOEvSQ0y/CWpQYa/JDXI8JekBhn+ktQgw1+SGtRX+Cd5a5L7kvw6yfhx+l2WZF+S/Umu72dMSVL/+j3zvxd4M3DrfB2SrAA+DVwOXARck+SiPseVJPXhjH4eXFX3AyQ5XrdLgP1V9VDX92bgKuBH/YwtSTp1w1jzXwc8PGv/QNcmSRqRE575J/kOcHaPQx+oqm8sYIxevxbUPGNtAbYAnH/++Qt4aknSqThh+FfV6/oc4wBw3qz9c4GpecbaDmwHGB8f7/kGIUnq3zCWfW4HLkxyQZIzgauBXUMYV5I0j34v9XxTkgPAq4B/TzLRta9Nshugqp4GrgMmgPuBf6qq+/orW5LUj36v9vk68PUe7VPAFbP2dwO7+xlLkjQ4fsNXkhpk+EtSgwx/SWqQ4S9JDTL8JalBhr8kNcjwl6QGGf6S1CDDX5IaZPhLUoMMf0lqkOEvSQ3q6x92W4x27jnItol9TB2eZu2qMbZu2sDmjf7HYZI027IK/517DnLDjr1MHzkKwMHD09ywYy+AbwCSNMuyWvbZNrHvN8F/zPSRo2yb2DeiiiRpcVpW4T91ePqk2iWpVcsq/NeuGjupdklq1bIK/62bNjC2csUz2sZWrmDrpg0jqkiSFqdl9YHvsQ91vdpHko5vWYU/zLwBGPaSdHzLatlHkrQwhr8kNcjwl6QGGf6S1CDDX5IaZPhLUoNSVaOuoackh4CfjrqOBVoN/GzURZyEpVYvWPOwLLWal1q9cPprfnFVrTlRp0Ub/ktJksmqGh91HQu11OoFax6WpVbzUqsXFk/NLvtIUoMMf0lqkOE/GNtHXcBJWmr1gjUPy1KreanVC4ukZtf8JalBnvlLUoMM/wVIcl6S7ye5P8l9Sf6mR59LkzyZ5K7u9sFR1Dqnpp8k2dvVM9njeJJ8Isn+JPckuXgUdc6qZ8Os+bsryVNJ3jenz8jnOcmNSR5Lcu+stucn+XaSB7r7s+Z57LVdnweSXDvCercl+XH3c/96klXzPPa4r6Eh1/zhJAdn/eyvmOexlyXZ172urx9xzbfMqvcnSe6a57HDn+eq8naCG3AOcHG3/Vzgv4CL5vS5FPi3Udc6p6afAKuPc/wK4JtAgFcCPxx1zbNqWwH8NzPXLC+qeQZeA1wM3Dur7e+A67vt64GP93jc84GHuvuzuu2zRlTvG4Azuu2P96p3Ia+hIdf8YeBvF/C6eRB4CXAmcPfcv6vDrHnO8b8HPrhY5tkz/wWoqkeq6s5u+3+A+4Hl8J8GXAV8qWbcBqxKcs6oi+q8FniwqhbdF/2q6lbgiTnNVwE3dds3AZt7PHQT8O2qeqKqfg58G7jstBXa6VVvVX2rqp7udm8Dzj3ddZyMeeZ4IS4B9lfVQ1X1K+BmZn42p93xak4S4C+Arw6jloUw/E9SkvXARuCHPQ6/KsndSb6Z5KVDLay3Ar6V5I4kW3ocXwc8PGv/AIvnTe1q5v+LstjmGeBFVfUIzJwsAC/s0Wexzvc7mfkNsJcTvYaG7bpuqerGeZbWFusc/xnwaFU9MM/xoc+z4X8Skvwe8C/A+6rqqTmH72RmieLlwCeBncOur4dXV9XFwOXAe5O8Zs7x9HjMyC//SnImcCXwzz0OL8Z5XqhFN99JPgA8DXxlni4neg0N02eA3wdeATzCzDLKXItujjvXcPyz/qHPs+G/QElWMhP8X6mqHXOPV9VTVfWLbns3sDLJ6iGXObemqe7+MeDrzPxKPNsB4LxZ++cCU8Op7rguB+6sqkfnHliM89x59NiSWXf/WI8+i2q+uw+c3wj8ZXULz3Mt4DU0NFX1aFUdrapfA5+bp5ZFNccASc4A3gzcMl+fUcyz4b8A3XrdF4D7q+of5ulzdtePJJcwM7ePD6/KZ9Xzu0mee2ybmQ/47p3TbRfwtu6qn1cCTx5buhixec+SFts8z7ILOHb1zrXAN3r0mQDekOSsbsniDV3b0CW5DHg/cGVV/XKePgt5DQ3NnM+j3jRPLbcDFya5oPsN8mpmfjaj9Drgx1V1oNfBkc3zMD9dXqo34E+Z+dXxHuCu7nYF8B7gPV2f64D7mLm64DbgT0Zc80u6Wu7u6vpA1z675gCfZubqiL3A+CKY699hJsyfN6ttUc0zM29MjwBHmDnTfBfwAuC7wAPd/fO7vuPA52c99p3A/u72jhHWu5+ZtfFjr+fPdn3XAruP9xoaYc1f7l6n9zAT6OfMrbnbv4KZK/IeHHXNXfsXj71+Z/Ud+Tz7DV9JapDLPpLUIMNfkhpk+EtSgwx/SWqQ4S9JDTL8JalBhr8kNcjwl6QG/T+EUKgrqEIliwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "v = torch.randn_like(t)\n",
    "plt.scatter(t, v);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(3.3606)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "((t-t.mean())*(v-v.mean())).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's generally more conveniently defined like so:\n",
    "\n",
    "$$\\operatorname{E}\\left[X Y\\right] - \\operatorname{E}\\left[X\\right] \\operatorname{E}\\left[Y\\right]$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(3.3606)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cov = (t*v).mean() - t.mean()*v.mean(); cov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From now on, you're not allowed to look at an equation (or especially type it in LaTeX) without also typing it in Python and actually calculating some values. Ideally, you should also plot some values.\n",
    "\n",
    "Finally, here is the Pearson correlation coefficient:\n",
    "\n",
    "$$\\rho_{X,Y}= \\frac{\\operatorname{cov}(X,Y)}{\\sigma_X \\sigma_Y}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.2978)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cov / (t.std() * v.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's just a scaled version of the same thing. Question: *Why is it scaled by standard deviation, and not by variance or mean or something else?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Softmax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's our final `logsoftmax` definition:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=2674)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def log_softmax(x): return x - x.exp().sum(-1,keepdim=True).log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which is:\n",
    "\n",
    "$$\\hbox{logsoftmax(x)}_{i} = x_{i} - \\log \\sum_{j} e^{x_{j}}$$ \n",
    "\n",
    "And our cross entropy loss is:\n",
    "$$-\\log(p_{i})$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Browsing source code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=1782)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Jump to tag/symbol by with (with completions)\n",
    "- Jump to current tag\n",
    "- Jump to library tags\n",
    "- Go back\n",
    "- Search\n",
    "- Outlining / folding"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}