004 Tensors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `004` Tensors\n",
    "\n",
    "Requirements: none.\n",
    "\n",
    "A tensor is a mathematical structure with an arbitrary number of dimensions. This abstraction can represent many different typical supports:\n",
    "* A zero-dimensional tensor is a scalar; e.g. `4` is a 0-d tensor.\n",
    "* A one-dimensional tensor is a vector; e.g. `[0, 1, 2]` is a 1-d tensor.\n",
    "* A two-dimensional tensor is a matrix\n",
    "* A three-dimensional tensor can be thought as a 3d volume or a matrix where each component is a vector.\n",
    "* A four-dimensional tensor can be thought as a matrix in which each component is another vector, or a volume in which each component is a vector, and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1, 2, 3, 4])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "torch.Size([4])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "a = torch.tensor([1, 2, 3, 4])\n",
    "print(a)\n",
    "a.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shape of a tensor indicates the size of each dimension. Each dimension (or axis) may have a certain size, known as its dimensionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1, 2, 3],\n",
      "        [4, 5, 6]])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "b = torch.tensor([[1, 2, 3], [4, 5, 6]])\n",
    "print(b)\n",
    "b.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access elements in a tensor using squared bracket notation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1, 2, 3],\n",
      "        [4, 5, 6]])\n",
      "tensor([1, 2, 3])\n",
      "tensor(1)\n",
      "tensor([1, 4])\n"
     ]
    }
   ],
   "source": [
    "a = torch.tensor([[1, 2, 3], [4, 5, 6]])\n",
    "print(a)\n",
    "print(a[0])\n",
    "print(a[0, 0])\n",
    "print(a[:, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the operations available in normal maths are also implemented for tensors. For instance, we can take two vectors and add or multiply them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([2, 4, 6, 8])\n",
      "tensor([3, 4, 5, 6])\n"
     ]
    }
   ],
   "source": [
    "a = torch.tensor([1, 2, 3, 4])\n",
    "b = torch.tensor([2, 2, 2, 2])\n",
    "print(a * b)\n",
    "print(a + b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also operate with tensors with different dimensions, like one 2 x 3 matrix and one 3-sized vector.\n",
    "\n",
    "When this happens, the following rule is applied:\n",
    "* First, dimensions are right aligned, and the missing ones become 1. Since our matrix has shape (2, 3) and two dimensions, our vector will become (1, 3).\n",
    "* Then, dimensions of the same size are left as they are, but dimensions with different size 1 are **broadcast**, or repeated. That makes our vector be repeated twice row-wise into a matrix.\n",
    "* Finally, the operation is performed.\n",
    "\n",
    "In some cases, this process will try to match two dimensions in which one of them is not 1 or the same as the other operator. In these cases we will get an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0.1470, -0.0082, -0.8585],\n",
       "         [-0.9397, -0.3412,  2.3592]]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = torch.randn(1, 2, 3)\n",
    "b = torch.ones(2, 3)\n",
    "c = torch.ones(1, 3)\n",
    "d = torch.ones(2, 5)\n",
    "\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[1.1470, 0.9918, 0.1415],\n",
       "         [0.0603, 0.6588, 3.3592]]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a + b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[1.1470, 0.9918, 0.1415],\n",
       "         [0.0603, 0.6588, 3.3592]]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a + c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "The size of tensor a (3) must match the size of tensor b (5) at non-singleton dimension 2",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[10], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43ma\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m+\u001b[39;49m\u001b[43m \u001b[49m\u001b[43md\u001b[49m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: The size of tensor a (3) must match the size of tensor b (5) at non-singleton dimension 2"
     ]
    }
   ],
   "source": [
    "a + d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Matrix multiplication is represented by the `@` operator. This operation is different from the element-wise multiplication, which is represented by the `*` operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[32]])\n",
      "tensor([[ 4,  8, 12],\n",
      "        [ 5, 10, 15],\n",
      "        [ 6, 12, 18]])\n"
     ]
    }
   ],
   "source": [
    "a = torch.tensor([[1, 2, 3]])\n",
    "b = torch.tensor([[4], [5], [6]])\n",
    "print(a @ b)\n",
    "print(b @ a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides these functions, there are a bunch of useful ones 1-ary operators:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 1.0283,  0.2767,  0.4558,  2.0266, -0.1950, -0.1028,  1.2314, -1.7533,\n",
      "        -0.0986, -0.3401])\n",
      "tensor(2.5290)\n",
      "tensor(0.2529)\n",
      "tensor([1.0283, 0.2767, 0.4558, 2.0266, 0.0000, 0.0000, 1.2314, 0.0000, 0.0000,\n",
      "        0.0000])\n"
     ]
    }
   ],
   "source": [
    "a = torch.randn(10)\n",
    "print(a)\n",
    "print(a.sum())\n",
    "print(a.mean())\n",
    "print(a.relu())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part of the secret power of torch tensors is that all the operations happen at a very low memory-operation level that makes things really fast. Plus, you can use the `device` parameter to send the tensor to the GPU if available."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}