058-k8s-distributed.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Distributed Data Parallel Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Distributed Stochastic Gradient Descent\n",
    "\n",
    "- core operation: global sum reduction (already seen this)\n",
    "- other issues:\n",
    "    - arrange for initialization with same weights\n",
    "    - arrange for different kinds of training data for each node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No resources found\n",
      "No resources found\n"
     ]
    }
   ],
   "source": [
    "# make sure we are starting with a clean slate\n",
    "kubectl delete jobs --all\n",
    "kubectl delete pods --all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Configuration\n",
    "\n",
    "- headless service enables DNS resolution in cluster\n",
    "- communications on port 9000\n",
    "- environment variables tell clients where to connect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "cat > kubetpl.yaml <<'EOF'\n",
    "image: gcr.io/research-191823/bigdata19\n",
    "memory: 4G\n",
    "cpu: 1\n",
    "app: bigdata19\n",
    "subdomain: bigdata19\n",
    "port:\n",
    "  - 9000\n",
    "config_map: files\n",
    "env:\n",
    "  - MASTER_ADDR=master.bigdata19\n",
    "  - MASTER_PORT=9000\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "service \"bigdata19\" deleted\n",
      "service/bigdata19 created\n"
     ]
    }
   ],
   "source": [
    "kubectl delete service/bigdata19 || true\n",
    "kubetpl service | kubectl apply -f -"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Few Changes Needed for Distributed Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10a11\n",
      "> import torch.distributed as dist\n",
      "25a27\n",
      "> parser.add_argument(\"-P\", \"--sample-probability\", type=float, default=1.0)\n",
      "26a29,30\n",
      "> parser.add_argument(\"--seed\", type=int, default=902842093840)\n",
      "> parser.add_argument(\"--dist\", default=\"-1/-1\")\n",
      "32a37,41\n",
      "> rank, world = [int(x) for x in args.dist.split(\"/\")]\n",
      "> \n",
      "> if world > 0:\n",
      ">     dist.init_process_group(\"gloo\", rank=rank, world_size=world)\n",
      "> \n",
      "61a71,73\n",
      "> if world > 0:\n",
      ">     torch.manual_seed(args.seed)\n",
      "> \n",
      "68a81,84\n",
      "> \n",
      "> if world > 0:\n",
      ">     model = nn.parallel.DistributedDataParallel(model)\n",
      ">     torch.manual_seed(args.seed+173434*rank)\n"
     ]
    }
   ],
   "source": [
    "diff training.py disttraining.py || true"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-- --from-file=reduce.py=reduce.py\n",
      "-- --from-file=disttraining.py=disttraining.py\n",
      "-- --from-file=training.py=training.py\n",
      "-- --from-file=helpers.py=helpers.py\n",
      "configmap \"files\" deleted\n",
      "configmap/files created\n"
     ]
    }
   ],
   "source": [
    "kubefcm files reduce.py disttraining.py training.py helpers.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Distributed Training\n",
    "\n",
    "- arrange for the same weights in each node by setting the same seed\n",
    "    - explicit distribution of weights might be better\n",
    "    - alternative: load same starting network in all of them\n",
    "- arrange for different training data in each node\n",
    "    - here we rely on different random shuffling in `WebDataset`\n",
    "    - PyTorch examples use explicit splitting (not necessary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Create the Master and Compute Nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kubectl delete pods --all || true\n",
    "kubetpl pod -n master -G 1 -c 'cp /files/*.py .; python3 disttraining.py --dist 0/4' | kubectl apply -f -\n",
    "for i in {1..3}; do\n",
    "    kubetpl pod -n node$i -G 1 -c \"cp /files/*.py .; python3 disttraining.py --dist $i/4\" | kubectl apply -f -\n",
    "done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "sleep 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME     READY   STATUS    RESTARTS   AGE\n",
      "master   1/1     Running   0          49s\n",
      "node1    1/1     Running   0          32s\n",
      "node2    1/1     Running   0          31s\n",
      "node3    1/1     Running   0          31s\n"
     ]
    }
   ],
   "source": [
    "kubectl get pods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/torchvision/io/_video_opt.py:17: UserWarning: video reader based on ffmpeg c++ ops not available\n",
      "  warnings.warn(\"video reader based on ffmpeg c++ ops not available\")\n",
      "Mon Dec  9 20:40:15 UTC 2019; master; root; /workspace; GPU 0: Tesla T4 (UUID: GPU-98cb0870-4950-111e-9140-5d7ed3a2c273); \n",
      "creating resnet50\n",
      "        0 bs   128 per sample loss 5.47e-02 loading 2.06e-02 training 2.30e-02\n",
      "      512 bs   128 per sample loss 5.50e-02 loading 1.70e-02 training 1.87e-02\n",
      "     1024 bs   128 per sample loss 5.50e-02 loading 1.46e-02 training 1.59e-02\n",
      "     1536 bs   128 per sample loss 5.51e-02 loading 1.31e-02 training 1.40e-02\n",
      "     2048 bs   128 per sample loss 5.51e-02 loading 1.21e-02 training 1.27e-02\n",
      "     2560 bs   128 per sample loss 5.51e-02 loading 1.14e-02 training 1.19e-02\n",
      "     3072 bs   128 per sample loss 5.50e-02 loading 1.09e-02 training 1.16e-02\n",
      "     3584 bs   128 per sample loss 5.48e-02 loading 1.06e-02 training 1.13e-02\n",
      "     4096 bs   128 per sample loss 5.46e-02 loading 1.05e-02 training 1.11e-02\n",
      "     4608 bs   128 per sample loss 5.45e-02 loading 1.03e-02 training 1.11e-02\n",
      "     5120 bs   128 per sample loss 5.45e-02 loading 1.03e-02 training 1.10e-02\n",
      "     5632 bs   128 per sample loss 5.45e-02 loading 1.03e-02 training 1.10e-02\n",
      "     6144 bs   128 per sample loss 5.44e-02 loading 1.02e-02 training 1.09e-02\n",
      "     6656 bs   128 per sample loss 5.43e-02 loading 1.02e-02 training 1.10e-02\n",
      "     7168 bs   128 per sample loss 5.43e-02 loading 1.01e-02 training 1.13e-02\n",
      "     7680 bs   128 per sample loss 5.42e-02 loading 9.95e-03 training 1.15e-02\n",
      "     8192 bs   128 per sample loss 5.42e-02 loading 9.92e-03 training 1.16e-02\n",
      "     8704 bs   128 per sample loss 5.42e-02 loading 1.04e-02 training 1.17e-02\n",
      "     9088 bs   128 per sample loss 5.42e-02 loading 1.03e-02 training 1.27e-02\n",
      "     9600 bs   128 per sample loss 5.42e-02 loading 1.02e-02 training 1.25e-02\n"
     ]
    }
   ],
   "source": [
    "kubectl logs master"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/torchvision/io/_video_opt.py:17: UserWarning: video reader based on ffmpeg c++ ops not available\n",
      "  warnings.warn(\"video reader based on ffmpeg c++ ops not available\")\n",
      "Mon Dec  9 20:40:15 UTC 2019; node3; root; /workspace; GPU 0: Tesla T4 (UUID: GPU-fd29201b-d663-6697-b413-a761dceb23c8); \n",
      "creating resnet50\n",
      "        0 bs   128 per sample loss 5.52e-02 loading 2.13e-02 training 2.26e-02\n",
      "      512 bs   128 per sample loss 5.53e-02 loading 1.75e-02 training 1.85e-02\n",
      "     1024 bs   128 per sample loss 5.52e-02 loading 1.50e-02 training 1.56e-02\n",
      "     1536 bs   128 per sample loss 5.52e-02 loading 1.34e-02 training 1.38e-02\n",
      "     2048 bs   128 per sample loss 5.52e-02 loading 1.23e-02 training 1.26e-02\n",
      "     2560 bs   128 per sample loss 5.49e-02 loading 1.16e-02 training 1.18e-02\n",
      "     3072 bs   128 per sample loss 5.48e-02 loading 1.10e-02 training 1.15e-02\n",
      "     3584 bs   128 per sample loss 5.48e-02 loading 1.07e-02 training 1.12e-02\n",
      "     4096 bs   128 per sample loss 5.47e-02 loading 1.05e-02 training 1.11e-02\n",
      "     4608 bs   128 per sample loss 5.46e-02 loading 1.06e-02 training 1.09e-02\n",
      "     5120 bs   128 per sample loss 5.45e-02 loading 1.04e-02 training 1.09e-02\n",
      "     5632 bs   128 per sample loss 5.45e-02 loading 1.03e-02 training 1.10e-02\n",
      "     6144 bs   128 per sample loss 5.45e-02 loading 1.02e-02 training 1.10e-02\n",
      "     6656 bs   128 per sample loss 5.44e-02 loading 1.04e-02 training 1.09e-02\n",
      "     7168 bs   128 per sample loss 5.44e-02 loading 1.04e-02 training 1.10e-02\n",
      "     7680 bs   128 per sample loss 5.43e-02 loading 1.04e-02 training 1.10e-02\n",
      "     8192 bs   128 per sample loss 5.42e-02 loading 1.05e-02 training 1.10e-02\n",
      "     8704 bs   128 per sample loss 5.41e-02 loading 1.06e-02 training 1.14e-02\n",
      "     9088 bs   128 per sample loss 5.42e-02 loading 1.05e-02 training 1.25e-02\n",
      "     9600 bs   128 per sample loss 5.41e-02 loading 1.05e-02 training 1.23e-02\n"
     ]
    }
   ],
   "source": [
    "kubectl logs node3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME     READY   STATUS      RESTARTS   AGE\n",
      "master   0/1     Completed   0          10m\n",
      "node1    0/1     Completed   0          10m\n",
      "node2    0/1     Completed   0          10m\n",
      "node3    0/1     Completed   0          10m\n"
     ]
    }
   ],
   "source": [
    "kubectl get pods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}