053-multinode.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Kubernetes Service Discovery"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "maybe() {\n",
    "    \"$@\" > .last_maybe 2>&1 || true\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Service Discovery\n",
    "\n",
    "- for large scale deep learning we need multiple processes that talk to each other\n",
    "- this requires\n",
    "    - service discovery\n",
    "    - networking\n",
    "    - name resolution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# K8s Service Discovery\n",
    "\n",
    "Simple:\n",
    "\n",
    "- every pod gets assigned a hostname and domain\n",
    "- you can simply connect directly to these well-known names\n",
    "\n",
    "Requirements:\n",
    "\n",
    "- create a \"headless service\" to start the name resolver\n",
    "- add ports, host name, and subdomain to your pods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Headless Service\n",
    "\n",
    "The `clusterIP: None` makes it headless. (Other services are load balancing, which we don't want.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "service/bigdata19 created\n"
     ]
    }
   ],
   "source": [
    "kubectl apply -f - <<'EOF'\n",
    "apiVersion: v1\n",
    "kind: Service\n",
    "metadata:\n",
    "  name: bigdata19\n",
    "spec:\n",
    "  clusterIP: None\n",
    "  ports:\n",
    "    - port: 7880\n",
    "      targetPort: 7880\n",
    "  selector:\n",
    "    app: bigdata19\n",
    "EOF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A Visible Pod\n",
    "\n",
    "This pod will be assigned the DNS name `shards.bigdata19`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pod/shards created\n"
     ]
    }
   ],
   "source": [
    "# nodes get assigned DNS names if they have a port and the app label matches the headless service\n",
    "\n",
    "maybe kubectl delete pod/shards\n",
    "kubectl apply -f - <<'EOF'\n",
    "apiVersion: v1\n",
    "kind: Pod\n",
    "metadata:\n",
    "  name: shards\n",
    "  labels:\n",
    "    app: bigdata19\n",
    "spec:\n",
    "  containers:\n",
    "  - name: shards\n",
    "    image: gcr.io/research-191823/bigdata19\n",
    "    command: [\"serve-imagenet-shards\", \"-b\", \"96\", \"zpub://0.0.0.0:7880\"]\n",
    "    ports:\n",
    "      - containerPort: 7880\n",
    "  restartPolicy: Never\n",
    "  hostname: shards\n",
    "  subdomain: bigdata19\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "sleep 15"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# DNS Debugging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME     READY   STATUS    RESTARTS   AGE\n",
      "shards   1/1     Running   0          4m26s\n"
     ]
    }
   ],
   "source": [
    "kubectl get pods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Server:\t\t10.64.0.10\n",
      "Address:\t10.64.0.10#53\n",
      "\n",
      "Name:\tshards.bigdata19.default.svc.cluster.local\n",
      "Address: 10.0.2.3\n",
      "\n",
      "nameserver 10.64.0.10\n",
      "search default.svc.cluster.local svc.cluster.local cluster.local c.research-191823.internal google.internal\n",
      "options ndots:5\n"
     ]
    }
   ],
   "source": [
    "# make sure resolution is working\n",
    "kubectl exec -ti shards -- nslookup shards.bigdata19\n",
    "\n",
    "# check resolv.conf file\n",
    "kubectl exec -ti shards -- cat /etc/resolv.conf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE\n",
      "default-http-backend   NodePort    10.64.6.85     <none>        80:32565/TCP    59m\n",
      "heapster               ClusterIP   10.64.10.206   <none>        80/TCP          59m\n",
      "kube-dns               ClusterIP   10.64.0.10     <none>        53/UDP,53/TCP   59m\n",
      "metrics-server         ClusterIP   10.64.7.249    <none>        443/TCP         59m\n",
      "NAME       ENDPOINTS                                             AGE\n",
      "kube-dns   10.0.2.134:53,10.0.2.2:53,10.0.2.134:53 + 1 more...   59m\n"
     ]
    }
   ],
   "source": [
    "# check service running\n",
    "kubectl get svc --namespace=kube-system\n",
    "\n",
    "# check endpoints\n",
    "kubectl get ep kube-dns --namespace=kube-system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                        READY   STATUS    RESTARTS   AGE\n",
      "kube-dns-79868f54c5-46q7h   4/4     Running   0          59m\n",
      "kube-dns-79868f54c5-tf275   4/4     Running   0          59m\n",
      "E1212 17:12:08.610570       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:192: Failed to list *v1.Service: Get https://10.64.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.64.0.1:443: i/o timeout\n",
      "E1212 17:12:08.611507       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:189: Failed to list *v1.Endpoints: Get https://10.64.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.64.0.1:443: i/o timeout\n",
      "I1212 17:57:33.512228       1 dns.go:601] Could not find endpoints for service \"bigdata19\" in namespace \"default\". DNS records will be created once endpoints show up.\n",
      "I1212 17:03:17.281524       1 nanny.go:146] dnsmasq[24]: read /etc/hosts - 7 addresses\n",
      "I1212 17:03:17.281450       1 nanny.go:149] \n",
      "W1212 17:03:17.281647       1 nanny.go:150] Got EOF from stdout\n",
      "I1212 17:03:17.803027       1 server.go:45] Starting server (options {DnsMasqPort:53 DnsMasqAddr:127.0.0.1 DnsMasqPollIntervalMs:5000 Probes:[{Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33} {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}] PrometheusAddr:0.0.0.0 PrometheusPort:10054 PrometheusPath:/metrics PrometheusNamespace:kubedns})\n",
      "I1212 17:03:17.803047       1 dnsprobe.go:75] Starting dnsProbe {Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}\n",
      "I1212 17:03:17.803219       1 dnsprobe.go:75] Starting dnsProbe {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}\n",
      "I1212 17:03:18.931340       1 main.go:124] Taking source configs from kubernetes api server\n",
      "I1212 17:03:18.931345       1 main.go:85] Built the following source configs: [{kubedns localhost 10054 /metrics [probe_kubedns_latency_ms probe_kubedns_errors dnsmasq_misses dnsmasq_hits] 0xc4203740f0}]\n",
      "I1212 17:03:18.931373       1 main.go:133] Running prometheus-to-sd, monitored target is kubedns localhost:10054\n"
     ]
    }
   ],
   "source": [
    "# when desperate, you can look through the kube-dns logs\n",
    "kubectl get pods --namespace=kube-system -l k8s-app=kube-dns\n",
    "kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | sed 1q |\n",
    "while read pod; do \n",
    "kubectl logs --tail=3 --namespace=kube-system $pod -c kubedns\n",
    "kubectl logs --tail=3 --namespace=kube-system $pod -c dnsmasq\n",
    "kubectl logs --tail=3 --namespace=kube-system $pod -c sidecar\n",
    "kubectl logs --tail=3 --namespace=kube-system $pod -c prometheus-to-sd\n",
    "done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "sleep 15"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Logs of the Running Server\n",
    "\n",
    "The server is chugging along nicely, sending out training batches to anybody who will listen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "serving zpub://0.0.0.0:7880\n",
      "0 rate 0.000000 msg/s throughput 0.00e+00 bytes/s\n",
      "10 rate 5.676849 msg/s throughput 8.20e+07 bytes/s\n",
      "20 rate 5.370448 msg/s throughput 7.76e+07 bytes/s\n",
      "30 rate 4.273988 msg/s throughput 6.18e+07 bytes/s\n",
      "40 rate 4.440511 msg/s throughput 6.42e+07 bytes/s\n",
      "50 rate 4.553441 msg/s throughput 6.58e+07 bytes/s\n",
      "60 rate 4.636682 msg/s throughput 6.70e+07 bytes/s\n",
      "70 rate 4.671525 msg/s throughput 6.75e+07 bytes/s\n",
      "80 rate 4.725854 msg/s throughput 6.83e+07 bytes/s\n"
     ]
    }
   ],
   "source": [
    "kubectl logs shards | sed 10q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME     READY   STATUS    RESTARTS   AGE\n",
      "shards   1/1     Running   0          4m55s\n"
     ]
    }
   ],
   "source": [
    "kubectl get pods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Starting a Client\n",
    "\n",
    "Here is a small network client that listens to training data and outputs statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pod/client created\n"
     ]
    }
   ],
   "source": [
    "maybe kubectl delete pod/client\n",
    "kubectl apply -f - <<'EOF'\n",
    "apiVersion: v1\n",
    "kind: Pod\n",
    "metadata:\n",
    "  name: client\n",
    "  labels:\n",
    "    app: bigdata19\n",
    "spec:\n",
    "  containers:\n",
    "  - name: client\n",
    "    image: gcr.io/research-191823/bigdata19\n",
    "    command: [\"tensormon\", \"zsub://shards.bigdata19:7880\"]\n",
    "    stdin: true\n",
    "    tty: true\n",
    "  restartPolicy: Never\n",
    "  hostname: client\n",
    "  subdomain: bigdata19\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "sleep 15"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Client Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input: ['zsub://shards.bigdata19:7880']\n",
      "zsub://shards.bigdata19:7880\n",
      "connected\n",
      "                  10    5.431 batches/s  521.379 samples/s (batchsize: 96)\n",
      "                  20    4.976 batches/s  477.705 samples/s (batchsize: 96)\n",
      "                  30    4.789 batches/s  459.767 samples/s (batchsize: 96)\n",
      "                  40    4.622 batches/s  443.675 samples/s (batchsize: 96)\n",
      "                  50    3.835 batches/s  368.196 samples/s (batchsize: 96)\n",
      "                  60    3.263 batches/s  313.282 samples/s (batchsize: 96)\n",
      "                  70    4.048 batches/s  388.614 samples/s (batchsize: 96)\n"
     ]
    }
   ],
   "source": [
    "kubectl logs client | sed 10q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Starting a DL Client on a GPU Node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "job.batch/myjob created\n"
     ]
    }
   ],
   "source": [
    "maybe kubectl delete job/myjob\n",
    "kubectl apply -f - <<'EOF'\n",
    "apiVersion: batch/v1\n",
    "kind: Job\n",
    "metadata:\n",
    "  name: myjob\n",
    "  labels:\n",
    "    app: bigdata19\n",
    "spec:\n",
    "  backoffLimit: 0\n",
    "  template:\n",
    "    spec:\n",
    "      containers:\n",
    "        - name: myjob\n",
    "          image: gcr.io/research-191823/bigdata19\n",
    "          command: \n",
    "            - \"/bin/bash\"\n",
    "            - \"-c\"\n",
    "            - |\n",
    "              cp /files/*.py .\n",
    "              python3 training.py --tensorcom zsub://shards.bigdata19:7880\n",
    "          stdin: true\n",
    "          tty: true\n",
    "          resources:\n",
    "            limits:\n",
    "              nvidia.com/gpu: \"1\"\n",
    "          volumeMounts:\n",
    "            - mountPath: /files\n",
    "              name: files\n",
    "      nodeSelector:\n",
    "        cloud.google.com/gke-accelerator: nvidia-tesla-t4\n",
    "      restartPolicy: Never\n",
    "      volumes:\n",
    "        - configMap:\n",
    "            name: files\n",
    "          name: files\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "sleep 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/torchvision/io/_video_opt.py:17: UserWarning: video reader based on ffmpeg c++ ops not available\n",
      "  warnings.warn(\"video reader based on ffmpeg c++ ops not available\")\n",
      "Thu Dec 12 18:21:30 UTC 2019; myjob-scs9r; root; /workspace; GPU 0: Tesla T4 (UUID: GPU-e3b63d8c-056b-140d-43e1-de274722818d); \n",
      "creating resnet50\n",
      "        0 bs    96 per sample loss 7.38e-02 loading 5.98e-04 training 1.88e-02\n"
     ]
    }
   ],
   "source": [
    "kubectl logs job/myjob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "sleep 60"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Training\n",
    "\n",
    "- Note that with distributed preprocessing, loading is very fast.\n",
    "- We will talk about the Tensorcom package late."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/torchvision/io/_video_opt.py:17: UserWarning: video reader based on ffmpeg c++ ops not available\n",
      "  warnings.warn(\"video reader based on ffmpeg c++ ops not available\")\n",
      "Thu Dec 12 18:21:30 UTC 2019; myjob-scs9r; root; /workspace; GPU 0: Tesla T4 (UUID: GPU-e3b63d8c-056b-140d-43e1-de274722818d); \n",
      "creating resnet50\n",
      "        0 bs    96 per sample loss 7.38e-02 loading 5.98e-04 training 1.88e-02\n",
      "     1152 bs    96 per sample loss 7.35e-02 loading 5.76e-04 training 7.51e-03\n",
      "     2304 bs    96 per sample loss 7.32e-02 loading 5.66e-04 training 4.33e-03\n",
      "     3360 bs    96 per sample loss 7.31e-02 loading 5.63e-04 training 3.50e-03\n",
      "     4416 bs    96 per sample loss 7.28e-02 loading 5.65e-04 training 3.29e-03\n",
      "     5472 bs    96 per sample loss 7.27e-02 loading 5.62e-04 training 3.23e-03\n",
      "     6528 bs    96 per sample loss 7.27e-02 loading 5.67e-04 training 3.26e-03\n"
     ]
    }
   ],
   "source": [
    "kubectl logs job/myjob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME    COMPLETIONS   DURATION   AGE\n",
      "myjob   0/1           75s        75s\n"
     ]
    }
   ],
   "source": [
    "kubectl get jobs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "job.batch \"myjob\" deleted\n",
      "pod \"client\" deleted\n",
      "pod \"myjob-scs9r\" deleted\n",
      "pod \"shards\" deleted\n"
     ]
    }
   ],
   "source": [
    "kubectl delete jobs --all\n",
    "kubectl delete pods --all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Kubernetes Service Discovery\n",
    "\n",
    "- it's like creating new server out of thin air\n",
    "- you can define your distributed application as a collection of pods\n",
    "- K8s also provides load balancing and more complex name spaces"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}