layout | title | permalink | redirect_from | ||
---|---|---|---|---|---|
post |
DOCKER MAIN |
/docs/docker-main |
|
AIStore can be run as a cluster of Docker containers. There are two modes of operation: development and quick-start. They can be found in the deploy/dev/docker
directory.
This mode is currently used for development purposes. All docker containers mount the same host's AIStore source directory, and then execute from this single source. Upon restart (of the AIStore cluster), all changes made in the host will, therefore, take an immediate effect.
The development mode is currently being maintained and updated.
Use this mode if you would like to set up a containerized, one-proxy, one-target deployment of AIStore within seconds. See quick start AIS with Docker. You just need Docker installed to quick-start AIS.
For an introduction to Docker, please watch Docker 101 youtube
Note: Using Docker requires one of the following versions of Ubuntu:
- Bionic 18.04 (LTS)
- Xenial 16.04 (LTS)
- Trusty 14.04 (LTS)
- Uninstall any old versions of docker:
$ sudo apt-get remove docker docker-engine docker.io
It’s OK if apt-get reports that none of these packages are installed.
- Update the apt package index:
$ sudo apt-get update
- Install packages to allow apt to use a repository over HTTPS:
$ sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
software-properties-common
- Install Docker
$ sudo apt-get install docker-ce
Note: For version 16.04
and up, docker-ce
is not in the default Ubuntu repository. See this guide for docker-ce
installation.
- Verify that Docker CE is installed correctly by running the "hello-world" image.
$ sudo docker run hello-world
- Add your current user to the Docker group (but only if you are not the root). After executing the command, restart your machine for it to take effect.
$ sudo usermod -aG docker $(whoami)
- Install Docker-Compose using python
pip
. Installpip
if you don't have it:
$ sudo apt-get install -y python-pip
$ sudo pip install docker-compose
OR
sudo apt install docker-compose
- Test the installation:
$ docker-compose --version
docker-compose version 1.23.1, build 1719ceb
- If you have any troubles with your installation, consider using the latest version of Docker and Docker-Compose.
- To uninstall Docker, run the following:
$ sudo apt-get purge docker-ce
$ sudo apt-get purge docker-ce-cli
- Ensure docker is completely uninstalled by running the following command:
$ dpkg -l | grep -i docker
There should be no docker-ce and docker-ce-cli packages listed.
- To uninstall Docker-Compose, run the following:
$ pip uninstall docker-compose
- Images, containers, volumes, or customized configuration files on your host are not automatically removed. To delete all images, containers, and volumes:
$ sudo rm -rf /var/lib/docker
If not already done, install Go and set $GOPATH environment variable.
The steps:
- Build
aisnode
.
$ go install github.com/NVIDIA/aistore/cmd/aisnode@latest
- To get all sources and use scripts, you can clone AIStore repo.
$ git clone https://github.com/NVIDIA/aistore.git
- Optionally, setup AWS configuration by running
aws configure
command. To run AIStore Docker containers, you will need to pass your AWS config and credential directory via flag-a=<aws directory>
CLI. By default, AWS stores config and credential files in~/.aws/
Example:
$ ./deploy_docker.sh -a=~/.aws/
Building and running "containerized" AIStore is further documented at deploy/dev/docker
.
Please note: the image building process will take some time but only for the first time; subsequent runs will use already cached images.
List all of the running containers using docker ps
. Many commands require the CONTAINER ID or NAME fields. Example output:
Lists all containers (not only the running ones).
$ docker ps -a
To view docker logs, use docker logs <container_name>
. Example:
$ docker logs ais0_proxy_1
I 21:23:56.400794 metasync.go:142] Starting metasyncer
I 21:24:06.415473 stats.go:422] {"err.n":0,"get.n":0,"del.n":0,"get.ns":0,"kalive.ns":0,"err.get.n":0,"err.list.n":0,"pst.n":0,"ren.n":0,
"lst.ns":0,"uptime.ns":0,"kalive.ns.max":0,"err.delete.n":0,"err.post.n":0,"err.range.n":0,"err.head.n":0,"put.n":0,"lst.n":0,
"kalive.ns.min":0,"err.put.n":0}
I 21:24:08.386182 proxy.go:2236] joined target 1463af8ddcd3 (num targets 1)
I 21:24:11.759453 proxy.go:2236] joined target 3c86e5e71978 (num targets 2)
I 21:24:12.411714 earlystart.go:262] Reached the expected 2/2 target registrations
I 21:24:12.412098 earlystart.go:228] 0e76b56086eb: merged local Smap (2/1)
I 21:24:12.412244 metasync.go:290] dosync: smaptag, action=early-start-have-registrations, version=2
I 21:24:12.412772 metasync.go:290] dosync: bucketmdtag, action=early-start-have-registrations, version=1
I 21:24:13.425617 earlystart.go:307] 0e76b56086eb: merging discovered Smap v2 (2, 1)
...
Note:
- You can obtain the container name by running command
docker ps
- The
deploy/dev/docker/
directory has a more comprehensive script namedlogs.sh
to view logs
$ docker exec -it CONTAINER_NAME /bin/bash
Note:
- In production mode, the logs are expected to be in
/var/log/ais/
.By default (development mode) the logs are undertmp/ais/log
- The
deploy/dev/docker/
directory has a script namedcontainer_shell.sh
that does the same thing
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
ais1_proxy latest ced2cbd2ff2f 27 minutes ago 676MB
ais1_target latest ced2cbd2ff2f 27 minutes ago 676MB
ais0_proxy latest 4c3fbcb54e4d 27 minutes ago 676MB
ais0_target latest 4c3fbcb54e4d 27 minutes ago 676MB
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
cb05b22edcb3 bridge bridge local
86517bef938d ais0_internal_control bridge local
51f49b56fe57 ais0_internal_data bridge local
2cee0651aa72 ais0_public bridge local
e11635deb15d ais1_internal_control bridge local
9eefba13018f ais1_internal_data bridge local
674fee80acac ais1_public bridge local
$ docker start CONTAINER_NAME
$ docker stop CONTAINER_NAME
$ docker restart CONTAINER_NAME
$ docker kill CONTAINER_NAME
$ docker stats
$ docker image prune -f
$ docker container prune -f
$ docker network prune -f
$ docker stop $(docker ps -a -q)
$ docker rm $(docker ps -a -q)
$ docker rmi $(docker images -q -a)
Following is a super-simple presentation to showcase some of the AIS capabilities.
In Docker playground, you can find the scripts to download different popular AI datasets (e.g., MNIST and ImageNet). The datasets are downloaded with the AIS-integrated Downloader that stores all downloaded objects directly into the AIStore.
During the download, you can monitor:
- number of requests made
- number of requests failed
- number of bytes transferred
In the example below, AIS downloads a handful of ImageNet images and collects/visualizes the corresponding statistics:
$ cd path_to/deploy/dev/docker
$ ./deploy_docker.sh -d=2 -p=2 -t=4 -c=1 -grafana -nocloud # start 2 proxies and 4 targets
$ ./playground/download_imagenet.sh # download some of ImageNet images into AIS and show stats
# once Downloader will save the files...
$ ./playground/stress_get.sh imagenet # do gets on saved files (requires jq command)
$ ./stop_docker.sh -l # stop docker