[Feature/playbooks] - How to use Minio Client as a task

[piyoki]

Objective

This discussion demonstrates how you can run Minio Client as a task to quickly run backup job against multiple (10+) remote servers.

Introduction

What is Minio

MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Amazon S3 cloud storage service. It can handle unstructured data such as photos, videos, log files, backups, and container images with the maximum supported object size of 5TB

Minio Client

MinIO Client (mc) provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff etc. It supports filesystems and Amazon S3 compatible cloud storage service (AWS Signature v2 and v4).

Steps to Reproduce

Step 0 - Setup Minio Client (Include Basic Operations)

# Install MC Client from the latest binary
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/bin
mc --help

# Connect to a S3 or Minio Server
mc alias set minio http://<HOST_IP>:<MINIO_SERVER_PORT> <ACCESS_KEY> <SECRET_KEY> --api s3v4
# Alternatively connect Minio Server with Service Account Credentials
mc config host add minio http://<HOST_IP>:<MINIO_SERVER_PORT> <ACCESS_KEY> <SECRET_KEY>
mc ls minio

# Copy content to Minio Server
mc cp --recursive <LOCAL_PATH>/ <SERVER_ALIAS>/<BUCKET_PATH>/
# e.g
mc cp -r nginx/ minio/nginx

# Copy content from Minio Server
mc cp --recursive <SERVER_ALIAS>/<BUCKET_PATH>/ <LOCAL_PATH>/
# e.g
mc cp -r minio/nginx/ nginx/

# Sync content from local dir to Minio Server (Directory ONLY)
mc mirror <LOCAL_PATH>/ <SERVER_ALIAS>/<BUCKET_PATH>/

Step 0 - Setup Minio Client with Ansible (Include Basic Operations)

export MINIO_ACCESS_KEY=<MINIO_ACCESS_KEY>
export MINIO_SECRET_KEY=<MINIO_SECRET_KEY>
ansible-playbook -K -i inventory/servers.yml ~/workspace/cloud-estate/playbooks/minio/minio-setup.yml

Step 1 - create a backup script

# backup.sh
#----------#
#!/bin/bash

mc cp -r <TARGET_LOCAL_PATH>/ minio/<BUCKET_NAME>/<SUB_PATH>/
#----------#

# Grant execute permission
sudo chmod +x $HOME/backup.sh

# Test execution
sudo sh backup.sh

Step 2 - Prepare an inventory list

Notes: make sure your publich is already copied to the target remote server, if not then run the following commands:

ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa <USER>@<SEVER_IP>

# e.g inventory/servers.yml
---
application-servers:
  hosts:
    portainer-instance:
      ansible_host: portainer-instance
      ansible_user: packer
      ansible_ssh_private_key_file: ~/.ssh/id_rsa

Step 3 - Run the playbook

To view the file structure, please review the PR - #69

export MINIO_ACCESS_KEY=<MINIO_ACCESS_KEY>
export MINIO_SECRET_KEY=<MINIO_SECRET_KEY>
ansible-playbook -K -i inventory/servers.yml ~/workspace/cloud-estate/playbooks/minio/minio-backup.yml

[piyoki]

Rework setup.ops in this PR - #166