Scooby is a build pipeline for creating bare-metal K3S clusters of one or more nodes.
Scooby attempts to automate the configuration and deployment of a K3S cluster into a single Linux image in a simple, repeatable fashion.
Scooby is designed to be easy to work with and requires a bare minimum of hardware and software configuration to get up and running.
Scooby uses Vagrant, GitHub and optionally, Amazon S3.
Features:
- From commit to bootable image in 15 minutes
- Versioned deployments
- Centralized configuration
- Network booted agents
- Ready-to-go build pipelines
- Github asset storage
- Optional S3 bucket uploader
To get going, you will need at least one Raspberry Pi 3A or higher, and one local storage device (e.g. USB stick) per Pi.
If you are using two or more Pis you will also need a physical Ethernet switch or VLAN group, and the master node will require a second ethernet device.
Several generic USB2 and USB3 devices using the RTL-8152 chipset have been tested and work well. Driverless Ethernet hats should also work but at the current time have not been tested.
Agent nodes perform well on Pi 3A+ but for performance reasons it is recommended to use a Pi 4B or greater for the master node, and a USB3 SSD for master node storage.
To create a three-node Scooby cluster a complete hardware list might be:
- 1x Raspberry Pi 4B
- 1x Raspberry Pi 3B
- 1x Raspberry Pi 3A
- 4x port Ethernet switch
- 1x 1TB USB3 SSD external storage
- 2x 8GB USB2 keys
- 1x RTL8152 USB3 to Ethernet adapter
- 4x Ethernet cables
Click the use this template button above, create a new GitHub repository, and clone it to your local machine.
The master node in any Scooby cluster of more than one node has two physical ethernet ports and acts as a network gateway for all traffic originating inside the cluster. A Scooby cluster therefore has a concept of internal and external networks.
The internal network is the physical network on which the master and agent nodes communicate, and to which the eth1 device of the master node is physically connected.
The external network is a separate physical network with access to the Internet, and to which the eth0 device of the master node is physically connected.
All agent nodes' eth0 devices are connected to the internal network.
The first file you will need to edit is /config, in the project root. This file contains information about the cluster's base operating system and network settings. There is an example configured with default values that you can uncomment and change to get you going. For example:
/config
# Base Raspberry Pi image and SHA
LC_IMAGE_HREF="http://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-01-28/2022-01-28-raspios-bullseye-arm64-lite.zip"
LC_IMAGE_SHA=sha256:d694d2838018cf0d152fe81031dba83182cee79f785c033844b520d222ac12f5
# Master node hostname
LC_HOSTNAME=buffy
# DNS providers
LC_PRIMARY_DNS=1.0.0.1
LC_SECONDARY_DNS=
# Master node external network configuration
LC_EXTERNAL_DEVICE=eth0
LC_EXTERNAL_IP=192.168.1.64
LC_EXTERNAL_NET=192.168.1.0/24
LC_EXTERNAL_GW=192.168.1.1
LC_EXTERNAL_DOMAIN=
# Master node cluster network configuration
LC_INTERNAL_DEVICE=eth1
LC_INTERNAL_IP=192.168.64.1
LC_INTERNAL_NET=192.168.64.0/24
LC_INTERNAL_DOMAIN=sunnydale
Scooby has been tested with both bullseye and bookworm releases of raspios.
Any files placed in the ./server directory in the project root will be merged into the master node's root filesystem.
For example, ./server/etc/dnsmasq.d/40-hemeryhigh.conf would be copied to /etc/dnsmasq.d/40-hemeryhigh.conf on the master node file system.
Files are persistent and available on first boot.
Kubernetes manifests should go in ./server/var/lib/rancher/k3s/server/manifests/ in the project root.
Each agent must be setup for network booting. This part cannot be automated and you should perform it manually on each agent. Fortunately, it needs only to be done once and does not need to be done at all on the Pi 4.
Mete Balci has written an excellent guide detailing how to configure Raspberry Pi for network boot 👏. The official Raspberry Pi documentation for debugging network book mode is also very useful.
Create one file for each of the agent nodes in the ./agents directory. This file contains information about the agent's network settings, storage and K3S arguments specific to that node. The name of the file should be the hostname of the agent with an extension of .agent.
For each agent, you will need to provide:
- the agent node's hardware address
- its IPv4 address
- a UUID for a partition containing an ext4 filesystem for
K3Spersistent local storage - an optional UUID for a swap partition
- its PXE client ID
- any extra args for K3s that the agent might need (optional)
For example:
./agents/willow.agent
AGENT_ETHERNET=b8:27:eb:81:1a:52
AGENT_IP=192.168.64.65
AGENT_RANCHER_PART_UUID=d5f9e6c2-493c-48da-baf2-0c63dd7a36b1
AGENT_PXE_ID=c6811a52
AGENT_K3S_ARGS=
- An incomplete or incorrect agent description will not result in a working agent node.
The Pi transmits its serial number for PXE boot. Find the serial number of any Raspberry Pi with the following command:
$ cat /sys/firmware/devicetree/base/serial-number
Although the agents' operating systems are served over the network, K3S requires some storage on each agent's local filesystem.
Scooby agents expect an ext4 partition on any local storage media, although USB is highly recommended due to the relative volatility of many SD cards.
You can find the UUID of any partition using blkid. To find the partition UUID of /dev/sda1, for example:
$ blkid --match-tag="PARTUUID" /dev/sda1
- Ensure you don't have any bootable physical media in the Pi when you power it up
- If you have several agents, it can be helpful to write the name of the intended agent on the USB key to ensure the correct key is matched with the correct agent host.
By default, a commit to main will trigger the build job. When the build has completed you will see a zipped asset available to download in the build workflow.
You can enable builds on branch commits by removing the following line
./github/workflows/branch.yml
if: false
To build Scooby locally, you must first install Vagrant. On Debian-like systems you can do this with the following command:
apt install vagrant
Then, execute the following command in the project root:
$ ./scripts/build.sh
The output disk image will be found in ./images/scooby.img. The image is typically in the region of 3.5GB.
Building with the defaults will produce a deployable image that:
- has only a master node
- contains no manifests
- cannot be accessed by SSH
Scooby has the following secrets:
LC_DEFAULT_USER name of your default user for the cluster (default: spike)
LC_SSH_AUTH_KEY an SSH auth key to access the cluster nodes (optional)
LC_PACKER_GITHUB_API_TOKEN your Packer API token (optional)
- If you are building locally, you will need to set the secret environment variables in a file called
.envin the project root. - If you are building using the GHA pipeline, you will need to set these as action secrets in your repository settings.
If you commit to main, Scooby can also attempt to upload the finished image to an AWS S3 bucket.
To enable S3 upload, create these secrets in your repository settings.
AWS_ACCESS_KEY_ID the Key ID for your AWS account
AWS_SECRET_ACCESS_KEY the Secret Access Key for your AWS account
AWS_BUCKET_S3_URI the S3 URI for your bucket account
AWS_REGION the AWS region in which your bucket is located
For convenience, Scooby is configured to use macos-latest by default. This is the only GitHub runner that currently provides Vagrant.
A self-hosted runner is recommended if you plan to make frequent changes.
If you wish to use a self-hosted runner, you will need to edit .github/workflows/build.yml and change runs-on: macos-latest to runs-on: self-hosted.
You will then need to follow these instructions to host your own runner.
Once you have built and downloaded the image, use dd to transfer it to the master node's storage device. If your master node's storage device appeared as /dev/sdf, for example:
$ dd oflag=sync bs=1M if=~/Downloads/scooby.img of=/dev/sdf
A typical Scooby network might look something like this:
graph TD;
PublicNetwork-->Buffy;
Buffy-->Xander;
Buffy-->Willow;
Buffy-->Cordy;
Buffy-->Giles;
As explained earlier, the master node acts as a gateway for all network traffic originating inside the cluster that is destined for external networks. NAT is enabled for the agents using iptables-persistent.
Scooby is configured such that master node's eth0 (the onboard Ethernet device) is the cluster's external or public interface, and eth1 (e.g. the RTL-8152) is the internal, or cluster interface. Agent node eth0 devices should exist on the same network as the master node's eth1 device.
Clients on the external network may access services on the cluster via the master node's eth0 device. Traefik cannot provide IP addresses by itself so it is recommended to provide them on the master node's eth0 device with a MetalLB deployment on the master node.
To enable network boot services for agent nodes, the master node uses dnsmasq to provide PXE and DHCP services to the cluster on its eth1 device. You should therefore not connect the master node's eth1 device to any network with existing DHCP services.
Scooby uses Ethernet only. Wifi is not currently supported.
Once all your Pis are plugged into the network and all your storage devices are attached, it is time to power up the master node.
A master node called buffy would be ready when you see the following in the master node console:
BUFFY WILL PATROL TONIGHT
Be patient; this process can about take 7 minutes on a Pi 4B. Allow the master node a few more minutes following the above message before powering up any agents. cloud-init config files are served to the agents by lighttpd and it takes a few moments to start responding.
You can watch the progress of your agent nodes coming online with the following command in a terminal session on the master node:
$ kubectl get nodes -Aw
Eventually you should be rewarded with something like:
NAME STATUS ROLES AGE VERSION
buffy Ready control-plane,master 25h v1.27.4+k3s1
willow Ready <none> 25h v1.27.4+k3s1
xander Ready <none> 24h v1.27.4+k3s1
cordy Ready <none> 25h v1.27.4+k3s1
Congratulations! Your Raspberry Pi supercomputer is now complete 😊
The master node will appear to go quiet on first boot.
There will be no console output after cloud-init has identified the network devices for about four minutes. This is a known issue, and is caused by cloud-init waiting for eth0 to become ready.
Cloud-init will eventually find the network interface and continue.
The default Raspberry Pi user pi is disabled for security reasons.
RTL-8152 USB to Ethernet devices are a good solution for adding extra connectivity to the Raspberry Pi but they can appear to the operating system as USB storage media when first inserted.
To ensure the device appears as an Ethernet device and not USB storage, add the following file to your project:
/server/etc/udev/rules.d/10-modeswitch.rules
ACTION=="add", SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="0bda", ENV{ID_MODEL_ID}=="8152", RUN+="/usr/bin/usb_modeswitch -K -v 0bda -p 8151;/usr/bin/usb_modeswitch -v 0bda -p 8152 -R"
Use MetalLB to provide IP addresses for all your K3S services. An example MetalLB manifest in a Scooby deployment might look like the following:
/server/var/lib/rancher/k3s/server/manifests/metallb.yml
apiVersion: v1
kind: Namespace
metadata:
name: metallb-system
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: metallb-system
namespace: kube-system
spec:
repo: https://charts.bitnami.com/bitnami
chart: metallb
targetNamespace: metallb-system
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: first-pool
namespace: metallb-system
spec:
addresses:
- 192.168.1.65-192.168.1.127
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: public-pool
namespace: metallb-system
spec:
interfaces: ["eth0"]
ipAddressPools:
- first-pool
nodeSelectors:
- matchLabels:
kubernetes.io/hostname: buffy
You can easily deploy Pihole to a Scooby cluster with a manifest like this:
/server/var/lib/rancher/k3s/server/manifests/pihole.yml
apiVersion: v1
kind: Namespace
metadata:
name: pihole
labels:
name: traffic
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: pihole
namespace: pihole
spec:
chart: pihole
targetNamespace: pihole
repo: https://mojo2600.github.io/pihole-kubernetes/
valuesContent: |-
...
Pihole works well for DNS, but load balancers aren't designed to respond to broadcast traffic like DHCP requests.
You might find that Scooby's dnsmasq instance is an easier solution for serving DHCP to your external network than Pihole. Here is an example configuration file that will do that:
/server/etc/dnsmasq.d/10-external-dhcp.conf
domain=magicbox,192.168.1.0/24
interface=eth0
dhcp-range=tag:eth0,192.168.1.128,192.168.1.240
dhcp-option=tag:eth0,option:router,192.168.1.1
dhcp-option=tag:eth0,6,192.168.1.66
dhcp-host=30:05:5C:8E:ED:C3,192.168.1.10,snyder
dhcp-host=08:00:20:AA:BB:CC,192.168.1.11,joycey
Scooby uses OverlayFS for agent filesystems. Under some circumstances you may wish to return an agent to its pristine state and re-deploy it without re-deploying the entire cluster.
To do this, you will need to SSH into the master node and execute the following as root, changing the hostname to match the agent node in question.
$ exportfs -u faith:/mnt/scooby/agents/faith
$ umount /mnt/scooby/agents/faith
$ rm -rf /var/lib/scooby/agents/faith/*
$ rm -rf /var/lib/scooby/overlay/faith/*
$ mount -a
$ exportfs -a
Raspberry Pis are often used for IoT applications. Scooby provides a very convenient method of remotely managing IoT devices when used with Smarter Device Manager published by Arm Research.
To access devices such as USB, GPIO pins, V4L and I2C and more from within K3S, download the following files from the Smarter Device Manager repo to ./server/var/lib/rancher/k3s/server/manifests/
Provide each agent hosting a device with a K3S node label of smarter-device-manager=enabled. An agent description file for a node with device access enabled might look like the following:
./agents/ted
AGENT_ETHERNET=b8:27:eb:86:53:3c
AGENT_IP=192.168.64.254
AGENT_RANCHER_PART_UUID=b357bda8-b641-41eb-9b5d-b3c5486e50e7
AGENT_PXE_ID=9686533c
AGENT_K3S_ARGS="--node-label 'smarter-device-manager=enabled'"