metropolis

Monogon Operating System

A cluster operating system. Codename: metropolis. Linux kernel, stateless userland, API-driven management, high integrity. Designed to run Kubernetes and other workload scheduling systems.

Documentation

The canonical documentation for Monogon OS is the Monogon OS Handbook.

Developer Quick Start

Follow the setup instructions in the top-level README.md. We recommend using nix-shell either via Nix installed on an existing distribution or on NixOS.

Start a test cluster by running:

$ bazel run //metropolis:launch-cluster

This will build all the required code and run a fully userspace test cluster consisting of four qemu VMs (three for Monogon OS nodes, one for a swich/router emulator). No root access on the host required.

.--------. .--------. .--------.
| node 0 | | node 1 | | node 2 |
'--------' '--------' '--------'
     ^          ^          ^
     |          |          | Virtual Ethernet (10.1.0.0/24)
     V          V          V
   .-------------------------.
   | nanoswitch              |
   |-------------------------|    .-------------------.
   | Router, switch,         |--->| Internet via host |
   | SOCKS proxy             |    '-------------------'
   '-------------------------'
     ^
     | gRPC over SOCKS to nodes
   .----------.
   | metroctl |
   '----------'

The launch tool will output information on how to connect to the cluster:

Launch: Cluster running!
  To access cluster use: metroctl --config /tmp/metroctl-3733429479 --proxy 127.0.0.1:42385 --endpoints 10.1.0.2 --endpoints 10.1.0.3 --endpoints 10.1.0.4
  Or use this handy wrapper: /tmp/metroctl-3733429479/metroctl.sh
  To access Kubernetes, use kubectl --context=launch-cluster

You can use the metroctl wrapper to then look at the node list per the Monogon OS cluster control plane:

$ alias metroctl=/tmp/metroctl-3733429479/metroctl.sh
$ metroctl node describe
NODE ID                                       STATE   ADDRESS    HEALTH    ROLES                                  TPM   VERSION                         HEARTBEAT   
metropolis-067651202d00b79fffe92df0001aabff   UP      10.1.0.4   HEALTHY                                          yes   v0.1.0-dev494.g0d8a8a4f.dirty   1s          
metropolis-7ccd2437c50696ea9a9b6543dc163f84   UP      10.1.0.3   HEALTHY                                          yes   v0.1.0-dev494.g0d8a8a4f.dirty   3s          
metropolis-ec101152c48c5f761534c1910cf66200   UP      10.1.0.2   HEALTHY   ConsensusMember,KubernetesController   yes   v0.1.0-dev494.g0d8a8a4f.dirty   3s       

We have a node running the Monogon OS control plane (ConsensusMember role) and Kubernetes control plane ( KubernetesController role), but no Kubernetes worker nodes. But changing that is a simple API call (or metroctl invocation) away:

$ metroctl node add role KubernetesWorker metropolis-067651202d00b79fffe92df0001aabff
2024/02/12 17:42:33 Updated node metropolis-067651202d00b79fffe92df0001aabff.
$ metroctl node add role KubernetesWorker metropolis-7ccd2437c50696ea9a9b6543dc163f84
2024/02/12 17:42:36 Updated node metropolis-7ccd2437c50696ea9a9b6543dc163f84.
$ metroctl node describe
NODE ID                                       STATE   ADDRESS    HEALTH    ROLES                                  TPM   VERSION                         HEARTBEAT   
metropolis-067651202d00b79fffe92df0001aabff   UP      10.1.0.4   HEALTHY   KubernetesWorker                       yes   v0.1.0-dev494.g0d8a8a4f.dirty   0s          
metropolis-7ccd2437c50696ea9a9b6543dc163f84   UP      10.1.0.3   HEALTHY   KubernetesWorker                       yes   v0.1.0-dev494.g0d8a8a4f.dirty   3s          
metropolis-ec101152c48c5f761534c1910cf66200   UP      10.1.0.2   HEALTHY   ConsensusMember,KubernetesController   yes   v0.1.0-dev494.g0d8a8a4f.dirty   3s     

And just like that, we can now see these nodes in Kubernetes, too:

$ kubectl --context=launch-cluster get nodes
NAME                                          STATUS   ROLES    AGE   VERSION
metropolis-067651202d00b79fffe92df0001aabff   Ready    <none>   15s   v1.24.2+mngn
metropolis-7ccd2437c50696ea9a9b6543dc163f84   Ready    <none>   13s   v1.24.2+mngn
$ kubectl --context=launch-cluster run -it --image=ubuntu:22.04 test -- bash
If you don't see a command prompt, try pressing enter.
root@test:/# uname -a
Linux test 6.1.69-metropolis #1 SMP PREEMPT_DYNAMIC Tue Jan 30 14:43:23 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@test:/# 

With the test launch tooling, you can now start iterating on the codebase. Regardless of whether you're changing the Linux kernel config or implementing a new RPC, testing your changes interactively is a single bazel command away.

End-to-end tests

We have an end-to-end test suite. It's run automatically on CI. Any new logic should be exercised there.

$ bazel run //metropolis/test/e2e:e2e_test

These tests operate on a fully virtualized cluster just like the launch tooling, so be patient.