Magus: Setting up package builds

Setting up a magus cluster requires a few things

PostgreSQL database (9.x or higher, tested up to 13) Subversion & git installed magus-utils from ports / package installed. (ports-mgmt/magus-utils) mports checked out on each build node and your master node. (you can just use one system as both) You then need to setup the database by loading the schema.sql and ready_ports.sql scripts.
Nodes must be able to communicate with a storage location. It can be available via NFS or scp. In the latter case, you must setup ssh keys without passwords. If you're running on one machine, it can just be a directory Code for magus is included in mports with most of it in:

/usr/mports/Tools/magus/

Once the database is created, you need to create a user for each node to communicate read/write with. Unfortunately, all nodes must be able to communicate with the database server directly. We'd like to move to a REST interface for this in the next version. You can use a shared account or make a unique one.

Create a config.yml for each node and master. This is how you set database settings, etc. A sample config is available in the Tools/magus directory

Each node should have a unique machine name. This will also need to be inserted into the machines table in the magus database. This needs the OS version and architecture. This is how magus knows what a node can build.

The example file contains a local path by default for packages. You can point this at an nfs mount or a scp friendly path

The config file also needs a path to fetch the "run tarball" which is what is created after indexing from the checked out mports directory.

PkgExtension: mport

MasterDataDir: ftp://stargazer.midnightbsd.org/pub/magus MportsSnapDir: runs

The database also needs a mirrors table entry for the location to download packages. This will be written into the index

Workflow

1. Indexing a checked out copy of mports. This checks it out, runs a perl script to go through all directories and create database entries for each port and it's dependencies as well as taking into account flavors.
2. Start one or more magus.pl scripts on each slave via /usr/mports/Tools/magus/slave/magus.pl (optionally use -f for foreground and -v for verbose to see what is going on) You can also use -j 2 to run two builds on the same node.
3. Each node will pull a job from postgres, create a lock in pg, download dependencies from shared storage, do the build, write the file generated back to the shared storage and remove the lock.
4. Once the run is done, you must create an index file (sqlite database) that contains all the build packages, minus restricted packages. This is done with the magus-bless utility.

Blessing

When a package build is done, a magus index must be generated for mport clients to download and use. The code for this is in /usr/mports/Tools/magus/bless

It's a C program that must link against PostgreSQL client and sqlite3 client.

./magus-bless <path/to/files>

Once done, you should get a file like magus-.db. You can open it with sqlite3 and then should vacuum, and verify everything looks good. Then rename the file to index.db and bzip2 it.