Clover is a control plane and web console program for managing virtual machines and other programs.
It's a Ruby program that connects to Postgres.
The source organization is based on the Roda-Sequel Stack, though a number of development choices have been modified. As the name indicates, this project uses Roda (for HTTP code) and Sequel (for database queries).
Web authentication is managed with Rodauth.
It communicates with servers using SSH, via the library net-ssh.
The tests are written using RSpec.
Code is automatically linted and formatted using RuboCop.
Web console is designed with Tailwind CSS based on components from Tailwind UI. It uses jQuery for interactivity.
I suggest using asdf-vm to manage
software versions. There is a .tool-versions file
that asdf-vm reads, and it is kept up to date.
In the case of Ruby, obtaining a matching version is most important, because it is constrained in the Gemfile.
Though, any method of obtaining Ruby and Postgres is adequate.
If using asdf-vm, follow the instructions at the Getting Started
Manual. There are
three general steps:
- Download some common dependencies,
gitandcurl. You may already have them. - Use
gitto cloneasdf-vminto your home directory - Source it into your shell automatically
Having done so, typing asdf will yield a bunch of help text:
$ asdf
version: v0.11.2-8eb11b8
MANAGE PLUGINS
asdf plugin add <name> [<git-url>] Add a plugin from the plugin repo OR,
[...]
I like to have these four plugins (you can paste these commands):
asdf plugin add ruby
asdf plugin add direnv
asdf plugin add postgres
asdf plugin add nodejs
Once you have the plugins, you can start to install the software the plugin supports. Let's first install Ruby.
First, install some system dependencies, e.g. a C and Rust compiler. There is documentation listing of commands you can use for each platform (e.g. Macintosh Homebrew, or Ubuntu).
After that, install Ruby. asdf will consult the .tool-versions
file to select the version.
asdf install ruby
Having done this, you can see if your $PATH finds the ruby "shim"
generated by asdf and consult the version:
$ which ruby
/home/fdr/.asdf/shims/ruby
$ ruby --version
ruby 3.2.1 (2023-02-08 revision 31819e82c8) [x86_64-linux]
I find use of asdf with
asdf-direnv almost
obligatory, for the reasons discussed in its README. Let's set it up
as a user-global tool, and not a project-local one:
echo "direnv 2.32.2" >> ~/.tool-versions
asdf direnv setup --version latest
After direnv setup you need to source your shell's startup files or
start a new shell.
Now, upon cd-ing into the clover directory -- which already has a
.envrc committed -- you should see something like:
direnv: error /home/fdr/code/clover/.envrc is blocked. Run `direnv allow` to approve its content
Okay, let's allow direnv in this directory:
$ asdf exec direnv allow .
direnv: loading ~/code/clover/.envrc
direnv: using asdf
direnv: Creating env file /home/fdr/.cache/asdf-direnv/env/2363097900-478416608-3909218245-3753665172
[...]
You should still be able to get the same Ruby version:
$ ruby --version
ruby 3.2.1 (2023-02-08 revision 31819e82c8) [x86_64-linux]
But, instead of being resolved through a "shim" program, the binary is referenced directly:
$ which ruby
/home/fdr/.asdf/installs/ruby/3.2.1/bin/ruby
You will need dependencies to compile Postgres installed on your system.
First, set some autoconf ./configure options to be passed to
Postgres:
$ echo "POSTGRES_EXTRA_CONFIGURE_OPTIONS='--with-uuid=e2fs --with-openssl'" > ~/.asdf-postgres-configure-options
Then run:
asdf install postgres
There are many alternative ways to get Postgres, e.g. via system package manager, Homebrew, Postgres.app, etc. They are all acceptable, our version requirements for Postgres are more relaxed than with Ruby.
Node.js is required to work with frontend tooling such as tailwind-cli.
Install Node.js. asdf will consult the .tool-versions
file to select the version.
asdf install nodejs
Having done this, you can see if your $PATH finds the node "shim"
generated by asdf and consult the version:
$ which node
/Users/enescakir/.asdf/installs/nodejs/20.2.0/bin/node
$ node --version
v20.2.0
You may need to re-generate your direnv cache when adding programs
that satisfy version requirements after having generated the direnv
directory cache. This can be done via touch .envrc:
$ touch .envrc
direnv: loading ~/code/clover/.envrc
direnv: using asdf
direnv: Creating env file /home/fdr/.cache/asdf-direnv/env/2363097900-478416608-196231759-3753665172
direnv: loading ~/.cache/asdf-direnv/env/2363097900-478416608-196231759-3753665172
direnv: using asdf direnv 2.32.2
direnv: using asdf postgres 15.1
direnv: loading ~/.asdf/plugins/postgres/bin/exec-env
direnv: using asdf ruby 3.2.1
direnv: loading ~/.asdf/plugins/ruby/bin/exec-env
direnv: export +LD_LIBRARY_PATH +PGDATA +PGHOST +PGPORT +RUBYLIB ~PATH
Although reading this output closely is seldom necessary, here we can
see all the paths exported by plugins activated by .tool-versions.
Note that $PGDATA is exported. Thus, we can start Postgres:
$ postgres -D $PGDATA
2023-03-01 13:22:43.682 PST [36002] LOG: starting PostgreSQL 15.1 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 12.2.1 20221121 (Red Hat 12.2.1-4), 64-bit
Also note LD_LIBRARY_PATH:
$ printenv LD_LIBRARY_PATH
/home/fdr/.asdf/installs/postgres/15.1/lib
This is important to be able to compile client drivers against the
correct libpq.
Clover uses one database per installation, but is developed using two
such installations, and thus, two databases. One environment is
called "development" and the other "test", and they each have a
database: clover_development and clover_test. Only one user is
used to connect to both databases, though, named clover.
Presuming you have set up Postgres using asdf-vm, run a server in a
dedicated terminal window with postgres -D $PGDATA set aside, and
then create the user and databases:
createuser -U postgres clover
createuser -U postgres clover_password
createdb -U postgres -O clover clover_test
psql -U postgres -c 'CREATE EXTENSION citext; CREATE EXTENSION btree_gist;' clover_test
createdb -U postgres -O clover clover_development
psql -U postgres -c 'CREATE EXTENSION citext; CREATE EXTENSION btree_gist;' clover_development
The clover_test database is used by automated tests, and is prone to
automatic truncation and the like. clover_development is the
database used by default, where the developer (you) manages the data.
For example, you might create records in clover_development
addressing a few hosts you bought on Hetzner and then experiment with
creating and destroying VMs this way. Looking at the clover_test
database is rare, usually when working on or debugging the testing
infrastructure itself.
You can read config.rb to see what environment variables are used.
CLOVER_DATABASE_URL and RACK_ENV are mandatory, but for running
tests, you will also need to set CLOVER_SESSION_SECRET and
CLOVER_COLUMN_ENCRYPTION_KEY. The former is necessary for web (but
not database model) tests, the latter is necessary for any test that
uses an encrypted column.
Our programs load a file .env.rb if present to run arbitrary Ruby
code to set up the environment. You can generate a sensible .env.rb
with rake overwrite_envrb:
$ rake overwrite_envrb
$ cat .env.rb
case ENV["RACK_ENV"] ||= "development"
when "test"
ENV["CLOVER_SESSION_SECRET"] ||= "mbvxopHlcCTWxT6E62weAT+9vxAr1BJp7X3OuQ4K+fFYOLwM20wBVHLuM5tITJDZcEMy2luUD9CDbfgU9okiCw=="
ENV["CLOVER_DATABASE_URL"] ||= "postgres:///clover_test?user=clover"
ENV["CLOVER_COLUMN_ENCRYPTION_KEY"] ||= "EWLXd9OzR7Rvs254gVOE9BeTv3fBoZeysOjcNReu5zw="
else
ENV["CLOVER_SESSION_SECRET"] ||= "/UBMRpwQ5NN3NmSM81FtqDfaaRWhqxbmfFXMxMA2fjcdUk53SZF5n4SKd+uAIpPgPWx1ItRGq/JW1yzQqx0PdQ=="
ENV["CLOVER_DATABASE_URL"] ||= "postgres:///clover_development?user=clover"
ENV["CLOVER_COLUMN_ENCRYPTION_KEY"] ||= "9sljUbAiMmH0uiYE6lM64Tix72ehGr0W7yFrbpD+l4s="
end
Here we can see that .env.rb chooses the database and keys in question
based on RACK_ENV, defaulting to development.
Note that these keys change with every execution of overwrite_envrb,
so generating a new .env.rb can result in encrypted data in your
clover_development database being indecipherable. You are unlikely
to generate this file often, and can probably use the same .env.rb
with minor modifications for years.
Like most programming environments, Ruby has an application-level
dependency management system, called
RubyGems. We manage those versions through
the program bundler, which itself we get
through the low-level gem command:
$ which gem
/home/fdr/.asdf/installs/ruby/3.2.1/bin/gem
$ gem install bundler
Fetching bundler-2.4.7.gem
[...]
$ bundle install
[...]
Bundle complete! 30 Gemfile dependencies, 75 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
Bundler's function is to solve complex gem version constraint upgrades
(when running bundle update) and to generate and interpret
Gemfile.lock to select the correct Gem versions to be
loaded when multiple versions are installed. This is done via bundle exec or loading bundler in application code, such as
loader.rb's call to Bundler.setup. In general, bundle exec is necessary when Clover does not control the entry point into
the program, such as rubocop (to lint code) or rspec (to run
tests):
$ bundle exec rubocop
But it's not necessary with programs in bin that we control and load
loader.rb right away, as a convenience:
$ ./bin/pry
It's harmless yet duplicative to run:
$ bundle exec bin/pry
RuboCop is a code linter and rewriter. It can take care of all minor
formatting issues automatically, e.g. indentation. You can run
auto-correction with bundle exec rubocop -a
If you ran overwrite_envrb, it generates a file that's prone to
correction by RuboCop:
$ bundle exec rubocop -a
Inspecting 68 files
C...................................................................
Offenses:
.env.rb:6:34: C: [Corrected] Style/StringLiterals: Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping.
ENV["CLOVER_DATABASE_URL"] ||= 'postgres:///clover_test?user=clover'
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
68 files inspected, 1 offense detected, 1 offense corrected
Some useful corrections are only made with bundle exec rubocop -A
(upper case A) which applies "unsafe" corrections that may alter the
semantics of the program.
Empty databases will cause Clover to crash. You can run database
migrations (presuming the database is running with e.g. postgres -D $PGDATA) with rake:
$ rake test_up
$ rake dev_up
The way this works is, the rake task for these sets RACK_ENV, and
.env.rb generated by the overwrite_envrb task interprets
RACK_ENV to find the right configuration set, including the database
name to migrate.
With the database running and the test database up to date with migrations, you can run the tests:
$ bundle exec rspec
or even just:
$ rake
As the default rake task runs all the tests.
You can collect coverage by setting:
$ COVERAGE=1 rake
You can run a specific file or line when using bundle exec rspec:
$ bundle exec rspec ./spec/model/strand_spec.rb
$ bundle exec rspec ./spec/model/strand_spec.rb:10
There is editor integration for RSpec that are very useful.
rspec-mode for emacs (as seen in M-x list-packages) has lisp
procedures rspec-verify to run rspec on the file where the point
is, rspec-verify-single to run it on the line the point is at, and
rspec-rerun to run rspec the same way as whatever came last, which
is excellent when editing code that should affect the outcome of a
test. There is also rspec-verify-all which runs all the specs, but
this is less essential than running one or a few specs with editor
integration.
Assuredly, there is all this and more in other editor environments.
Web Console is designed with Tailwind CSS. Tailwind CSS works by scanning all of your HTML files, JavaScript components, and any other templates for class names, generating the corresponding styles and then writing them to a static CSS file. You need to generate CSS file before running web console if you do not want to see HTML files without any style.
We manage node module versions through npm. It's
installed with nodejs package.
$ which npm
/Users/enescakir/.asdf/installs/nodejs/20.2.0/bin/npm
$ npm install
[...]
added 86 packages, and audited 87 packages in 1s
14 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
Now we can build CSS file. If you do development on UI, you can run
npm run watch on separate terminal window to see changes realtime.
$ npm run prod
> prod
> npx tailwindcss -o assets/css/app.css --minify
Rebuilding...
Done in 767ms.
assets/css/app.css should be created. Let's start our web server.
bundle exec rackup
And then visiting http://localhost:9292, you can create an account. Check the rackup log for the verification link to navigate to, in production, we would send that output as email. Having verified, log in. You'll see the "Getting Started" page.
When you change any template file, format them with erb-formatter:
rake linter:erb_formatter
That's everything there is to know. As exercise, you can consider
inserting a crash into some source under test (e.g. strand.rb) and
try to make the tests fail with a backtrace:
An edited strand.rb:
[...]
def self.lease(id)
fail "my first crash"
affected = DB[<<SQL, id].first
[...]
And, the crash:
bundle exec rspec ./spec/model/strand_spec.rb
Randomized with seed 60335
Strand
can load a prog
can run a label (FAILED - 1)
can take leases (FAILED - 2)
Failures:
1) Strand can run a label
Failure/Error: st.run
RuntimeError:
my first crash
# ./model/strand.rb:15:in `lease'
# ./model/strand.rb:9:in `lease'
# ./model/strand.rb:44:in `run'
# ./spec/model/strand_spec.rb:24:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:41:in `block (3 levels) in <top (required)>'
# ./spec/spec_helper.rb:40:in `block (2 levels) in <top (required)>'