Added documentation and instructions for nixOS jail

[templehasfallen]

Added documentation and instructions for nixOS jail

• Added documentation for deploying a nixOS based jail
• Updated compatibility table

[Jip-Hop]

Cool! Looks good. By the way I think you can add the bridge interface in the create statement: jlmkr create --distro=nixos --release=23.11 nixos-jail --network-bridge=br1.

[templehasfallen]

Cool! Looks good. By the way I think you can add the bridge interface in the create statement: jlmkr create --distro=nixos --release=23.11 nixos-jail --network-bridge=br1.

You're right, I missed it. Updated

[Jip-Hop]

Thanks for the update!

My idea behind the templates are to provide, if possible, a config template file which can help the user setup a jail in a specific, reproducible, way with as little user interaction as possible.

See for example the docker template: https://github.com/Jip-Hop/jailmaker/tree/main/templates/docker

The readme is almost empty. It only contains the create command.

Do you think you could setup the nixos jail by using the features of the config file? Particularly the initial_setup= comes to mind.

[templehasfallen]

Thanks for the update!

My idea behind the templates are to provide, if possible, a config template file which can help the user setup a jail in a specific, reproducible, way with as little user interaction as possible.

See for example the docker template: https://github.com/Jip-Hop/jailmaker/tree/main/templates/docker

The readme is almost empty. It only contains the create command.

Do you think you could setup the nixos jail by using the features of the config file? Particularly the initial_setup= comes to mind.

While a part of it could be done (replacing/changing the configuration.nix), the OS will still have to be rebuilt manually. The issue with that is because, since host networking is used in the initial setup phase, it will either:

1. Fail to rebuild if CAP_NET_ADMIN is disabled
2. Mess with the hosts iptables table if CAP_NET_ADMIN is enabled (guaranteed to be destructive for the host).

What could be done is editing/adding/replacing the configuration.nix (maybe pulling it from this repo and replacing it?) and once set up, just having to call nixos-rebuild switch inside the jail.

I'll have a look into the above scenario and do some tests next week unless someone gets to it before me.

[Lockszmith-GH]

I was curious so I tried this just now on my Cobia system. This is the first jail I'm installing on this one.
I was able to start the jail

Following the instruction the installation output is:

USE THIS SCRIPT AT YOUR OWN RISK!
IT COMES WITHOUT WARRANTY AND IS NOT SUPPORTED BY IXSYSTEMS.
Creating jail nixos-jail with default config.
Overriding distro config value with nixos.
Overriding release config value with 23.11.
Overriding systemd_nspawn_user_args config value with ['--network-bridge=br0'].
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created an Nixos 23.11 x86_64 (20240419_01:00) container.

WARNING: DISTRO NOT SUPPORTED

Chosen distro appears not to use systemd...

You probably will not get a shell with:
machinectl shell nixos-jail

You may get a shell with this command:
nsenter -t $(machinectl show nixos-jail -p Leader --value) -a /bin/sh -l

Read about the downsides of nsenter:
https://github.com/systemd/systemd/issues/12785#issuecomment-503019081

Using this distro with jlmkr.py is NOT recommended.

Autostart has been disabled.
You need to start this jail manually.

I think it would be wise to include an explanation about the DISTRO NOT SUPPORTED and that it's expected. Or even provide an example of the output so it will look familiar.

Thanks for sharing this, I've been really interested exploring NixOS, and this is the perfect starting point for me.

[Jip-Hop]

I think it would be wise to include an explanation about the DISTRO NOT SUPPORTED and that it's expected.

Thanks for pointing this out!

Jailmaker checks if the init system is systemd. In the case of nixOS the init system is also systemd:

[root@nixos:~]# /sbin/init --version 

<<< NixOS Stage 2 >>>

booting system configuration /nix/store/901sbqb9ys72xjps9c9y6yy5s3l4fxch-nixos-system-nixos-23.11.6418.e402c3eb6d88
running activation script...
setting up /etc...
starting systemd...
systemd 254 (254.10)
+PAM +AUDIT -SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified

[root@nixos:~]# ls -la /sbin/init
lrwxrwxrwx 1 root root 91 Apr 19 16:21 /sbin/init -> /nix/store/901sbqb9ys72xjps9c9y6yy5s3l4fxch-nixos-system-nixos-23.11.6418.e402c3eb6d88/init

But it can't be determined from the filepath that the init system is indeed systemd. So the current check fails and throws this warning while it shouldn't...

If we provide a template for nixOS then I think this check also requires some work...

[Jip-Hop]

just having to call nixos-rebuild switch inside the jail.

Perhaps the initial_setup could also insert a task which would execute nixos-rebuild switch the first time the jail starts? I don't know how nixOS works and if that's possible but here's a generic example of how to add a first boot task via systemd. At least then nixos-rebuild switch will run with --network-bridge=br1 in effect.

[Jip-Hop]

I also ran into NixOS/nixpkgs#63028 when removing the nixOS jail:

jlmkr remove nixos-jail

CAUTION: Type "nixos-jail" to confirm jail deletion!

nixos-jail

Wait for nixos-jail to stop.
Cleaning up: jails/nixos-jail.
Traceback (most recent call last):
  File "/mnt/data/jailmaker/jlmkr.py", line 2122, in <module>
    main()
  File "/mnt/data/jailmaker/jlmkr.py", line 2117, in main
    sys.exit(func(**args))
             ^^^^^^^^^^^^
  File "/mnt/data/jailmaker/jlmkr.py", line 1553, in remove_jail
    cleanup(jail_path)
  File "/mnt/data/jailmaker/jlmkr.py", line 768, in cleanup
    shutil.rmtree(jail_path, onerror=_onerror)
  File "/usr/lib/python3.11/shutil.py", line 732, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.11/shutil.py", line 660, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.11/shutil.py", line 660, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.11/shutil.py", line 666, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/mnt/data/jailmaker/jlmkr.py", line 765, in _onerror
    raise exc_value
  File "/usr/lib/python3.11/shutil.py", line 664, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
PermissionError: [Errno 1] Operation not permitted: 'empty'

My jails are created in plain directories, not datasets.

I had to run chattr -i jails/nixos-jail/rootfs/var/empty/ before I could remove the nixOS jail.

[templehasfallen]

I also ran into NixOS/nixpkgs#63028 when removing the nixOS jail:

I had to run chattr -i jails/nixos-jail/rootfs/var/empty/ before I could remove the nixOS jail.

Interesting. I would say making v1.1.4 with datasets a requirement would make sense. Possibly even a check within the script. Or alternatively an additional step for the remove command if its directories & nixOS?

just having to call nixos-rebuild switch inside the jail.

Perhaps the initial_setup could also insert a task which would execute nixos-rebuild switch the first time the jail starts? I don't know how nixOS works and if that's possible but here's a generic example of how to add a first boot task via systemd. At least then nixos-rebuild switch will run with --network-bridge=br1 in effect.

This is kinda tricky, specifically for nixos, because whatever task of this type you'd add would only be applied once you actually run nixos-rebuild switch.

Thanks for sharing this, I've been really interested exploring NixOS, and this is the perfect starting point for me.

Happy to do so. I'm also just starting to learn NixOS :)

If we provide a template for nixOS then I think this check also requires some work...

Yes I agree. Couldn't have been that easy after all

[templehasfallen]

Its more trickier than thought so far.

The following is the issue.

The nixOS image pulled is barebones, with directories and configuration created @ first boot. This means stuff such as /root, /usr don't exist yet. Attempting to run any kind of initial_setup will fail from systemd-nspawn because /usr does not exist yet.

The error is:

Directory /mnt/path/to/jails/nixostest7/rootfs doesn't look like it has an OS tree (/usr/ directory is missing). Refusing.

which seems to be a safety check from systemd-nspawn as mentioned in paragraph 6

As a safety check systemd-nspawn will verify the existence of /usr/lib/os-release or /etc/os-release in the container tree before booting a container (see os-release(5)). It might be necessary to add this file to the container tree manually if the OS of the container is too old to contain this file out-of-the-box.

The directories in question and the config /etc/nixos/configuration.nix are created during first boot (thanks to the people @ nixOS discord).

The main problem is bypassing/finding a way around the systemd-nspawn safety check.
I thought about placing the initial/setup file in a different location maybe (from what I see it is placed in /root/initial_setup) but I don't understand why the container will start when you don't pass and initial setup and won't if you do. Maybe because the rest of the systemd-run and systemd-nspawn arguments aren't specified during initial setup?

From what I can see there isn't really any option to run it via ExecPreStart only once directly via systemd.

Only option I see to get this working currently is actually adding support to jlmkr via a function/if block.

But I'll dig into it more to see if I can find a workaround.

[Jip-Hop]

You may be able to modify the extracted rootfs from the pre_start_hook. If you can figure out, by looking inside the rootfs, if it's the first boot, then I think you may be able to modify the nixOS configuration even before it booted and you may not even need to run nixos-rebuild switch. This is me brainstorming. Pure speculation, I have 0 experience with nixOS.

[Jip-Hop]

Attempting to run any kind of initial_setup will fail from systemd-nspawn because /usr does not exist yet.

Since jailmaker v1.4.0 the initial_setup runs once the jail has started for the first time and the init is ready.

[templehasfallen]

Attempting to run any kind of initial_setup will fail from systemd-nspawn because /usr does not exist yet.

Since jailmaker v1.4.0 the initial_setup runs once the jail has started for the first time and the init is ready.

I've tan some with the bew version, still unable to get it to fully work but making progress.

I'm trying to edit the config in place with the initial setup and then running nixos-rebuild switch which seems to complete successfully, but then config doesn't add up.

[dkowis]

Thanks for the update!
My idea behind the templates are to provide, if possible, a config template file which can help the user setup a jail in a specific, reproducible, way with as little user interaction as possible.
See for example the docker template: https://github.com/Jip-Hop/jailmaker/tree/main/templates/docker
The readme is almost empty. It only contains the create command.
Do you think you could setup the nixos jail by using the features of the config file? Particularly the initial_setup= comes to mind.

While a part of it could be done (replacing/changing the configuration.nix), the OS will still have to be rebuilt manually. The issue with that is because, since host networking is used in the initial setup phase, it will either:

1. Fail to rebuild if `CAP_NET_ADMIN` is disabled

2. Mess with the hosts iptables table if `CAP_NET_ADMIN` is enabled (guaranteed to be destructive for the host).

What could be done is editing/adding/replacing the configuration.nix (maybe pulling it from this repo and replacing it?) and once set up, just having to call nixos-rebuild switch inside the jail.

Is this something that would be accepted into nixos-hardware so it can just be used as a module for this "hardware"?

I don't know if it'd accept the bits from the config file, but maybe something could be made to do that...

There's a lot I don't know, and I'm experimenting with it also :)