Jails on boot drive

[spusuf]

TrueNAS is notorious for disallowing any use of the boot drive, leaving 100gb unused on a basic 120gb drive.
Would it be feasible to use jails in this space?
Are there any drawbacks or limitations?
Does the install script work from the boot drive (root of filesystem) as opposed to within a dataset?

I am planning to test this myself, but wanted a thread for anyone with similar testing. I'm not going to do this on my main server, just waiting for the time to reset my second server.

[Jip-Hop]

Why would you want to do that? The root filesystem is completely overwritten when you upgrade to the next (minor) release. All your jail files and config would be lost. It's kind of the whole point of jailmaker to prevent / work around this. Otherwise you could just use systemd-nspawn directly...

[Jip-Hop]

In that case just play with systemd-nspawn directly. It should be pre-installed (again). Jailmaker doesn't and will not support this.

[Lockszmith-GH]

Totally agree with Jip-Hop, if your stung up about the unused disk space, repartition the drive. That at least survives an upgrade.

[Lockszmith-GH]

@spusuf check out this and/or this reddit post

[spusuf]

I didnt see this until now, but partitioning during boot can (and did for the few times I tried) lead to non boot. I'd much rather have a config file on a persistent location and have jailmaker recreate the jail if necessary (a la docker compose). But I understand this is out of scope.

This is one of the main reasons I have switched to Debian. Losing an entire drive for such little operations and storage that it could be done by emmc on a small appliance/low end server with only 4x sata + occupied PCIe, is entirely unreasonable.

[Lockszmith-GH]

I use boot drive partitioning, but not for extra storage, mostly for performance.
I would agree that IX got the right idea of having the boot drives disposable, I just tweak them because small SSD drives are no longer a feasable thing to buy, and so not utilizing the rest of the drives seems like a waste.
So I'm agree with you @spusuf, I'd rather not put any precious storage on the boot drives either.

I can share my experience though of this working for over a year on multiple systems, surviving IX upgrades. Luckily never had a boot drive hardware failure or corruption, but I did test failover which gave me both peace of mind + I got the performance boost for the mechanical drives, which is what I wanted.

But my point is, repartitioning the boot drive correctly survives IX system upgrades, and does not affect the boot process.

While I understand why IX doesn't offer this out of the box, This has become my standard method of operation with TrueNAS SCALE.
It's a hassle to setup, but once it's setup - no further tinkering is required.

My setup, in case you are interested

(click to expand)

I did not come up with this design, stumbled on it in the old TrueNAS forums a while back.

NOTE: I've simplified the output to use short device name, but I recommend using PartUUIDs instead, which is what I do.

My boot-pool pool is a mirror of 2x 46GB partitions (sdc3, sdd3) out of 2x 256 GB SSD drives (sdc, sdd).
My tank pool is a mirror of 2x 1TB mechanical drives (sda, sdb).

Each of the boot drives are partitioned the same way, contianing an EFI boot parition (sdc2, sdd2), a swap partitoin (sdc4, sdd4).
The remainder of the SSD boot drivers (~170Gb) are assigned to the tank pool as L2ARC cache (sdd5) and slog for ZIL (sdc5), improving the performance of ZFS on the mechanical drives.

# zpool status
  pool: boot-pool
 state: ONLINE
...
config:
        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdc3    ONLINE       0     0     0
            sdd3    ONLINE       0     0     0
...
  pool: tank
 state: ONLINE
...
config:
        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb2    ONLINE       0     0     0
            sda2    ONLINE       0     0     0
        logs
          sdc5      ONLINE       0     0     0
        cache
          sdd5      ONLINE       0     0     0
          
# lsblk --output=NAME,SIZE,TYPE,FSTYPE,LABEL
NAME          SIZE TYPE  FSTYPE            LABEL
sda         931.5G disk
├─sda1          2G part
└─sda2      929.5G part  zfs_member        tank
sdb         953.9G disk
├─sdb1          2G part
└─sdb2      951.9G part  zfs_member        tank
sdc         232.9G disk
├─sdc1          1M part
├─sdc2          2G part  vfat              EFI
├─sdc3         46G part  zfs_member        boot-pool
├─sdc4         16G part  linux_raid_member swap0
│ └─md127      16G raid1
│   └─md127    16G crypt swap
└─sdc5      168.9G part  zfs_member        tank
sdd         232.9G disk
├─sdd1          1M part
├─sdd2          2G part  vfat              EFI
├─sdd3         46G part  zfs_member        boot-pool
├─sdd4         16G part  linux_raid_member swap0
│ └─md127      16G raid1
│   └─md127    16G crypt swap
└─sdd5      168.9G part  zfs_member        tank
zd0            32G disk
zd16           32G disk