Skip to content

My Nix ecosystem journey, from the first commit to currently running configs

Notifications You must be signed in to change notification settings

nazarewk-iac/nix-configs

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

nix-configs

Repository containing my personal Nix (NixOS, Home Manager etc.) configurations.

Basic structure:

  • modules/ - all modules live here, they MUST be turned off by default (side-effect free imports),
    • modules/default.nix - holds imports to all the modules and basic Nix package manager configuration,
    • modules/profile/machine - large NixOS configuration bundles
    • modules/profile/user - user specific profiles
  • packages/, some personal/in-house/out-of-band tools

Generally I aim to hide as much as possible behind *.enable options.

Overview

This is an incomplete list of incorporated software/systems:

  • disks: ZFS on LUKS through disko
  • desktop: Sway (Wayland), as much as possible through Home Manager
  • shell: Fish
  • users: Home Manager
  • MISSING: development environment

Notes

Custom ISO installer

see https://bmcgee.ie/posts/2022/12/setting-up-my-new-laptop-nix-style/

# building the installer from packages/install-iso
sudo dd if="$(nom build '.#install-iso' --no-link --print-out-paths --print-build-logs)/iso/nixos.iso" of=/dev/disk/by-id/usb-SanDisk_Cruzer_Blade_02000515031521144721-0:0 status=progress
sudo dd if="$(nom build '.#install-iso' --no-link --print-out-paths --print-build-logs)/iso/nixos.iso" of=/dev/disk/by-id/usb-_Patriot_Memory_070133F17AC22052-0:0 status=progress
# boot the machine and ssh into it
ssh -o StrictHostKeyChecking=no kdn@nixos

Golden path for bootstrapping new physical machine

  1. build the install-iso and boot it

  2. prepare a new config at modules/profile/host/${HOSTNAME}/default.nix

    • put kdn.profile.machine.baseline.enable = true
    • look through kdn.hardware.{gpu,cpu}.{intel,amd}
    • set up zramSwap & boot.tmp.tmpfsSize
    • figure out missing boot.initrd.{availableK,k}ernelModules
  3. set up a disk configuration:

    (
      # hashtag comment denotes defaults
      # /* */ are just comments
      let
        cfg = config.kdn.hardware.disks;
        d1 = "<DISK_1_NAME>-${config.networking.hostName}";
        d1Cfg = cfg.luks.volumes."${d1}";
        d2 = "<DISK_2_NAME>-${config.networking.hostName}";
        d2Cfg = cfg.luks.volumes."${d2}";
      in
      {
        kdn.hardware.disks.enable = true;
        #kdn.hardware.disks.zpool-main.name = "<HOSTNAME>-main";
        #kdn.hardware.disks.devices."boot" = { type = "gpt"; content.partitions.ESP = { size = "4096M"; ... }; };
        #kdn.hardware.disks.zpools."${cfg.zpool-main.name}" = { };
        #disko.devices.zpool."${cfg.zpool-main.name}".datasets = {
        # "${hostname}/nix-system/nix-store" = { mountpoint = "/nix/store"; ... };
        # "${hostname}/nix-system/nix-var" = { mountpoint = "/nix/var"; ... };
        #}
    
        /* point it at the right detached `/boot` disk (USB flash drive) */
        kdn.hardware.disks.devices."boot".path = "/dev/disk/by-id/usb-<CORRECT_IDENTIFIER>";
        #disko.devices.disk.boot = { content.type = "gpt"; content.partitions = { ... }; ... };
        #kdn.hardware.disks.devices."${d1}" = { type = "luks"; path = d1Cfg.targetSpec.path; };
        #disko.devices.disk."${d1}" = { type = "luks"; name = "${d1}-crypted"; ... };
        #kdn.hardware.disks.devices."${d2}" = { type = "luks"; path = d2Cfg.targetSpec.path; };
        #disko.devices.disk."${d1}" = { type = "luks"; name = "${d1}-crypted"; ... };
    
        kdn.hardware.disks.luks.volumes."${d1}" = {
          #target.deviceKey = d1;
          targetSpec.path = "/dev/disk/by-id/<DISK_PATH>";
          uuid = "<uuidgen result>";
          #keyFile = "/tmp/${d1}.key";
          #header.deviceKey = "boot";
          #header.partitionKey = d1;
          headerSpec.num = 2;
          #zpool.name = cfg.zpool-main.name;
        };
    
        kdn.hardware.disks.luks.volumes."${d2}" = {
          #target.deviceKey = d2;
          targetSpec.path = "/dev/disk/by-id/<DISK_PATH>";
          uuid = "<uuidgen result>";
          #keyFile = "/tmp/${d1}.key";
          #header.deviceKey = "boot";
          #header.partitionKey = d2;
          headerSpec.num = 3;
          #zpool.name = cfg.zpool-main.name;
        };
    
        #kdn.hardware.disks.impermanence."sys/config".snapshots = true;
        #kdn.hardware.disks.impermanence."sys/cache".snapshots = false;
        #kdn.hardware.disks.impermanence."sys/data".snapshots = true;
        #kdn.hardware.disks.impermanence."sys/state".snapshots = false;
        #kdn.hardware.disks.impermanence."usr/config".snapshots = true;
        #kdn.hardware.disks.impermanence."usr/cache".snapshots = false;
        #kdn.hardware.disks.impermanence."usr/data".snapshots = true;
        #kdn.hardware.disks.impermanence."usr/state".snapshots = false;
        
        /* just a single impermanence example goes below */
        #kdn.hardware.disks.impermanence."usr/data" = {
        #  #zpool.name = cfg.zpool-main.name;
        #  #zfsPrefix = "${config.networking.hostName}/impermanence";
        #  #zfsPath = "${zfsPrefix}/${"usr/data"}";
        #  imp.directories = [
        #    "/var/lib/libvirt/images"
        #  ];
        #  imp.users.root.directories = [
        #    # ...
        #  ];
        #  imp.users.kdn.directories = [
        #    ".local/share/atuin"
        #    ".local/share/nix"
        #    ".local/share/containers"
        #    "dev"
        #  ];
        #};
        #environment.persistence."/nix/persist/usr/data" = {
        #  enable = true; 
        #  hideMounts = true; 
        #  users.root.home = "/root";
        #} // cfg.impermanence."usr/data".imp;
        #disko.devices.zpool."${cfg.zpool-main.name}".datasets."${cfg.impermanence."usr/data".zfsPath}" = { ... };
    
        #kdn.hardware.disks.tmpfs.size = "16M";
        #disko.devices.nodev."/" = { fsType = "tmpfs"; mountPoints = ["size=${cfg.tmpfs.size}" ... ]; };
        #disko.devices.nodev."/" = { fsType = "tmpfs"; mountPoints = ["size=${cfg.tmpfs.size}" ... ]; };
      }
    )
  4. add all your required environment.persistence entries

  5. set up keyfiles for each disk:

    dd if=/dev/random bs=1 count=2048 of=/dev/stdout | pass insert --force --multiline luks/${DISK_NAME}-${HOST_NAME}/keyfile
  6. boot the install-iso

  7. run nixos-anywhere to deploy

    nixos-anywhere --no-reboot --disk-encryption-keys /tmp/${DISK_NAME}-${HOST_NAME}.key "$(pass show luks/${DISK_NAME}-${HOST_NAME}/keyfile | psub)" --flake '.#${HOST_NAME}' nixos.lan.
  8. set up either of for each disk:

    • (unattended) TPM2 unlock:
      ssh nixos.lan. sudo systemd-cryptenroll --unlock-key-file=/tmp/${DISK_NAME}-${HOST_NAME}.key --tpm2-device=auto /dev/disk/by-partlabel/${DISK_NAME}-${HOST_NAME}-header 
    • (attended) YubiKey FIDO2 (touch required, without PIN) unlock:
      ssh nixos.lan. sudo systemd-cryptenroll --unlock-key-file=/tmp/${DISK_NAME}-${HOST_NAME}.key --fido2-device=auto --fido2-with-client-pin=false --fido2-with-user-verification=false /dev/disk/by-partlabel/${DISK_NAME}-${HOST_NAME}-header 
  9. add SSH key to /.sops.yaml

    • ssh-to-age </etc/ssh/ssh_host_ed25519_key.pub
    • reboot

Building on Hetzner Cloud from NixOS installer image

  1. mount the NixOS installer image
  2. run the build:
    APPLY=1 HOST="<HOST>" bash <(curl -L 'https://raw.githubusercontent.com/nazarewk-iac/nix-configs/main/hetzner.sh')
    

Interaction between NixOS and Home Manager

How to find out what uses the specific store path?

Find immediate parents/reverse dependencies: nix-store --query --referrers <paths...>.

Find the root using paths: nix-store --query --roots <paths...>.

Fix nix store errors

Fix errors like /nix/store/*-source not found: sudo nix-store --repair --verify --check-contents.

GitHub rate-limiting unauthenticated requests

see https://discourse.nixos.org/t/flakes-provide-github-api-token-for-rate-limiting/18609

add token to ~/.config/nix/nix.conf:

access-tokens = github.com=github_pat_XXXX

example of standalone Nix module

https://gist.github.com/nazarewk/8988facb6118f73d2db3d28b64463cba

systemd-cryptenroll --fido2-with-user-presence=false doesn't work with YubiKey

YubiKey's FIDO2 applet seems to ALWAYS require touch, so it cannot be used for unattended unlock of LUKS volume:

> systemd-cryptenroll --unlock-key-file=$(cat /tmp/emmc-etra.key | psub) --fido2-device=auto --fido2-with-client-pin=false --fido2-with-user-verification=false --fido2-with-user-presence=false /dev/disk/by-partlabel/disk-boot-emmc-etra-header
Initializing FIDO2 credential on security token.
πŸ‘† (Hint: This might require confirmation of user presence on security token.)
Generating secret key on FIDO2 security token.
πŸ‘† Locking without user presence test requested, but FIDO2 device /dev/hidraw1 requires it, enabling.
New FIDO2 token enrolled as key slot 1.

Keeping nixpkgs fork with arbitrary patches up to date

How to use https://github.com/katrinafyi/nix-patcher to maintain nixpkgs fork:

  1. Create required nixpkgs inputs, see nix-patcher repo for more details:
    • nixpkgs-upstream - desired upstream nixpkgs branch
    • nixpkgs - the fork you have access to that will be managed by nix-patcher
    • nixpkgs-patch-<num> - numbered patches to be applied in specific order on top of nixpkgs-upstream and pushed to nixpkgs fork
  2. run nix flake update && GITHUB_TOKEN="$(pass show python-keyring/git/github.com/nazarewk)" nix-patcher --update "$@"
  3. In your root flake.nix create the nixpkgs input pointing to the same input as the sub-flake.
  4. Run nix flake update nixpkgs in your root flake.

Most of it is wrapped in /nixpkgs-update.sh and top entries of /flake.nix.

Note this can be used to patch arbitrary flake, not just nixpkgs! Just create <input>-upstream, <input> & <input>-patch-<num> inputs.

About

My Nix ecosystem journey, from the first commit to currently running configs

Topics

Resources

Stars

Watchers

Forks