Automating Infrastructure with Edge Image Builder

SUSE's Edge Image Builder (EIB) skips config management entirely for edge deployments: you describe a cluster in one YAML definition, and it hands back a self-installing ISO that partitions the disk, installs the OS, and bootstraps RKE2 on first boot — no PXE server, no Ansible run, no manual kubeadm join. The definition I started from was a reference architecture for a full SUSE Observability stack: five nodes — three HA servers and two agents — with cert-manager, Longhorn, MetalLB, ingress-nginx, and the observability Helm charts themselves all baked directly into the image.

The Build Pipeline

There's only one build. The exact same ISO that gets validated in a disposable VM is the one that ends up on the five physical boxes — nothing is rebuilt for hardware. The VM exists purely to answer "does this image even install and boot" somewhere free to fail, before trusting that same file to real disks:

Repository Layout

Every file named throughout this post lives in one flat-ish repo. Worth calling out: the two Helm-chart directories under kubernetes/ are empty, not missing — they held values files and manifests before the strip-down and were never deleted, just emptied. EIB's own _build/ directory (its per-run logs and extracted artefacts) and the multi-gigabyte base ISO and build outputs are real, but collapsed here since they're binary and not something you'd read:

eib-base-lenny/
├── edge-cluster.yaml            the EIB cluster definition
├── build.sh                     podman-wrapped `eib build` invocation
├── test-vm.sh                   builds + boots the KVM test VM
├── smart-test.sh                SMART extended self-test for real disks
├── nodes.txt                    MAC / IP reservations, lenny1-5
├── CHANGES.md                   history of the strip-down and scale-up
├── custom/scripts/
│   ├── 01-fix-growfs.sh         SUSE bugzilla #1217430 workaround
│   └── 02-tmpfs-cni.sh          tmpfs mounted over /etc/cni
├── network/
│   ├── lenny1.shed.local.yaml   NMState, one file per node
│   ├── lenny2.shed.local.yaml
│   ├── lenny3.shed.local.yaml
│   ├── lenny4.shed.local.yaml
│   └── lenny5.shed.local.yaml
├── kubernetes/
│   ├── config/
│   │   ├── server.yaml          RKE2 server config
│   │   └── agent.yaml           RKE2 agent config
│   ├── helm/values/             empty — stripped with the Helm charts
│   └── manifests/               empty — stripped with the Helm charts
├── os-files/etc/profile.local
├── rpms/gpg-keys/rancher-public.key
├── base-images/
│   └── SL-Micro...install.iso   unmodified base image, 1.6G
├── eib-lenny.iso                build output, 3.0G
├── eib-test.org                 raw test-VM disk image, 5.4G
└── _build/                      EIB's own build logs/artefacts (transient)

The Original Definition

The reference architecture this started from wasn't a toy — it was a genuine five-node edge deployment meant to run SUSE Observability in production, with every dependency that implies:

On top of the charts, the definition also embedded an embeddedArtifactRegistry block listing every Longhorn container image (so a first boot doesn't need to pull them from the internet), installed the open-iscsi package Longhorn needs for its volume backend, and dropped a sysctl.d drop-in tuning kernel network buffer sizes for the observability stack's data ingestion workload. Three server nodes gave the control plane and etcd quorum, two agents ran the workload, and network.apiVIP / network.apiHost fronted the three servers with a kube-vip virtual IP so the API stayed reachable if any one server went down.

That's six Helm charts, an embedded image registry, custom kernel tuning, and HA networking — a lot of surface area to debug simultaneously if the ISO doesn't even install cleanly. Before touching any of it on real hardware, I stripped the definition down to see if the basic imaging mechanism worked at all.

Strip It to the Studs First

Out went every Helm chart — cert-manager, Longhorn and its CRDs, MetalLB, ingress-nginx, the observability charts — along with their values files (certmanager.yaml, longhorn.yaml, ingress-nginx.yaml, suse-observability.yaml) and the manifests that configured them (cert-issuer.yaml, metallb-config.yaml). The embeddedArtifactRegistry block and open-iscsi package went with Longhorn, since neither had any purpose without it, and the network-tuning sysctl.d drop-in went too — it existed purely to help the observability stack ingest data faster, and there was no observability stack left to tune for.

The cluster topology went from three servers and two agents down to a single type: server node, which also meant dropping network.apiVIP and network.apiHost — kube-vip's HA virtual IP has nothing to front once there's only one API server to reach.

What was left was about as close to "just RKE2 and Calico" as the definition could get:

apiVersion: 1.2
image:
  arch: x86_64
  imageType: iso
  baseImage: SL-Micro.x86_64-6.2-Default-SelfInstall-GM.install.iso
  outputImageName: eib-test.iso
operatingSystem:
  isoConfiguration:
    installDevice: /dev/sda
  packages:
    packageList:
      - jq
      - openssh-server-config-rootlogin
kubernetes:
  version: v1.33.3+rke2r1
  nodes:
    - hostname: node1.shed.local
      initializer: true
      type: server

Building the ISO

EIB itself ships as a container, not a package, so building the ISO is just a podman run against it with the working directory bind-mounted in and the definition file named on the command line. The wrapper script exists mostly to fail fast and clearly if podman isn't there, rather than a minute into a container pull:

#!/bin/bash
set -euo pipefail

EIB_IMAGE="registry.suse.com/edge/3.4/edge-image-builder:1.3.1"
DEFINITION_FILE="edge-cluster.yaml"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

echo "==> Edge Image Builder"
echo "    Config : ${SCRIPT_DIR}/${DEFINITION_FILE}"
echo "    Image  : ${EIB_IMAGE}"
echo ""

if ! command -v podman &>/dev/null; then
  echo "ERROR: podman not found" >&2
  exit 1
fi

podman run --rm -it \
  --privileged \
  -v "${SCRIPT_DIR}":/eib \
  "${EIB_IMAGE}" \
  build --definition-file "${DEFINITION_FILE}"

echo ""
echo "==> Build complete. Output ISO: ${SCRIPT_DIR}/$(grep outputImageName ${SCRIPT_DIR}/${DEFINITION_FILE} | awk '{print $2}')"

--privileged is there because EIB needs to loop-mount and manipulate the base ISO and the raw disk image it builds on top of it — ordinary container capabilities aren't enough for that. The last line just greps the definition file for whatever outputImageName is currently set to, so the script never has to hardcode the filename it just produced.

The Installer That Wouldn't Say Anything

First boot in a KVM test VM produced nothing on the console — not an error, just silence. The kernel args inherited from the observability definition included quiet, which was doing exactly what it says and suppressing all boot output, including whatever the kiwi imaging step was doing to the disk underneath. Getting a usable picture took a full rework of kernelArgs:

kernelArgs:
  - net.ifnames=1              # predictable interface naming (kept from the original)
  - console=ttyS0,115200n8     # route kernel output to the serial console
  - nomodeset                  # no framebuffer, keep output on the text console
  - textmode=1                 # force the SUSE installer itself into text mode
  - rd.kiwi.debug               # log kiwi's disk-imaging phase in detail

quiet was simply deleted rather than replaced — there was no reason to keep suppressing output on a definition that existed purely to be watched fail and be fixed. console=ttyS0,115200n8 matches the serial console virt-install --autoconsole text attaches to; without it the kernel still writes to the (invisible, over a serial link) framebuffer console by default. nomodeset stops the kernel from switching to a GPU-driven framebuffer mode partway through boot, which would otherwise blank the serial output the moment a driver loaded. textmode=1 is SUSE's own installer flag forcing its text-mode UI instead of the graphical Cockpit-based installer, which has no business trying to render over a serial line. rd.kiwi.debug is the one that actually mattered most: it turns on verbose logging for kiwi, the tool that does the actual partitioning and image-writing work during a self-install, and that's where the real failures were hiding.

The systemd units disabled in the definition are also worth calling out, since none of them are obviously related to a test image at first glance: rebootmgr, transactional-update.timer, transactional-update-cleanup.timer, fstrim, and time-sync.target. SUSE Linux Micro is a transactional, immutable-root OS by design — it periodically checks for updates, applies them transactionally, and reboots itself to activate them. That's exactly the kind of background behavior you don't want on a box you're actively trying to get a clean, repeatable first boot out of; an unplanned reboot mid-debugging session looks identical to a crash until you've lost twenty minutes figuring out it wasn't one.

None of this changes what the image does — it only makes visible what it was already doing, which is the whole point when the alternative is guessing why an install silently stalled.

NMState Doesn't Know It's a VM

The network config for the first node was written against real hardware and named the interface em1 — a biosdevname-style name. KVM's virtio NICs don't get biosdevname names; they come up as predictable enp* names instead, so em1 in the NMState YAML matched nothing and the interface never came up. Fixing it meant renaming the interface to enp1s0 and swapping in the MAC address of the VM's virtual NIC for the duration of testing. Once the image behaved in the VM, the same file flipped once more to match the real physical NIC's enumerated name and MAC on the actual lenny1 box, which is the version live today — static IPv4 address, default route, and DNS resolvers unchanged throughout:

routes:
  config:
  - destination: 0.0.0.0/0
    metric: 100
    next-hop-address: 172.16.0.1
    next-hop-interface: enp0s31f6
    table-id: 254
dns-resolver:
  config:
    server:
    - 172.16.0.1
    - 172.16.0.3
    - 1.1.1.1
interfaces:
- name: enp0s31f6
  type: ethernet
  state: up
  mac-address: 6c:4b:90:66:32:bb
  ipv4:
    enabled: true
    address:
    - ip: 172.16.0.11
      prefix-length: 24
    dhcp: false
  ipv6:
    enabled: true

Two Real Bugs, Baked Permanently into the Image

EIB lets you drop custom scripts into the image to run at first boot, which is where two SUSE-specific workarounds live. Both were things I only discovered by watching a first boot go wrong, then wrote once and never had to think about again.

The root partition that wouldn't grow

SUSE Linux Micro images ship with a root filesystem sized for the base image, expecting to grow to fill whatever disk they're installed onto. On this install path, that growth didn't happen automatically — a known issue tracked as SUSE bugzilla #1217430. The workaround resolves the root device back to its parent disk and partition number (so it works whether root ends up on /dev/sda3 or /dev/nvme0n1p3), then runs growpart to extend the partition and systemd-growfs to extend the filesystem on top of it:

#!/bin/bash
# Bugzilla - https://bugzilla.suse.com/show_bug.cgi?id=1217430
growfs() {
  mnt="$1"
  dev="$(findmnt --fstab --target ${mnt} --evaluate --real --output SOURCE --noheadings)"
  # /dev/sda3 -> /dev/sda, /dev/nvme0n1p3 -> /dev/nvme0n1
  parent_dev="/dev/$(lsblk --nodeps -rno PKNAME "${dev}")"
  # Last number in the device name: /dev/nvme0n1p42 -> 42
  partnum="$(echo "${dev}" | sed 's/^.*[^0-9]\([0-9]\+\)$/\1/')"
  ret=0
  growpart "$parent_dev" "$partnum" || ret=$?
  [ $ret -eq 0 ] || [ $ret -eq 1 ] || exit 1
  /usr/lib/systemd/systemd-growfs "$mnt"
}
growfs /

growpart returning exit code 1 (meaning "nothing to do, already the right size") is treated as success rather than failure — on a re-run, or on a disk where the partition table already matches the disk size, that's the expected outcome, not an error.

Calico's CNI config surviving reboots it shouldn't

The second script is much smaller, and mounts a tmpfs directly over /etc/cni at boot, persisting the mount in /etc/fstab so it survives every subsequent reboot too:

#!/bin/bash
mkdir -p /etc/cni
mount -t tmpfs -o mode=0700,size=5M tmpfs /etc/cni
echo "tmpfs /etc/cni tmpfs defaults,size=5M,mode=0700 0 0" >> /etc/fstab

The effect is that Calico's CNI configuration directory starts genuinely empty on every single boot rather than accumulating whatever config a previous boot's CNI plugin wrote there. On an immutable-root OS where most of the filesystem resets to its image state anyway, a handful of writable directories like this one are exactly where subtly stale state likes to survive across reboots undetected.

Neither script is visible anywhere in the running cluster once RKE2 is up — they're the kind of thing you only find out you need after watching a first boot fail or a CNI misbehave, then baking the fix into the image so it never has to be diagnosed again.

Building and Booting the Test VM

The whole point of stripping the definition down was to get a fast, disposable iteration loop, so the test VM was built to match: 2 vCPUs, 8 GB of memory, and a 20 GB SCSI qcow2 disk, booted with UEFI firmware (OVMF) and an emulated CRB TPM 2.0 via swtpm — SUSE Linux Micro's installer expects both, and skipping them just moves the failure later instead of avoiding it.

#!/bin/bash
set -euo pipefail

VM_NAME="eib-test-node1"
VCPUS=2
MEMORY=8192
DISK_SIZE=20
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ISO="${SCRIPT_DIR}/eib-test.iso"
DISK="/var/lib/libvirt/images/${VM_NAME}.qcow2"

NET_NAME="eib-test"
NET_BRIDGE="virbr-eib"
NET_ADDR="10.110.0.1"
NET_PREFIX="24"
VM_MAC="52:54:00:20:79:d2"

# NOTE: NMState config uses interface 'em1'. In a KVM virtio NIC with
# net.ifnames=1 the interface will be named 'enp1s0'. Update
# network/node1.shed.local.yaml before building if you need networking.

if ! command -v virt-install &>/dev/null; then
  echo "ERROR: virt-install not found" >&2
  exit 1
fi

if [ ! -f "$ISO" ]; then
  echo "ERROR: ISO not found at $ISO — run ./build.sh first" >&2
  exit 1
fi

if virsh --connect qemu:///system dominfo "$VM_NAME" &>/dev/null; then
  echo "ERROR: VM '$VM_NAME' already exists — remove it first:" >&2
  echo "  virsh destroy $VM_NAME; virsh undefine $VM_NAME --remove-all-storage" >&2
  exit 1
fi

# Create NAT network if it doesn't exist
if ! virsh --connect qemu:///system net-info "$NET_NAME" &>/dev/null; then
  echo "==> Creating libvirt network: ${NET_NAME} (${NET_ADDR}/${NET_PREFIX})"
  virsh --connect qemu:///system net-define <(cat <<EOF
<network>
  <name>${NET_NAME}</name>
  <forward mode='nat'/>
  <bridge name='${NET_BRIDGE}' stp='on' delay='0'/>
  <ip address='${NET_ADDR}' prefix='${NET_PREFIX}'/>
</network>
EOF
)
  virsh --connect qemu:///system net-autostart "$NET_NAME"
  virsh --connect qemu:///system net-start "$NET_NAME"
fi

echo "==> Creating VM: ${VM_NAME}"
echo "    ISO  : ${ISO}"
echo "    Disk : ${DISK} (${DISK_SIZE}GB)"
echo "    Net  : ${NET_NAME} (${NET_ADDR}/${NET_PREFIX})"
echo ""

virt-install \
  --connect qemu:///system \
  --name "$VM_NAME" \
  --vcpus=$VCPUS \
  --memory $MEMORY \
  --cpu host \
  --os-variant sle15sp6 \
  --virt-type kvm \
  --boot loader=/usr/share/qemu/ovmf-x86_64-4m-code.bin,loader.readonly=on,loader.secure=off,loader.type=pflash \
  --disk path="${DISK}",bus=scsi,size=${DISK_SIZE},format=qcow2 --check disk_size=off \
  --graphics vnc,listen=127.0.0.1,port=-1 \
  --serial pty --console pty,target_type=serial --rng random --tpm emulator,model=tpm-crb,version=2.0 \
  --noreboot --cdrom "$ISO" \
  --network network="${NET_NAME}",model=virtio,mac="${VM_MAC}" \
  --autoconsole text

echo ""
echo "==> Install complete. To boot the installed system:"
echo "    virsh start ${VM_NAME} && virsh console ${VM_NAME}"
echo ""
echo "==> To connect to the VNC display:"
echo "    virt-viewer ${VM_NAME}"
echo ""
echo "==> To remove VM:"
echo "    virsh destroy ${VM_NAME}; virsh undefine ${VM_NAME} --remove-all-storage"
echo ""
echo "==> To remove network:"
echo "    virsh net-destroy ${NET_NAME}; virsh net-undefine ${NET_NAME}"

The script is defensive on purpose — it checks for virt-install, checks the ISO actually exists (pointing back at build.sh if not), and refuses to run if a VM of the same name already exists rather than silently colliding with it. The libvirt network it creates is a small NAT'd 10.110.0.0/24 — deliberately unrelated to the real 172.16.0.0/24 lab subnet, which is exactly why the NMState interface config earlier had to be adjusted per-target rather than shared as-is. virsh console after boot drops straight into the same serial stream the kernel args above were fought over making visible in the first place.

From One Throwaway VM to Five Real Boxes

Once the ISO installed and booted cleanly in the disposable qcow2, that exact same file — not a rebuild, not a second definition — is what got written to each of the five physical boxes. The definition already described the real topology: RKE2 v1.33.3+rke2r1 with Calico, five nodes across three servers and two agents, with an apiVIP fronting the HA control plane. What the VM test actually exercised was whether that image's boot mechanics worked at all — the growfs and CNI scripts, the kernel args, the installer itself — not a smaller stand-in definition that then had to be scaled up separately afterwards. The cluster is called Lenny because the machines are Lenovo M910q tiny-form-factor PCs — the name is just what's left of "Lenovo" once you're naming five of them after it. The final kubernetes.nodes list is where the generic node1.shed.local from the stripped-down single-node stage disappears for good, replaced by five real hostnames with real roles:

kubernetes:
  version: v1.33.3+rke2r1
  network:
    apiVIP: 172.16.0.10
    apiHost: api.lenny.shed.local
  nodes:
    - hostname: lenny1.shed.local
      initializer: true
      type: server
    - hostname: lenny2.shed.local
      type: server
    - hostname: lenny3.shed.local
      type: server
    - hostname: lenny4.shed.local
      type: agent
    - hostname: lenny5.shed.local
      type: agent

Every box's MAC and static IP live in one plain nodes.txt, alongside the shared gateway and DNS servers, so the DHCP reservations and each node's own NMState config are guaranteed to agree with each other and with the physical hardware — no KVM MAC standing in for a real one anymore, and no risk of the network config file and the DHCP server silently disagreeing about which IP belongs to which box.

The RKE2 config that ships inside the image is deliberately minimal — Calico as the CNI, a world-readable kubeconfig for convenience on a lab cluster, SELinux left enforcing, and debug logging on while the deployment is still being shaken out:

cni:
  - calico
write-kubeconfig-mode: '0644'
selinux: true
debug: true
token: <shared cluster join token, not published here>

The only things that were ever VM-specific were the 10.110.0.0/24 libvirt network and the KVM MAC standing in on lenny1's NMState file for the duration of testing — both swapped back out for the real subnet and real hardware MAC before anything touched a physical disk. Everything else about the ISO, the same file, carried straight through. The VM answered "does this image actually install and boot" in a few minutes per iteration, against a disk that costs nothing to wipe and retry, before trusting that same answer to hardware that takes longer to reimage and is considerably less forgiving of a bad guess.

Trusting Real Disks Before Real Data Goes on Them

The one place the VM detour doesn't help at all is the disks themselves. A qcow2 file doesn't have a failure mode worth testing for; five physical drives that are about to hold an actual etcd quorum do. Before trusting any of them, each one gets a SMART extended self-test:

sktest "$DEVICE" "$TEST_TYPE"    # kicks off the self-test (short|extended)

while true; do
    REMAINING=$(skdump "$DEVICE" | grep 'Percent Self-Test Remaining' | grep -o '[0-9]*')
    STATUS=$(skdump "$DEVICE" | grep 'Self-Test Execution Status' | cut -d'[' -f2 | tr -d ']')
    echo "[$(date '+%H:%M:%S')] $STATUS — ${REMAINING}% remaining"
    [ "$REMAINING" = "0" ] && break
    sleep 30
done

skdump "$DEVICE" | grep -E \
  'Overall Status|Bad Sectors|Self-Test Execution Status|Temperature|Powered On|reallocated-sector|current-pending|offline-uncorrectable'

An extended self-test walks the entire surface of the drive rather than sampling it, so it can run for hours on a large disk — the polling loop just reports progress every 30 seconds rather than blocking silently. The three attributes at the end are the ones that actually predict failure: reallocated sectors (already-failed sectors the drive has quietly remapped), current pending sectors (suspect sectors awaiting reallocation on next write), and offline uncorrectable sectors (ones the drive gave up correcting entirely). Any non-zero count on a drive that's about to join a cluster is a reason to swap it before it joins, not after.

Completing the Definition

Getting five real, SMART-tested nodes running RKE2 was the point of everything above, but the definition wasn't actually finished there. A few real gaps got closed afterward, verified against the live cluster rather than guessed.

MetalLB and Longhorn come back

Both were part of the original six-chart definition, stripped for the single-node test image, and both are genuinely needed for real workloads — MetalLB for LoadBalancer IPs, Longhorn for persistent block storage backing everything this cluster actually runs (immich, jellyfin, nextcloud, photoprism, pdf-webapp). They're back in kubernetes.helm, pinned to the same chart versions the original definition had:

kubernetes:
  helm:
    repositories:
      - name: rancher-charts
        url: https://charts.rancher.io
    charts:
      - name: longhorn-crd
        version: 109.3.0+up1.11.1
        repositoryName: rancher-charts
        targetNamespace: longhorn-system
        createNamespace: true
        installationNamespace: kube-system
      - name: longhorn
        version: 109.3.0+up1.11.1
        repositoryName: rancher-charts
        targetNamespace: longhorn-system
        createNamespace: true
        installationNamespace: kube-system
        valuesFile: longhorn.yaml
      - name: metallb
        version: 0.15.3
        repositoryName: rancher-charts
        targetNamespace: metallb-system
        createNamespace: true
        installationNamespace: kube-system

kubernetes/helm/values/longhorn.yaml is a reconstruction, not a recovered copy — the original was deleted along with everything else in the strip-down and was never recoverable, so this uses plain small-cluster defaults rather than whatever tuning the original actually had:

defaultSettings:
  defaultDataPath: /var/lib/longhorn
persistence:
  defaultClassReplicaCount: 3
  defaultClass: true

kubernetes/manifests/metallb-config.yaml, on the other hand, isn't a guess — it matches the IPAddressPools actually running on the live cluster today, read directly off it:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: api-ip
  namespace: metallb-system
spec:
  addresses:
    - 172.16.0.10/32
  autoAssign: true
  avoidBuggyIPs: true
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: general-pool
  namespace: metallb-system
spec:
  addresses:
    - 172.16.100.10-172.16.100.200
  autoAssign: true
  avoidBuggyIPs: false

The apiVIP needed a TLS SAN, and a second disk needed mounting

Two gaps existed even after the definition already listed all five real nodes. kubernetes.network.apiVIP had come back, but not its usual partner apiHost — and without the VIP and its hostname listed in RKE2's tls-san, any client connecting through the VIP instead of a node's own IP hits a TLS certificate mismatch, a classic HA gotcha:

tls-san:
  - 172.16.0.10
  - api.lenny.shed.local

Separately, checking each physical node directly (kubectl debug node/... -- chroot /host lsblk, run against all five) turned up something the definition never declared: every lenny box actually has a second disk, /dev/nvme0n1 at roughly 932G, already mounted at /var/lib/longhorn — separate from the OS disk on /dev/sda. Someone had set that up by hand; nothing in the image provisioned it. A first-boot script now does, written defensively so a reinstall can never wipe the real data already sitting on that disk:

DISK="/dev/nvme0n1"
MOUNT_POINT="/var/lib/longhorn"

mkdir -p "$MOUNT_POINT"

if ! blkid "$DISK" &>/dev/null; then
  mkfs.ext4 -F -L longhorn "$DISK"
fi

grep -q "^${DISK} " /etc/fstab || \
  echo "${DISK} ${MOUNT_POINT} ext4 defaults 0 2" >> /etc/fstab

mountpoint -q "$MOUNT_POINT" || mount "$MOUNT_POINT"

The blkid check is the whole point of the script: it only formats the disk if it doesn't already have a filesystem, so re-running the install never touches a disk that's already holding real Longhorn volumes.

One last thing: operatingSystem.users had only ever listed root. There's a second user now, authenticated by SSH key rather than a password, in the wheel group for sudo — and sccRegistrationCode, it turns out, had never actually been removed at all; it survived the entire strip-down untouched.

What Actually Mattered

None of the individual pieces here were exotic on their own — suppressed boot output, a VM NIC naming mismatch, a couple of first-boot scripts, a set of DHCP reservations, a disk health check. What mattered was the order: strip a six-chart, five-node definition down to the smallest thing that could possibly boot, prove the imaging mechanism itself worked in a VM where failure is free, then scale it back up one real constraint at a time — real NICs, real disks, real HA topology — instead of debugging all of it, on real hardware, at once.