debug-node

Debug Hetzner Node via Rescue Console

Overview

When a Hetzner Cloud server is unreachable (SSH hangs, provisioning stuck, cloud-init failure), this skill uses Hetzner's rescue mode to mount the node's filesystem and inspect everything from the outside — no working SSH required.

Usage

/debug-node

When invoked, ask for:

1. The server name or IP (can be found from hcloud server list)
2. What symptom they're seeing (SSH timeout, provisioning hang, etc.)

Prerequisites

• hcloud CLI installed and configured with a valid token
• The server must exist in Hetzner Cloud

hcloud server list

Leap Micro Filesystem Model

Leap Micro uses a transactional-update system on btrfs. This is the mental model for everything below.

Layer	Writable?	Persists reboot?	Persists Hetzner snapshot?
/usr (snapshot)	No (read-only)	Yes	Yes
/etc via transactional-update shell	Yes (new snapshot)	Yes (after reboot)	Yes
/etc via direct edit on running system	Yes (current snapshot overlay)	Yes, unless a pending snapshot changed the same path	Current-snapshot content only
/var (separate subvolume)	Yes	Yes	Yes

Rule: Ordinary /etc edits persist across a normal reboot. However, after transactional-update creates a pending snapshot, concurrent edits to the current snapshot's /etc are not merged when the pending snapshot changed the same path. Put image-building and package-coupled /etc changes inside transactional-update --continue shell, or reboot into the pending snapshot before editing.

Packer build phases:

1. Rescue mode: Write qcow2 to disk, reboot
2. install_packages: Inside transactional-update — changes persist
3. clean_up: Volatile overlay — /etc changes are lost in the Hetzner snapshot

Step 1: Identify the Server

hcloud server list -o columns=id,name,status,ipv4 | grep <pattern>

Step 2: Enable Rescue Mode & Reboot

hcloud server enable-rescue <SERVER_ID> --type linux64
hcloud server reboot <SERVER_ID>
sleep 30

Save the rescue root password from the output (usually key auth works, but just in case).

Step 3: SSH into Rescue

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@<SERVER_IP>

Step 4: Mount the Filesystem

Leap Micro / MicroOS (btrfs)

# Mount btrfs top-level
mount -o subvolid=5 /dev/sda3 /mnt

# List snapshots — highest number is active
ls /mnt/@/.snapshots/

Layout:

/mnt/@/.snapshots/N/snapshot/    latest active snapshot — /etc lives here
/mnt/@/root/                     /root home (/root/.ssh/authorized_keys)
/mnt/@/var/                      /var (logs, cloud-init state, journal)

Key: /etc is inside the snapshot. /var and /root are separate subvolumes at @/var and @/root.

Unsupported/custom ext4 images

mount /dev/sda1 /mnt

Step 5: Diagnostic Checklist

Set this once and use throughout:

SNAP=/mnt/@/.snapshots/N/snapshot   # replace N with highest snapshot number

5a. Cloud-Init Status

Start here — most provisioning failures trace back to cloud-init.

cat /mnt/@/var/lib/cloud/data/result.json
cat /mnt/@/var/lib/cloud/data/status.json
cat /mnt/@/var/lib/cloud/instance/datasource

# What Terraform actually sent
cat /mnt/@/var/lib/cloud/instance/user-data.txt
zcat /mnt/@/var/lib/cloud/instance/user-data.txt.i 2>/dev/null

# Logs
tail -100 /mnt/@/var/log/cloud-init.log
tail -100 /mnt/@/var/log/cloud-init-output.log

Expected: DataSourceHetzner, no errors. Watch for: Skipping modules — means cloud-init already ran for this instance-id.

Cloud-init facts on Hetzner + Leap Micro:

• Datasource: DataSourceHetzner (metadata API)
• Terraform's cloudinit_config → gzip+base64 multipart MIME → user_data
• disable_root: false prevents cloud-init from disabling root but does NOT unlock a locked account
• ssh_authorized_keys writes keys to /root/.ssh/authorized_keys

5b. SSH Keys

cat /mnt/@/root/.ssh/authorized_keys

Compare with your local pubkey. If missing, cloud-init failed to inject — check 5a logs.

5c. SSHD Configuration

Config loading order (first match wins):

1. /etc/ssh/sshd_config.d/40-kube-hetzner-authorized-keys-command.conf
2. /etc/ssh/sshd_config.d/50-cloud-init.conf
3. /etc/ssh/sshd_config.d/kube-hetzner.conf           (MaxAuthTries 2)
4. /usr/etc/ssh/sshd_config.d/40-suse-crypto-policies.conf
5. /usr/etc/ssh/sshd_config                            (UsePAM yes)

ls $SNAP/etc/ssh/sshd_config.d/
cat $SNAP/etc/ssh/sshd_config.d/*.conf
cat $SNAP/usr/etc/ssh/sshd_config
ls -la $SNAP/etc/ssh/ssh_host_*

5d. Account Status

grep '^root:' $SNAP/etc/shadow

Pattern	Meaning	SSH pubkey works?
root:*:...	Unlocked, no password	Yes
root:!*:... or root:!:...	Locked	No (PAM rejects with UsePAM yes)

This is fixed in packer (usermod -p '*' root inside transactional-update) with a cloud-init bootcmd safety net. If you see a locked account on a fresh node, the packer snapshot needs rebuilding.

5e. Journal Logs

journalctl -D /mnt/@/var/log/journal/ -u sshd --no-pager | tail -50
journalctl -D /mnt/@/var/log/journal/ -u k3s --no-pager | tail -30
journalctl -D /mnt/@/var/log/journal/ -u rke2-server --no-pager | tail -30
journalctl -D /mnt/@/var/log/journal/ -u rke2-agent --no-pager | tail -30

5f. Network

ls $SNAP/etc/NetworkManager/system-connections/
cat $SNAP/etc/NetworkManager/system-connections/*.nmconnection 2>/dev/null

5g. Kubernetes

cat $SNAP/etc/rancher/k3s/config.yaml 2>/dev/null
cat $SNAP/etc/rancher/rke2/config.yaml 2>/dev/null
cat /mnt/@/var/lib/rancher/k3s/server/token 2>/dev/null

5h. SELinux

cat $SNAP/etc/selinux/config
chroot $SNAP rpm -qa | grep -iE 'selinux|k3s|rke2'
grep -i 'avc:.*denied' /mnt/@/var/log/audit/audit.log | tail -50
journalctl -D /mnt/@/var/log/journal/ --no-pager | grep -i 'avc:.*denied' | tail -50

For workload denials, follow docs/selinux.md: collect the AVC lines, workload name/version, k3s/RKE2 distribution, OS image, and udica result before proposing upstream policy changes. Do not globally disable SELinux as the first answer; use per-pool selinux = false only as the last resort for a workload or nodepool that cannot run under policy.

Step 6: Apply a Fix

Edit files in the active snapshot ($SNAP), not in @/ base.

# Unlock root account (if locked)
sed -i 's/^root:!*/root:*/' $SNAP/etc/shadow

# Fix authorized_keys
mkdir -p /mnt/@/root/.ssh
echo "ssh-ed25519 AAAA..." > /mnt/@/root/.ssh/authorized_keys
chmod 700 /mnt/@/root/.ssh && chmod 600 /mnt/@/root/.ssh/authorized_keys

# Regenerate host keys
mount --bind /proc $SNAP/proc && mount --bind /sys $SNAP/sys && mount --bind /dev $SNAP/dev
chroot $SNAP ssh-keygen -A
umount $SNAP/proc $SNAP/sys $SNAP/dev

Note: Rescue-mode edits are immediate fixes. The proper long-term fix belongs in the packer template or cloud-init.

Step 7: Reboot to Normal

umount /mnt 2>/dev/null
exit

hcloud server disable-rescue <SERVER_ID>
hcloud server reboot <SERVER_ID>
sleep 60
ssh -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 root@<SERVER_IP> 'echo ok'

Common Diagnoses

Symptom	Likely Cause	Check	Fix
SSH timeout	Firewall or network	Hetzner firewall rules	Open port 22
SSH "Connection refused"	sshd not running	Journal logs	Fix sshd config syntax
SSH key rejected	Keys not injected	authorized_keys empty	Check cloud-init logs
SSH "Too many auth failures"	Agent offers too many keys	MaxAuthTries 2	Use -o IdentitiesOnly=yes
SSH "unable to authenticate"	Root locked, or key mismatch	/etc/shadow, authorized_keys	Rebuild packer snapshot
Provisioner hangs "Still creating"	SSH can't connect	All above	Fix underlying SSH issue
Cloud-init skips modules	Already ran for instance-id	cloud-init.log	Clean /var/lib/cloud/instance
k3s/rke2 not starting	Config or SELinux	Journal + audit.log	Fix config or policy
Workload denied by SELinux	Missing workload policy	AVC lines in audit/journal	Follow docs/selinux.md; try udica before disabling a pool
Network/subnet destroy hangs	Autoscaler-created server outside Terraform state	hcloud server list for cluster-name or kh-ci-* leftovers	Delete only after control plane is dead, or scale autoscaler min_nodes = 0 first
/etc change vanished after transactional reboot	Current and pending snapshots changed the same path	Inspect transactional-update status and /var/lib/overlay	Reapply in the active snapshot or move image-build changes into the transaction

Debugging SSH Manually

# Verbose with specific key (avoids agent key spray hitting MaxAuthTries 2)
ssh -vvv -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 root@<SERVER_IP>

# In -vvv output:
# "Offering public key: ..."              → key was offered
# "Server accepts key: ..."               → success path
# "Authentications that can continue: ..."  → key was REJECTED
# "Too many authentication failures"       → agent sent too many keys

Pro Tips

1. Mount with subvolid=5 — gets the real btrfs root, navigate to @/.snapshots/N/snapshot/
2. Highest snapshot = active — that's where /etc lives
3. /var is separate — logs and cloud-init are at /mnt/@/var/, not inside the snapshot
4. Journal without a running system — journalctl -D /path/to/journal/
5. Use -o IdentitiesOnly=yes — kube-hetzner sets MaxAuthTries 2
6. Volatile overlay trap — if rescue shows different content than the live system did, it was running on a volatile overlay that never got committed
7. After fixing packer, rebuild snapshots — verify build logs show changes inside the transactional-update output
8. Rescue mode is non-destructive — you're just reading/writing files on the disk
9. Destroy with the wrapper — for full-cluster teardown, run scripts/destroy.sh from the Terraform root; it retries only the known ingress-LB detach race and prints a read-only orphan report. Use scripts/cleanup.sh only as the forceful fallback; it treats the token's entire HCloud project as cluster-dedicated, so review the dry run first.