Blog
Hetzner and rescueing a freebsd root server
One of my machines is a Hetzner root server where I run FreeBSD
current (updated once a month) that servers as a virtualization host for couple VMs (that runn FreeBSD current
themselves, updated usually once a week).
The host boots of ZFS. Initially I deployed that machine in 2017
and for many years Hetzner had some sort of FreeBSD support in their root server products. Support as in: one could
boot a FreeBSD install image to install the machine and there was a FreeBSD based rescue system.
Due to the nature of running current, over the years I’ve used the rescue system every now and then
(typically, once per year).
Last night I rebuilt current on the host and also upgraded the zpool to use the latest features (haven’t
done that in a while). zpool upgrade zroot did remind me that I needed to update the bootblocks,
since some features where enabled that the installed bootblocks did not have yet:
# zpool upgrade zroot
This system supports ZFS pool feature flags.
Enabled the following features on 'zroot':
hole_birth
If you boot from pool 'zroot', don't forget to update boot code.
Assuming you use GPT partitioning and ada0 is your boot disk
the following command will do it:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
This morning I rebooted into the new kernel and the machine did not come back. jeiks. Immediately I reminded myself that, even though I was told too, I did not upgrade the bootblocks. doh!
I confirmed this by requesting a remote console and resetted the server. The bootloader showed this message:
ZFS: unsupported feature: com.delphix:hole_birth
ZFS: pool zroot is not supported
gptzfsboot: failed to mount default pool zroot
“No worries” I thought, I will just boot into the FreeBSD rescue system and fix that…however there is no more FreeBSD rescue system.
Apparently Hetzner
silently dropped support for FreeBSD. Their support confirmed that in a quick mail exchange.
Confronted with that, I had to find a different way.
Thankfully the linux rescue system is very well equipped, so I thought: Why not boot some bsd rescue system
via qemu-system-x86_64 and access the freebsd disks from there.
There is a reddit post that suggests to use mfsbsd (github project site). However mfsbsd is currently based on FreeBSD-13.1 (the stable version of FreeBSD) and as such does not support the ZFS featureset of FreeBSD-current. Filis, a friend of mine, reminded me then that the installer iso of FreeBSD is able to serve as a rescue image as well. Thankfully the rescue system boots with a 32 gb large memory filesystem, so enough space to downloads isos.
Download the disc1.iso. Make sure to grab the latest available:
# wget http://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/14.0/FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso
Then:
qemu-system-x86_64 \
-cdrom FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso \
-drive format=raw,file=/dev/nvme0n1 \
-drive format=raw,file=/dev/nvme1n1 \
-nic user,hostfwd=tcp::2222-:22 \
-curses \
-boot d \
-m 8G
The parameters for the harddisk drives of course depend on your hardware. This will boot the installer iso and allows you to drop in a shell.
To access the data on your zpool. You can import the zpool.
My zpool is named zroot:
# mkdir -p /tmp/mnt/zroot
# zpool import -R /tmp/mnt/zroot zroot
If you need to update your bootblocks, depending on wether you have UEFI boot or legacy boot, there are a few simple commands to be issues.
For legacy boot:
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
This is for legacy boot. If you have a UEFI system:
# mount -t msdosfs -o longnames /dev/ada0p1 /mnt
# cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi
# umount /mnt
Beware: if you have more than one disk that you could boot of, you will have to do this for each and every disk.
Further Resources
Installing a fresh system @ Hetzner with FreeBSD
There is a good blogpost by Martin Matuška on FreeBSD on Hetzner dedicated servers.