One of my machines is a Hetzner root server where I run FreeBSD current (updated once a month) that servers as a virtualization host for couple VMs (that runn FreeBSD current themselves, updated usually once a week). The host boots of ZFS. Initially I deployed that machine in 2017 and for many years Hetzner had some sort of FreeBSD support in their root server products. Support as in: one could boot a FreeBSD install image to install the machine and there was a FreeBSD based rescue system. Due to the nature of running current, over the years I’ve used the rescue system every now and then (typically, once per year). Last night I rebuilt current on the host and also upgraded the zpool to use the latest features (haven’t done that in a while). zpool upgrade zroot did remind me that I needed to update the bootblocks, since some features where enabled that the installed bootblocks did not have yet:

# zpool upgrade zroot
This system supports ZFS pool feature flags.

Enabled the following features on 'zroot':

If you boot from pool 'zroot', don't forget to update boot code.
Assuming you use GPT partitioning and ada0 is your boot disk
the following command will do it:

    gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

This morning I rebooted into the new kernel and the machine did not come back. jeiks. Immediately I reminded myself that, even though I was told too, I did not upgrade the bootblocks. doh!

I confirmed this by requesting a remote console and resetted the server. The bootloader showed this message:

ZFS: unsupported feature: com.delphix:hole_birth
ZFS: pool zroot is not supported
gptzfsboot: failed to mount default pool zroot

“No worries” I thought, I will just boot into the FreeBSD rescue system and fix that…however there is no more FreeBSD rescue system.

Apparently Hetzner silently dropped support for FreeBSD. Their support confirmed that in a quick mail exchange. Confronted with that, I had to find a different way. Thankfully the linux rescue system is very well equipped, so I thought: Why not boot some bsd rescue system via qemu-system-x86_64 and access the freebsd disks from there.

There is a reddit post that suggests to use mfsbsd (github project site). However mfsbsd is currently based on FreeBSD-13.1 (the stable version of FreeBSD) and as such does not support the ZFS featureset of FreeBSD-current. Filis, a friend of mine, reminded me then that the installer iso of FreeBSD is able to serve as a rescue image as well. Thankfully the rescue system boots with a 32 gb large memory filesystem, so enough space to downloads isos.

Download the disc1.iso. Make sure to grab the latest available:

# wget


qemu-system-x86_64 \ 
	-cdrom FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso \ 
	-drive format=raw,file=/dev/nvme0n1 \ 
	-drive format=raw,file=/dev/nvme1n1 \ 
	-nic user,hostfwd=tcp::2222-:22 \ 
	-curses \ 
	-boot d \ 
	-m 8G

The parameters for the harddisk drives of course depend on your hardware. This will boot the installer iso and allows you to drop in a shell.

To access the data on your zpool. You can import the zpool. My zpool is named zroot:

# mkdir -p /tmp/mnt/zroot
# zpool import -R /tmp/mnt/zroot zroot

If you need to update your bootblocks, depending on wether you have UEFI boot or legacy boot, there are a few simple commands to be issues.

For legacy boot:

# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

This is for legacy boot. If you have a UEFI system:

# mount -t msdosfs -o longnames /dev/ada0p1 /mnt
# cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi
# umount /mnt

Beware: if you have more than one disk that you could boot of, you will have to do this for each and every disk.

Further Resources

Blogpost by Staf Wagemakers

RootOnZFS Wiki Page @ FreeBSD

Installing a fresh system @ Hetzner with FreeBSD

There is a good blogpost by Martin Matuška on FreeBSD on Hetzner dedicated servers.