Hetzner and rescueing a freebsd root server
One of my machines is a Hetzner root server where I run FreeBSD
current (updated once a month) that servers as a virtualization host for couple VMs (that runn FreeBSD current
themselves, updated usually once a week).
The host boots of ZFS. Initially I deployed that machine in 2017
and for many years Hetzner had some sort of FreeBSD support in their root server products. Support as in: one could
boot a FreeBSD install image to install the machine and there was a FreeBSD based rescue system.
Due to the nature of running current, over the years I’ve used the rescue system every now and then
(typically, once per year).
Last night I rebuilt current on the host and also upgraded the zpool to use the latest features (haven’t
done that in a while).
zpool upgrade zroot did remind me that I needed to update the bootblocks,
since some features where enabled that the installed bootblocks did not have yet:
# zpool upgrade zroot This system supports ZFS pool feature flags. Enabled the following features on 'zroot': hole_birth If you boot from pool 'zroot', don't forget to update boot code. Assuming you use GPT partitioning and ada0 is your boot disk the following command will do it: gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
This morning I rebooted into the new kernel and the machine did not come back. jeiks. Immediately I reminded myself that, even though I was told too, I did not upgrade the bootblocks. doh!
I confirmed this by requesting a remote console and resetted the server. The bootloader showed this message:
ZFS: unsupported feature: com.delphix:hole_birth ZFS: pool zroot is not supported gptzfsboot: failed to mount default pool zroot
“No worries” I thought, I will just boot into the FreeBSD rescue system and fix that…however there is no more FreeBSD rescue system.
silently dropped support for FreeBSD. Their support confirmed that in a quick mail exchange.
Confronted with that, I had to find a different way.
Thankfully the linux rescue system is very well equipped, so I thought: Why not boot some bsd rescue system
qemu-system-x86_64 and access the freebsd disks from there.
There is a reddit post that suggests to use mfsbsd (github project site). However mfsbsd is currently based on FreeBSD-13.1 (the stable version of FreeBSD) and as such does not support the ZFS featureset of FreeBSD-current. Filis, a friend of mine, reminded me then that the installer iso of FreeBSD is able to serve as a rescue image as well. Thankfully the rescue system boots with a 32 gb large memory filesystem, so enough space to downloads isos.
disc1.iso. Make sure to grab the latest available:
# wget http://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/14.0/FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso
qemu-system-x86_64 \ -cdrom FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso \ -drive format=raw,file=/dev/nvme0n1 \ -drive format=raw,file=/dev/nvme1n1 \ -nic user,hostfwd=tcp::2222-:22 \ -curses \ -boot d \ -m 8G
The parameters for the harddisk drives of course depend on your hardware. This will boot the installer iso and allows you to drop in a shell.
To access the data on your zpool. You can import the zpool.
My zpool is named
# mkdir -p /tmp/mnt/zroot # zpool import -R /tmp/mnt/zroot zroot
If you need to update your bootblocks, depending on wether you have UEFI boot or legacy boot, there are a few simple commands to be issues.
For legacy boot:
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
This is for legacy boot. If you have a UEFI system:
# mount -t msdosfs -o longnames /dev/ada0p1 /mnt # cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi # umount /mnt
Beware: if you have more than one disk that you could boot of, you will have to do this for each and every disk.
Installing a fresh system @ Hetzner with FreeBSD
There is a good blogpost by Martin Matuška on FreeBSD on Hetzner dedicated servers.