Tag: zfs

Categories
Linux

GRUB: unknown filesystem on ZFS-based Proxmox

Error: unknown filesystem.
grub rescue>

This was the nice error that greeted me when I tried to (re)boot my Proxmox server.
What unknown filesystem? The server was still supposed to boot from the same ZFS disk it always booted from.

TL;DR it was due to the large_dnode getting enabled on a dataset I had manually created. GRUB does not support that feature. Create a new dataset with that feature explicitly disabled, move the data to it, delete the original and rename the new one. Done.

Some background

Being space-constrained in my apartment, my Proxmox server is actually my everything server. It is my virtualization server, my router (it runs pfSense in a VM) and my NAS (it runs Samba in an LXC container).

For NAS duties, I decided a while back to create a dedicated dataset rpool/nas, which I then mounted in the Samba container. Without much thinking, I just ran a simple:

zfs create rpool/nas

Due to ZFS defaults, it had the feature dnodesize set to auto. It was never an issue, until the other day.

What happened

Some file must have triggered a non-legacy (512 bytes) dnode size in the dataset, which meant that GRUB could no longer read the drive.

I came to that conclusion by installing Proxmox in a VM on my Mac (with an ext4 boot drive, to avoid having to rename the pool), and attaching the server’s SSD through a SATA-USB adapter.

I changed the mountpoint of my server’s boot partition from / to /recovery, mounted /dev /sys and /proc in there and then chrooted into /recovery:

zfs set mountpoint=/recovery rpool/ROOT/pve-1
mount --rbind /dev /recovery/dev
mount --rbind /sys /recovery/sys
mount -t proc /proc /recovery/proc
chroot /recovery

I ran grub-probe -vvvv / to get some insight on why GRUB was failing and one line was interesting:

grub-core / fs / zfs / zfs.c: 2112: zap: name = org.zfsonlinux: large_dnode, value = 1, cd = 0

I read a bit online and I found out about the dnodesize thing. I usually like to link back to all the places that were useful to find a solution to any problem I blog about, but this time it was just too difficult to keep track of everything, except for this discussion on the german Proxmox forums.

The solution

I ran zfs get -r dnodesize rpool to get a sense of what were the different dnodesize values for all the datasets in the pool. They were all set to legacy, except for my rpool/nas dataset.
So I made a new dataset for my NAS data with dnodesize explicitly set to legacy, then I rsync’d everything from the old dataset into the new one, destroyed the old dataset and renamed the new one.

zfs create -o dnodesize=legacy rpool/nas2
rsync -av --progress /rpool/nas/ /rpool/nas2/
zfs destroy rpool/nas
zfs rename rpool/nas2 rpool/nas

Last step: moving back the mountpoint for `rpool/ROOT/pve-1`:

zfs set mountpoint=/rpool/ROOT/pve-1

And then I moved the SSD back into my server.

That’s it. It just took an afternoon of cursing.

Categories
Linux

Moving Proxmox ZFS boot drive to a new disk

When I assembled my little Proxmox home server, mainly used for pfSense, Home Assistant, Nextcloud and a few other apps, I underestimated the amount of storage I needed. I went with a cheap 120 GB SSD, but it was pretty much always full. I then found a deal on a 960 GB Kingston A400 SSD, so I got it.

My Kettop Mi3865L6 runs Proxmox on ZFS

The thing is I didn’t really want to go through a complete reinstall of Proxmox, restore of my VMs/CTs and reinstallation of Zabbix Agent on the host itself. Thankfully, my whole drive is ZFS formatted, so I have access to the great zfs send and zfs receive commands to move stuff around.

The steps

  1. Connect the SSD with a USB-SATA adapter to a Virtual Machine running on my Mac, and install Proxmox on it. This takes care of the GRUB bootloader.
  2. On the main Proxmox install, shutdown all CTs and VMs, and take a snapshot
  3. Connect the drive to Proxmox and import the pool with a different name
  4. ZFS send/receive the snapshot (recursively) to the new pool
  5. Export the pool
  6. Shutdown Proxmox and swap the drives
  7. Power on Proxmox, fix the pool name, reboot
  8. Fix the bootloader and initial ramdisk
  9. Profit (?)

1. Proxmox install on the new SSD

So the first step is to install Proxmox on the new SSD, and the easiest thing I could think was to use a simple USB3 to SATA adapter to connect it to my Mac, and then pass it to a VM in Parallels with the Proxmox ISO mounted. I then proceded with a regular install, choosing ZFS as the SSD’s filesystem. I could have done it all on a VM on the Proxmox server, but I didn’t bother.

2. Snapshot the old drive

Then I moved to the Proxmox server, shut down every VM and every CT, and took a recursive snapshot of the main pool (rpool is the default pool name on Proxmox):

sudo zfs snapshot -r rpool@newSSD

3. Connect the new SSD to Proxmox

After shutting down the VM I used to install Proxmox on the new SSD, I moved the USB3-SATA adapter to the Proxmox server.

First I needed to import the pool with a new name (rpoolUSB), since of course rpool was already taken.

sudo zpool import -d /dev
sudo zpool import [ID-OF-THE-POOL-ON-THE-NEW-SSD] rpoolUSB

4. Clone the old SSD onto the NEW one

Having just taken the snapshot on the old drive, It was just a metter of a ZFS send/receive, with the -F to overwrite the pool. This operation left the bootloader intact, which is great.

 sudo zfs send -R rpool@newSSD | sudo zfs recv -F rpoolUSB

5. Export the new pool

sudo zpool export rpoolUSB

6. Shutdown Proxmox and swap the drives

Connect a display to your Proxmox server if you don’t have one, or connect through KVM if your server has IPMI capabilities.

7. Fix the pool name

Remember how we renamed the pool to rpoolUSB in step 3? Proxmox doesn’t like that. Or rather, it doesn’t know about that. So the boot process with fail leaving you at a Busybox shell. Just import the pool giving it the usual rpool name and exit.

zpool import -d /dev
zpool import rpoolUSB rpool
exit

8. Fix the bootloader and initial ramdisk

The boot process now works fine, but it complains about some missing things. What’s needed is a fix of the initial ramdisk and possibly of the GRUB bootloader, I did both just to be on the safe side.

sudo update-grub2
sudo update-initramfs -u -k all

9. Profit

Let me know if you actually profited from this. I think you owe me 1% of your profits 😁

Categories
FreeBSD

How to use a USB drive with FreeNAS

First of all, a disclaimer: I use this setup just as a secondary backup, I do not recommend you rely on USB drives as storage media for your FreeNAS server.

Overview

I have a few spare 2 TB drives that used to be in my homeserver, and I wanted to connect them to my FreeNAS every few weeks to store some backups on them, then disconnect them and put them in a drawer.

Preparing the USB hard drive

If your drives are not brand new, chances are you already have some data on them, so you’ll need to wipe them first. To do so, connect your drives to FreeNAS, login to the webinterface, go to the Storage tab, click on “View Disks”, select the correct one from the list, then click on the “Wipe” button on the bottom of the screen.

Everything will turn red now. That’s because you’re doing something potentially dangerous. Make sure you really select the correct drive, otherwise you might destroy all of your data.

A Quick wipe is usually enough. Select that and proceed.

Once the wipe is completed, you can turn the disk into a Volume you can use on FreeNAS, just like your main one(s). Head over to the Storage tab, click on the “Volume Manager” button. You will see your newly wiped disk there, give it a volume name, tick the “Encryption” checkbox if you’d like, then click on “Add Volume”. I’ll call mine “ColdStorage” for the purpose of this tutorial.

Do NOT select any “Volume to extend”. That would mean you’d be adding a single (USB!) hard drive striped to your main pool, and that drive would become the single point of failure for all the data in that pool.

After the disk has been initialized, you can start adding datasets, clone other ones, setup network sharing, whatever.

Disconnecting the USB drive

Once you’re done with it, you will want to disconnect your USB drive to store it somewhere. To do so, once again go to the Storage Tab, select your pool from the list (ColdStorage in this example), then click on the “Detach Volume” button. You will only see this button if you select the pool, while if you select the dataset with the same name you will get a different set of buttons.

I’m not sure that I have to point that out, but let’s do it anyway: do NOT select “Mark the disks as new (destroy data)”. Once the pool has been detached, you can disconnect the USB drive.

Reconnecting the USB drive

For the thousandth time, go back to your trusty Storage tab, then click on the “Import Volume” button. A box will appear, select wether you have an encrypted volume or not, then click OK. FreeNAS will stop a few seconds to reflect on the good it’s made in its life, then it will present you a dropdown from which you’ll be able to select your USB volume. Click OK and it will be once again available in FreeNAS.