ZFS root on Debian

From PrgmrWiki

Converting the prgmr.com Debian 9 Stretch installation to ZFS on root: the plan is to add the ZFS kernel modules, move the existing system to a tmpfs, chroot into that tmpfs root, repartition and format the disk with ZFS filesystems, then copy the system back to the new ZFS root.

  • Pros: ZFS!
  • Cons: if anything goes wrong in the future, the prgmr.com console rescue images won't help you as they can't mount your ZFS filesystem.
  • Requirements: HVM, and you probably need at least 2GB of RAM to hold the tmpfs. Maybe less if you rm all non-essential files and replace them after?

The two references have a great description of the processes. Please read them through before starting this run-through. This description will be rather terse. If everything goes right, expect this to take about 20 minutes.

Start with a fresh Debian 9 Stretch installation from the console (option 9, then 6). Add your ssh key, then log in as root.

Add contrib and update:

# echo "deb http://ftp.debian.org/debian stretch main contrib" >> /etc/apt/sources.list
# apt update
# apt install --yes debootstrap gdisk dpkg-dev linux-headers-$(uname -r)

Install the zfs and zpool utilities:

# apt install --yes zfsutils-linux

I received these zfs-mount warnings at this point, but ignored them:

Job for zfs-mount.service failed because the control process exited with error code.
See "systemctl status zfs-mount.service" and "journalctl -xe" for details.
zfs-mount.service couldn't start.
Job for zfs-share.service failed because the control process exited with error code.
See "systemctl status zfs-share.service" and "journalctl -xe" for details.
zfs-share.service couldn't start.

Install and build the modules (about 8 minutes for me!), with a click through license warning:

# apt install --yes zfs-dkms zfs-initramfs

Load the ZFS module:

# modprobe zfs

Kill off anything using the ext4 root filesystem:

# systemctl stop anacron.service  atd.service cron.service dbus.service \
   exim4.service getty@tty1.service ntp.service rsyslog.service \
   serial-getty@hvc0.service serial-getty@ttyS0.service ssh.service \
   systemd-journald.service systemd-logind.service systemd-udevd.service \
   user@0.service

# umount -a

Create the new tmpfs root, move everything over, and pivot /:

# mkdir /tmp/tmproot
# mount -t tmpfs none /tmp/tmproot
# mkdir /tmp/tmproot/{proc,sys,dev,run,usr,var,tmp,oldroot,boot,bin,etc,home,lib,lib64,opt,root,sbin,srv}
# cp -ax /{bin,boot,etc,home,lib,lib64,opt,root,sbin,srv,usr,var} /tmp/tmproot/
# mount --make-rprivate /
# pivot_root /tmp/tmproot /tmp/tmproot/oldroot
# for i in dev proc sys run; do mount --move /oldroot/$i /$i; done

Restart ssh to move it to the new /:

# systemctl restart sshd
# systemctl status sshd

Test ssh by starting a new ssh connection. Once you're in, log out of the first session. (The old session was using the old root, the new session will be using the tmpfs root.)

See what's still using the old / filesystem:

# fuser -vm /oldroot

And kill it off:

# killall exim4 rsyslogd systemd-journald

Check again, and kill -9 anything left (except PID 1):

# fuser -vm /oldroot
# kill -9 23575        # (your PID will be different)

Restart systemd on the tmpfs and unmount /:

# systemctl daemon-reexec
# umount /oldroot

Wipe the disk and re-partition it:

# dd if=/dev/zero of=/dev/xvda bs=1M
# sgdisk --zap-all /dev/xvda
# sgdisk -a1 -n2:34:2047  -t2:EF02 /dev/xvda
# sgdisk     -n1:0:0      -t1:BF01 /dev/xvda

Create the ZFS pool:

# zpool create -f -o ashift=12 \
   -O atime=off -O canmount=off -O compression=lz4 -O normalization=formD \
   -O xattr=sa -O mountpoint=/ -R /mnt \
   rpool /dev/xvda1

And the filesystems:

# zfs create -o canmount=off -o mountpoint=none rpool/ROOT
# zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/debian
# zfs mount rpool/ROOT/debian
# zpool set bootfs=rpool/ROOT/debian rpool
# zfs create                 -o setuid=off              rpool/home
# zfs create -o mountpoint=/root                        rpool/home/root
# zfs create -o canmount=off -o setuid=off  -o exec=off rpool/var
# zfs create -o com.sun:auto-snapshot=false             rpool/var/cache
# zfs create                                            rpool/var/log
# zfs create                                            rpool/var/spool
# zfs create -o com.sun:auto-snapshot=false -o exec=on  rpool/var/tmp
# zfs create                                            rpool/srv
# zfs create                                            rpool/var/mail
# zfs create -o com.sun:auto-snapshot=false \
   -o setuid=off                              rpool/tmp
# chmod 1777 /mnt/tmp

Copy the files to the new ZFS filesystems:

# cp -ax /{bin,boot,etc,home,lib,lib64,opt,root,sbin,srv,usr,var} /mnt

Prepare to pivot back:

# mkdir /mnt/dev
# mkdir /mnt/proc
# mkdir /mnt/sys
# mount --rbind /dev  /mnt/dev
# mount --rbind /proc /mnt/proc
# mount --rbind /sys  /mnt/sys
# chroot /mnt /bin/bash --login

Fix some mount points:

# zfs set mountpoint=legacy rpool/var/log
# zfs set mountpoint=legacy rpool/var/tmp
# cat > /etc/fstab << EOF
# rpool/var/log /var/log zfs noatime,nodev,noexec,nosuid 0 0
# rpool/var/tmp /var/tmp zfs noatime,nodev,nosuid 0 0
# EOF

# zfs set mountpoint=legacy rpool/tmp
# cat >> /etc/fstab << EOF
# rpool/tmp /tmp zfs noatime,nodev,nosuid 0 0
# EOF

Install GRUB (refer to the ZFS reference for important notes!) and rebuild the initrd:

# grub-probe /
# update-initramfs -u -k all
# update-grub
# grub-install /dev/xvda

Verify the GRUB ZFS module is present:

# ls /boot/grub/*/zfs.mod

Make a snapshot of the bare system:

# zfs snapshot rpool/ROOT/debian@install

Recreate the symbolic links:

# ln -fs /boot/vmlinuz-4.9.0-7-amd64 /vmlinuz
# ln -fs /boot/initrd.img-4.9.0-7-amd64 /initrd.img

Should be done! Power off, restart, and see if it boots:

# exit
# poweroff

Note: after booting into the new installation, apt-get full-upgrade installed a new kernel but failed to rebuild the ZFS modules:

Error! Your kernel headers for kernel 4.9.0-8-amd64 cannot be found.
Please install the linux-headers-4.9.0-8-amd64 package,
or use the --kernelsourcedir option to tell DKMS where it's located

But this seemed to fix it for me:

# apt-get -y install linux-headers-amd64

or

# apt-get -y install linux-headers-4.9.0-8-amd64