Remote root disk encryption

This document contains notes about setting up remote root disk encryption on a Linux server. My experiences are based on Ubuntu 12.04, 14.04, 16.04 and 18.04 but I’m guessing most of these things will work very similar on Debian and its derivatives. In fact some of these notes apply to all Linux distributions.

We will configure the system to enable network connectivity in the initramfs so that we can connect using SSH and inject the root disk encryption pass phrase into the boot process. This makes it possible to set this up for headless servers.

Contents

Set up root disk encryption

It’s much easier to start an Ubuntu installation with root disk encryption than to set this up later [1] because the installer takes care of a lot of the details involved. Of course doing this on a headless server assumes you have access to something like a VNC connection.

[1]It is possible to add root disk encryption to an existing system, but it can be a bit tricky to get all the details right and not end up with a system that won’t boot :-).

Enable SSH in the initramfs

Dropbear is a lightweight SSH server and BusyBox is a lightweight collection of command line utilities. We’ll need both of these in our initramfs. They can be installed on Debian/Ubuntu using the following command:

$ sudo apt-get install busybox dropbear

This should [2] automatically configure Dropbear to start in the initramfs using the shell script /usr/share/initramfs-tools/hooks/dropbear. Depending on the version of the dropbear package this may or may not generate an SSH public/private key pair during installation:

  • If a key pair is generated then the private key file will be located at /etc/initramfs-tools/root/.ssh/id_rsa. You can download this file and remove it from the server, because it won’t be needed there.
  • If a key pair isn’t generated, or if you prefer to use your own key pair, you can install your own public key into the initramfs (see the next section).
[2]The script /usr/share/initramfs-tools/hooks/dropbear should automatically include Dropbear in the initramfs image when it detects root disk encryption. If this doesn’t work you can force Dropbear to be included by adding (or uncommenting) the line DROPBEAR=y in /usr/share/initramfs-tools/conf-hooks.d/dropbear.

Install your own SSH key

You can install your own public SSH key using the following steps:

On Ubuntu >= 18.04

  1. Open the file /etc/dropbear-initramfs/authorized_keys in a text editor of your choosing to add your public key:

    $ sudo vim /etc/dropbear-initramfs/authorized_keys
    
  2. Update your initramfs images to include the key you added:

    $ sudo update-initramfs -uk all
    

On Ubuntu < 18.04

  1. First make sure the /etc/initramfs-tools/root/.ssh directory exists:

    $ sudo mkdir -p /etc/initramfs-tools/root/.ssh
    
  2. Open the file /etc/initramfs-tools/root/.ssh/authorized_keys in a text editor of your choosing to add your public key:

    $ sudo vim /etc/initramfs-tools/root/.ssh/authorized_keys
    
  3. Update your initramfs images to include the key you added:

    $ sudo update-initramfs -uk all
    

Enable networking in the initramfs

The network interfaces of Debian/Ubuntu servers are usually configured in /etc/network/interfaces which is not available in initramfs images. What we can do instead is to provide a suitable (static) network configuration directly to the kernel via a kernel parameter. On Debian/Ubuntu this is done by changing the GRUB_CMDLINE_LINUX variable defined in the configuration file /etc/default/grub. Here’s an example based on my VPS:

GRUB_CMDLINE_LINUX="ip=149.210.193.173::149.210.193.1:255.255.255.0::eth0:none"

As you can see this compresses an entire static network interface configuration into a single ip=… kernel parameter, with fields separated by colons. The kernel documentation contains details on the ‘ip’ kernel parameter, but for your convenience here is an overview of the supported fields:

ip=<client-ip>:<server-ip>:<gw-ip>:<netmask>:<hostname>:<device>:<autoconf>:<dns0-ip>:<dns1-ip>

To elaborate even further, here is how the values in the example above map to the named fields:

  • client-ip is 149.210.193.173
  • server-ip is empty (we’re not using an NFS root)
  • gw-ip is the gateway IP, in this case 149.210.193.1 (if you hadn’t already guessed, my VPS is hosted at TransIP 😉)
  • netmask is 255.255.255.0
  • hostname is empty (we’re not using DHCP so there’s no point in configuring a DHCP client id)
  • device is eth0
  • autoconf is none (to disable autoconfiguration)

Make sure to run the following command after editing /etc/default/grub:

$ sudo update-grub

Consistent device naming

The example above uses the device name eth0 however with the introduction of consistent network device naming the eth* names were retired. If you need or want them back you can add the kernel parameters biosdevname=0 and net.ifnames=0 to the $GRUB_CMDLINE_LINUX variable.

On a Raspberry Pi

In June 2018 I got Ubuntu 16.04 with root disk encryption running on a Raspberry Pi. Because the boot process of a Raspberry Pi is very different from a regular computer, the way you configure it is also different. In this case I needed to add the ip=… kernel parameter to the file /boot/firmware/cmdline.txt.

Slow boot issues

When the ip= kernel parameter and /etc/network/interfaces both define a static network interface configuration, you may encounter slow boot issues. If you were to look at the messages emitted by the boot process you would most likely see a message along the lines of:

Waiting for network configuration..

This can slow down the boot process by two or three minutes, making you doubt whether a server is going to come back online! Fortunately there’s an easy way to avoid this problem. Open /etc/network/interfaces in your favorite text editor and add the line pre-up ip addr flush dev eth0, similar to this:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address 149.210.193.173
  netmask 255.255.255.0
  gateway 149.210.193.1
  pre-up ip addr flush dev eth0

I originally found this trick on the Ubuntu Forums in October 2014 when I created my first headless server with root disk encryption based on Ubuntu 12.04 and I still need the workaround at the time of writing, on that same server, which has since been upgraded to 14.04 and then to 16.04.

External references

  • The cryptsetup package on Debian/Ubuntu contains notes on how to setup remote unlocking in /usr/share/doc/cryptsetup/README.remote.gz, this is how I initially got started back in 2014.
  • The StackExchange question SSH to decrypt encrypted LVM during headless server boot? received some interesting answers including a honorable mention of Mandos [3].
[3]You would not believe how much time I’ve invested in getting Mandos to unlock my servers unattended, I even went so far as to (cross) compile the “latest & greatest” versions for multiple CPU architectures in a desperate attempt to get it to work. I never did.