Gentoo with EFIStub, encrypted BTRFS, swap, dracut or genkernel initramfs, open-rc.

Posted on 12 November 2021 by AJ Ianozi

Hello! It’s been a while since I wrote a post; life has been busy I’m working on a better blog management system and I finally got a desktop! Hopefully more updates to come both here and on Seldom.Travel! Stay tuned for more updates.

Also, I must apologize for the lack of wrapping on this blog entry. This blog will be 11 years old next month, and I plan on updating the theme soon. Please bear with me.

Recently, I built a Gentoo system with an efistub (no bootloader) and ZFS natively-encrypted rootfs. It was great until I ran into a bug on kernels below 5.14 where the zfs system would get currupted. OpenZFS’s answer was “Update to 5.14” (apparently Gentoo’s stable was 5.10), and if I’m updating out of whatever is stable I’m going all the way and trying 5.15… which isn’t supported yet for ZFS. Result? I scrapped the whole thing and decided to just use btrfs!

This is heavily inspired by this post (shout out to William for putting that together) except I plan on using btrfs with zlib compression instead of ext4, I plan on having a module-free kernel, I want an encrypted swap partition, and we can no longer use eudev (because they’re retiring it). I also don’t want to use lvm, it’s just another layer of complexity that I don’t need, and I’ll be showing how to use dracut as well as genkernel depending on the route you wanted to go. We’ll be extracting the resulting initramfs, and building it into the kernel.

Boot into a Linux environment

I decided to use an ubuntu live CD, but anything with the standard tools will work. I also plugged as many devices into the computer as I thought I would need, so I can take advantage of localyesconfig.

Bring up a terminal and find your HDD/SSD

This is a good way to do it:

root@ubuntu:~$ sudo fdisk -l | grep GiB
Disk /dev/loop0: 2.3 GiB, 2470006784 bytes, 4824232 sectors
Disk /dev/nvme0n1: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk /dev/sda: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk /dev/sdb: 111.79 GiB, 120034123776 bytes, 234441648 sectors
Disk /dev/sdd: 14.32 GiB, 15376000000 bytes, 30031250 sectors
root@ubuntu:~$ 

I’m using the nvme, but others are probably using sdX.

Partition out the drive

We’ll be setting the EFI partition with about 1GB of space (more than we’ll ever need), along with plenty of swap and leave the rest for the rootfs:

root@ubuntu:~# parted -a optimal /dev/nvme0n1
GNU Parted 3.4
Using /dev/nvme0n1
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit mib
(parted) mklabel gpt
(parted) mkpart primary 1 1025
(parted) name 1 "EFI System Partition"                                    
(parted) set 1 esp on                                                     
(parted) set 1 boot on                                                    
(parted) mkpart primary 1025 33280                                        
(parted) name 2 "swap"
(parted) mkpart primary 33280 100%
(parted) name 3 "rootfs"
(parted) print                                                            
Model: WD Green SN350 1TB (nvme)
Disk /dev/nvme0n1: 953870MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start     End        Size       File system  Name                  Flags
 1      1.00MiB   1025MiB    1024MiB                 EFI System Partition  boot, esp
 2      1025MiB   33280MiB   32255MiB                swap
 3      33280MiB  953869MiB  920589MiB               rootfs

(parted) quit                                                             
Information: You may need to update /etc/fstab.

root@ubuntu:~# fdisk -l | grep nvme
Disk /dev/nvme0n1: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
/dev/nvme0n1p1     2048    2099199    2097152    1G EFI System
/dev/nvme0n1p2  2099200   68157439   66058240 31.5G Linux filesystem
/dev/nvme0n1p3 68157440 1953523711 1885366272  899G Linux filesystem
root@ubuntu:~# 

Encrypt / format the root partition

I’m taking a page out of Sakaki’s handbook and choosing the serpent cipher with the whirlpool cipher. The reason being is that serpent is a technically better cipher and has actually managed to outperform AES when using xts. While I do have an intel processor that could take advantage of AES-NI, not everyone does. As for whirlpool, it has been shown to slow brute force attacks.

root@ubuntu:~# cryptsetup --cipher serpent-xts-plain64 --key-size 512 --hash whirlpool luksFormat /dev/nvme0n1p3

Since I use dvorak, I then switched my layout to US, and added an additional key, re-typing the same passphrase that I used on dvorak (in case I need to unlock this partition on a qwerty layout):

root@ubuntu:~# cryptsetup luksAddKey /dev/nvme0n1p3

What about swap? Don’t worry, we’ll get there. That’s later down the road. Let’s open, map, and format the root partition:

root@ubuntu:~# cryptsetup luksOpen /dev/nvme0n1p1 root
root@ubuntu:~# mkfs.btrfs -L "root" /dev/mapper/root
root@ubuntu:~# mkdir /mnt/gentoo
root@ubuntu:~# mount -o noatime,nodiratime,ssd,compress=zlib /dev/mapper/root /mnt/gentoo

Now that the root partition is added, don’t forget the EFI partition!

root@ubuntu:~# mkfs.vfat -F32 /dev/nvme0n1p1 
root@ubuntu:~# mkdir -p /mnt/gentoo/boot
root@ubuntu:~# mount /dev/nvme0n1p1 /mnt/gentoo/boot
root@ubuntu:~# mkdir /mnt/gentoo/boot/efi

At this point, go ahead and continue following the Gentoo HandBook through Installing the Gentoo installation files and come back here after you finish that page and extract the stage3.

Encrypt swap

We’re going to be handling swap a little differently, by encrypting it with a keyfile. First, let’s generate that file:

root@ubuntu:~# mkdir /mnt/gentoo/etc/keys
root@ubuntu:~# dd if=/dev/random of=/mnt/gentoo/etc/keys/swap.key bs=8388607 count=1
1+0 records in
1+0 records out
8388607 bytes (8.4 MB, 8.0 MiB) copied, 0.0694414 s, 121 MB/s
root@ubuntu:~# 

Now we’ll use that file to encrypt and create the swap partition. Make sure you’re using the right partition!

root@ubuntu:~# cryptsetup --cipher serpent-xts-plain64 --key-size 512 --hash whirlpool luksFormat /dev/nvme0n1p2 /mnt/gentoo/etc/keys/swap.key

WARNING!
========
This will overwrite data on /dev/nvme0n1p2 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
root@ubuntu:~# cryptsetup luksOpen -d /mnt/gentoo/etc/keys/swap.key /dev/nvme0n1p2 swap
root@ubuntu:~# mkswap /dev/mapper/swap
Setting up swapspace version 1, size = 31.5 GiB (33805037568 bytes)
root@ubuntu:~# swapon /dev/mapper/swap

Finally, get the UUID for the crypt root/swap partitions, /boot, mapper/root, and mapper/swap. We’ll need them later:

root@ubuntu:~# blkid | grep '/dev/mapper\|nvme'
/dev/mapper/root: LABEL="root" UUID="2c98ff1c-a40a-4b8d-9131-e5fbc0446091" UUID_SUB="0927d47e-97c0-491b-8654-d4c3f939d1d7" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/nvme0n1p3: UUID="47e7fa0b-2472-4015-96fb-e92ff1c21df7" TYPE="crypto_LUKS" PARTLABEL="rootfs" PARTUUID="aa3322ba-70fd-447a-a353-642a5ad9d1a9"
/dev/nvme0n1p1: UUID="3B87-3ED8" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="e26dee0c-13fe-4b9b-95f7-4001547424a9"
/dev/mapper/swap: UUID="3303e028-df54-424a-876d-1f0c9197d965" TYPE="swap"
/dev/nvme0n1p2: UUID="87833367-6fe9-4118-87b8-9accbe4a09e6" TYPE="crypto_LUKS" PARTLABEL="swap" PARTUUID="d74d55e7-f97f-4007-8d97-181d248ce233"
root@ubuntu:~# 

Now we can proceed with Installing the Gentoo base system up to the Configuring the Kernel: Manual configuration step.

Sidenote: Configuring /etc/portage/make.conf

Here’s some special stuff that I’m doing with my make.conf. First, I installed app-portage/cpuid2cpuflags, to get my supported CPU flags:

(chroot) ubuntu /usr/src/linux # emerge --quiet app-portage/cpuid2cpuflags
>>> Verifying ebuild manifests
>>> Emerging (1 of 1) app-portage/cpuid2cpuflags-11::gentoo
>>> Installing (1 of 1) app-portage/cpuid2cpuflags-11::gentoo
>>> Recording app-portage/cpuid2cpuflags in "world" favorites file...
(chroot) ubuntu /usr/src/linux # cpuid2cpuflags 
CPU_FLAGS_X86: aes avx avx2 avx512f avx512dq avx512cd avx512bw avx512vl f16c fma3 mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3
(chroot) ubuntu /usr/src/linux # 

Then I added them toto my make.conf like so:

CPU_FLAGS_X86="aes . . .ssse3"

Then, enter the command nproc to find out how many cores you have.

(chroot) ubuntu /usr/src/linux # nproc
40
(chroot) ubuntu /usr/src/linux # 

I recommend setting MAKEOPTS="-j<nproc>/2" and EMERGE_DEFAULT_OPTS to have "--jobs <nproc> --load-average <nproc+1>", so we’ll end up with in the make.conf something like:

COMMON_FLAGS="-O2 -pipe -march=native -fomit-frame-pointer"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"

PORTDIR="/var/db/repos/gentoo"
DISTDIR="/var/cache/distfiles"
PKGDIR="/var/cache/binpkgs"

# This sets the language of build output to English.
LINGUAS="en en_US"
# Please keep this setting intact when reporting bugs.
LC_MESSAGES=C

MAKEOPTS="-j20"
EMERGE_DEFAULT_OPTS="--with-bdeps y --complete-graph y --jobs 40 --load-average 41"
# This is where the real fun begins
USE="offensive"

People probably won’t like that I also do this:

# Get off my lawn
ACCEPT_LICENSE="*"

On fstab

We’ll be more heavily modifying /etc/fstab later, but for now, be sure to add the boot entry:

(chroot) ubuntu /usr/src/linux # echo 'UUID="3B87-3ED8" /boot vfat defaults 0 2' >> /etc/fstab

This will come in handy later so we can just run “mount /boot” if it isn’t mounted (though for right now it already is).

Configuring the kernel

The change I’m going to make from the Gentoo Handbook is that we’re going to run a couple extra config commands. Remember when I said to plug everything you think you’ll need into the computer when you boot up? We’re going to leverage that now:

(chroot) ubuntu / # eselect kernel list
Available kernel symlink targets:
  [1]   linux-5.15.1-gentoo
(chroot) ubuntu / # eselect kernel set 1
(chroot) ubuntu / # cd /usr/src/linux
(chroot) ubuntu /usr/src/linux # make defconfig
(chroot) ubuntu /usr/src/linux # make localyesconfig

This gives us a sane baseline to what we’re probably going to need. Now all we have to do is customize!

In addition to whatever the Gentoo handbook recommends, we’ll also be enabling all of the following to get luks, btrfs, efi support, and initramfs:

General setup  --->
  [*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
  (/usr/src/initramfs) Initramfs source file(s)
Processor type and features  --->
    [*] EFI runtime service support 
    [*]   EFI stub support
    [ ]     EFI mixed-mode support
    [*] Built-in kernel command line
    (root=UUID=<UUID OF /dev/mapper> rd.luks.uuid=<UUID of /dev/nvme0n1p3>)
[*] Enable loadable module support
Device Drivers  --->
  Generic Driver Options --->
    [*] Maintain a devtmpfs filesystem to mount at /dev 
    [*]   Automount devtmpfs at /dev, after the kernel mounted the rootfs
  [*] Multiple devices driver support (RAID and LVM)  --->
    {*}   RAID support
    [*]     Autodetect RAID arrays during kernel boot
    <*>     Linear (append) mode
    <*>     RAID-0 (striping) mode
    {*}     RAID-1 (mirroring) mode
    {*}     RAID-10 (mirrored striping) mode
    {*}     RAID-4/RAID-5/RAID-6 mode
    <*> Device mapper support
    <*> Crypt target support
    <*> Snapshot target
    <*> Mirror target
    <*> Multipath target
    <*> Multipath target
      <*> I/O Path Selector based on the number of in-flight I/Os
      <*> I/O Path Selector based on the service time
Firmware Drivers ---> (may be in Device Drivers ---> Firmware Drivers)
  EFI (Extensible Firmware Interface) Support  --->
    <*> EFI Variable Support via sysfs
    [*] Export efi runtime maps to sysfs
File systems  --->
  <*> Btrfs filesystem support
  [*]   Btrfs POSIX Access Control Lists
  <*> FUSE (Filesystem in Userspace) support
  DOS/FAT/EXFAT/NT Filesystems  --->
    <*> MSDOS fs support
    <*> VFAT (Windows-95) fs support
    (437) Default codepage for FAT
    (iso8859-1) Default iocharset for FAT 
    [*] Enable FAT UTF-8 option by default
    <*> exFAT filesystem support
    (utf8) Default iocharset for exFAT
    # Do this if you're on 5.15!
    <*> NTFS Read-Write file system support
      [ ]   64 bits per NTFS clusters
      [*]   activate support of external compressions lzx/xpress
      [ ]   NTFS POSIX Access Control Lists
  Pseudo filesystems --->
    <*> EFI Variable filesystem # This is needed for Efibootmgr
Cryptographic API --->
    <*> XTS support
    <*> RIPEMD-160 digest algorithm
    <*> SHA224 and SHA256 digest algorithm
    <*> Whirlpool digest algorithms 
    <*> LRW support
    <*> AES cipher algorithms
    <*> AES cipher algorithms (x86_64)
    <*> Serpent cipher algorithm 
    <*> Serpent cipher algorithm (x86_64/SSE2) # If your CPU supports these
    <*> Serpent cipher algorithm (x86_64/AVX)  # If your CPU supports these
    <*> Serpent cipher algorithm (x86_64/AVX2) # If your CPU supports these
    <*> Twofish cipher algorithm
    <*> User-space interface for hash algorithms
    <*> User-space interface for symmetric key cipher algorithms

For the kernel command line, we’ll want to pass the uuid of the decrypted kernel (/dev/mapper) to the kernel and the uuid encrypted drive to the initramfs.

If using genkernel for the initramfs:

"dobtrfs crypt_root=UUID=47e7fa0b-2472-4015-96fb-e92ff1c21df7 root=UUID=2c98ff1c-a40a-4b8d-9131-e5fbc0446091"

If using dracut for the initramfs:

"rd.luks.uuid=47e7fa0b-2472-4015-96fb-e92ff1c21df7 root=UUID=2c98ff1c-a40a-4b8d-9131-e5fbc0446091"

I don’t recommend compiling the kernel yet, so once you get to Compiling and installing in the handbook, just hold off.

Configuring the Initramfs

The next steps change depending on if you compile your kernel with modules or not, and if you’re using dracut.

With modules?

If you did have any modules compiled into your kernel (you can verify by running grep “=m” .config) then you’ll have to build the kernel first. In the below command, replace “-j41” with your corecount plus one.

(chroot) ubuntu /usr/src/linux # mkdir -p /usr/src/initramfs # so we can precompile kernel
(chroot) ubuntu /usr/src/linux # make -j41 && make modules_install

If you do not have modules, then continue on.

The plan is to generate an initramfs, but then we’ll actually be compiling it into the kernel. So first, let’s install the required apps:

(chroot) ubuntu /usr/src/linux # cd
(chroot) ubuntu ~ # emerge -avq sys-fs/cryptsetup sys-fs/btrfs-progs sys-apps/busybox sys-kernel/dracut sys-boot/efibootmgr

And add dmcrypt to boot:

(chroot) ubuntu ~ # rc-update add dmcrypt boot
 * service dmcrypt added to runlevel boot

Create the /usr/src/initramfs symbolic link:

(chroot) ubuntu ~ # KERNVERN=$(basename $(cd -P /usr/src/linux && pwd) | cut -c 7-)
(chroot) ubuntu ~ # mount /boot
(chroot) ubuntu ~ # rm /usr/src/initramfs # this should be a symbolic link if exists
(chroot) ubuntu ~ # mkdir /usr/src/initramfs-$KERNVERN
(chroot) ubuntu ~ # ln -s /usr/src/initramfs-$KERNVERN initramfs 
(chroot) ubuntu ~ # cd /usr/src/initramfs

Generate the initramfs and extract it to /usr/src/initramfs

If using genkernel:

(chroot) ubuntu /usr/src/initramfs # genkernel --btrfs --luks --microcode --makeopts=-j41 --no-ramdisk-modules initramfs
(chroot) ubuntu /usr/src/initramfs # xzcat /boot/initramfs-$KERNVERN.img | cpio -imdv
(chroot) ubuntu /usr/src/initramfs # rm /boot/initramfs-$KERNVERN.img

If using dracut:

(chroot) ubuntu /usr/src/initramfs # mkdir -p /lib/modules/$KERNVERN
(chroot) ubuntu /usr/src/initramfs # dracut --hostonly --kver $KERNVERN -f /usr/src/initramfs-$KERNVERN.img
(chroot) ubuntu /usr/src/initramfs # cpio -imdv  < /usr/src/initramfs-$KERNVERN.img

The reason for that intial mkdir is a workaround to a dracut bug when kernels are built without modules. The KERNVERN is just a quick way of getting the current kernel version (in my case, 5.15.1-gentoo).

Now /usr/src/initramfs should contain everything we need to build the kernel!

(chroot) ubuntu /usr/src/initramfs # cd /usr/src/linux
(chroot) ubuntu /usr/src/linux # make clean
(chroot) ubuntu /usr/src/linux # make -j41 && make modules_install

(chroot) ubuntu /usr/src/linux # mkdir -p /boot/EFI/Gentoo
(chroot) ubuntu /usr/src/linux # cp arch/x86_64/boot/bzImage /boot/EFI/Gentoo/bzImage-$KERNVERN.efi
(chroot) ubuntu /usr/src/linux # efibootmgr --create --part 1 --disk /dev/nvme0n1 --label "Gentoo" --loader "\EFI\Gentoo\bzImage-$KERNVERN.efi"

Mounting /, swap

Next step is to configure dm-crypt to automatically unlock the swap partition. Add the following to /etc/conf.d/dmcrypt:

# Definition for /dev/mapper/swap
target=swap
source=UUID="87833367-6fe9-4118-87b8-9accbe4a09e6"
key=/etc/keys/swap.key

(replace the UUID with the one for the swap partition, e.g. /dev/nvme0n1p2)

And finally, add this to /etc/fstab under the boot entry we made earlier:

# Swap partition (/dev/mapper/swap)
UUID="3303e028-df54-424a-876d-1f0c9197d965" none swap sw 0 0
# rootfs (/dev/mapper/root)
UUID="2c98ff1c-a40a-4b8d-9131-e5fbc0446091" / btrfs noatime,nodiratime,ssd,compress=zlib 0 1

At this point, it should be safe to continue with the Gentoo Handbook, either at the Kernel Modules section if you built modules, or Configuring the System. Remember, we’re not using a bootloader, so you can skip that part! (feel free to read the entry on efibootmgr though)

If everything worked, you should get a prompt asking you to enter your password to decrypt the root filesystem. Unfortuanetly you’ll probably also see other “noise” happening (like init systems loading), not much we can do about that without installing something like Plymouth, which I may write another guide for later.

I’ll also look at writing a short script for kernel upgrades, once I do, I can amend this blog post.