Unlocking at boot

full disk encryption for administrators for Ubuntu Core developers

At every boot snap-bootstrap measures the model and then unlocks ubuntu-data with the sealed run key and unlocks ubuntu-save with a key stored inside ubuntu-data and checks that the two belong together and locks the sealed keys again. Recover mode falls back through the fallback keys and the recovery key and records what it had to use.

On the map: ubuntu core full disk encryption

▶ Watch this deck — 8 scenes, ~6 min

01 Nobody types a password

CORE AND HYBRID · UNLOCKING AT BOOT cabinetthe gatewaynobody on sitepower back on, 3 a.m.it boots on its own; no password is typed snap-bootstrap, in the initramfs1 · measurewhat it will trust2 · unlockdata and save3 · checkthey belong together4 · lockthe sealed keysubuntu-dataubuntu-save 2 · unlockdata and saveubuntu-dataubuntu-save 3 · checkthey belong together 4 · lockthe sealed keyssealed keysuntil the next boot
A gateway in a cabinet with nobody on site powers back on; snap-bootstrap measures, unlocks ubuntu-data and ubuntu-save, checks that they belong together, and locks the sealed keys until the next boot

snap-bootstrap measures, unlocks ubuntu-data and ubuntu-save, checks that they are paired, and locks the sealed keys for the rest of the boot.

The power comes back at three in the morning, and your gateway starts on its own. Nobody's there to type a password, and nobody needs to. A small program in the initramfs, snap-bootstrap, does four things in a strict order. It measures what it's about to trust. It unlocks the two encrypted partitions, data and save. It checks that those two belong to each other. And then it locks the sealed keys again, so nothing that runs later in this boot can ask for them a second time. The rest of this deck is those four moves, and what happens when the second one fails.

02 A normal boot, in order for administrators for Ubuntu Core developers for snapd developers

CORE AND HYBRID · RUN MODE snap-bootstrap ubuntu-boot ubuntu-seed ubuntu-data ubuntu-save ubuntu-bootmountedTPMmeasure the modelubuntu-seedmounted run keyubuntu-dataone unlock callrecovery key allowed · 3 tries ubuntu-save.keyubuntu-saveno TPM callkey missing: the boot stops same marker?different: the boot stops unlocked.jsonubuntu-data unlock-key: runubuntu-save unlock-key: run
Run mode: snap-bootstrap mounts ubuntu-boot, measures the model into the TPM, mounts ubuntu-seed, unlocks ubuntu-data with the run key in one call, unlocks ubuntu-save with ubuntu-save.key from ubuntu-data, compares the markers and writes unlocked.json

Run mode mounts ubuntu-boot, measures the model, mounts ubuntu-seed, unlocks ubuntu-data in one call, unlocks ubuntu-save with ubuntu-save.key from inside ubuntu-data, compares the pairing markers, and writes unlocked.json.

Here's a normal boot, one step at a time. snap-bootstrap mounts the boot partition, measures the model into the TPM, and mounts the seed. Then a single call unlocks data. If the measurements match, the run key opens it and you notice nothing. If they don't, the same call can still take a recovery key, with three tries. Save comes next, and it doesn't go to the TPM at all. Its key is a file inside the data partition that's just been opened. Then the two markers written at install are compared, and if they differ, the boot stops. Last, it writes down which key opened what.

03 Locking the keys behind you for Ubuntu Core developers for snapd developers for administrators

CORE AND HYBRID · LOCK ON EVERY EXIT initramfs: snap-bootstrapsealed keys: can be unsealedthe running system the lock is registered firstunlockederrorpaniclock sealed keys: locked TPM: block the policies that use PCR 12hook: fde-reveal-key gets op "lock"; the helper decidesOP-TEE: its own lock PCR 12 only: the other registersstay as measured, for attestation
Every exit from snap-bootstrap in the initramfs, unlocked, error or panic, passes through one lock; after it the sealed keys are locked for the running system. The TPM policies that use PCR 12 are blocked, a hook gets a lock request, OP-TEE has its own lock

A deferred call locks the sealed keys whenever snap-bootstrap returns, with an error or not, or panics. With a TPM it blocks the PCR protection policies using PCR 12 only; with hooks it sends {"op":"lock"} to fde-reveal-key, and what that does is up to the helper; OP-TEE has its own lock. What blocking the policies does inside the TPM is decided by secboot.

The disks are open. Now, would you leave the keys in the door? The sealed keys are still sitting on disk, and a process that gets root an hour from now could try them. So snap-bootstrap registers the lock first, and the lock runs on every way out: success, error or panic. With a TPM, the lock blocks the sealing policies that use PCR 12 for the rest of this boot. Only twelve, because the other registers have to stay as measured for attestation. With hooks, the helper gets a lock request, and it's up to the helper to honour it. And OP-TEE has its own lock.

04 Recover mode, three keys deep for administrators for Ubuntu Core developers for snapd developers

CORE AND HYBRID · RECOVER MODE ubuntu-datarun keyon ubuntu-bootdoes not open fallback keyon ubuntu-seedopensoffered together in one callunlock-key: fallback recovery keytyped inonly if both fail ubuntu-savekey filefrom ubuntu-datafallback keyon ubuntu-seedrecovery keytyped ineach tried only if the one before fails degraded.jsonwritten when any step left the first choice
Recover mode: ubuntu-data is offered the run key and the fallback key in one call, then the recovery key; ubuntu-save gets the key file from ubuntu-data, then its fallback key, then the recovery key; degraded.json records which key was used

Recover mode offers ubuntu-data the run key and the fallback key in one call, then accepts the recovery key; ubuntu-save gets the key from ubuntu-data, then its fallback key, then the recovery key. Any step off the first choice is recorded in degraded.json. Factory-reset mode only unlocks ubuntu-save, with the fallback key.

Suppose your gateway's run key stops opening, say after a firmware update, and you boot it into recover mode. Recover mode doesn't give up at the first refusal. For data, one call offers both the run key and the fallback key from the seed partition, and if neither opens it, you're asked for the recovery key. For save, it tries the key file from data, then save's fallback key, then the recovery key. Any step off the first choice makes the boot degraded, and degraded.json records which key opened which partition. That file is how you, and snapd's system-mode report, tell what happened.

05 A data partition you cannot trust for administrators for Ubuntu Core developers for snapd developers

CORE AND HYBRID · THE PAIRING CHECK ubuntu-savethe gateway's owndevice identity and FDE secrets ubuntu-data (someone else's)their own accounts on itopened with a recovery key they know BAA ≠ B mounted-untrustedits logins and passwordsthe recovery system safe defaults: nobody can log inrun mode: the same mismatch stops the boot
A foreign ubuntu-data partition with marker B is swapped in next to the gateway's ubuntu-save with marker A; the markers differ, so ubuntu-data is mounted untrusted, its logins are not copied and the recovery system gets safe defaults that block logins

When ubuntu-data and ubuntu-save carry different markers, recover mode marks ubuntu-data mounted-untrusted, does not copy user authentication from it, and sets safe defaults that block logins. Run mode aborts the boot instead.

Here's the attack this check is for. Someone prepares a data partition of their own, with their own accounts on it and a recovery key they know, and swaps it into your gateway. Recover mode would unlock it. If it then copied the logins from that partition, they could sign in and reach the secrets on save. So after unlocking, the two markers are compared. If they differ, data is still mounted, but it's marked untrusted: its user accounts aren't copied, and the recovery system gets safe defaults that block logins. In run mode, the same mismatch simply stops the boot.

06 Power cuts and the lockout counter for administrators for Ubuntu Core developers for snapd developers

CORE AND HYBRID · THE LOCKOUT COUNTER TPMlockout counterlimit: 32set in secboot power cutpower cutpower cut a clean shutdown does not count snapd, at each start in run moderesetlockout auth, kept on ubuntu-save back to 0 a device that keeps losing power before snapd starts keeps counting
The TPM's lockout counter fills one notch for each power cut, not for a clean shutdown; snapd in run mode resets it each time it starts with the lockout authorisation kept on ubuntu-save, and a device that loses power before snapd starts keeps counting

Each unclean shutdown raises the TPM dictionary-attack lockout counter towards a limit set in secboot, 32 according to snapd's comments. Each time snapd starts in run mode, on installs stamped tpm, it resets the counter with the lockout authorisation stored on ubuntu-save, then considers auto-repair.

Your gateway lives on a site with bad power, and the TPM is keeping count. Every time the power drops without a clean shutdown, the TPM counts it towards its dictionary attack lockout. The limit is set inside secboot; snapd's notes put it at thirty two. Reach it and the TPM stops unsealing, and you're at the recovery key prompt. So each time snapd starts in run mode, it resets that counter, using an authorisation value saved on the save partition at install. Note the condition: snapd has to get that far. A device that loses power early in boot, again and again, keeps counting.

07 A different shape: confidential VMs for Ubuntu Core developers for snapd developers for administrators

CLASSIC CONFIDENTIAL VMS · CLOUDIMG-ROOTFS confidential VMUEFI (the ESP)cloudimg-rootfsvTPM tpm2-srk.tmplfirst boot: provisionthen deleted device/fde/cloudimg-rootfs.sealed-keyunsealed by the vTPM · recovery key allowed no model to measure:a fixed "signed, classic" stand-in EFI/ubuntu/manifest.jsonoptional: dm-verity read-only and writable
A confidential VM with a vTPM and a disk holding the ESP, labelled UEFI, and cloudimg-rootfs; the vTPM is provisioned on first boot from tpm2-srk.tmpl, which is deleted; the root partition is unlocked with a sealed key from device/fde on the ESP, with no model

In cloudimg-rootfs mode the ESP, labelled UEFI, acts as the seed. The TPM is provisioned on first boot from tpm2-srk.tmpl, which is then deleted, and the root partition is unlocked with <label>.sealed-key from device/fde on the ESP, without a model.

One more shape, because it's the same code doing the unlocking. A confidential VM boots a classic image, not Ubuntu Core, in a mode named after its root partition. There's no seed, and no model to measure, so a fixed stand-in takes its place. The EFI system partition plays the seed's part. On first boot, the virtual TPM is provisioned from a template file on that partition, and the template is deleted. After that, the root partition is unlocked with one sealed key, also on the ESP, and a recovery key still works. An optional manifest can describe dm-verity partitions instead of a single root.

08 Where to go from here

ROADS Unlocking at boot Recovery keys and repairstatus after a degraded boot Sealed keyswhy a boot may not match The boot modesrun, recover, factory reset The boot pathfrom power-on to snapd FDE hooksunlocking without a TPM Recovery systemswhere fallbacks are off
Roads from unlocking at boot: recovery keys and repair, sealing, the boot modes, the boot path, FDE hooks without a TPM, and recovery systems

Roads: keys and repair, sealing, the modes, the boot path, hooks, recovery systems and the encryption overview.

If that boot came up degraded, or asked you for a recovery key, the keys deck is where you go next: where that key comes from, how to read the status, and what snapd repairs on its own. If you want to know why a boot didn't match, go back to sealing. The modes deck has the whole of recover mode; here you've only seen its keys. The boot path deck puts this unlock between the bootloader and snapd. If your board has no TPM, the hooks deck shows the same unlock there. And recovery systems covers the one case where the fallbacks are switched off.

Read more

Run mode, step by step

StepWhat happensIf it fails
1mount ubuntu-boot, measure the model, mount ubuntu-seedthe boot fails
2unlock ubuntu-data in one call; a recovery key is acceptedthe boot fails
3unlock ubuntu-save with ubuntu-save.key from ubuntu-data"cannot find ubuntu-save encryption key"
4compare the pairing markers"cannot validate boot: ubuntu-save and ubuntu-data are not marked as from the same install"
5write unlocked.json

The order in which keyslots are tried inside the single unlock call is decided by secboot. When the initramfs fails it enters the emergency target and reboots after 10 seconds, unless dangerous is on the kernel command line. See From power-on to snapd.

Tries

The unlock prompts allow 3 passphrase tries, 3 PIN tries and 3 recovery key tries.

Which key is tried, by mode

Modeubuntu-dataubuntu-save
runone call: sealed run key, recovery key acceptedubuntu-save.key from ubuntu-data
recoverone call offering the run key and the fallback key, recovery key acceptedkey from ubuntu-data, then fallback key, then recovery key
factory-resetnot unlockedfallback key
trying a recovery systemno fallbacks; a degraded result fails the try and rebootsno fallbacks

unlocked.json and degraded.json

Both describe ubuntu-boot, ubuntu-data and ubuntu-save.

FieldValues
mount-statemounted, mounted-untrusted, error-mounting, absent-but-optional
unlock-stateunlocked, error-unlocking
unlock-keyrun, fallback, recovery
mount-locationwhere the partition was mounted (recover mode only)

unlocked.json is written in every mode that unlocks disks; in run mode only after every partition is open. degraded.json is written in recover mode when any step left the first choice.

Untrusted data

  • Recover mode compares the markers of ubuntu-data and ubuntu-save after unlocking.
  • On a mismatch ubuntu-data is mounted-untrusted: user authentication is not copied and safe defaults block logins. The log line is "cannot trust ubuntu-data, ubuntu-save and ubuntu-data are not marked as from the same install".
  • The same result follows when ubuntu-save could not be unlocked at all; the code marks this as something to refine.

for Ubuntu Core developers for snapd developers

Locking the sealed keys

BackendLock
TPMblock the PCR protection policies, using PCR 12 only; the effect inside the TPM is decided by secboot
fde-setup hookfde-reveal-key with {"op":"lock"}; what it does is up to the vendor helper
OP-TEEthe trusted application's lock

The lock is a deferred call registered before any unlock. It runs when snap-bootstrap returns, with or without an error, and on a panic; not on a hard kill such as a signal.

The TPM lockout counter

  • Each unclean shutdown raises the dictionary-attack lockout counter. The limit is set in secboot; snapd's comments give 32.
  • A clean shutdown does not raise it.
  • snapd resets it once each time it starts in run mode, with ubuntu-save/device/fde/tpm-lockout-auth, and only on installs stamped tpm. TPM installs with an empty (legacy) stamp are not reset.

for Ubuntu Core developers for snapd developers

Confidential VMs

ItemValue
Boot modecloudimg-rootfs
Seed directorythe ESP, labelled UEFI
TPM provisioningfirst boot, from tpm2-srk.tmpl on the ESP, deleted afterwards
Sealed keydevice/fde/<label>.sealed-key on the ESP
Modelnone; a fixed "signed, classic" stand-in
Optional manifestEFI/ubuntu/manifest.json: dm-verity read-only and writable partitions

Validation of the manifest's signature is not implemented yet.

Further reading

  • Full disk encryption: the official account of the initramfs unlocking ubuntu-data and ubuntu-save.
  • Use a recovery mode: the exact recovery key prompt shown at boot.
  • Troubleshooting: the "TPM is in DA Lockout Mode" case with logs and how to clear the TPM. It says the TPM stores the keys; the sealed keys are files or LUKS2 tokens and the TPM holds what unseals them.
  • Test on QEMU: resetting an emulated TPM.

Roads from here

UBUNTU CORE · boot

UBUNTU CORE · install and recovery

UBUNTU CORE · full disk encryption