How a TPM system is launched

swtpm per launch for CI operators for contributors

The .run script of a +tpm system checks for swtpm, its own directory and the TPM state next to the disk, starts swtpm on that state (a copy under -snapshot) with its socket in a runtime directory, runs QEMU with a TPM device on that socket, and a watcher removes the directory when swtpm exits.

On the map: runtime

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 A TPM that is a process

A TPM FOR EVERY BOOT ubuntu-core-24+tpm.x86_64.qcow2 ubuntu-core-24+tpm.x86_64.efi-vars.qcow2 ubuntu-core-24+tpm.x86_64.run ubuntu-core-24+tpm.x86_64.tpm/ the TPM state the disk keys are sealed to every boot: the same TPM, or the disk stays locked swtpm an emulator, a process socket QEMU ubuntu-core-24+tpm…run starts swtpm first, then QEMU, and leaves a helper to clean up
Next to ubuntu-core-24+tpm.x86_64.qcow2 and its UEFI variables sits the TPM state directory ubuntu-core-24+tpm.x86_64.tpm/, to which the disk keys are sealed. Every boot needs that TPM. The .run script starts swtpm, an emulator process, and then QEMU, which talks to swtpm over a socket.

ubuntu-core-24+tpm.x86_64.run starts swtpm on the state in ubuntu-core-24+tpm.x86_64.tpm/, then QEMU with a TPM device on swtpm's socket.

Take ubuntu-core-24+tpm. Its first boot encrypted the data partitions and sealed the keys to a TPM, and that TPM's state now sits in a directory next to the disk image. Every later boot needs the same TPM again, or the disk won't unlock. A real machine has the chip on its board. Here, the TPM is swtpm, an emulator that runs as a separate process, and QEMU talks to it through a socket. So the .run script of such a system does more than start QEMU: it starts swtpm first, and leaves a helper to clean up after it.

Details and sources

Details

  • The +tpm systems are ubuntu-core-24+tpm, 24+dangerous+tpm, 26+tpm and 26+dangerous+tpm, for x86_64 only; their keys are sealed to the TPM state in ubuntu-core-R.x86_64.tpm/, which "belongs with the disk image and the UEFI variables" (050-ubuntu-core.mk:9-16).
  • Their launcher is written by EMIT_QEMU_SYSTEM_X86_64_EFI_RUN_COMMAND_WITH_TPM with the .tpm directory as its fifth argument (050-ubuntu-core.mk:63-68, 010-qemu-x86_64.mk:53-61).
  • QEMU gets the TPM as -chardev socket,id=chrtpm,path=SOCKET -tpmdev emulator,id=tpm0,chardev=chrtpm -device tpm-crb,tpmdev=tpm0 (010-qemu-x86_64.mk:132-133).
  • The README: copy or remove the .tpm/ directory together with the .qcow2 and .efi-vars.qcow2 files (README.md:711-718).
  • Nothing on this page was run: swtpm was not installed where the deck was written. Every statement is read in the code.

02 Three checks first

THREE CHECKS swtpm installed? …: swtpm, the TPM emulator, is required: install the swtpm package run from its own directory? …: run this launcher from its own directory, DIR: its disk is sealed to the TPM state there disk, UEFI variables and TPM state are opened from ./ : the keys are sealed to all three TPM state there? …: the TPM state, DIR, is missing: make the .qcow2 image again each: exit status 1, before swtpm or QEMU starts
The launcher checks that swtpm is installed, that it runs from its own directory, because the disk, the UEFI variables and the TPM state are opened from there and the keys are sealed to those three, and that the TPM state directory exists. Each failure has its own message and exit status 1.

Missing swtpm, another working directory, or a missing .tpm/ directory each stop the launcher with exit status 1 and a message.

Before anything starts, the script checks three things, each with its own message. Is swtpm installed? If not, it says swtpm, the TPM emulator, is required, and to install the swtpm package. Are you running it from its own directory? The disk, the firmware variables and the TPM state are opened relative to where you are, and the keys are sealed to exactly those three, so it refuses to run anywhere else. And is the TPM state there? If it's gone, the script tells you to make the qcow2 image again, because the disk can't be unlocked without it.

Details and sources

Details

  • command -v $(SWTPM), else "$0: swtpm, the TPM emulator, is required: install the swtpm package" and exit 1 (010-qemu-x86_64.mk:154-158, message 010-qemu-x86_64.mk:120).
  • The launcher's directory, from readlink -f "$0" and pwd -P, must be the current directory, else "$0: run this launcher from its own directory, DIR: its disk is sealed to the TPM state there" (010-qemu-x86_64.mk:159-167). The disk and the UEFI variables are named relative to it in the QEMU options (010-qemu-x86_64.mk:67-68).
  • The state directory is DIR/ubuntu-core-R.x86_64.tpm; without it: "$0: the TPM state, DIR, is missing: make the .qcow2 image again" (010-qemu-x86_64.mk:168-172).
  • --query is answered before these checks, so it works without swtpm (010-qemu.mk:57-90).
  • In the snap, making a +tpm system without the swtpm component stops earlier, with "install the swtpm component to use systems with a TPM: snap install image-garden+swtpm" (010-qemu-x86_64.mk:124-131).

03 A directory, and maybe a copy

WHERE SWTPM WORKS $XDG_RUNTIME_DIR/image-garden/ socket paths must be short allocate: port.5555.tpm for discard otherwise: tpm.XXXXXXXX a new one for arg in "$@"; do … -snapshot? … done allocate passes -snapshot with -snapshot cp -a ….tpm RUN_DIR/state swtpm gets the copy without it --tpmstate dir=….tpm swtpm changes the real state
swtpm's socket goes in a directory under $XDG_RUNTIME_DIR/image-garden, because socket paths must be short: port.5555.tpm when allocate names it with GARDEN_TPM_RUN_DIR, otherwise a new tpm.XXXXXXXX. If the arguments contain -snapshot, the state is copied into it and swtpm gets the copy; without it swtpm uses the state directory next to the disk.

The run directory is $GARDEN_TPM_RUN_DIR (allocate: port.PORT.tpm) or a new tpm.XXXXXXXX. With -snapshot swtpm gets a copy of the state in it.

Next, swtpm needs a place for its socket. Socket paths must be short, so it isn't next to the disk but under your runtime directory. When allocate starts the script, that place has a name tied to the port, so discard can find it later. Otherwise the script makes a fresh one with a random name. Then it looks at its arguments. With -snapshot, as allocate passes, QEMU won't change the disk or the variables, so the TPM must not change either: the script copies the state into that directory and gives swtpm the copy. Without it, swtpm works on the real state.

Details and sources

Details

  • With GARDEN_TPM_RUN_DIR set the launcher runs mkdir -m 700 on it; otherwise it makes $XDG_RUNTIME_DIR/image-garden/tpm.XXXXXXXX with mktemp -d, or ${TMPDIR:-/tmp}/image-garden-tpm.XXXXXXXX without a runtime directory (010-qemu-x86_64.mk:173-189).
  • A launcher made by the snap and run outside it uses $XDG_RUNTIME_DIR/snap.image-garden/image-garden, which the snap's confined swtpm and QEMU can reach (010-qemu-x86_64.mk:177-181).
  • allocate sets GARDEN_TPM_RUN_DIR=$XDG_RUNTIME_DIR/image-garden/port.PORT.tpm and removes a leftover one first (image-garden:825-831, image-garden:1108-1112).
  • The comment gives the reasons: "UNIX socket paths are short", and with -snapshot "swtpm must not change the TPM state either: it gets a copy" (010-qemu-x86_64.mk:137-152). The copy is cp -a STATE RUN_DIR/state when any argument is -snapshot (010-qemu-x86_64.mk:190-197).
  • allocate passes -snapshot unless QEMU_SNAPSHOT_OPTION is set (image-garden:1154). Run without it, the launcher lets swtpm change the real state, as QEMU changes the real disk and variables.

04 Start, watch, run

START, WATCH, RUN swtpm socket --tpm2 --tpmstate dir="$tpm_state_dir" --ctrl type=unixio,path="$tpm_run_dir/swtpm.sock" --pid file="$tpm_run_dir/swtpm.pid" --terminate --daemon --daemon: in the background --terminate: exits when QEMU does (tail --pid="$swtpm_pid" -f /dev/null; rm -rf "$tpm_run_dir") & the watcher qemu-system-x86_64 … -chardev socket,id=chrtpm,path=…/swtpm.sock -tpmdev emulator,id=tpm0,chardev=chrtpm -device tpm-crb,tpmdev=tpm0 "${qemu_cmd[@]}" || { kill "$swtpm_pid"; exit "$rc"; } QEMU failed: stop swtpm
The launcher runs swtpm socket --tpm2 with the state, a control socket and a pid file in the run directory, --terminate so it exits when QEMU disconnects and --daemon so it runs in the background; starts a watcher that removes the run directory when swtpm ends; runs QEMU with a TPM device on the socket; and kills swtpm if QEMU fails.

swtpm socket --tpm2 --tpmstate dir=… --ctrl type=unixio,path=…/swtpm.sock --pid file=… --terminate --daemon, a watcher, then QEMU with -tpmdev emulator and -device tpm-crb.

Now swtpm starts, as a TPM 2.0, on that state, with its control socket in the directory and its process ID in a pid file. Two flags matter. --daemon puts it in the background. --terminate makes it exit as soon as QEMU disconnects. Right after, the script starts a small watcher in the background, which waits for swtpm to end and then deletes the directory. Last, QEMU runs with a TPM device backed by that socket. If QEMU fails, the script stops swtpm and exits with QEMU's status.

Details and sources

Details

  • $(SWTPM) socket --tpm2 --tpmstate dir=STATE --ctrl type=unixio,path=RUN/swtpm.sock --pid file=RUN/swtpm.pid --terminate --daemon; if it fails, the run directory is removed and the launcher exits 1 (010-qemu-x86_64.mk:198-201).
  • The watcher: (tail --pid=SWTPM_PID -f /dev/null; rm -rf RUN) </dev/null >/dev/null 2>&1 &, so it also works when QEMU detaches with -daemonize (010-qemu-x86_64.mk:202-203, comment :151-152).
  • The QEMU options: QEMU_TPM_OPTION, default from QEMU_TPM_OPTION_FOR on the socket, and -global driver=cfi.pflash01,property=secure,value=on on the firmware flash (010-qemu-x86_64.mk:67, :71, :80, :104, :133-135).
  • EMIT_SWTPM_RUN_QEMU runs QEMU without exec and, if it fails, kills swtpm and exits with QEMU's status (010-qemu-x86_64.mk:207-216).
  • That swtpm with --terminate exits when its client disconnects is from swtpm's options as the comment describes them (010-qemu-x86_64.mk:140-142).

Read more

05 When it ends

WHEN IT ENDS QEMU exits powered off, or discarded swtpm terminates the watcher port.5555.tpm/ and the copied state next boot: from ….x86_64.tpm/ next to the disk, unchanged image-garden discard: wait up to 5 s, then rm -rf image-garden allocate: rm -rf port.PORT.tpm first three removers, one directory
QEMU exits, because the guest powered off or discard stopped it; swtpm terminates; the watcher removes the run directory with the copied state, so the next boot starts from the .tpm directory again. discard also removes the directory after waiting up to five seconds for swtpm, and allocate clears any directory left on its port first.

QEMU exits, swtpm terminates, the watcher removes the run directory and the copied state. discard and the next allocate on the port remove it too.

When the machine stops, the pieces fall away in order. QEMU exits, whether the guest powered off or discard stopped it. swtpm sees the disconnect and terminates. The watcher notices and removes the directory, and with it the copy of the TPM state, so the next boot starts again from the state next to the disk. For an allocated machine, discard removes the directory as well, after giving swtpm up to five seconds to go. And allocate clears any directory left on its port before it starts a new machine there.

Details and sources

Details

  • The watcher removes the run directory once swtpm's process ends (010-qemu-x86_64.mk:203).
  • discard_tpm_run_dir: waits up to 5 seconds while swtpm runs and is not a zombie, kills it, then retries rm -rf for 2 seconds because the watcher may remove the directory at the same time (image-garden:833-865). allocate calls it too when the machine dies before SSH or fails to start (image-garden:1186-1190, image-garden:1208-1210).
  • allocate runs rm -rf on port.PORT.tpm before it starts the launcher (image-garden:1108-1112).
  • A launcher started without allocate uses a tpm.XXXXXXXX directory that only the watcher removes.

Read more

06 In the makefiles for contributors

FOR CONTRIBUTORS EMIT_QEMU_SYSTEM_X86_64_EFI_RUN_COMMAND_WITH_TPM $(1) disk $(2) code $(3) vars $(4) .run $(5) TPM dir EMIT_QEMU_SYSTEM_X86_64_EFI_RUN_COMMAND passes $(5) empty +tpm rules, 050-ubuntu-core.mk $(5) = ubuntu-core-R.x86_64.tpm EMIT_SWTPM_START QEMU_SECURE_PFLASH_OPTION QEMU_TPM_OPTION_FOR EMIT_SWTPM_RUN_QEMU, not exec QEMU_SYSTEM_X86_64_MACHINE = q35,smm=on the provisioning boot, in the qcow2 rule: its own swtpm, a new state directory
EMIT_QEMU_SYSTEM_X86_64_EFI_RUN_COMMAND_WITH_TPM takes the disk, the firmware code and variables, the launcher and, as $(5), the TPM state directory. The plain EFI launcher passes it empty; the +tpm rules in 050-ubuntu-core.mk pass ubuntu-core-R.x86_64.tpm. With it the template adds EMIT_SWTPM_START, QEMU_SECURE_PFLASH_OPTION, QEMU_TPM_OPTION_FOR and EMIT_SWTPM_RUN_QEMU. The rules set q35,smm=on, and the qcow2 rule starts its own swtpm on a new state directory.

EMIT_QEMU_SYSTEM_X86_64_EFI_RUN_COMMAND_WITH_TPM with $(5) set adds EMIT_SWTPM_START, QEMU_SECURE_PFLASH_OPTION, QEMU_TPM_OPTION_FOR and EMIT_SWTPM_RUN_QEMU; the +tpm rules set q35,smm=on.

In the makefiles, a TPM launcher is the ordinary x86 UEFI launcher with a fifth argument: the TPM state directory. The plain UEFI launcher passes it empty. The +tpm rules in the Ubuntu Core makefile pass the directory next to the disk. With it, the template writes the swtpm start, adds the secure flash property and the TPM options, and runs QEMU with a check instead of exec. Those rules also set the machine to q35 with SMM on, which Secure Boot firmware needs. The provisioning boot isn't a launcher: the qcow2 rule starts its own swtpm, on a new, empty state directory.

Details and sources

Details

  • EMIT_QEMU_SYSTEM_X86_64_EFI_RUN_COMMAND calls ..._WITH_TPM with an empty fifth argument (010-qemu-x86_64.mk:46-51); with a fifth argument the template adds EMIT_SWTPM_START, the secure pflash option, default_qemu_tpm_option, QEMU_TPM_OPTION and EMIT_SWTPM_RUN_QEMU instead of EMIT_EXEC_QEMU (010-qemu-x86_64.mk:53-113).
  • The +tpm rules: QEMU_SYSTEM_X86_64_MACHINE = q35,smm=on for their .run and .qcow2, Secure Boot firmware copies converted with $(QEMU_IMG) from efi-secboot-code.x86_64.img and efi-secboot-vars.x86_64.img, and the launcher with ubuntu-core-R.x86_64.tpm (050-ubuntu-core.mk:55-68).
  • The qcow2 rule removes the state directory, makes it again with mkdir -m 700, starts swtpm on it in a tpm.XXXXXXXX run directory, boots with the seed, fails when the log says QEMU was stopped by a signal, then waits up to 10 s for swtpm and removes the run directory (050-ubuntu-core.mk:77-106).
  • SWTPM is swtpm, or, in a launcher made by the snap and run outside it, image-garden.swtpm (010-qemu-x86_64.mk:115-119).
  • The +tpm qcow2 rule runs under bash with set -o pipefail, so a QEMU that fails gives the recipe its status through tee (050-ubuntu-core.mk:77-99); the Ubuntu Core rules without a TPM do the same (050-ubuntu-core.mk:37-53), and .DELETE_ON_ERROR removes the half-made disk (000-garden.mk:4-7).

07 Where to go from here

ROADS swtpm per launch Secure Boot and TPM the four +tpm systems discard and ps stopping allocated machines swtpm in the snap a component and an app Full-disk encryption on the snapd map
Four roads from swtpm per launch: Secure Boot and TPM systems, discard and ps, swtpm in the snap, and full-disk encryption on the snapd map

Next: the +tpm systems, discard, swtpm in the snap, and full-disk encryption on the snapd map.

That's a TPM launch: three checks, a short-lived directory for swtpm's socket, a copy of the state when the machine runs with a snapshot, a watcher that tidies up, and QEMU with a TPM device. From here, the Secure Boot and TPM deck shows the four systems that use this and what their first boot does. The discard deck shows how allocated machines are stopped, and the snap's swtpm deck shows how the snap supplies swtpm. The snapd map explains what the sealed keys protect.

Details and sources

Read more

Basis of the statements on this page

Read in the image-garden source at the commit in garden.commit ("code"). Nothing was run: swtpm was not installed where this deck was written. What swtpm's own options do is taken from the comments in the code.

The launcher of a +tpm system, in order

StepWhatSource
1--query: answer and exit010-qemu.mk:57-90
2swtpm on PATH, or exit 1010-qemu-x86_64.mk:154-158
3run from the launcher's directory, or exit 1010-qemu-x86_64.mk:159-167
4SYSTEM.x86_64.tpm/ exists, or exit 1010-qemu-x86_64.mk:168-172
5run directory: GARDEN_TPM_RUN_DIR, or a new tpm.XXXXXXXX010-qemu-x86_64.mk:173-189
6with -snapshot: copy the state into RUN/state010-qemu-x86_64.mk:190-197
7swtpm socket … --terminate --daemon010-qemu-x86_64.mk:198-202
8the watcher, which removes the run directory when swtpm exits010-qemu-x86_64.mk:203
9QEMU with -machine q35,smm=on, secure pflash and a tpm-crb device; kill swtpm if it fails010-qemu-x86_64.mk:59-108, :207-216

Files

PathWhatLifetime
ubuntu-core-R.x86_64.tpm/the TPM state the disk keys are sealed to, next to the diskmade by the first boot, removed by clean
$XDG_RUNTIME_DIR/image-garden/port.PORT.tpm/run directory of an allocated machineuntil swtpm exits, discard, or the next allocate on the port
$XDG_RUNTIME_DIR/image-garden/tpm.XXXXXXXX/run directory of any other launchuntil swtpm exits
RUN/swtpm.sock, RUN/swtpm.pidswtpm's control socket and PIDwith the run directory
RUN/state/the copy used under -snapshotwith the run directory

Sources: 050-ubuntu-core.mk:314-318, image-garden:825-865.

Environment

VariableEffect
GARDEN_TPM_RUN_DIRthe run directory to create; allocate sets it
XDG_RUNTIME_DIRwhere run directories go; without it ${TMPDIR:-/tmp}
QEMU_TPM_OPTIONreplaces the TPM options of the launcher
SWTPM (make)the swtpm command written into the launcher

Roads from here

UBUNTU CORE · all snaps

RUNTIME · make rules and the CLI

SNAP · snap-only features

Snapd map ↗