The kernel command line

boot mechanism for Ubuntu Core developers for administrators

On GRUB devices snapd composes the kernel command line from the mode argument and a static part and the gadget and two system options. A change is first recorded in modeenv next to the old line and the keys are resealed for both. After the next boot the line in use must match one of them.

On the map: ubuntu core boot

▶ Watch this deck — 7 scenes, ~5 min

01 One line, several authors for Ubuntu Core developers for administrators

COMMAND LINE $ cat /proc/cmdlineacme-gateway, run mode snapd_recovery_mode=runthe boot scriptmode argument console=ttyS0,115200n8 console=tty1 panic=-1snapdstatic part quietthe gadget loglevel=7administratorsystem option TPMmeasured at every bootthe disk key is sealed to the line snapd expects
The output of cat /proc/cmdline on the gateway split into four segments with their authors: snapd_recovery_mode=run from the boot script, console=ttyS0,115200n8 console=tty1 panic=-1 as snapd's static part, quiet from the gadget and loglevel=7 from the administrator through a system option. A bracket under the whole line leads to the TPM: the line is measured and the disk key is sealed to it.

The run-mode command line is the mode argument, snapd's static part, the gadget's arguments and the arguments from system options. With a TPM the whole line is measured and the disk key is sealed against it.

Type cat /proc/cmdline on the gateway and you get one line of text. It looks like one person wrote it. In fact four parties did. The boot script puts the mode first: snapd_recovery_mode=run. Next comes a static part, built into snapd's GRUB script. Then the gadget adds what this board needs, here quiet. Last, an administrator can append more with a system option, here loglevel=7. And because the gateway has a TPM, the whole line is measured at boot, and the disk key is sealed to the line snapd expects. Change one word carelessly and the next boot can't unlock the disk.

02 The parts you cannot set for Ubuntu Core developers for administrators

FIXED PARTS grub.cfg, Snapd-Boot-Config-Edition: 3 menuentry "Run Ubuntu Core" {chainloader $prefix/$kernelsnapd_recovery_mode=run$cmdline_args} set snapd_static_cmdline_args='panic=-1'if [ "$grub_cpu" = "x86_64" ]; thenset snapd_static_cmdline_args='console=ttyS0,115200n8 console=tty1 panic=-1'fi x86-64console=ttyS0,115200n8 console=tty1panic=-1other architecturespanic=-1 before edition 3: console=ttyS0, no baud rate recover mode, from the recovery grub.cfg on ubuntu-seed:snapd_recovery_mode=recover snapd_recovery_system=<label>
A shortened grub.cfg from snapd, edition 3: it sets the static arguments to panic=-1, and on x86_64 to console=ttyS0,115200n8 console=tty1 panic=-1, and the run entry passes snapd_recovery_mode=run with the arguments. On the right the results per architecture; below, the recover mode arguments from the recovery grub.cfg.

snapd's grub.cfg passes snapd_recovery_mode=run and a static part: console=ttyS0,115200n8 console=tty1 panic=-1 on x86-64, panic=-1 elsewhere. Recover mode passes snapd_recovery_mode=recover and the system label.

Start with the parts that aren't yours to change. They're written into the GRUB script that snapd installs. The run entry always passes snapd_recovery_mode=run. The static part is set in the same script: on x86-64, two console arguments and panic=-1; elsewhere, only the panic argument. The serial console gained its baud rate in edition three, so if you've seen it without one, that was right for older editions. Recover mode comes from the other GRUB, on the seed, and passes its own mode and system label instead.

03 The gadget's say for Ubuntu Core developers

GADGET pc: meta/gadget.yaml run-mode line, after the mode argument console=ttyS0,115200n8console=tty1panic=-1 kernel-cmdline: append:- quietquietappended remove:- console=ttyS0,115200n8removed from the static parta board with no serial port allow:- loglevel=*what an administrator may append later* matches any value older form: a file cmdline.extra is appended, cmdline.full replaces the static partboth files: an errorread only when append and remove are both empty a gadget may not setroot nfsroot init snapd*except snapd_system_disk and snapd.debug
A gadget.yaml fragment with kernel-cmdline append quiet, remove console=ttyS0,115200n8 and allow loglevel=*. On the right the static part with quiet appended and the serial console struck out, and the allow list labelled as what a system option may add later. Below, the older files cmdline.extra and cmdline.full, and the arguments a gadget may not set: root, nfsroot, init and snapd*, except snapd_system_disk and snapd.debug.

gadget.yaml kernel-cmdline has append, remove and allow. The older files cmdline.extra and cmdline.full are still read when the section sets neither append nor remove; having both files is an error. A gadget may not set root, nfsroot, init or snapd* arguments other than snapd_system_disk and snapd.debug. A gadget refresh that changes the line ends in a reboot.

The gadget knows the board, so it gets the first say. In gadget.yaml, kernel-cmdline has three lists. Append adds arguments after the static part; ours adds quiet. Remove takes matching arguments out of the static part, so a board with no serial-port could drop that console. Allow is what an administrator may append later; a star means any value. Older gadgets use files: cmdline.extra is appended, cmdline.full replaces the static part. Both is an error, and neither is read once append or remove is set. Some arguments are off limits: root, nfsroot, init, and anything starting with snapd, apart from two.

04 The administrator's say for Ubuntu Core developers for administrators

SYSTEM OPTION $ sudo snap set system system.kernel.cmdline-append="loglevel=7"accepted: the gadget allows loglevel=* $ snap changesDone Update kernel command line due to change in system configurationone change with one task, which works out the new line no reboot is requested: the new line is used from the next boot $ sudo snap set system system.kernel.cmdline-append="mitigations=off""mitigations=off" is not allowed in the kernel command line by the gadget $ sudo snap set system system.kernel.dangerous-cmdline-append="..."no allow list, but used only when the model's grade is dangerous acme-gateway is secured: accepted and ignored, with a warning in snapd's log
A terminal on the gateway. snap set system system.kernel.cmdline-append=loglevel=7 is accepted because the gadget allows it, and snap changes shows one change, Update kernel command line due to change in system configuration, with no reboot. Setting mitigations=off is refused by the gadget's allow list. Setting system.kernel.dangerous-cmdline-append is ignored on this secured model with a warning in the log.

snap set system system.kernel.cmdline-append=... appends arguments that the gadget's allow list permits; a disallowed one is refused. No reboot is requested. system.kernel.dangerous-cmdline-append applies only on dangerous-grade models.

You're the administrator, you want more kernel logging, and you don't build gadgets. This is your door. Set system.kernel.cmdline-append, and snapd checks each argument against the gadget's allow list. If they pass, you get a change with one task that works out the new line. It doesn't reboot; the line is used from the next boot, whenever that is. The docs and the code agree on that. One disallowed argument and the whole setting is refused. system.kernel.dangerous-cmdline-append skips the allow list, but only on a dangerous model. The gateway is secured, so it's ignored, with a warning in the log.

05 Two lines in modeenv for Ubuntu Core developers for administrators

THE HANDSHAKE 1 · recordmodeenvcurrent_kernel_command_lines=["snapd_recovery_mode=run ... quiet","snapd_recovery_mode=run ... quiet loglevel=7"] 2 · resealTPMthe keys accept either line 3 · write the variablegrubenv on ubuntu-bootsnapd_full_cmdline_args=console=... panic=-1 quiet loglevel=7 4 · rebootwhenever it comes/proc/cmdline: snapd_recovery_mode=run ... quiet loglevel=7 5 · checkmatches one entry exactly[ "... quiet loglevel=7" ] matches neither: marking the boot successful fails
Five steps. Record: modeenv current_kernel_command_lines holds the old and the new line. Reseal: the TPM keys accept either line. Write the boot variable snapd_full_cmdline_args in the grubenv on ubuntu-boot. Reboot: /proc/cmdline shows the new line. Check: the line in use matches one entry and modeenv is trimmed to it; if it matches neither, marking the boot successful fails.

A pending change puts two entries in current_kernel_command_lines in modeenv and reseals before the bootloader variable is written. After the reboot, /proc/cmdline must equal one of them; modeenv is then trimmed to it.

Here's the delicate part. The disk key is sealed to the old line, so snapd keeps the old line valid until the new one has booted. First, modeenv gets a second entry in its current_kernel_command_lines: old and new. Then the keys are resealed, so either line can open the disk. Only then does snapd write the boot variable that carries the new line. Whenever the reboot comes, the kernel starts with it. When snapd marks that boot successful, the line in use must be exactly one of the two, and the list is cut back to it. If it matches neither, that step fails.

06 Where the line is kept, and where it is not for Ubuntu Core developers for administrators

STORAGE GRUB every other bootloader grubenv on ubuntu-bootsnapd_full_cmdline_args=console=ttyS0,115200n8console=tty1 panic=-1 quiet loglevel=7snapd_extra_cmdline_args= grub.cfgif [ -n "$snapd_full_cmdline_args" ]; thenset cmdline_args="$snapd_full_cmdline_args"nothing left at all? snapd stores a single space snapd's own fragment: a keyboard layout (xkb)for typing a passphrase before the disk is open pibootU-Bootlkandroidboot not managed by snapdthe gadget's files own the command linethe system options change nothing
Left, GRUB: the grubenv on ubuntu-boot holds the composed line in snapd_full_cmdline_args and an empty snapd_extra_cmdline_args, and grub.cfg uses the full variable when it is not empty; an empty result is stored as a single space; snapd's own fragment is a keyboard layout. Right: piboot, U-Boot, lk and androidboot do not have a managed command line, the gadget owns it and the system options change nothing.

snapd writes the composed arguments to snapd_full_cmdline_args in the ubuntu-boot grubenv; a single space stands for "empty". Only GRUB has a managed command line: on other bootloaders the gadget owns it and the system options have no effect.

Where does the finished line live between boots? On the gateway, in one variable. snapd works out the result and stores it in snapd_full_cmdline_args, in the environment file on ubuntu-boot. The script uses it whenever it isn't empty, so if nothing's left, snapd writes a single space. snapd can add a fragment of its own, today only a keyboard layout, so a passphrase can be typed before the disk is open. All this needs a bootloader whose assets snapd manages, and that's only GRUB. On a Raspberry Pi, a U-Boot board or lk, the gadget owns the line, and the system options change nothing.

07 Where to go from here

ROADS The kernel command line Sealed keyswhy the line is measured GRUB on Ubuntu Coregrubenv and editions Rebootswhich changes restart Configurationsystem options and snap set Model, gadget and kernelthe kernel-cmdline section Ubuntu Coreback to the overview
Roads from the kernel command line: sealed keys, GRUB on Ubuntu Core, reboots, configuration and system options, the model and gadget, and the Ubuntu Core overview

Roads: sealing, GRUB, reboots, system options, the gadget and the overview.

One line of text, and it touches the gadget, the configuration system, the bootloader and the TPM. Here are the decks for each. Sealing explains why the line is part of what the keys are sealed to. The GRUB deck has the environment files and the script editions. Reboots shows which changes restart the device and which wait. Configuration covers system options and snap set. The model deck has the gadget and its kernel section. And the Ubuntu Core overview takes you back to the whole boot.

Read more

How the run-mode line is composed (GRUB)

OrderPartOwnerExample
1mode argumentsnapd's grub.cfgsnapd_recovery_mode=run
2static part, minus the gadget's remove matchessnapd's grub.cfg editionconsole=ttyS0,115200n8 console=tty1 panic=-1
3gadget append, or cmdline.extragadgetquiet
4snapd's own fragmentssnapdkeyboard layout (xkb)
5system.kernel.cmdline-append, filtered by the gadget's allowadministratorloglevel=7
6system.kernel.dangerous-cmdline-appendadministrator, dangerous grade only

A gadget cmdline.full replaces parts 2 and 3. Parts 2 to 6 are stored together in snapd_full_cmdline_args in the grubenv on ubuntu-boot, and snapd_extra_cmdline_args is cleared. An empty result is stored as a single space, so that grub.cfg still uses the variable.

Static part

Config and editionx86-64Other architectures
run grub.cfg, edition 3console=ttyS0,115200n8 console=tty1 panic=-1panic=-1
run grub.cfg, earlier editionsconsole=ttyS0 console=tty1 panic=-1panic=-1
recovery grub-recovery.cfg, edition 2console=ttyS0 console=tty1 panic=-1panic=-1

Recover mode passes snapd_recovery_mode=recover snapd_recovery_system=<label> instead of the run mode argument. snapd does not update the recovery command line at runtime.

What a gadget may not set

root, nfsroot, init, and any argument whose name starts with snapd, except snapd_system_disk and snapd.debug. The rule applies to append and to the cmdline.extra and cmdline.full files. The error is "kernel parameter '...' is not allowed". A gadget with both cmdline.extra and cmdline.full: "cannot support both extra and full kernel command lines". The files are read only when kernel-cmdline has no append and no remove entries.

System options

OptionChecked againstApplies when
system.kernel.cmdline-appendthe gadget's kernel-cmdline allow list (values may use *)Ubuntu Core with a managed command line
system.kernel.dangerous-cmdline-appendnothingonly when the model's grade is dangerous
  • A disallowed argument: "<argument>" is not allowed in the kernel command line by the gadget.
  • On a classic system: "changing the kernel command line is not supported on a classic system".
  • On a model that is not dangerous, system.kernel.dangerous-cmdline-append is accepted and ignored, and snapd logs a warning.
  • Setting either option creates a change, "Update kernel command line due to change in system configuration", with one update-gadget-cmdline task. It does not request a reboot.
  • If the gadget's allow list changes later, arguments that are no longer allowed are filtered out and a message is logged.

When a command line change reboots

Source of the changeReboot requested
a system optionno
snapd's own fragments (keyboard layout)no
a gadget refresh that changes the lineyes
a snapd refresh that installs a newer grub.cfg editionyes

The handshake

1. modeenv current_kernel_command_lines gets two entries: current and candidate. 2. The keys are resealed. 3. The bootloader variables are written. 4. After the next boot, the line in /proc/cmdline must equal one entry; otherwise marking the boot successful fails with "current command line content ... not matching any expected entry". 5. modeenv is trimmed to the entry in use.

The list is stored as JSON because arguments may contain commas.

Other bootloaders

Only GRUB manages its boot assets, so only GRUB has a command line composed by snapd. On piboot, U-Boot, lk and androidboot the gadget's boot files own the command line and the command line update reports that nothing changed.

for Ubuntu Core developers

gadget.yaml

kernel-cmdline:
  allow:
    - loglevel=*
  append:
    - quiet
  remove:
    - console=ttyS0,115200n8

Further reading

Roads from here

RUNNING SNAPS

UBUNTU CORE · boot

UBUNTU CORE · bootloaders

UBUNTU CORE · install and recovery

UBUNTU CORE · full disk encryption