The kernel command line
boot mechanism for Ubuntu Core developers for administrators
On GRUB devices snapd composes the kernel command line from the mode argument and a static part and the gadget and two system options. A change is first recorded in modeenv next to the old line and the keys are resealed for both. After the next boot the line in use must match one of them.
On the map: ubuntu core boot
01 One line, several authors for Ubuntu Core developers for administrators
The run-mode command line is the mode argument, snapd's static part, the gadget's arguments and the arguments from system options. With a TPM the whole line is measured and the disk key is sealed against it.
Type cat /proc/cmdline on the gateway and you get one line of text. It looks like one person wrote it. In fact four parties did. The boot script puts the mode first: snapd_recovery_mode=run. Next comes a static part, built into snapd's GRUB script. Then the gadget adds what this board needs, here quiet. Last, an administrator can append more with a system option, here loglevel=7. And because the gateway has a TPM, the whole line is measured at boot, and the disk key is sealed to the line snapd expects. Change one word carelessly and the next boot can't unlock the disk.
02 The parts you cannot set for Ubuntu Core developers for administrators
snapd's grub.cfg passes snapd_recovery_mode=run and a static part: console=ttyS0,115200n8 console=tty1 panic=-1 on x86-64, panic=-1 elsewhere. Recover mode passes snapd_recovery_mode=recover and the system label.
Start with the parts that aren't yours to change. They're written into the GRUB script that snapd installs. The run entry always passes snapd_recovery_mode=run. The static part is set in the same script: on x86-64, two console arguments and panic=-1; elsewhere, only the panic argument. The serial console gained its baud rate in edition three, so if you've seen it without one, that was right for older editions. Recover mode comes from the other GRUB, on the seed, and passes its own mode and system label instead.
03 The gadget's say for Ubuntu Core developers
gadget.yaml kernel-cmdline has append, remove and allow. The older files cmdline.extra and cmdline.full are still read when the section sets neither append nor remove; having both files is an error. A gadget may not set root, nfsroot, init or snapd* arguments other than snapd_system_disk and snapd.debug. A gadget refresh that changes the line ends in a reboot.
The gadget knows the board, so it gets the first say. In gadget.yaml, kernel-cmdline has three lists. Append adds arguments after the static part; ours adds quiet. Remove takes matching arguments out of the static part, so a board with no serial-port could drop that console. Allow is what an administrator may append later; a star means any value. Older gadgets use files: cmdline.extra is appended, cmdline.full replaces the static part. Both is an error, and neither is read once append or remove is set. Some arguments are off limits: root, nfsroot, init, and anything starting with snapd, apart from two.
04 The administrator's say for Ubuntu Core developers for administrators
snap set system system.kernel.cmdline-append=... appends arguments that the gadget's allow list permits; a disallowed one is refused. No reboot is requested. system.kernel.dangerous-cmdline-append applies only on dangerous-grade models.
You're the administrator, you want more kernel logging, and you don't build gadgets. This is your door. Set system.kernel.cmdline-append, and snapd checks each argument against the gadget's allow list. If they pass, you get a change with one task that works out the new line. It doesn't reboot; the line is used from the next boot, whenever that is. The docs and the code agree on that. One disallowed argument and the whole setting is refused. system.kernel.dangerous-cmdline-append skips the allow list, but only on a dangerous model. The gateway is secured, so it's ignored, with a warning in the log.
05 Two lines in modeenv for Ubuntu Core developers for administrators
A pending change puts two entries in current_kernel_command_lines in modeenv and reseals before the bootloader variable is written. After the reboot, /proc/cmdline must equal one of them; modeenv is then trimmed to it.
Here's the delicate part. The disk key is sealed to the old line, so snapd keeps the old line valid until the new one has booted. First, modeenv gets a second entry in its current_kernel_command_lines: old and new. Then the keys are resealed, so either line can open the disk. Only then does snapd write the boot variable that carries the new line. Whenever the reboot comes, the kernel starts with it. When snapd marks that boot successful, the line in use must be exactly one of the two, and the list is cut back to it. If it matches neither, that step fails.
06 Where the line is kept, and where it is not for Ubuntu Core developers for administrators
snapd writes the composed arguments to snapd_full_cmdline_args in the ubuntu-boot grubenv; a single space stands for "empty". Only GRUB has a managed command line: on other bootloaders the gadget owns it and the system options have no effect.
Where does the finished line live between boots? On the gateway, in one variable. snapd works out the result and stores it in snapd_full_cmdline_args, in the environment file on ubuntu-boot. The script uses it whenever it isn't empty, so if nothing's left, snapd writes a single space. snapd can add a fragment of its own, today only a keyboard layout, so a passphrase can be typed before the disk is open. All this needs a bootloader whose assets snapd manages, and that's only GRUB. On a Raspberry Pi, a U-Boot board or lk, the gadget owns the line, and the system options change nothing.
07 Where to go from here
Roads: sealing, GRUB, reboots, system options, the gadget and the overview.
One line of text, and it touches the gadget, the configuration system, the bootloader and the TPM. Here are the decks for each. Sealing explains why the line is part of what the keys are sealed to. The GRUB deck has the environment files and the script editions. Reboots shows which changes restart the device and which wait. Configuration covers system options and snap set. The model deck has the gadget and its kernel section. And the Ubuntu Core overview takes you back to the whole boot.
Read more
How the run-mode line is composed (GRUB)
| Order | Part | Owner | Example |
|---|---|---|---|
| 1 | mode argument | snapd's grub.cfg | snapd_recovery_mode=run |
| 2 | static part, minus the gadget's remove matches | snapd's grub.cfg edition | console=ttyS0,115200n8 console=tty1 panic=-1 |
| 3 | gadget append, or cmdline.extra | gadget | quiet |
| 4 | snapd's own fragments | snapd | keyboard layout (xkb) |
| 5 | system.kernel.cmdline-append, filtered by the gadget's allow | administrator | loglevel=7 |
| 6 | system.kernel.dangerous-cmdline-append | administrator, dangerous grade only |
A gadget cmdline.full replaces parts 2 and 3. Parts 2 to 6 are stored together in snapd_full_cmdline_args in the grubenv on ubuntu-boot, and snapd_extra_cmdline_args is cleared. An empty result is stored as a single space, so that grub.cfg still uses the variable.
Static part
| Config and edition | x86-64 | Other architectures |
|---|---|---|
run grub.cfg, edition 3 | console=ttyS0,115200n8 console=tty1 panic=-1 | panic=-1 |
run grub.cfg, earlier editions | console=ttyS0 console=tty1 panic=-1 | panic=-1 |
recovery grub-recovery.cfg, edition 2 | console=ttyS0 console=tty1 panic=-1 | panic=-1 |
Recover mode passes snapd_recovery_mode=recover snapd_recovery_system=<label> instead of the run mode argument. snapd does not update the recovery command line at runtime.
What a gadget may not set
root, nfsroot, init, and any argument whose name starts with snapd, except snapd_system_disk and snapd.debug. The rule applies to append and to the cmdline.extra and cmdline.full files. The error is "kernel parameter '...' is not allowed". A gadget with both cmdline.extra and cmdline.full: "cannot support both extra and full kernel command lines". The files are read only when kernel-cmdline has no append and no remove entries.
System options
| Option | Checked against | Applies when |
|---|---|---|
system.kernel.cmdline-append | the gadget's kernel-cmdline allow list (values may use *) | Ubuntu Core with a managed command line |
system.kernel.dangerous-cmdline-append | nothing | only when the model's grade is dangerous |
- A disallowed argument:
"<argument>" is not allowed in the kernel command line by the gadget. - On a classic system: "changing the kernel command line is not supported on a classic system".
- On a model that is not dangerous,
system.kernel.dangerous-cmdline-appendis accepted and ignored, and snapd logs a warning. - Setting either option creates a change, "Update kernel command line due to change in system configuration", with one
update-gadget-cmdlinetask. It does not request a reboot. - If the gadget's allow list changes later, arguments that are no longer allowed are filtered out and a message is logged.
When a command line change reboots
| Source of the change | Reboot requested |
|---|---|
| a system option | no |
| snapd's own fragments (keyboard layout) | no |
| a gadget refresh that changes the line | yes |
a snapd refresh that installs a newer grub.cfg edition | yes |
The handshake
1. modeenv current_kernel_command_lines gets two entries: current and candidate. 2. The keys are resealed. 3. The bootloader variables are written. 4. After the next boot, the line in /proc/cmdline must equal one entry; otherwise marking the boot successful fails with "current command line content ... not matching any expected entry". 5. modeenv is trimmed to the entry in use.
The list is stored as JSON because arguments may contain commas.
Other bootloaders
Only GRUB manages its boot assets, so only GRUB has a command line composed by snapd. On piboot, U-Boot, lk and androidboot the gadget's boot files own the command line and the command line update reports that nothing changed.
for Ubuntu Core developers
gadget.yaml
kernel-cmdline:
allow:
- loglevel=*
append:
- quiet
remove:
- console=ttyS0,115200n8Further reading
- Modify kernel options: static and dynamic changes with worked results per mode, and how to repack a gadget.
- Kernel boot parameters: the Ubuntu Core specific parameters such as
snapd.debugandubuntu_core.bootchart. - Gadget snap format:
kernel-cmdlinewithallow,appendandremove, and the wildcard rule. - System options:
system.kernel.cmdline-appendandsystem.kernel.dangerous-cmdline-append. - Debug install mode: a commented
cmdline.fullthat opens a debug shell.
Roads from here
RUNNING SNAPS
- Configuration — system options and snap set
UBUNTU CORE · boot
- From power-on to snapd — how the mode arguments are read during boot
- Reboots and snap reboot — which command line changes reboot the device
- Ubuntu Core: how a device boots and recovers — back to the Ubuntu Core overview
UBUNTU CORE · bootloaders
- GRUB on Ubuntu Core — the grubenv variables and the edition that sets the static part
- Bootloaders: one interface, six implementations — why only GRUB has a managed command line
UBUNTU CORE · install and recovery
- Model, gadget and kernel — the gadget and its kernel-cmdline section
UBUNTU CORE · full disk encryption
- Sealed keys, boot chains and resealing — why the command line is part of what the keys are sealed to