Reboots and snap reboot

boot mechanism for administrators for users for Ubuntu Core developers

On Ubuntu Core a refresh of the kernel or base or gadget needs a reboot. snapd schedules it one minute ahead with a ten-minute fallback and lets the change continue after the device is back. snap reboot restarts at once and can select the mode and recovery system for the next boot.

On the map: ubuntu core boot

▶ Watch this deck — 8 scenes, ~6 min

01 Why is it rebooting?

REFRESH $ sudo snap refresh…reboot scheduled to update the systemyou didn't ask for a reboot the device boots from thesekernelbasegadgetrefresh = reboot gadget: only if its assetsor the kernel command line changed everything elseappsother snapsrestart the app, no reboot snapd itself: a reboot only when it rewrites the boot configuration it manages
A terminal on the gateway shows the message reboot scheduled to update the system; the kernel, base and gadget snaps the device boots from are completed by a reboot, while apps and other snaps only restart the app

On Ubuntu Core a refresh of the kernel or the base is completed by a reboot, and so is a gadget refresh that changes its assets or the kernel command line. App refreshes do not reboot the device; a snapd refresh reboots only when it updates the boot configuration that snapd manages.

You're logged in to the gateway and a line appears on your terminal: reboot scheduled to update the system. You didn't ask for a reboot, so who did? A refresh. The kernel, base and gadget are special: the gateway is running from them and can't swap them underneath itself. A new kernel or base has to be tried by booting it. A gadget reboots only when its update changed something on disk or the kernel command line. Everything else is gentler. When an app is refreshed, snapd restarts the app, and the device carries on. Even snapd reboots only when it rewrites the boot configuration it manages.

02 One minute, and a ten-minute safety net

THE DELAY ≈now+1 min+10 minnot to scale fallback rebootscheduled first snapd stops cleanly the real reboottime to read the messageshutdown -r +1 "reboot scheduled to update the system" replaced snap reboot: +0, at once
A time line from now to ten minutes: snapd first schedules a fallback reboot at ten minutes, stops cleanly, then schedules the real reboot one minute ahead, which replaces the fallback; snap reboot uses zero minutes

snapd first schedules a fallback reboot ten minutes ahead, then, once it has stopped cleanly, the real one: one minute ahead, or at once for an immediate request such as snap reboot.

The first broadcast you see may say the gateway is going down in ten minutes. Don't go and make coffee: it'll be gone in one. Here's why. As soon as a change asks for a reboot, snapd schedules one ten minutes ahead. That's a safety net, in case snapd gets stuck while it shuts itself down. Then snapd stops cleanly and replaces that schedule with the real one, one minute ahead, so anyone logged in has time to read the message. A request marked immediate, like snap reboot, uses zero minutes instead. So the ten-minute message is only the safety net being set.

03 The change waits for the reboot

ACROSS THE REBOOT before after $ snap changes14 Wait … Refresh "pc-kernel" snap $ snap change 14Wait … Make snap "pc-kernel" (2950) available to the system reboota new boot $ snap change 14Done … Make snap "pc-kernel" (2950) available to the system $ snap changes14 Done … Refresh "pc-kernel" snap a kernel that did not boot shows up here as a rollback
Before the reboot snap changes shows the refresh of pc-kernel in Wait and its link task in Wait; after the reboot both are Done, because snapd sees a new boot and releases the waiting task

A task that needs the reboot is put in Wait, and so is its change. After the reboot snapd sees a new boot, releases the task and completes the change; snap changes shows the result. The revision and change number here are examples.

The gateway's about to go down in the middle of a refresh. So what happens to the half-finished work? Look at snap changes before the reboot. The refresh is still open, in Wait. The task that made the new kernel the one to boot is in Wait too, and nothing after it can run until the device restarts. When the gateway comes back, snapd sees that this is a new boot, releases the waiting task and finishes the change. If the new kernel didn't boot, this is where the rollback shows up; the try deck explains it. After a reboot, snap changes tells you how the refresh ended.

04 One reboot for several snaps for Ubuntu Core developers for administrators

ONE REBOOT baselink-snap gadgetupdate-gadget-assetsupdate-gadget-cmdline kernellink-snapkernel waits for gadget restart?restart?change: pending-system-restartstored, not acted on yet link-snaprestart boundaryreboot DoneDoneDoneorder: OS, base, gadget, kernel; then the reboot, then the rest of each lane
Three lanes, base, gadget and kernel, run their pre-reboot tasks in order; the kernel waits for the gadget; restart requests are stored on the change; only the kernel's link-snap is a restart boundary, so one reboot comes before every lane finishes

Tasks record a pending restart on the change. Essential snaps are ordered OS, base, gadget, kernel, and only the last task before the reboot is a restart boundary, so one reboot covers them all.

Imagine the gateway gets a new base, a new gadget and a new kernel on the same night. Three reboots would be three chances to go wrong, so snapd plans one. The base goes first. Then the gadget updates its boot assets and the kernel command line. The kernel waits for the gadget, because a gadget can refer to assets inside the kernel, and the other order could deadlock. Any of these tasks may ask for a restart; the request is parked on the change. Only the kernel's link-snap is a restart boundary, so that's where the change stops and the one reboot happens. Afterwards every lane finishes.

05 If the reboot does not happen for administrators for Ubuntu Core developers

STILL THE SAME BOOT snapd startsa new boot? yescontinue the change noschedule the reboot againat once, then waitsnapd restartsattempt 1/3 2/33/3 give up: carry on without the rebootsnapd was restarted while a system restart was expected, snapd retried… 3 times and is giving upa refresh may roll back classic: snapd only records that a reboot is required, and waits for you
snapd starts and asks whether this is a new boot; if yes it continues the change; if no it schedules the reboot again at once and waits, up to three attempts, then gives up and carries on; on classic snapd only records that a reboot is required

If snapd restarts and the boot has not changed, it reschedules the reboot, up to three attempts, then gives up and continues. On classic systems snapd only notifies that a reboot is required.

What if someone cancels the shutdown, or snapd gets restarted before the device goes down? snapd comes back up, looks around, and realises it's still in the same boot it was trying to leave. So it asks for the reboot again, this time at once, and waits. It'll do that up to three times. After the third, it gives up. It clears the pending reboot and carries on, and the code warns that a refresh in progress might then be rolled back. On a classic system none of this applies. There, snapd only records that a reboot is required, and waits for you to do it.

06 "snap reboot: choosing the next boot yourself"

SNAP REBOOT $ sudo snap rebootReboota plain restart, now $ sudo snap reboot --recoverReboot into "recover" mode.a temporary system; your data untouched $ sudo snap reboot --factory-resetReboot into "factory-reset" mode.rebuilds the system; your data is lost$ sudo snap reboot --installReboot into "install" mode.reinstalls from the seed; the rest is erasedneither asks for confirmation $ sudo snap reboot --recover 20261112Reboot into "20261112" "recover" mode.this recovery system no label: the default recovery system,otherwise the current one can't log in? hold the 1 key while the device starts, for a menu
Five snap reboot commands and their consequences: a plain restart now; recover mode, a temporary system with data untouched; factory reset, which loses the data; install, which erases everything; and recover with a label for a specific recovery system. Without a shell, holding the 1 key at start-up gives a menu

snap reboot restarts at once. --recover, --factory-reset, --install and --run select the mode for the next boot; a label selects the recovery system, otherwise the default recovery system or the current one is used. --run only applies to the current system. The 1-key menu is console-conf's chooser, so it needs console-conf on the device and a keyboard attached.

Sometimes you want the reboot yourself, and you want the gateway to come back as something other than its usual self. Plain snap reboot is just a restart, with no minute of warning. Add recover, and the next boot is a temporary system; your data stays as it is. factory-reset and install are destructive, and the command doesn't ask twice. Put a label from snap recovery after the flag to pick a recovery system. Without one, snapd uses the default recovery system if you've set one, otherwise the current one. And if you can't log in at all, hold the 1 key while the gateway starts, for a menu.

07 Two variables and a key press for administrators for Ubuntu Core developers

WHAT IT WRITES snap reboot --recover 20261112POST /v2/systems/20261112action: reboot, mode: recoversnapd ubuntu-seed boot variablessnapd_recovery_system=20261112snapd_recovery_mode=recover restart nowseed bootloader reads themsnap debug boot-vars--root-dir /run/mnt/ubuntu-seed 1snap-bootstraprecovery-chooser-triggerwaits 10 s,hold for 2 s markerchooser (console-conf)action: do in install or factory-reset mode: no system or mode can be picked
snap reboot posts a reboot action to snapd, and a held 1 key at start-up leads through the chooser trigger and the chooser menu to the same snapd; snapd writes snapd_recovery_system and snapd_recovery_mode to the ubuntu-seed bootloader environment and restarts; the seed bootloader reads them at the next boot

snap reboot posts action: reboot to /v2/systems; the chooser posts action: do. Either way snapd writes snapd_recovery_system and snapd_recovery_mode to the ubuntu-seed bootloader environment and restarts. The chooser is triggered by holding the 1 key for two seconds within the ten seconds the trigger service waits. In install and factory-reset modes a mode or system request is refused; a plain reboot still works.

A command flag and a key held at start-up look different. Underneath, both write two words on the seed. snap reboot asks snapd's systems endpoint for a reboot. snapd writes the label and the mode into the seed bootloader's environment and restarts at once. Next boot, that bootloader reads them; snap debug boot-vars, pointed at the seed, shows them too. The key takes a detour. A small service waits ten seconds for the 1 key, held for two, and leaves a marker. A chooser then lists what snapd offers and sends your choice back. In install or factory-reset mode, neither can pick a system or a mode.

08 Where to go from here

ROADS Reboots and snap reboot Try, trying, donewhat the reboot tests The boot modesrecover, install, factory reset Recovery systemsthe labels snap reboot takes Changes and taskshow a change waits Refreshwhen refreshes happen, holds Ubuntu Corethe overview
Roads from reboots and snap reboot: try and rollback, the boot modes, recovery systems, changes and tasks, refresh scheduling, and the Ubuntu Core overview

Roads: try and rollback, the modes, recovery systems, changes and tasks, refresh scheduling and the Ubuntu Core overview.

So a reboot on Ubuntu Core is a step in a change, not an accident. Here's where to follow it. The try deck shows what the reboot is actually testing, and how a bad kernel is rolled back. The modes deck shows what recover, install and factory-reset do once you're there. Recovery systems explains the labels that snap reboot accepts. Changes and tasks covers how a change waits. If you'd rather choose when these reboots happen, the refresh deck shows how to schedule and hold refreshes. And the overview puts it all on one page.

Read more

What requests a reboot

Refresh ofReboots when
kernelalways: the new kernel must be tried
base of the modelalways: the new base must be tried
gadgetany gadget asset or the kernel command line changed; an update with nothing to change does not reboot
snapdit replaced the boot configuration it manages
an app or any other snapno reboot; the snap's own apps and services restart
a system option that changes the kernel command lineno reboot; it applies at the next one

Timings

WhatValue
fallback reboot, scheduled first10 minutes
normal reboot delay1 minute
immediate requests (snap reboot)0
attempts when the reboot did not happen3
chooser trigger wait10 seconds
how long the 1 key must be held2 seconds

The command is shutdown -r +N "reboot scheduled to update the system". Halt and power off use the messages "system halt scheduled" and "system poweroff scheduled". On a classic system snapd does not reboot; it records that a reboot is required.

snap reboot

CommandRecovery system usedEffect
snap rebootnonerestart at once
snap reboot --runthe current systemnext boot in run mode
snap reboot --recover [label]the label, else the default recovery system, else the current onenext boot in recover mode
snap reboot --factory-reset [label]same rulenext boot in factory-reset mode
snap reboot --install [label]same rulenext boot in install mode

The current system is the one the device was last seeded from: the one it was installed or factory reset from, or remodelled to. More than one mode flag gives "Please specify a single mode". The printed confirmation is Reboot, Reboot into "<mode>" mode., Reboot into "<label>". or Reboot into "<label>" "<mode>" mode. See Recovery systems for the labels and the modes for what each mode does.

Actions offered per system

SystemActions
any system that is not the current oneInstall, Recover, Factory reset
the current system, in run modeReinstall, Recover, Factory reset, Run normally
the current system, in recover modeReinstall, Factory reset, Run normally

In install and factory-reset modes no system or mode can be requested; a plain snap reboot still works. The boot-time chooser shows these actions; it starts when the 1 key is held for two seconds within the ten seconds that the trigger service waits after start-up. The menu is console-conf's recovery chooser, so it needs console-conf on the device; the trigger service runs only when an input device is present.

for administrators for Ubuntu Core developers for snapd developers

Looking at the variables

$ sudo snap debug boot-vars --root-dir /run/mnt/ubuntu-seed
snapd_recovery_mode=run
...

The output is one name=value line per variable. snap debug boot-vars without options shows the run bootloader's variables. snap reboot with a mode writes snapd_recovery_system and snapd_recovery_mode to the seed bootloader; it posts action: reboot to /v2/systems/<label>, and the chooser posts action: do.

for Ubuntu Core developers for snapd developers

How a change waits and resumes

  • A task ends with a restart request; the request is stored on the change as pending-system-restart.
  • The task goes to Wait only if it is a restart boundary.
  • The reboot is requested when the change's status changes, for example to Wait.
  • system-restart-immediate on the change makes the request immediate.
  • After the reboot, waiting tasks are released by comparing boot ids.
  • Task kinds that can request a reboot here: link-snap, update-gadget-assets, update-gadget-cmdline, update-managed-boot-config.
  • Essential snaps are ordered OS, base, gadget, kernel. The last pre-reboot task of the last of them is the restart boundary; when the refresh also refreshes the seed, the boundary is the recovery system creation task instead. UC16 systems put a boundary on every essential snap and may reboot more than once.
  • If the reboot has to be retried, the retry carries no 0 tryboot, so a Raspberry Pi kernel refresh rolls back (daemon/daemon.go:663-672).

Further reading

Roads from here

RUNNING SNAPS

  • The overlord — changes and tasks and how they wait
  • Refresh — when refreshes happen and how to hold them

SNAPCTL · the snap's API into snapd

UBUNTU CORE · boot

UBUNTU CORE · install and recovery