Reboots and snap reboot
boot mechanism for administrators for users for Ubuntu Core developers
On Ubuntu Core a refresh of the kernel or base or gadget needs a reboot. snapd schedules it one minute ahead with a ten-minute fallback and lets the change continue after the device is back. snap reboot restarts at once and can select the mode and recovery system for the next boot.
On the map: ubuntu core boot
01 Why is it rebooting?
On Ubuntu Core a refresh of the kernel or the base is completed by a reboot, and so is a gadget refresh that changes its assets or the kernel command line. App refreshes do not reboot the device; a snapd refresh reboots only when it updates the boot configuration that snapd manages.
You're logged in to the gateway and a line appears on your terminal: reboot scheduled to update the system. You didn't ask for a reboot, so who did? A refresh. The kernel, base and gadget are special: the gateway is running from them and can't swap them underneath itself. A new kernel or base has to be tried by booting it. A gadget reboots only when its update changed something on disk or the kernel command line. Everything else is gentler. When an app is refreshed, snapd restarts the app, and the device carries on. Even snapd reboots only when it rewrites the boot configuration it manages.
02 One minute, and a ten-minute safety net
snapd first schedules a fallback reboot ten minutes ahead, then, once it has stopped cleanly, the real one: one minute ahead, or at once for an immediate request such as snap reboot.
The first broadcast you see may say the gateway is going down in ten minutes. Don't go and make coffee: it'll be gone in one. Here's why. As soon as a change asks for a reboot, snapd schedules one ten minutes ahead. That's a safety net, in case snapd gets stuck while it shuts itself down. Then snapd stops cleanly and replaces that schedule with the real one, one minute ahead, so anyone logged in has time to read the message. A request marked immediate, like snap reboot, uses zero minutes instead. So the ten-minute message is only the safety net being set.
03 The change waits for the reboot
A task that needs the reboot is put in Wait, and so is its change. After the reboot snapd sees a new boot, releases the task and completes the change; snap changes shows the result. The revision and change number here are examples.
The gateway's about to go down in the middle of a refresh. So what happens to the half-finished work? Look at snap changes before the reboot. The refresh is still open, in Wait. The task that made the new kernel the one to boot is in Wait too, and nothing after it can run until the device restarts. When the gateway comes back, snapd sees that this is a new boot, releases the waiting task and finishes the change. If the new kernel didn't boot, this is where the rollback shows up; the try deck explains it. After a reboot, snap changes tells you how the refresh ended.
04 One reboot for several snaps for Ubuntu Core developers for administrators
Tasks record a pending restart on the change. Essential snaps are ordered OS, base, gadget, kernel, and only the last task before the reboot is a restart boundary, so one reboot covers them all.
Imagine the gateway gets a new base, a new gadget and a new kernel on the same night. Three reboots would be three chances to go wrong, so snapd plans one. The base goes first. Then the gadget updates its boot assets and the kernel command line. The kernel waits for the gadget, because a gadget can refer to assets inside the kernel, and the other order could deadlock. Any of these tasks may ask for a restart; the request is parked on the change. Only the kernel's link-snap is a restart boundary, so that's where the change stops and the one reboot happens. Afterwards every lane finishes.
05 If the reboot does not happen for administrators for Ubuntu Core developers
If snapd restarts and the boot has not changed, it reschedules the reboot, up to three attempts, then gives up and continues. On classic systems snapd only notifies that a reboot is required.
What if someone cancels the shutdown, or snapd gets restarted before the device goes down? snapd comes back up, looks around, and realises it's still in the same boot it was trying to leave. So it asks for the reboot again, this time at once, and waits. It'll do that up to three times. After the third, it gives up. It clears the pending reboot and carries on, and the code warns that a refresh in progress might then be rolled back. On a classic system none of this applies. There, snapd only records that a reboot is required, and waits for you to do it.
06 "snap reboot: choosing the next boot yourself"
snap reboot restarts at once. --recover, --factory-reset, --install and --run select the mode for the next boot; a label selects the recovery system, otherwise the default recovery system or the current one is used. --run only applies to the current system. The 1-key menu is console-conf's chooser, so it needs console-conf on the device and a keyboard attached.
Sometimes you want the reboot yourself, and you want the gateway to come back as something other than its usual self. Plain snap reboot is just a restart, with no minute of warning. Add recover, and the next boot is a temporary system; your data stays as it is. factory-reset and install are destructive, and the command doesn't ask twice. Put a label from snap recovery after the flag to pick a recovery system. Without one, snapd uses the default recovery system if you've set one, otherwise the current one. And if you can't log in at all, hold the 1 key while the gateway starts, for a menu.
07 Two variables and a key press for administrators for Ubuntu Core developers
snap reboot posts action: reboot to /v2/systems; the chooser posts action: do. Either way snapd writes snapd_recovery_system and snapd_recovery_mode to the ubuntu-seed bootloader environment and restarts. The chooser is triggered by holding the 1 key for two seconds within the ten seconds the trigger service waits. In install and factory-reset modes a mode or system request is refused; a plain reboot still works.
A command flag and a key held at start-up look different. Underneath, both write two words on the seed. snap reboot asks snapd's systems endpoint for a reboot. snapd writes the label and the mode into the seed bootloader's environment and restarts at once. Next boot, that bootloader reads them; snap debug boot-vars, pointed at the seed, shows them too. The key takes a detour. A small service waits ten seconds for the 1 key, held for two, and leaves a marker. A chooser then lists what snapd offers and sends your choice back. In install or factory-reset mode, neither can pick a system or a mode.
08 Where to go from here
Roads: try and rollback, the modes, recovery systems, changes and tasks, refresh scheduling and the Ubuntu Core overview.
So a reboot on Ubuntu Core is a step in a change, not an accident. Here's where to follow it. The try deck shows what the reboot is actually testing, and how a bad kernel is rolled back. The modes deck shows what recover, install and factory-reset do once you're there. Recovery systems explains the labels that snap reboot accepts. Changes and tasks covers how a change waits. If you'd rather choose when these reboots happen, the refresh deck shows how to schedule and hold refreshes. And the overview puts it all on one page.
Read more
What requests a reboot
| Refresh of | Reboots when |
|---|---|
| kernel | always: the new kernel must be tried |
| base of the model | always: the new base must be tried |
| gadget | any gadget asset or the kernel command line changed; an update with nothing to change does not reboot |
| snapd | it replaced the boot configuration it manages |
| an app or any other snap | no reboot; the snap's own apps and services restart |
| a system option that changes the kernel command line | no reboot; it applies at the next one |
Timings
| What | Value |
|---|---|
| fallback reboot, scheduled first | 10 minutes |
| normal reboot delay | 1 minute |
immediate requests (snap reboot) | 0 |
| attempts when the reboot did not happen | 3 |
| chooser trigger wait | 10 seconds |
| how long the 1 key must be held | 2 seconds |
The command is shutdown -r +N "reboot scheduled to update the system". Halt and power off use the messages "system halt scheduled" and "system poweroff scheduled". On a classic system snapd does not reboot; it records that a reboot is required.
snap reboot
| Command | Recovery system used | Effect |
|---|---|---|
snap reboot | none | restart at once |
snap reboot --run | the current system | next boot in run mode |
snap reboot --recover [label] | the label, else the default recovery system, else the current one | next boot in recover mode |
snap reboot --factory-reset [label] | same rule | next boot in factory-reset mode |
snap reboot --install [label] | same rule | next boot in install mode |
The current system is the one the device was last seeded from: the one it was installed or factory reset from, or remodelled to. More than one mode flag gives "Please specify a single mode". The printed confirmation is Reboot, Reboot into "<mode>" mode., Reboot into "<label>". or Reboot into "<label>" "<mode>" mode. See Recovery systems for the labels and the modes for what each mode does.
Actions offered per system
| System | Actions |
|---|---|
| any system that is not the current one | Install, Recover, Factory reset |
| the current system, in run mode | Reinstall, Recover, Factory reset, Run normally |
| the current system, in recover mode | Reinstall, Factory reset, Run normally |
In install and factory-reset modes no system or mode can be requested; a plain snap reboot still works. The boot-time chooser shows these actions; it starts when the 1 key is held for two seconds within the ten seconds that the trigger service waits after start-up. The menu is console-conf's recovery chooser, so it needs console-conf on the device; the trigger service runs only when an input device is present.
for administrators for Ubuntu Core developers for snapd developers
Looking at the variables
$ sudo snap debug boot-vars --root-dir /run/mnt/ubuntu-seed snapd_recovery_mode=run ...
The output is one name=value line per variable. snap debug boot-vars without options shows the run bootloader's variables. snap reboot with a mode writes snapd_recovery_system and snapd_recovery_mode to the seed bootloader; it posts action: reboot to /v2/systems/<label>, and the chooser posts action: do.
for Ubuntu Core developers for snapd developers
How a change waits and resumes
- A task ends with a restart request; the request is stored on the change as
pending-system-restart. - The task goes to
Waitonly if it is a restart boundary. - The reboot is requested when the change's status changes, for example to
Wait. system-restart-immediateon the change makes the request immediate.- After the reboot, waiting tasks are released by comparing boot ids.
- Task kinds that can request a reboot here:
link-snap,update-gadget-assets,update-gadget-cmdline,update-managed-boot-config. - Essential snaps are ordered OS, base, gadget, kernel. The last pre-reboot task of the last of them is the restart boundary; when the refresh also refreshes the seed, the boundary is the recovery system creation task instead. UC16 systems put a boundary on every essential snap and may reboot more than once.
- If the reboot has to be retried, the retry carries no
0 tryboot, so a Raspberry Pi kernel refresh rolls back (daemon/daemon.go:663-672).
Further reading
- Use a recovery mode:
snap rebootwith each flag, and the chooser menu. - Updating gadget boot assets: the gadget updates that are followed by a reboot.
- Manage updates: refresh timers and holds, which decide when a reboot can happen.
- Troubleshooting: messages seen while a refresh waits for its reboot.
Roads from here
RUNNING SNAPS
- The overlord — changes and tasks and how they wait
- Refresh — when refreshes happen and how to hold them
SNAPCTL · the snap's API into snapd
- snapctl: device and system — snapctl reboot inside the install-device hook
UBUNTU CORE · boot
- Try, trying, done — what the reboot is testing
- Run, install, recover, factory reset — what recover install and factory reset do after the reboot
- Ubuntu Core: how a device boots and recovers — back to the Ubuntu Core overview
UBUNTU CORE · install and recovery
- Recovery systems — the labels that snap reboot accepts