Preparing images and preseeding
build-time workflow for Ubuntu Core developers for snapd developers
snap prepare-image runs on a build host. It reads a model assertion then fetches and verifies the snaps the model needs and writes a seed with boot files for Ubuntu Core into a directory. Preseeding then runs snapd's first-boot seeding ahead of time in a chroot and stops before hooks so that the first real boot has less to do.
On the map: inside snapd district
01 Before any device boots
At build time snapd assembles a seed from a model assertion, and can also run first-boot seeding in advance. Both happen on the build host; the device only sees the result. Turning the directory into a disk image is the job of ubuntu-image.
Acme is about to build a thousand gateways. None of them has booted yet, and snapd has already done a good part of their first boot. Here's where. On a build machine, snapd takes the model, works out which snaps it needs, and fetches and checks them with their assertions. Then it lays out a seed and the boot files in a directory. Another tool, usually the ubuntu-image tool, runs this step and turns the directory into a disk image. Optionally, snapd also runs the first boot's seeding ahead of time, so each gateway has less to do when it's switched on.
02 What you hand to prepare-image for Ubuntu Core developers for snapd developers
snap prepare-image <model-assertion> <target-dir> takes the snaps from the model. --snap and --comp add more, --channel sets the channel, --revisions and --write-revisions pin and record exact revisions, and --assert adds assertions.
The ubuntu-image tool usually runs this for you. But when a build picks up a revision you didn't expect, you'll want to know which knob it had and didn't turn. Two arguments are required: the model and a target directory. The model already names snapd, the kernel, the base, the gadget and the required snaps, so most of the image is decided. the --snap option adds more, from the store or a local file. Channel picks the channel. --write-revisions records which revision of each snap went in, and the --revisions option feeds that record back, for the next build. Assert adds assertions, within limits.
03 Fetch, check, copy for Ubuntu Core developers for snapd developers
prepare-image refuses a model that does not match --classic and a target that already holds a system. It then fetches each snap, verifies its assertions and the validation sets, copies the snaps, writes the seed and, for Ubuntu Core, makes the directory bootable. The help text gives the --validation default as ignore but the code uses enforce: pass --validation=enforce or --validation=ignore explicitly.
Nothing lands in the seed just because it was downloaded. Watch the order: two refusals, then fetch, then check, and only then copy. A classic model without the classic option stops it, and so does the reverse, or a target that already holds a system. It opens an assertion database in memory with the trusted keys, downloads each snap, and cross-checks its assertions. It checks the model's validation sets, in the mode it printed at the start. Pass the --validation option yourself: the help text says ignore, but the code enforces. Only then does it copy the snaps and write the seed. For Ubuntu Core, it also makes the result bootable.
04 Three layouts in the target directory for Ubuntu Core developers for snapd developers
Classic: the seed is written to var/lib/snapd/seed under the target. Core 16 and 18: under image/, with gadget/ and kernel/ beside it. Core 20 and later: system-seed/, with gadget/ and kernel/ beside it, and one system labelled by the UTC build date, made bootable as a recovery system.
Run it for the gateway's model and look in the output directory. There's no root filesystem in there. What you find depends on the model. For a classic model, the target is the root filesystem, and the seed goes under var/lib/snapd/seed. For Ubuntu Core 16 or 18, the root is a subdirectory called image, with the gadget and kernel unpacked beside it. For Ubuntu Core 20 and later, our gateway, there's system-seed, with a pool of snaps and one system named after the build's UTC date, and the gadget and kernel unpacked beside it. It's made bootable as a recovery system, and a system-seed that already holds a system is refused.
05 Seeding ahead of time for Ubuntu Core developers for snapd developers for administrators
snap-preseed <chroot> runs as root, requires /proc, /dev and /sys/kernel/security mounted in the chroot and no state.json, mounts the seed's snapd at /tmp/snapd-preseed, and runs snapd with SNAPD_PRESEED=1. Seeding stops at mark-preseeded, before any hook.
Every instance started from your cloud image does the same seeding on its first boot. Do it once, on your own machine. As root, point the snap-preseed tool at the image's root. It wants proc, dev and the security filesystem mounted inside, and no state file yet. It enters the chroot, mounts the seed's snapd snap, and runs the newer of that and the packaged snapd, with SNAPD_PRESEED=1. It runs the ordinary seeding change, mounting snaps, writing units and profiles, connecting. At mark-preseeded it records that, asks to be stopped and leaves the task unfinished. No hook has run; first boot picks up there.
06 The same for Ubuntu Core for Ubuntu Core developers for snapd developers
snap prepare-image --preseed works for models with base core20 or later. It runs snapd with SNAPD_PRESEED=1 in a chroot built from the seed's base and snapd snaps and writes preseed.tgz and a signed preseed assertion for the system. --preseed-sign-key names the signing key; what install does with them is in the seed deck.
Scene four left us with a puzzle. The gateway's image has no root filesystem at build time. So where would a preseeding snapd run? For Ubuntu Core, preseeding is an option of snap prepare-image, run after the seed is written. It's refused for a classic model and for a base older than core20. It builds a temporary chroot from the base and snapd snaps in the seed, and runs that snapd in preseed mode, as before. A preseed.tgz and a preseed assertion, signed with the key you named, land in the system's directory. Building on another architecture? The error suggests qemu-user-static.
07 Undoing it: what reset removes for Ubuntu Core developers for snapd developers for administrators
A second run fails with the system at ... appears to be preseeded, pass --reset flag to clean it up. snap-preseed --reset <chroot> removes the state, units, profiles, snap files and assertion database that seeding created and keeps the seed. It is refused for Ubuntu Core image directories.
You've preseeded the image, and then someone hands you a newer snap for the seed. Run the tool again and it stops you: a state file means the image looks preseeded. Good. the --reset option is the way back, and its list of deletions is the best inventory of what preseeding made. The state file and the system key. The snaps' mount, service, timer and socket units. The AppArmor and seccomp profiles. The snap files, desktop files, assertion database and device directory. The seed isn't listed, so you can preseed again. Ubuntu Core directories refuse a reset; there, run prepare-image again into an empty directory.
08 Where to go from here
Roads: the model and snap assertions, the seed and install mode, snap debug, and the decks about the daemon.
Everything here happened on a build machine. Once the image leaves it, the rest of the journey is elsewhere on the map. The model deck explains the document that drives all of this, and the snap assertions deck the checks made on every snap. The seed deck picks up where this one stops: what a recovery system holds, how first boot checks it, and how a preseeded system is applied. The install deck shows the device's side. Snap debug can read the state file of a preseeded image without booting it. And the restart deck shows the stop request that ends a preseed run.
Read more
snap prepare-image options
| Option | Effect |
|---|---|
<model-assertion> <target-dir> | required: the model file and the directory to write |
--classic | prepare a classic image; required for a classic model and refused for a core model |
--arch | the architecture for --classic when the model has none |
--channel | the channel to use |
--snap <snap>[=<channel>] | add a snap from the store or a local file, or set the channel of a snap |
--comp <snap>+<comp> | add a component from the store or a local file |
--revisions <file> | install exactly the revisions listed in a seed manifest |
--write-revisions[=<path>] | write a manifest of the revisions used; default ./seed.manifest |
--validation=ignore or enforce | how validation sets are treated; see the note below |
--assert <file> | add an assertion from a local file |
--allow-snapd-kernel-mismatch | accept a snapd snap that differs from the snapd inside the kernel snap |
--preseed | preseed the image; Ubuntu Core 20 and later |
--preseed-sign-key | the key that signs the preseed assertion; needs --preseed |
--apparmor-features-dir | an AppArmor kernel features directory to use while preseeding |
--sysfs-overlay | a sysfs overlay to use while preseeding; needs --preseed |
The environment variable UBUNTU_STORE_COHORT_KEY sets a cohort key for the downloads.
Validation: the help text of --validation says the default is ignore, but the code uses enforce when the option is absent and prints INFO: validation mode: enforce. The default changed on purpose and the help text was not updated. Pass the option explicitly and check that line.
Extra assertions: only some assertion types are accepted (assertion type ... is not allowed for extra assertions). A system-user assertion is accepted only for a model of grade dangerous and only without a password.
What it refuses
| Message | Cause |
|---|---|
--classic mode is required to prepare the image for a classic model | classic model, no --classic |
cannot prepare the image for a core model with --classic mode specified | core model with --classic |
cannot override model architecture: ... | --arch differs from the model |
cannot have snaps for a classic image without an architecture in the model or from --arch | classic image with snaps and no architecture |
model with series ... unsupported | the model's series is not the supported one |
cannot prepare seed over existing system or an already booted image, detected state file ... | the target already has a state file |
expected empty snap dir in rootdir, got: ... | the target already holds snap files |
expected empty systems dir in system-seed, got: ... | the target already has a recovery system |
--preseed-sign-key cannot be used without --preseed | as it says; the same for --sysfs-overlay |
cannot preseed the image for a classic model | --preseed with a classic model |
cannot preseed the image for older base than core20 | --preseed with a core or core18 base |
It also warns: WARNING: ... installed from local snaps disconnected from a store cannot be refreshed subsequently!
What is written
| Model | Seed | Also written |
|---|---|---|
| classic | <target>/var/lib/snapd/seed | nothing else; a warning if the content is not owned by root |
| Core 16 and 18 | <target>/image/var/lib/snapd/seed | <target>/gadget, <target>/kernel, boot files, the gadget's cloud configuration and defaults |
| Core 20 and later | <target>/system-seed, system label YYYYMMDD (UTC) | <target>/gadget, <target>/kernel, boot files for the recovery system |
What a recovery system contains is explained in ubuntu-seed and first boot.
snap-preseed
The program is /usr/lib/snapd/snap-preseed. It must run as root and takes the chroot directory as its argument; / is refused.
| Invocation | What runs |
|---|---|
snap-preseed <dir> where <dir> contains system-seed | Ubuntu Core 20 and later preseeding, as prepare-image --preseed does |
snap-preseed <dir> | classic preseeding |
snap-preseed --reset <dir> | classic reset; refused for an Ubuntu Core directory |
snap-preseed --hybrid --system-label <label> <dir> | hybrid preseeding; each of the two options needs the other |
snap-preseed --hybrid --system-label <label> --reset <dir> | hybrid reset |
Requirements for a classic chroot: no state file yet; /proc, /dev and /sys/kernel/security mounted inside it; AppArmor's directory present under /sys/kernel/security. The snapd that will run must be at least 2.43.3.
| Message | Cause |
|---|---|
must be run as root | not root |
need chroot path as argument | no directory given |
cannot run snap-preseed against / | the directory is the running system |
the system at ... appears to be preseeded, pass --reset flag to clean it up | a state file exists |
cannot preseed without the following mountpoints: | a required mount is missing |
cannot snap-preseed --reset for Ubuntu Core | reset on a directory with system-seed |
error running snapd, please try installing the "qemu-user-static" package: ... | the image has a different architecture |
--sysfs-overlay accepts only these paths: sys/class/backlight, sys/class/bluetooth, sys/class/gpio, sys/class/leds, sys/class/ptp, sys/class/pwm, sys/class/rtc, sys/class/video4linux, sys/devices/platform, sys/devices/pci0000:00.
Looking at a preseeded classic image
$ snap debug state /mnt/cloudimg/var/lib/snapd/state.json --change=1
The task mark-preseeded is in Doing; the tasks before it are Done. Hooks run on the first boot, after that task completes.
What reset removes
- Files: the state file, the system key, the snap files, snap udev rules, D-Bus system policy files of snaps, the system units of snaps (services, timers, sockets and mount units with their links), user units of snaps with their links, and run-inhibit locks.
- Directory contents: snap data, the cache, the AppArmor cache, desktop files, D-Bus service files, and everything under the snap mount directory.
- Whole directories: the assertion database, feature flags, desktop icons, the device directory, cookies, mount profiles, AppArmor profiles, sequence files and seccomp profiles.
- Shell completion links that point at snapd's completion script.
The seed is not removed.
for snapd developers
Preseed mode inside snapd
- Preseed mode is the ordinary daemon started with
SNAPD_PRESEED=1; hybrid addsSNAPD_PRESEED_HYBRID=1. - The daemon skips re-execution in this mode.
- At
mark-preseededthe handler unmounts all snaps, storespreseeded,preseed-system-keyandpreseed-timein the state, requests thestop-daemonrestart type and returns a retry, so the task stays unfinished until a normal start. - For reset, a target snapd of 2.59 or later is asked to reset itself: the host tool enters the chroot and runs the target's
snap-preseed --reset-chroot.
Further reading
Only the preseed how-to covers this area; no published page explains snap prepare-image beyond it.
- Optimise boot speed:
snap prepare-image --preseedand its requirements. - Preseed performance: measured first-boot times with and without preseeding.
- Use ubuntu-image: the tool that calls the same image code and writes the disk image.
- model assertion: the main input of
prepare-image. - Add a system user: an extra assertion added at image time.
Roads from here
ASSERTIONS · signed statements snapd trusts
- Assertions: snaps and validation — the assertions fetched and checked for every snap
TOOLS
- snap debug: tools for looking inside — snap debug state for looking inside a preseeded image
UBUNTU CORE · install and recovery
- ubuntu-seed and first boot — what is inside the seed and how first boot uses it
- Model, gadget and kernel — the model assertion that drives prepare-image
- Install mode — what install mode does with the finished image
INSIDE SNAPD · the daemon itself
- How snapd restarts and re-executes — the stop-daemon restart that ends a preseed run
- The snapd REST API — back to the daemon's front door