Preparing images and preseeding

build-time workflow for Ubuntu Core developers for snapd developers

snap prepare-image runs on a build host. It reads a model assertion then fetches and verifies the snaps the model needs and writes a seed with boot files for Ubuntu Core into a directory. Preseeding then runs snapd's first-boot seeding ahead of time in a chroot and stops before hooks so that the first real boot has less to do.

On the map: inside snapd district

▶ Watch this deck — 8 scenes, ~6 min

01 Before any device boots

AT THE BUILD HOST build host modelsigned bythe brand prepare the imagesnapdkernelacme-agentfetched and verified a directoryseed and boot files turning it into a disk imageis another tool's job preseedoptional: first-boot work, done now devicefirst boot ofevery unitless left to do
On the build host a signed model goes into a prepare step that fetches and verifies snapd, the kernel and acme-agent and writes a directory with the seed and boot files; another tool turns it into a disk image; an optional preseed step does first-boot work on that directory; across the divider every unit's first boot has less left to do

At build time snapd assembles a seed from a model assertion, and can also run first-boot seeding in advance. Both happen on the build host; the device only sees the result. Turning the directory into a disk image is the job of ubuntu-image.

Acme is about to build a thousand gateways. None of them has booted yet, and snapd has already done a good part of their first boot. Here's where. On a build machine, snapd takes the model, works out which snaps it needs, and fetches and checks them with their assertions. Then it lays out a seed and the boot files in a directory. Another tool, usually the ubuntu-image tool, runs this step and turns the directory into a disk image. Optionally, snapd also runs the first boot's seeding ahead of time, so each gateway has less to do when it's switched on.

02 What you hand to prepare-image for Ubuntu Core developers for snapd developers

INPUTS $ snap prepare-image --channel stable --snap acme-agent \--write-revisions acme-gateway.model ./outprepare-image model assertionnames snapd, kernel,base, gadget andrequired snaps extra snaps--snap, --compa store name,name=channel or a file channel--channel stable revisions--revisionsreads a manifest--write-revisionswrites ./seed.manifest extra assertions--assert <file>only some types
The command snap prepare-image --channel stable --snap acme-agent --write-revisions acme-gateway.model ./out, broken into five inputs that flow into prepare-image: the model assertion, extra snaps, the channel, a revisions manifest read with --revisions and written with --write-revisions, and extra assertions from --assert

snap prepare-image <model-assertion> <target-dir> takes the snaps from the model. --snap and --comp add more, --channel sets the channel, --revisions and --write-revisions pin and record exact revisions, and --assert adds assertions.

The ubuntu-image tool usually runs this for you. But when a build picks up a revision you didn't expect, you'll want to know which knob it had and didn't turn. Two arguments are required: the model and a target directory. The model already names snapd, the kernel, the base, the gadget and the required snaps, so most of the image is decided. the --snap option adds more, from the store or a local file. Channel picks the channel. --write-revisions records which revision of each snap went in, and the --revisions option feeds that record back, for the next build. Assert adds assertions, within limits.

03 Fetch, check, copy for Ubuntu Core developers for snapd developers

THE STEPS model matches --classic?--classic mode is required to prepare the image for a classic model target holds no system?cannot prepare seed over existing system ... fetchlocal snaps:used as givenFetching acme-agent (42) verify assertionsand validation setsINFO: validation mode: enforcehelp says ignore, code enforces: pass --validation copy, write the seedCopying "/.../acme-agent_42.snap" (acme-agent) make it bootableUbuntu Core only
prepare-image as a pipeline: it refuses a model that does not match --classic and a target that already holds a system, then fetches each snap, verifies assertions and validation sets, copies the snaps and writes the seed, and for Ubuntu Core makes it bootable; a terminal shows INFO: validation mode: enforce, Fetching acme-agent (42) and a Copying line

prepare-image refuses a model that does not match --classic and a target that already holds a system. It then fetches each snap, verifies its assertions and the validation sets, copies the snaps, writes the seed and, for Ubuntu Core, makes the directory bootable. The help text gives the --validation default as ignore but the code uses enforce: pass --validation=enforce or --validation=ignore explicitly.

Nothing lands in the seed just because it was downloaded. Watch the order: two refusals, then fetch, then check, and only then copy. A classic model without the classic option stops it, and so does the reverse, or a target that already holds a system. It opens an assertion database in memory with the trusted keys, downloads each snap, and cross-checks its assertions. It checks the model's validation sets, in the mode it printed at the start. Pass the --validation option yourself: the help text says ignore, but the code enforces. Only then does it copy the snaps and write the seed. For Ubuntu Core, it also makes the result bootable.

04 Three layouts in the target directory for Ubuntu Core developers for snapd developers

OUTPUT classic model./out/var/lib/snapd/seed/the target is theroot filesystem Core 16 and 18./out/image/var/lib/snapd/seed/./out/gadget/./out/kernel/image/ is the root Core 20 and later./out/system-seed/snaps/systems/20260301/./out/gadget/./out/kernel/no root filesystem label: the build date, UTC bootable as a recovery systemexpected empty systems dir in system-seedif a system is already there
Three target directory layouts: for a classic model the seed is in ./out/var/lib/snapd/seed; for Core 16 and 18 it is in ./out/image/var/lib/snapd/seed beside ./out/gadget and ./out/kernel; for Core 20 and later there is ./out/system-seed with snaps and systems/20260301, labelled by the UTC build date, bootable as a recovery system, and a non-empty systems directory is refused

Classic: the seed is written to var/lib/snapd/seed under the target. Core 16 and 18: under image/, with gadget/ and kernel/ beside it. Core 20 and later: system-seed/, with gadget/ and kernel/ beside it, and one system labelled by the UTC build date, made bootable as a recovery system.

Run it for the gateway's model and look in the output directory. There's no root filesystem in there. What you find depends on the model. For a classic model, the target is the root filesystem, and the seed goes under var/lib/snapd/seed. For Ubuntu Core 16 or 18, the root is a subdirectory called image, with the gadget and kernel unpacked beside it. For Ubuntu Core 20 and later, our gateway, there's system-seed, with a pool of snaps and one system named after the build's UTC date, and the gadget and kernel unpacked beside it. It's made bootable as a recovery system, and a system-seed that already holds a system is refused.

05 Seeding ahead of time for Ubuntu Core developers for snapd developers for administrators

PRESEED $ sudo /usr/lib/snapd/snap-preseed /mnt/cloudimg chroot: /mnt/cloudimg/proc/dev/sys/kernel/securitymounted;no state.json yet snapd snap from the seed/tmp/snapd-preseedthe newer of this and the packaged snapd runs snapdSNAPD_PRESEED=1starting to preseed root: /mnt/cloudimgusing snapd binary: /tmp/snapd-preseed/usr/lib/snapd/snapd (2.78) the seeding changeDoneDoneDone mark-preseededDoingstop-daemonmounts removed; no hook has run
sudo snap-preseed /mnt/cloudimg: inside the chroot /proc, /dev and /sys/kernel/security must be mounted and there is no state.json; the seed's snapd snap is mounted at /tmp/snapd-preseed and the newer snapd runs with SNAPD_PRESEED=1; the seeding change runs its tasks until mark-preseeded, which stays in Doing, and a stop-daemon request ends the run before any hook

snap-preseed <chroot> runs as root, requires /proc, /dev and /sys/kernel/security mounted in the chroot and no state.json, mounts the seed's snapd at /tmp/snapd-preseed, and runs snapd with SNAPD_PRESEED=1. Seeding stops at mark-preseeded, before any hook.

Every instance started from your cloud image does the same seeding on its first boot. Do it once, on your own machine. As root, point the snap-preseed tool at the image's root. It wants proc, dev and the security filesystem mounted inside, and no state file yet. It enters the chroot, mounts the seed's snapd snap, and runs the newer of that and the packaged snapd, with SNAPD_PRESEED=1. It runs the ordinary seeding change, mounting snaps, writing units and profiles, connecting. At mark-preseeded it records that, asks to be stopped and leaves the task unfinished. No hook has run; first boot picks up there.

06 The same for Ubuntu Core for Ubuntu Core developers for snapd developers

--PRESEED $ snap prepare-image --preseed --preseed-sign-key acme-key \acme-gateway.model ./out cannot preseed the image for a classic modelcannot preseed the image for older base than core20 temporary chrootbasesnapdfrom system-seed snapdSNAPD_PRESEED=1starting to preseed UC20+ system: ...stops at mark-preseeded, as on classic system-seed/systems/20260301/preseed.tgzpreseed assertionsigned with the key you namedapplied at install: see the seed deck error running snapd, please try installing the "qemu-user-static" package
snap prepare-image --preseed --preseed-sign-key acme-key acme-gateway.model ./out; refused for a classic model or a base older than core20; a temporary chroot built from the seed's base and snapd snaps runs snapd with SNAPD_PRESEED=1; preseed.tgz and a preseed assertion signed with the named key land in system-seed/systems/20260301; on another architecture the error suggests qemu-user-static

snap prepare-image --preseed works for models with base core20 or later. It runs snapd with SNAPD_PRESEED=1 in a chroot built from the seed's base and snapd snaps and writes preseed.tgz and a signed preseed assertion for the system. --preseed-sign-key names the signing key; what install does with them is in the seed deck.

Scene four left us with a puzzle. The gateway's image has no root filesystem at build time. So where would a preseeding snapd run? For Ubuntu Core, preseeding is an option of snap prepare-image, run after the seed is written. It's refused for a classic model and for a base older than core20. It builds a temporary chroot from the base and snapd snaps in the seed, and runs that snapd in preseed mode, as before. A preseed.tgz and a preseed assertion, signed with the key you named, land in the system's directory. Building on another architecture? The error suggests qemu-user-static.

07 Undoing it: what reset removes for Ubuntu Core developers for snapd developers for administrators

RESET $ sudo /usr/lib/snapd/snap-preseed /mnt/cloudimgerror: the system at "/mnt/cloudimg" appears to be preseeded, pass --reset flagto clean it up $ sudo /usr/lib/snapd/snap-preseed --reset /mnt/cloudimg/mnt/cloudimgstate.json and the system keymount, service, timer and socket unitsAppArmor and seccomp profilessnap files, desktop files, assertion databasethe seed the seedkept: preseed again a directory with system-seed (Ubuntu Core): reset refused; run prepare-image again into an empty directory
A second snap-preseed run fails because the image appears to be preseeded; snap-preseed --reset removes state.json and the system key, the mount, service, timer and socket units, the AppArmor and seccomp profiles, and the snap files, desktop files and assertion database; the seed is kept so the image can be preseeded again; reset is refused for Ubuntu Core image directories

A second run fails with the system at ... appears to be preseeded, pass --reset flag to clean it up. snap-preseed --reset <chroot> removes the state, units, profiles, snap files and assertion database that seeding created and keeps the seed. It is refused for Ubuntu Core image directories.

You've preseeded the image, and then someone hands you a newer snap for the seed. Run the tool again and it stops you: a state file means the image looks preseeded. Good. the --reset option is the way back, and its list of deletions is the best inventory of what preseeding made. The state file and the system key. The snaps' mount, service, timer and socket units. The AppArmor and seccomp profiles. The snap files, desktop files, assertion database and device directory. The seed isn't listed, so you can preseed again. Ubuntu Core directories refuse a reset; there, run prepare-image again into an empty directory.

08 Where to go from here

ROADS Preparing imagesand preseeding what goes inModel, gadget and kernelthe document that drives itAssertions: snaps and validationchecks made on every snap what the device does with itubuntu-seed and first bootrecovery systems and preseedingInstall modethe device's side of first boot looking insidesnap debugan image's state, without booting the daemonRestart and re-execthe stop that ends a preseed runThe REST APIthe daemon's front door
Roads from Preparing images and preseeding: what goes in (the model, snap assertions), what the device does with it (ubuntu-seed and first boot, install mode), looking inside (snap debug) and the daemon (restart and re-exec, the REST API)

Roads: the model and snap assertions, the seed and install mode, snap debug, and the decks about the daemon.

Everything here happened on a build machine. Once the image leaves it, the rest of the journey is elsewhere on the map. The model deck explains the document that drives all of this, and the snap assertions deck the checks made on every snap. The seed deck picks up where this one stops: what a recovery system holds, how first boot checks it, and how a preseeded system is applied. The install deck shows the device's side. Snap debug can read the state file of a preseeded image without booting it. And the restart deck shows the stop request that ends a preseed run.

Read more

snap prepare-image options

OptionEffect
<model-assertion> <target-dir>required: the model file and the directory to write
--classicprepare a classic image; required for a classic model and refused for a core model
--archthe architecture for --classic when the model has none
--channelthe channel to use
--snap <snap>[=<channel>]add a snap from the store or a local file, or set the channel of a snap
--comp <snap>+<comp>add a component from the store or a local file
--revisions <file>install exactly the revisions listed in a seed manifest
--write-revisions[=<path>]write a manifest of the revisions used; default ./seed.manifest
--validation=ignore or enforcehow validation sets are treated; see the note below
--assert <file>add an assertion from a local file
--allow-snapd-kernel-mismatchaccept a snapd snap that differs from the snapd inside the kernel snap
--preseedpreseed the image; Ubuntu Core 20 and later
--preseed-sign-keythe key that signs the preseed assertion; needs --preseed
--apparmor-features-diran AppArmor kernel features directory to use while preseeding
--sysfs-overlaya sysfs overlay to use while preseeding; needs --preseed

The environment variable UBUNTU_STORE_COHORT_KEY sets a cohort key for the downloads.

Validation: the help text of --validation says the default is ignore, but the code uses enforce when the option is absent and prints INFO: validation mode: enforce. The default changed on purpose and the help text was not updated. Pass the option explicitly and check that line.

Extra assertions: only some assertion types are accepted (assertion type ... is not allowed for extra assertions). A system-user assertion is accepted only for a model of grade dangerous and only without a password.

What it refuses

MessageCause
--classic mode is required to prepare the image for a classic modelclassic model, no --classic
cannot prepare the image for a core model with --classic mode specifiedcore model with --classic
cannot override model architecture: ...--arch differs from the model
cannot have snaps for a classic image without an architecture in the model or from --archclassic image with snaps and no architecture
model with series ... unsupportedthe model's series is not the supported one
cannot prepare seed over existing system or an already booted image, detected state file ...the target already has a state file
expected empty snap dir in rootdir, got: ...the target already holds snap files
expected empty systems dir in system-seed, got: ...the target already has a recovery system
--preseed-sign-key cannot be used without --preseedas it says; the same for --sysfs-overlay
cannot preseed the image for a classic model--preseed with a classic model
cannot preseed the image for older base than core20--preseed with a core or core18 base

It also warns: WARNING: ... installed from local snaps disconnected from a store cannot be refreshed subsequently!

What is written

ModelSeedAlso written
classic<target>/var/lib/snapd/seednothing else; a warning if the content is not owned by root
Core 16 and 18<target>/image/var/lib/snapd/seed<target>/gadget, <target>/kernel, boot files, the gadget's cloud configuration and defaults
Core 20 and later<target>/system-seed, system label YYYYMMDD (UTC)<target>/gadget, <target>/kernel, boot files for the recovery system

What a recovery system contains is explained in ubuntu-seed and first boot.

snap-preseed

The program is /usr/lib/snapd/snap-preseed. It must run as root and takes the chroot directory as its argument; / is refused.

InvocationWhat runs
snap-preseed <dir> where <dir> contains system-seedUbuntu Core 20 and later preseeding, as prepare-image --preseed does
snap-preseed <dir>classic preseeding
snap-preseed --reset <dir>classic reset; refused for an Ubuntu Core directory
snap-preseed --hybrid --system-label <label> <dir>hybrid preseeding; each of the two options needs the other
snap-preseed --hybrid --system-label <label> --reset <dir>hybrid reset

Requirements for a classic chroot: no state file yet; /proc, /dev and /sys/kernel/security mounted inside it; AppArmor's directory present under /sys/kernel/security. The snapd that will run must be at least 2.43.3.

MessageCause
must be run as rootnot root
need chroot path as argumentno directory given
cannot run snap-preseed against /the directory is the running system
the system at ... appears to be preseeded, pass --reset flag to clean it upa state file exists
cannot preseed without the following mountpoints:a required mount is missing
cannot snap-preseed --reset for Ubuntu Corereset on a directory with system-seed
error running snapd, please try installing the "qemu-user-static" package: ...the image has a different architecture

--sysfs-overlay accepts only these paths: sys/class/backlight, sys/class/bluetooth, sys/class/gpio, sys/class/leds, sys/class/ptp, sys/class/pwm, sys/class/rtc, sys/class/video4linux, sys/devices/platform, sys/devices/pci0000:00.

Looking at a preseeded classic image

$ snap debug state /mnt/cloudimg/var/lib/snapd/state.json --change=1

The task mark-preseeded is in Doing; the tasks before it are Done. Hooks run on the first boot, after that task completes.

What reset removes

  • Files: the state file, the system key, the snap files, snap udev rules, D-Bus system policy files of snaps, the system units of snaps (services, timers, sockets and mount units with their links), user units of snaps with their links, and run-inhibit locks.
  • Directory contents: snap data, the cache, the AppArmor cache, desktop files, D-Bus service files, and everything under the snap mount directory.
  • Whole directories: the assertion database, feature flags, desktop icons, the device directory, cookies, mount profiles, AppArmor profiles, sequence files and seccomp profiles.
  • Shell completion links that point at snapd's completion script.

The seed is not removed.

for snapd developers

Preseed mode inside snapd

  • Preseed mode is the ordinary daemon started with SNAPD_PRESEED=1; hybrid adds SNAPD_PRESEED_HYBRID=1.
  • The daemon skips re-execution in this mode.
  • At mark-preseeded the handler unmounts all snaps, stores preseeded, preseed-system-key and preseed-time in the state, requests the stop-daemon restart type and returns a retry, so the task stays unfinished until a normal start.
  • For reset, a target snapd of 2.59 or later is asked to reset itself: the host tool enters the chroot and runs the target's snap-preseed --reset-chroot.

Further reading

Only the preseed how-to covers this area; no published page explains snap prepare-image beyond it.

Roads from here

ASSERTIONS · signed statements snapd trusts

TOOLS

UBUNTU CORE · install and recovery

INSIDE SNAPD · the daemon itself