Quota groups

managing snaps for administrators for Ubuntu Core developers

A quota group is a named set of snaps or of services of one snap with limits on memory and CPU and threads and journal use. snapd writes a systemd slice for the group and places the snaps' service units in it. Groups nest and a sub-group may hold single services. Memory and thread limits can be raised but not lowered and a snap leaves a group only when the group is removed.

On the map: administration district

▶ Watch this deck — 7 scenes, ~5 min

01 A ceiling for the agent

QUOTA GROUPS gateway: 1 GB of memoryacme-agentmemorymemory keeps growing 256MBa name, some limits, some snapsgroup: agents $ sudo snap set-quota agents \--memory=256MB --cpu=50% acme-agentneeds root $ snap quotasQuotaParentConstraintsCurrentagentsmemory=256MB,cpu=50%memory=41.2MB six limits: memory, cpu, cpu-set, threads, journal-size, journal-rate-limit · at least one is required
A gateway with 1 GB of memory runs acme-agent, whose memory use keeps growing. A dashed outline around the snap marks the quota group agents with a ceiling at 256MB. The command sudo snap set-quota agents --memory=256MB --cpu=50% acme-agent creates it, and snap quotas lists agents with constraints memory=256MB,cpu=50% and current memory=41.2MB. A group has up to six limits and needs at least one

snap set-quota <group> <limits> <snap>... creates or updates a quota group; snap quotas lists groups with their constraints and current use. Type the CPU value exactly as 50% or 2x100% and check it with snap quota <group>: text around a valid value is ignored rather than rejected. The gateway's 1 GB and four cores are invented.

The gateway has a gigabyte of memory, and acme-agent has a leak. Left alone it takes the lot, and the device stops answering. What you want is a ceiling. That's a quota group: a name, some limits, and the snaps they apply to. Think of a fence with the limits written on it. One command makes it. snap set-quota, the group name agents, two hundred and fifty six megabytes and half the CPU, then the snap. It needs root. Snap quotas lists the group, its limits and what it's using right now. There are six limits in all, and a group needs at least one.

02 A group becomes a slice for administrators for Ubuntu Core developers

WHAT SNAPD WRITES snapdwrites/etc/systemd/system/snap.agents.slice[Unit]Description=Slice for snap quota group agentsBefore=slices.targetX-Snappy=yes [Slice]CPUAccounting=trueCPUQuota=200%50% x 4 cores MemoryAccounting=trueMemoryMax=256000000MemoryLimit=256000000256MB in bytesfor older systemd TasksAccounting=trueTasksMax= with --threadsAllowedCPUs= with --cpu-set accounting: always onsystemdcontrol group $ systemctl show --property=ActiveState snap.agents.sliceActiveState=active
snapd writes /etc/systemd/system/snap.agents.slice. Under [Slice] it sets CPUAccounting=true and CPUQuota=200%, which is 50% times 4 cores, MemoryAccounting=true, MemoryMax=256000000 and MemoryLimit=256000000 for older systemd, and TasksAccounting=true; TasksMax= appears with --threads and AllowedCPUs= with --cpu-set. systemd loads the slice into a control group, and systemctl show --property=ActiveState snap.agents.slice prints ActiveState=active

A top-level group is the unit snap.<group>.slice in /etc/systemd/system, with CPUQuota=, AllowedCPUs=, MemoryMax= and TasksMax= taken from the group's limits. A plain percentage is multiplied by the number of CPUs (or of CPUs in the set). Sizes are decimal: 256MB is 256000000 bytes. The gateway's 1 GB and four cores are invented.

So who actually holds acme-agent under two hundred and fifty six megabytes? Not snapd. Its part is one file: a systemd slice unit named after the group, snap.agents.slice. The limits you typed become the slice's settings. CPUQuota= is your fifty percent times the gateway's four cores. MemoryMax= is in bytes, written twice, once for older systemd. A thread limit would add TasksMax=, and a CPU set AllowedCPUs=. Accounting is always on. From there it's systemd and the kernel's control groups that do the work, so you can look at a group with a tool you already know: systemctl show.

03 The services move in for administrators for Ubuntu Core developers

MEMBERSHIP snap.agents.sliceacme-agent.agentacme-agent.reportbefore: in system.slice like any service snap.acme-agent.agent.service[Service]Slice=snap.agents.slice restartedacme-agent.agentacme-agent.reportservices that are not disabled restart $ sudo snap install --quota-group=agents acme-agentjoins before its services start $ systemctl show --property=ControlGroup snap.acme-agent.agent.serviceControlGroup=/snap.agents.slice/snap.acme-agent.agent.service acme-agent, run by handsnap run: a scope outsidethe slice; not limited
The services acme-agent.agent and acme-agent.report start outside the empty slice snap.agents.slice, in system.slice like any service. snapd adds Slice=snap.agents.slice to the [Service] section of each unit and restarts the services that are not disabled, which moves them into the slice; snap install --quota-group=agents joins a snap before its services start. systemctl show --property=ControlGroup prints /snap.agents.slice/snap.acme-agent.agent.service. A command run by hand with snap run, or a hook, gets a scope of its own outside the slice and is not limited

snapd adds Slice=snap.<group>.slice to every service unit of a member snap and restarts the services that are not disabled. snap install --quota-group <group> joins at install. Commands started with snap run and hooks run in their own scope outside the slice and are not limited.

A slice with nothing in it limits nothing. So how does acme-agent get inside? snapd rewrites each of its service units with one more line: Slice=snap.agents dot slice. A running process isn't moved by editing a file, so snapd restarts the snap's services that aren't disabled. Plan for that blip on a live gateway. Install with the --quota-group flag and the snap joins before its services start. Afterwards systemctl shows the agent's control group under the slice. That's services, though. A command someone runs by hand, or a hook, gets a scope of its own outside the slice, so the group's limits don't reach it.

04 Groups inside groups for administrators for Ubuntu Core developers

NESTING agents · memory=256MB,cpu=50%acme-agentagentreport $ sudo snap set-quota reports \--parent=agents --cpu=10% \acme-agent.report reports · cpu=10%snap.agents-reports.slice sub-group limits must fit the parentsub-group cpu limit of ...% is too large to fit inside group "agents"remaining quota space ...% a service's snap is in the direct parent one level under a group with snaps,and it holds services, not snapsgroups without snaps can nest deeper
The group agents, memory=256MB and cpu=50%, holds the snap acme-agent with its services agent and report. sudo snap set-quota reports --parent=agents --cpu=10% acme-agent.report creates the sub-group reports around the report service only, with the slice snap.agents-reports.slice. Three rules: sub-group limits must fit the parent, a service's snap must be in the direct parent group, and under a group with snaps there is one level of sub-groups holding services only; groups without snaps can nest deeper

snap set-quota <sub> --parent=<group> <limits> <snap.service> makes a service sub-group; its slice is snap.<parent>-<sub>.slice. Sub-group limits must fit the parent, and a group that holds snaps has one level of sub-groups, for services only.

It's really the nightly report that's greedy, and it shares a snap with the agent. Can you cap one service and not the other? Yes, with a sub-group. snap set-quota again: a group called reports, parent agents, ten percent of CPU, and the service, acme-agent.report. The sub-group gets a slice of its own inside the parent's, named after both. Three rules keep this consistent. What the sub-groups are given has to fit inside the parent. A service goes only under the group that holds its snap. And under a group with snaps there's one level, for services only. Groups without snaps can nest deeper.

05 A journal of its own for administrators for Ubuntu Core developers

JOURNAL LIMITS acme-agent.agentsystem journalone chatty snap fills the journal for everyone $ sudo snap set-quota agents \--journal-size=16MB \--journal-rate-limit=100/1s journal namespacesnap-agents /etc/systemd/journald@snap-agents.conf[Journal]Storage=autoSystemMaxUse=16000000RuntimeMaxUse=16000000RateLimitIntervalSec=1000000usRateLimitBurst=100 Environment=SNAPD_LOG_NAMESPACE=snap-agentsadded to the group's service units size: 64 KiB to 4 GiB · systemd 245 · no experimental flag · then check snap logsa service sub-group cannot have its own journal limit; it uses its parent's namespace
acme-agent.agent logs into the shared system journal, so one chatty snap fills it for everyone. sudo snap set-quota agents --journal-size=16MB --journal-rate-limit=100/1s moves its log stream into the journal namespace snap-agents. /etc/systemd/journald@snap-agents.conf holds Storage=auto, SystemMaxUse=16000000, RuntimeMaxUse=16000000, RateLimitIntervalSec=1000000us and RateLimitBurst=100, and the service unit gets Environment=SNAPD_LOG_NAMESPACE=snap-agents. The size is 64 KiB to 4 GiB, systemd 245 is needed, no experimental flag is needed, and snap logs should be checked afterwards

A journal limit creates the journal namespace snap-<group> with /etc/systemd/journald@snap-<group>.conf holding SystemMaxUse=, RuntimeMaxUse=, RateLimitIntervalSec= and RateLimitBurst=. The services get SNAPD_LOG_NAMESPACE=snap-<group>; a service sub-group uses its parent's namespace.

With debugging on, acme-agent writes a thousand lines a second, and the journal is shared. The kernel messages you wanted from last night are gone. A journal limit gives the group a journal of its own. snap set-quota with a journal size, a rate limit, or both. snapd creates a journal namespace called snap-agents, writes its configuration with the size and the rate, and points the group's services at it. The size must be between 64 KiB and 4 GiB, and you need systemd 245. No experimental flag, whatever older guides say. The help warns that this changes the log command, so check snap logs afterwards.

06 What you can change later

CHANGING A GROUP $ sudo snap set-quota agents --memory=128MBerror: cannot update quota group: cannot update group "agents":cannot decrease memory limit, remove and re-create it to decrease the limit memorythreads cpucpu-setjournal a limit, once set, cannot be taken off the groupa snap, once added, cannot be taken out $ sudo snap remove-quota agents$ sudo snap set-quota agents --memory=128MB --cpu=50% acme-agentremove, then create again both steps restart the servicesraising memory restarts nothing · remove sub-groups first · a sub-group cannot move to another parent
sudo snap set-quota agents --memory=128MB fails with: cannot decrease memory limit, remove and re-create it to decrease the limit. Memory and threads limits can only go up; cpu, cpu-set and journal limits can go up and down. A limit, once set, cannot be taken off, and a snap, once added, cannot be taken out. The way back is sudo snap remove-quota agents followed by set-quota with the new limit; both steps restart the services. Raising memory restarts nothing, sub-groups must be removed first, and a sub-group cannot move to another parent

Memory and thread limits can be raised but not lowered; CPU, CPU-set and journal limits can change both ways. Limits and snaps cannot be removed from a group: use snap remove-quota and create the group again. A group with sub-groups cannot be removed until they are.

A month later the leak is fixed and you'd like to tighten the ceiling to a hundred and twenty eight megabytes. snapd says no. Memory and thread limits only go up. CPU, CPU set and journal limits move both ways. Two things hold for every group: you can add a limit but not take one off, and you can add a snap but not take one out. The way back is to remove the group and create it again. Both steps restart the services, so pick a quiet moment. Raising memory restarts nothing, so if in doubt, start low. Sub-groups go first, and a sub-group can't move to another parent.

07 Where to go from here

ROADS Quota groups Servicesunits in a slice, snap logs The overlordone change, one task, no undo Configurationsystem options The REST APIread: anyone · write: root First bootno removal while preseeding Administrationback to the district
Roads from quota groups: services, the overlord, configuration, the REST API, first boot and preseeding, and the administration district

Roads: services, the overlord, configuration, the REST API, first boot, and the administration district.

A quota group is a thin layer over systemd, so most roads lead to what it sits on. Services, for the units inside a slice and for snap logs. The overlord, because each quota command is a change with a single task, and that task has no undo. Configuration, for the system options. The REST API, where anyone can read the quotas and only root can change them. First boot and preseeding, during which groups can't be removed. And back to the administration district.

Read more

Commands

CommandEffect
snap set-quota <group> [limits] [--parent=<group>] [<snap> or <snap.service>...]create a group or update it
snap quota <group>show one group: name, parent, constraints, current, subgroups, snaps, services
snap quotaslist groups: Quota, Parent, Constraints, Current
snap remove-quota <group>remove a group that has no sub-groups
snap install --quota-group <group> <snap>add the snap to a group as it is installed

Writing needs root. With no groups, snap quotas prints No quota groups defined.

Limits

FlagFormSlice or journal settingAfter creationNeeds
--memory<number><unit>, for example 64MB, 1GB; more than 640 KiBMemoryMax=, MemoryLimit=raise onlythe memory control group
--cpu<percentage>% or <count>x<percentage>%, for example 50%, 2x100%; percentage 1 to 100CPUQuota=raise or lower
--cpu-setcomma-separated core indices, for example 0,1,3AllowedCPUs=add or remove coressystemd 243, cgroup v2
--threadsa positive integerTasksMax=raise only
--journal-size<number><unit>, 64 KiB to 4 GiBSystemMaxUse=, RuntimeMaxUse=raise or lowersystemd 245
--journal-rate-limit<count>/<period>, for example 100/1sRateLimitBurst=, RateLimitIntervalSec=raise or lowersystemd 245

Any quota needs systemd 230. Sizes are decimal: 256MB is 256000000 bytes. CPUQuota= is the count times the percentage, capped at the number of CPUs times 100. A plain percentage such as 50% uses the number of CPUs, or of CPUs in the group's cpu-set when that is smaller, as the count: 50% on four CPUs is CPUQuota=200%.

snap quotas prints the constraints as memory=, cpu=NxM% or cpu=M%, cpu-set=, threads=, journal-size=, journal-rate=; Current shows memory= and threads= only.

Type the --cpu value exactly in one of the two forms. The command line looks for a valid part anywhere in the value and ignores text around it, so a typo can produce a group with a limit you did not intend: 0.5x100% is read as 5x100% and -50% as 50%. Check with snap quota <group> after setting it.

Rules

  • Group names: 2 to 40 characters in snap-name syntax; root, system, snapd and user are reserved.
  • A group needs at least one limit.
  • A snap is in at most one group. Snaps and limits cannot be removed from a group.
  • The limits of sub-groups must fit inside the parent's.
  • A group that holds snaps may have one level of sub-groups, and those hold services only. A service's snap must be in the direct parent group. Groups without snaps nest deeper.
  • Services and snaps, or services and sub-groups, cannot be mixed in one group.
  • A service sub-group cannot have a journal limit; it uses the parent's journal namespace.
  • A sub-group cannot be moved to another parent.
  • A group with sub-groups cannot be removed. Groups cannot be removed while preseeding.
  • Adding a snap restarts its services that are not disabled. Removing a group restarts them too. Raising the memory limit does not restart anything.
  • The limits apply to the snaps' services only; commands and hooks run in their own scopes outside the slice.

for administrators for Ubuntu Core developers for snapd developers

Files

FileContent
/etc/systemd/system/snap.<group>.slicethe slice of a top-level group
/etc/systemd/system/snap.<parent>-<group>.slicethe slice of a sub-group; all parent names are joined with -
/etc/systemd/journald@snap-<group>.confthe journal namespace configuration
/etc/systemd/system/systemd-journald@snap-<group>.service.d/00-snap.confa drop-in for the namespace's journald instance
the snap's service unitsgain Slice= and, with a journal limit, Environment=SNAPD_LOG_NAMESPACE=snap-<group>

Group names in unit names are escaped the way systemd-escape --path escapes them.

Messages

MessageWhen
cannot create quota group without any limitset-quota on a new group with no limit
quota group must have at least one resource limit setthe same, from the daemon
cannot decrease memory limit, remove and re-create it to decrease the limitlowering memory; the same for threads
cannot remove memory limit from quota grouptaking a limit off
cannot add snap "S" to group "G": snap already in quota group "H"a snap in two groups
cannot add snaps to group "G": only services are allowed in this sub-groupa snap into a service sub-group
only one level of sub-groups are allowed for groups with snapsnesting too deep
cannot mix services and snaps in the same quota groupboth kinds given
journal quotas are not supported for individual servicesa journal limit on a service sub-group
cannot remove quota group "G" with sub-groups, remove the sub-groups firstremoval order
group "G" cannot be moved to a different parent (re-parenting not yet supported)--parent changed
group name "G" reservedroot, system, snapd, user
cannot use quotas with incompatible systemd: ...systemd older than 230
cannot use CPU set with cgroup version N--cpu-set on cgroup v1
cpu quota percentage must be between 1 and 100--cpu with a larger percentage

for administrators for Ubuntu Core developers for snapd developers

Change, task and state

  • Change kind quota-control, summary Create quota group "G", Update quota group "G" or Remove quota group "G", with one task of kind quota-control. The task has no undo.
  • When services must restart, the task queues service-control tasks with the summary Restarting services for snap "S".
  • snap install --quota-group adds a quota-add-snap task: Add snap "S" to quota group "G".
  • A quota change conflicts with other changes on the same group and on its snaps.
  • State key quotas: group name to memory-limit, cpu-limit, task-limit, journal-limit, parent-group, sub-groups, snaps, services.

for administrators for Ubuntu Core developers for snapd developers

API

  • GET /v2/quotas and GET /v2/quotas/{group}: open to any local user.
  • POST /v2/quotas: root only. Body: action (ensure or remove), group-name, parent, snaps, services, constraints. ensure creates the group or updates it.

Where the documentation differs

  • The how-to says journal quotas need experimental.quota-groups=true. The feature has graduated and no flag is needed.
  • The how-to gives 4KB as the memory minimum; the code requires more than 640 KiB.
  • A sample shows cpu=2x,cpu=100%; the command prints cpu=2x100%.
  • The how-to does not mention service sub-groups or the journal size range.

Further reading

Roads from here

RUNNING SNAPS

  • Services — the services and units that a quota group contains
  • The overlord — quota commands are changes with one quota-control task
  • Configuration — system options such as the journal settings

UBUNTU CORE · install and recovery

ADMINISTRATION · managing a system of snaps

  • Aliases — back to the administration district

INSIDE SNAPD · the daemon itself