Quota groups
managing snaps for administrators for Ubuntu Core developers
A quota group is a named set of snaps or of services of one snap with limits on memory and CPU and threads and journal use. snapd writes a systemd slice for the group and places the snaps' service units in it. Groups nest and a sub-group may hold single services. Memory and thread limits can be raised but not lowered and a snap leaves a group only when the group is removed.
On the map: administration district
01 A ceiling for the agent
snap set-quota <group> <limits> <snap>... creates or updates a quota group; snap quotas lists groups with their constraints and current use. Type the CPU value exactly as 50% or 2x100% and check it with snap quota <group>: text around a valid value is ignored rather than rejected. The gateway's 1 GB and four cores are invented.
The gateway has a gigabyte of memory, and acme-agent has a leak. Left alone it takes the lot, and the device stops answering. What you want is a ceiling. That's a quota group: a name, some limits, and the snaps they apply to. Think of a fence with the limits written on it. One command makes it. snap set-quota, the group name agents, two hundred and fifty six megabytes and half the CPU, then the snap. It needs root. Snap quotas lists the group, its limits and what it's using right now. There are six limits in all, and a group needs at least one.
02 A group becomes a slice for administrators for Ubuntu Core developers
A top-level group is the unit snap.<group>.slice in /etc/systemd/system, with CPUQuota=, AllowedCPUs=, MemoryMax= and TasksMax= taken from the group's limits. A plain percentage is multiplied by the number of CPUs (or of CPUs in the set). Sizes are decimal: 256MB is 256000000 bytes. The gateway's 1 GB and four cores are invented.
So who actually holds acme-agent under two hundred and fifty six megabytes? Not snapd. Its part is one file: a systemd slice unit named after the group, snap.agents.slice. The limits you typed become the slice's settings. CPUQuota= is your fifty percent times the gateway's four cores. MemoryMax= is in bytes, written twice, once for older systemd. A thread limit would add TasksMax=, and a CPU set AllowedCPUs=. Accounting is always on. From there it's systemd and the kernel's control groups that do the work, so you can look at a group with a tool you already know: systemctl show.
03 The services move in for administrators for Ubuntu Core developers
snapd adds Slice=snap.<group>.slice to every service unit of a member snap and restarts the services that are not disabled. snap install --quota-group <group> joins at install. Commands started with snap run and hooks run in their own scope outside the slice and are not limited.
A slice with nothing in it limits nothing. So how does acme-agent get inside? snapd rewrites each of its service units with one more line: Slice=snap.agents dot slice. A running process isn't moved by editing a file, so snapd restarts the snap's services that aren't disabled. Plan for that blip on a live gateway. Install with the --quota-group flag and the snap joins before its services start. Afterwards systemctl shows the agent's control group under the slice. That's services, though. A command someone runs by hand, or a hook, gets a scope of its own outside the slice, so the group's limits don't reach it.
04 Groups inside groups for administrators for Ubuntu Core developers
snap set-quota <sub> --parent=<group> <limits> <snap.service> makes a service sub-group; its slice is snap.<parent>-<sub>.slice. Sub-group limits must fit the parent, and a group that holds snaps has one level of sub-groups, for services only.
It's really the nightly report that's greedy, and it shares a snap with the agent. Can you cap one service and not the other? Yes, with a sub-group. snap set-quota again: a group called reports, parent agents, ten percent of CPU, and the service, acme-agent.report. The sub-group gets a slice of its own inside the parent's, named after both. Three rules keep this consistent. What the sub-groups are given has to fit inside the parent. A service goes only under the group that holds its snap. And under a group with snaps there's one level, for services only. Groups without snaps can nest deeper.
05 A journal of its own for administrators for Ubuntu Core developers
A journal limit creates the journal namespace snap-<group> with /etc/systemd/journald@snap-<group>.conf holding SystemMaxUse=, RuntimeMaxUse=, RateLimitIntervalSec= and RateLimitBurst=. The services get SNAPD_LOG_NAMESPACE=snap-<group>; a service sub-group uses its parent's namespace.
With debugging on, acme-agent writes a thousand lines a second, and the journal is shared. The kernel messages you wanted from last night are gone. A journal limit gives the group a journal of its own. snap set-quota with a journal size, a rate limit, or both. snapd creates a journal namespace called snap-agents, writes its configuration with the size and the rate, and points the group's services at it. The size must be between 64 KiB and 4 GiB, and you need systemd 245. No experimental flag, whatever older guides say. The help warns that this changes the log command, so check snap logs afterwards.
06 What you can change later
Memory and thread limits can be raised but not lowered; CPU, CPU-set and journal limits can change both ways. Limits and snaps cannot be removed from a group: use snap remove-quota and create the group again. A group with sub-groups cannot be removed until they are.
A month later the leak is fixed and you'd like to tighten the ceiling to a hundred and twenty eight megabytes. snapd says no. Memory and thread limits only go up. CPU, CPU set and journal limits move both ways. Two things hold for every group: you can add a limit but not take one off, and you can add a snap but not take one out. The way back is to remove the group and create it again. Both steps restart the services, so pick a quiet moment. Raising memory restarts nothing, so if in doubt, start low. Sub-groups go first, and a sub-group can't move to another parent.
07 Where to go from here
Roads: services, the overlord, configuration, the REST API, first boot, and the administration district.
A quota group is a thin layer over systemd, so most roads lead to what it sits on. Services, for the units inside a slice and for snap logs. The overlord, because each quota command is a change with a single task, and that task has no undo. Configuration, for the system options. The REST API, where anyone can read the quotas and only root can change them. First boot and preseeding, during which groups can't be removed. And back to the administration district.
Read more
Commands
| Command | Effect |
|---|---|
snap set-quota <group> [limits] [--parent=<group>] [<snap> or <snap.service>...] | create a group or update it |
snap quota <group> | show one group: name, parent, constraints, current, subgroups, snaps, services |
snap quotas | list groups: Quota, Parent, Constraints, Current |
snap remove-quota <group> | remove a group that has no sub-groups |
snap install --quota-group <group> <snap> | add the snap to a group as it is installed |
Writing needs root. With no groups, snap quotas prints No quota groups defined.
Limits
| Flag | Form | Slice or journal setting | After creation | Needs |
|---|---|---|---|---|
--memory | <number><unit>, for example 64MB, 1GB; more than 640 KiB | MemoryMax=, MemoryLimit= | raise only | the memory control group |
--cpu | <percentage>% or <count>x<percentage>%, for example 50%, 2x100%; percentage 1 to 100 | CPUQuota= | raise or lower | |
--cpu-set | comma-separated core indices, for example 0,1,3 | AllowedCPUs= | add or remove cores | systemd 243, cgroup v2 |
--threads | a positive integer | TasksMax= | raise only | |
--journal-size | <number><unit>, 64 KiB to 4 GiB | SystemMaxUse=, RuntimeMaxUse= | raise or lower | systemd 245 |
--journal-rate-limit | <count>/<period>, for example 100/1s | RateLimitBurst=, RateLimitIntervalSec= | raise or lower | systemd 245 |
Any quota needs systemd 230. Sizes are decimal: 256MB is 256000000 bytes. CPUQuota= is the count times the percentage, capped at the number of CPUs times 100. A plain percentage such as 50% uses the number of CPUs, or of CPUs in the group's cpu-set when that is smaller, as the count: 50% on four CPUs is CPUQuota=200%.
snap quotas prints the constraints as memory=, cpu=NxM% or cpu=M%, cpu-set=, threads=, journal-size=, journal-rate=; Current shows memory= and threads= only.
Type the --cpu value exactly in one of the two forms. The command line looks for a valid part anywhere in the value and ignores text around it, so a typo can produce a group with a limit you did not intend: 0.5x100% is read as 5x100% and -50% as 50%. Check with snap quota <group> after setting it.
Rules
- Group names: 2 to 40 characters in snap-name syntax;
root,system,snapdanduserare reserved. - A group needs at least one limit.
- A snap is in at most one group. Snaps and limits cannot be removed from a group.
- The limits of sub-groups must fit inside the parent's.
- A group that holds snaps may have one level of sub-groups, and those hold services only. A service's snap must be in the direct parent group. Groups without snaps nest deeper.
- Services and snaps, or services and sub-groups, cannot be mixed in one group.
- A service sub-group cannot have a journal limit; it uses the parent's journal namespace.
- A sub-group cannot be moved to another parent.
- A group with sub-groups cannot be removed. Groups cannot be removed while preseeding.
- Adding a snap restarts its services that are not disabled. Removing a group restarts them too. Raising the memory limit does not restart anything.
- The limits apply to the snaps' services only; commands and hooks run in their own scopes outside the slice.
for administrators for Ubuntu Core developers for snapd developers
Files
| File | Content |
|---|---|
/etc/systemd/system/snap.<group>.slice | the slice of a top-level group |
/etc/systemd/system/snap.<parent>-<group>.slice | the slice of a sub-group; all parent names are joined with - |
/etc/systemd/journald@snap-<group>.conf | the journal namespace configuration |
/etc/systemd/system/systemd-journald@snap-<group>.service.d/00-snap.conf | a drop-in for the namespace's journald instance |
| the snap's service units | gain Slice= and, with a journal limit, Environment=SNAPD_LOG_NAMESPACE=snap-<group> |
Group names in unit names are escaped the way systemd-escape --path escapes them.
Messages
| Message | When |
|---|---|
cannot create quota group without any limit | set-quota on a new group with no limit |
quota group must have at least one resource limit set | the same, from the daemon |
cannot decrease memory limit, remove and re-create it to decrease the limit | lowering memory; the same for threads |
cannot remove memory limit from quota group | taking a limit off |
cannot add snap "S" to group "G": snap already in quota group "H" | a snap in two groups |
cannot add snaps to group "G": only services are allowed in this sub-group | a snap into a service sub-group |
only one level of sub-groups are allowed for groups with snaps | nesting too deep |
cannot mix services and snaps in the same quota group | both kinds given |
journal quotas are not supported for individual services | a journal limit on a service sub-group |
cannot remove quota group "G" with sub-groups, remove the sub-groups first | removal order |
group "G" cannot be moved to a different parent (re-parenting not yet supported) | --parent changed |
group name "G" reserved | root, system, snapd, user |
cannot use quotas with incompatible systemd: ... | systemd older than 230 |
cannot use CPU set with cgroup version N | --cpu-set on cgroup v1 |
cpu quota percentage must be between 1 and 100 | --cpu with a larger percentage |
for administrators for Ubuntu Core developers for snapd developers
Change, task and state
- Change kind
quota-control, summaryCreate quota group "G",Update quota group "G"orRemove quota group "G", with one task of kindquota-control. The task has no undo. - When services must restart, the task queues
service-controltasks with the summaryRestarting services for snap "S". snap install --quota-groupadds aquota-add-snaptask:Add snap "S" to quota group "G".- A quota change conflicts with other changes on the same group and on its snaps.
- State key
quotas: group name tomemory-limit,cpu-limit,task-limit,journal-limit,parent-group,sub-groups,snaps,services.
for administrators for Ubuntu Core developers for snapd developers
API
GET /v2/quotasandGET /v2/quotas/{group}: open to any local user.POST /v2/quotas: root only. Body:action(ensureorremove),group-name,parent,snaps,services,constraints.ensurecreates the group or updates it.
Where the documentation differs
- The how-to says journal quotas need
experimental.quota-groups=true. The feature has graduated and no flag is needed. - The how-to gives 4KB as the memory minimum; the code requires more than 640 KiB.
- A sample shows
cpu=2x,cpu=100%; the command printscpu=2x100%. - The how-to does not mention service sub-groups or the journal size range.
Further reading
- Use resource quotas: the four commands with examples for every limit. See the differences above.
- Control services: the services a group contains and
snap logs. - Use the REST API: a
curlexample on/v2/quotas. - snapd REST API:
/v2/quotasand/v2/quotas/{group-name}.
Roads from here
RUNNING SNAPS
- Services — the services and units that a quota group contains
- The overlord — quota commands are changes with one quota-control task
- Configuration — system options such as the journal settings
UBUNTU CORE · install and recovery
- ubuntu-seed and first boot — preseeding during which groups cannot be removed
ADMINISTRATION · managing a system of snaps
- Aliases — back to the administration district
INSIDE SNAPD · the daemon itself
- The snapd REST API — the quotas endpoints and their access levels