SSH logins and apt sources

passwords, root, NoCloud, http for users

The common user-data turns on SSH password logins, lets root log in, limits any later cloud-init run to the NoCloud data source, and rewrites https:// to http:// in apt's .list and .sources files so that a caching proxy can see the downloads; Gentoo and Ubuntu Core differ.

On the map: first boot

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 A cloud image says no

SSH AND APT spread, the CI tools SSH with a password, often as root a stock cloud image password logins: off root: with a key at most image-garden's user-data password logins on root may log in cloud-init: NoCloud only apt: https to http then the two that differ: Gentoo and Ubuntu Core
spread and the CI tools log in over SSH with a password, often as root; a stock cloud image has password logins off and root on a key at most; the user-data makes four changes: password logins on, root may log in, cloud-init limited to NoCloud, apt sources from https to http; Gentoo and Ubuntu Core differ

Four changes in the common user-data: SSH password logins, root logins, cloud-init limited to NoCloud, and apt sources moved to plain http.

spread logs in to every test machine over SSH, with a user name and a password, and the tools built on image-garden often log in as root. A stock cloud image allows neither. Password logins are off, and root may log in with a key at most. So the user-data changes four things. It turns password logins on. It lets root log in. It pins cloud-init to the NoCloud data source. And while it's at it, it switches apt's sources from https to plain http, so that a caching proxy can see what's fetched. Each gets a scene, and then come the two kinds of system where SSH works differently.

Details and sources

Details

  • All four changes are in the common user-data template (010-qemu.mk:187-247): ssh_pwauth: true (197), the SSH snippet with UsePam yes and PermitRootLogin yes (214-220), 99-image-garden.cfg with datasource_list: [NoCloud] (221-230), and the sed over /etc/apt (244).
  • image-garden's own spread.yaml lists a user name and a password for each system, such as ubuntu/ubuntu (spread.yaml:60-72). The snapcraft, rockcraft and charmcraft components log in as system.users.0, which is root (snapcraft-vm:374-381).
  • Derived, not from the tree: distribution cloud images usually ship with SSH password authentication off and root logins limited to keys.

02 Password logins

PASSWORDS the user-data ssh_pwauth: true cloud-init: password logins on, now /etc/cloud/cloud.cfg.d/99-image-garden.cfg ssh_pwauth: true kept, should cloud-init run again the SSH snippet UsePam yes logins go through PAM $ ssh -p 5123 ubuntu@localhost ubuntu@localhost's password: ubuntu@ubuntu-24-04:~$ password: ubuntu
ssh_pwauth: true in the user-data turns on SSH password authentication in this boot; the same setting is in /etc/cloud/cloud.cfg.d/99-image-garden.cfg for a later run; the SSH snippet adds UsePam yes; then ssh -p PORT ubuntu@localhost asks for a password and ubuntu logs in

ssh_pwauth: true at the top and in 99-image-garden.cfg, plus UsePam yes in the SSH snippet. Then ssh -p PORT ubuntu@localhost takes the password ubuntu.

First, passwords. Near the top, the user-data sets ssh_pwauth to true, which tells cloud-init to switch on password authentication in the SSH server during this boot. The same setting appears again in a small cloud-init config file, so it would hold if cloud-init ever ran again. And the SSH snippet the user-data writes adds UsePam yes, so logins go through PAM, where Linux normally checks passwords. With that in place, ssh to ubuntu at localhost, on the port allocate printed, asks for a password, and ubuntu gets you in.

Details and sources

Details

  • ssh_pwauth: true at the top level of the user-data (010-qemu.mk:197) and again in /etc/cloud/cloud.cfg.d/99-image-garden.cfg, written with mode 0600 (010-qemu.mk:221-230).
  • The SSH snippet /root/sshd_config.APPEND holds UsePam yes and PermitRootLogin yes (010-qemu.mk:214-220); the next scene follows it into /etc/ssh/sshd_config.
  • allocate forwards 127.0.0.1:PORT to the guest's port 22 (image-garden:1105-1107); the port 5123 on the slide is an example from the range 5000 to 9999.
  • Derived, from cloud-init's and OpenSSH's behaviour: ssh_pwauth: true makes cloud-init set PasswordAuthentication yes for the SSH server; sshd's keywords are case-insensitive, so UsePam is UsePAM.

Read more

03 Letting root log in

ROOT /root/sshd_config.APPEND UsePam yes PermitRootLogin yes in root's home: the config directory may be read-only cat … >> /etc/ssh/sshd_config then the copy is removed /etc/ssh/sshd_config PermitRootLogin prohibit-password … UsePam yes PermitRootLogin yes sshd: the first value it reads wins #PermitRootLogin prohibit-password a sed comments that line out root, password root: in over SSH
The SSH snippet with UsePam yes and PermitRootLogin yes is written to /root/sshd_config.APPEND, because the configuration directory may be read-only; runcmd appends it to /etc/ssh/sshd_config and removes the copy; sshd uses the first value it reads, so a sed comments out an active PermitRootLogin prohibit-password line; root then logs in over SSH with password root

PermitRootLogin yes goes to /root/sshd_config.APPEND, then runcmd appends it to /etc/ssh/sshd_config and comments out PermitRootLogin prohibit-password.

Root needs one more step. The SSH snippet also says PermitRootLogin yes. image-garden doesn't write that into the SSH configuration directly. It writes the snippet to a file in root's home first, in case the system's configuration directory is read-only at that point, and a command near the end appends it and deletes the copy. There's a catch: the SSH server takes the first value it reads for a setting. An active PermitRootLogin prohibit-password line earlier in the file would win, so another command comments it out. Now root, with the password root, gets in over SSH too.

Details and sources

Details

  • write_files puts the snippet at /root/sshd_config.APPEND, mode 0600, under the comment "Use /root/foo.APPEND in case /etc is a read-only fs." (010-qemu.mk:213-220).
  • runcmd then runs cat /root/sshd_config.APPEND >> /etc/ssh/sshd_config, rm /root/sshd_config.APPEND, and sed -i -e 's/^PermitRootLogin prohibit-password/#PermitRootLogin prohibit-password/' /etc/ssh/sshd_config || true (010-qemu.mk:241-243).
  • The sed changes only a line that starts with PermitRootLogin prohibit-password; a commented default is left alone. The line at the top of the panel is an example of what it changes, not the contents of a particular image.
  • Derived, from OpenSSH's sshd_config(5): for most keywords the first value obtained is used, so a setting appended at the end loses to an earlier one, including one in a file included near the top.

04 cloud-init looks only for NoCloud

NOCLOUD ONLY cloud-init cloud metadata services with no cloud: wait, then time out /etc/cloud/cloud.cfg.d/99-image-garden.cfg ssh_pwauth: true datasource_list: - NoCloud provisioning boot: the seed is found this file is for later cloud-init, a later run straight to NoCloud
On a machine with no cloud, cloud-init's probes for cloud metadata services only wait and time out; the user-data writes /etc/cloud/cloud.cfg.d/99-image-garden.cfg with ssh_pwauth: true and datasource_list: NoCloud; in the provisioning boot the seed is already found, so the file is for later: a later run of cloud-init goes straight to NoCloud

/etc/cloud/cloud.cfg.d/99-image-garden.cfg sets datasource_list: [NoCloud] and ssh_pwauth: true for any later run of cloud-init.

The third change is about where cloud-init looks for its data. On a real cloud it probes for metadata services, and on a machine with no cloud those probes can only wait and time out. So the user-data writes a config file of its own into cloud-init's config directory. It sets the data source list to just NoCloud, and repeats the password setting. In the provisioning boot the seed has already been found, so this file is for later. If cloud-init ever runs again in this image, say because you enabled it again, it goes straight to NoCloud instead of hunting for a cloud.

Details and sources

Details

  • write_files creates /etc/cloud/cloud.cfg.d/99-image-garden.cfg, mode 0600, with ssh_pwauth: true and datasource_list: [NoCloud], under the comments "Use only none and no-cloud data sources to avoid probing cloud-specific network services" and "Instruct cloud-init not to look for cloud APIs" (010-qemu.mk:221-230).
  • The last runcmd command disables cloud-init for later boots (010-qemu.mk:246); see provisioning runs once. Removing /etc/cloud/cloud-init.disabled in a guest is one way cloud-init would run again (derived).
  • Derived, from cloud-init's behaviour: the data source is chosen early in the boot, before write_files runs, so the file does not change the provisioning boot; files in cloud.cfg.d are read in name order, and 99- sorts late.
  • The Ubuntu Core template sets datasource_list: [NoCloud] in its user-data instead (050-ubuntu-core.mk:131-133).

Read more

05 apt sources over plain http

APT OVER HTTP $ find /etc/apt -type f \( -name '*.list' -o -name '*.sources' \) \ -exec sed -i -e 's|https://|http://|g' {} \; *.list: deb https://… *.list: deb http://… *.sources: URIs: https://… *.sources: URIs: http://… pkgproxy http: seen and cached https: a tunnel, nothing kept apt checks signatures either way recent Ubuntu, deb822 Signed-By: -----BEGIN PGP PUBLIC KEY BLOCK-----  mQINBF… an inline key: only https:// is replaced, never letters inside it
A find and sed over /etc/apt turns https:// into http:// in every .list and .sources file; pkgproxy can cache plain http, while https passes through as a tunnel and nothing is kept; apt checks signatures either way; .sources files are the newer format of recent Ubuntu and may carry a signing key inline, so only https:// at the start of an address is replaced, never letters inside a key

find /etc/apt -type f \( -name '*.list' -o -name '*.sources' \) -exec sed -i -e 's|https://|http://|g' {} \;: plain http lets a proxy cache packages. Only https:// is replaced, so an inline key is left as it is.

The last change is for apt. A command walks the apt directory and, in every file whose name ends in .list or .sources, turns each https address into plain http. Why give up https? Because a caching proxy such as pkgproxy can only cache what it can see. An https download passes through it as a sealed tunnel, and nothing is kept. apt checks the signatures of what it fetches either way. The .sources files are the newer format recent Ubuntu releases use, and they can carry a signing key inline. So the command replaces https only where an address starts, never letters inside a key.

Details and sources

Details

  • The runcmd command: if [ -d /etc/apt ]; then find /etc/apt -type f \( -name '*.list' -o -name '*.sources' \) -exec sed -i -e 's|https://|http://|g' {} \; ; fi (010-qemu.mk:244). It runs on every system and does nothing where there is no /etc/apt.
  • The pattern is https://, not https, because a deb822 .sources file may hold an inline armoured key, whose text can contain those letters (the reason given in the message of the commit that added .sources; NEWS says that guests with deb822 sources "fetch packages over http, so that pkgproxy can cache them", NEWS:33-34). The key lines on the slide are illustrative.
  • The reason for plain http is stated in the CI components: "pkgproxy caches http:// archives only; an https:// one is a tunnel." snapcraft-vm also rewrites https://(archive|security|ports).ubuntu.com to http:// in *.sources and *.list under /etc/apt (snapcraft-vm:388-391); rockcraft-vm and charmcraft-vm run the same command. On a garden image those files are already rewritten.
  • pkgproxy's own apt setup runs from bootcmd on Debian and Ubuntu when a pkgproxy is reachable (010-qemu.mk:127-130); see what pkgproxy adds to user-data.
  • The README's security notice: "Guest apt sources are switched from HTTPS to HTTP, so that pkgproxy can cache packages; package signatures still protect their integrity" (README.md:32-33).
  • Derived, not from the tree: apt verifies the signed release files and package checksums whatever the transport; recent Ubuntu releases, such as 24.04, keep their sources in deb822 .sources files.

06 Where SSH works differently

WHERE IT DIFFERS Gentoo: sshd off out of the box $(BASE_CLOUD_INIT_USER_DATA_TEMPLATE) - systemctl enable --now sshd.service upstream spread: SSH crypto mismatch Ubuntu Core: its own template ubuntu / ubuntu no root password login Alpine to Rocky: the common changes
Gentoo ships with sshd off, so its template adds systemctl enable --now sshd.service, and upstream spread still cannot use its SSH server; Ubuntu Core has its own template with only ubuntu/ubuntu and no root password login; every other system gets the common changes

Gentoo's template adds systemctl enable --now sshd.service; upstream spread cannot use its SSH. Ubuntu Core has only ubuntu/ubuntu.

Two kinds of system don't fit this picture. Gentoo ships with its SSH server switched off, so image-garden's Gentoo template adds one command that enables and starts it. Even then, the README warns that upstream spread can't talk to Gentoo's SSH server, because their crypto algorithms don't match. Ubuntu Core has a template of its own. It creates only the ubuntu user, with the password ubuntu, and root can't log in with a password, so there you log in as ubuntu. Everything else, from Alpine to Rocky, gets the changes you've just seen.

Details and sources

Details

  • mk/050.gentoo.mk: "Gentoo ships ssh disabled out of the box, so enable it.", and GENTOO_CLOUD_INIT_USER_DATA_TEMPLATE is the common template plus - systemctl enable --now sshd.service (050.gentoo.mk:4-8).
  • The README on Gentoo: "Supported by image-garden but not compatible with upstream Spread (SSH crypto algorithm mismatch)" (README.md:651-655).
  • The Ubuntu Core template sets ssh_pwauth: true and creates the user with useradd … --extrausers and chpasswd (050-ubuntu-core.mk:131-177); the README says "Ubuntu Core systems do not allow root password authentication" (README.md:708), and its --query lists only ubuntu (QUERY:40-53).
  • The Ubuntu Core template has no sshd_config snippet and no apt command.

07 Where to go from here

ROADS SSH and apt The users root and the distribution user allocate where the SSH port comes from Per-distribution quirks Gentoo and the others Why pkgproxy what plain http buys Insecure on purpose where to run garden VMs
Roads from this deck: the users every system gets, allocate, what differs per distribution, why pkgproxy, and test images insecure on purpose

Next: the users, allocate, the per-distribution quirks, pkgproxy, and safety.

Password logins on, root let in, cloud-init held to NoCloud, and apt over plain http. The users deck has the accounts these logins are for, and allocate is where your SSH port comes from. The quirks deck shows the Gentoo template and the other exceptions in full. To see what plain http buys you, follow the road to pkgproxy. And the safety deck says what all these open doors mean for where you run garden VMs.

Details and sources

Read more

Basis of the statements on this page

Read in mk/010-qemu.mk, mk/050.gentoo.mk and mk/050-ubuntu-core.mk of image-garden at the commit this map describes and in bin/snapcraft-vm of snapcraft-gitlab-ci ("code"). What cloud-init, OpenSSH and apt do with these settings is their behaviour, not image-garden's ("derived").

What the common user-data changes

SettingWhereWhen it acts
ssh_pwauth: truetop level of the user-data (010-qemu.mk:197)the provisioning boot
UsePam yes, PermitRootLogin yes/root/sshd_config.APPEND, appended to /etc/ssh/sshd_config by runcmd (010-qemu.mk:214-220, 010-qemu.mk:241-242)from the next start of sshd (derived)
#PermitRootLogin prohibit-passwordsed over /etc/ssh/sshd_config (010-qemu.mk:243)same
ssh_pwauth: true, datasource_list: [NoCloud]/etc/cloud/cloud.cfg.d/99-image-garden.cfg (010-qemu.mk:221-230)any later run of cloud-init
https:// to http://every *.list and *.sources file under /etc/apt (010-qemu.mk:244)the next apt update (derived)

Logging in

HowUserPassword
ssh -p PORT root@localhost, with the port allocate printedrootroot
ssh -p PORT ubuntu@localhostthe group's name, such as ubuntuthe same word
Ubuntu Coreubuntuubuntu

allocate forwards 127.0.0.1:PORT to the guest's port 22 (image-garden:1105-1107). For tools, ./X.run --query prints the same users; see the users every system gets.

Exceptions

SystemDifference
Gentoosystemctl enable --now sshd.service appended to runcmd (050.gentoo.mk:4-8); not usable with upstream spread (README.md:651-655)
Ubuntu Coreits own template: ssh_pwauth: true, user ubuntu in extrausers, no root password login (050-ubuntu-core.mk:131-177)
systems without /etc/aptthe apt command does nothing

Roads from here

START HERE

RUNTIME · make rules and the CLI

SPREAD · tests

FIRST BOOT · cloud-init changes

PKGPROXY · caching for guests