Testing AppArmor
parser, library, utils for AppArmor developers
make check runs the parser's C unit tests and parser/tst suites, libapparmor's automake and DejaGnu tests and the utilities' unittest files, without root or an AppArmor kernel. The tests that load policy run under spread on image-garden VMs of several distributions, with build dependencies baked into the images by .image-garden.mk and, in GitLab CI, prepared images cached by a key on that file.
On the map: testing
▶ Watch this deck — 14 scenes, ~10 min
Worth seeing first
- From profile text to kernel — what the parser does with the profiles the tests feed it
Part of journeys
- Testing AppArmor on many systems, step 1 of 6 · next: Running spread on image-garden
01 You changed the parser, did anything break?
make check in parser/, libraries/libapparmor/ and utils/ needs neither root nor AppArmor in the kernel. The suites under tests/ load policy, so they run in spread on image-garden VMs.
You've changed how the parser compiles one rule, and before you open a merge request you want to know that nothing else broke. AppArmor's tests live next to the code they test. The parser, libapparmor and the Python utilities each have their own suite, and make check runs it as you, with no root and no AppArmor in your kernel. The tests directory holds the rest: tests that load policy into a real kernel and watch what it does. Those need a machine you can break, so AppArmor runs them in virtual machines of several distributions, with spread and image-garden. This deck follows both halves.
Details and sources
Details
- The README's "AppArmor Testsuites" section lists how to run each suite and where its documentation lives (README.md:290-400). Each component's own
make checkis listed with its build steps (README.md:141-216). make checkinparser/,libraries/libapparmor/andutils/drives the in-tree parser and library; none of them writes to the kernel. The parser suites compile with-Sor-QKSqand never load policy.tests/regression/apparmor/needs root: the README runs it withsudo make tests(README.md:341-367).tests/profiles/andtests/snapd/exist only as spread tasks.- The README also says the test steps gain little from
make -j: the harnesses spawn a few long-lived runners that mostly run sequentially (README.md:133-139). - Other suites not covered here:
binutils(make -C binutils check),changehat/mod_apparmor,profiles(make -C profiles check),tests/stress.
Read more
- parser/tst/, tests/regression/apparmor/, wiki: Integrating-Test-plan.
- From profile text to kernel: what the parser does with the profiles these tests feed it.
- Running spread on image-garden, on the image-garden map.
02 What make check runs in the parser
make -C parser check runs check_pod_files, the seven tst_* unit programs and make -C tst tests. Build tst_binaries with -j first to save time.
Make check in the parser does three things. It checks the manual pages. It builds seven small C programs from the parser's own sources, compiled with UNIT_TEST defined, so each source file can carry a main function full of checks. A failed check prints FAIL, and the program exits non-zero. Then it runs make tests in parser/tst, the slower suites that drive the parser binary you just built: the error messages, the cache, minimization, equality, loading a directory, and the big one, parser sanity. Building the seven programs is the step that runs in parallel, so the README builds them first, with make -j.
Details and sources
Details
check: check_pod_files tests(Makefile:438-445).check_pod_filesrunspodchecker -warning -warning *.pod(Make.rules:145-147).TESTS = tst_regex tst_misc tst_symtab tst_variable tst_lib tst_interface tst_network, compiled withTEST_CXXFLAGS = ... -DUNIT_TESTand linked with the parser's objects minus the one under test (Makefile:186-187, Makefile:417-425).tst_libcomes fromlib.candtst_networkfromnetwork.cc; the others fromparser_NAME.c.MY_TEST(statement, error)printsFAIL: errorand setsrc = 1(unit_test.h:49-53). Each file'smain()sits under#ifdef UNIT_TEST, as in parser_regex.c (parser_regex.c:1481, parser_regex.c:1884-1910).- The
teststarget runs each binary, then$(MAKE) -s -C tst tests. - In parser/tst:
tests: error_output caching minimize equality dirtest parser_sanity(Makefile:33).error_outputrunserrors.py,cachingrunspytest -n auto caching.py,minimize,equalityanddirtestrun their.shscripts,parser_sanityrunsproveonsimple.plaftergen_tests(Makefile:49-75). parser_sanity-no-genskips the generated profiles for faster manual runs;valgrindruns every simple test under valgrind (Makefile:57-60, Makefile:77-78).- Without
USE_SYSTEM, the parser/tst makefile pointsLD_LIBRARY_PATHat the in-tree libapparmor (Makefile:11-21);VERBOSE=1makesproveand pytest verbose. - The README builds the unit programs with
make -j $(nproc) tst_binariesbeforemake check(README.md:165-173).
Read more
- parser/tst/.
- The backend pipeline and Minimisation: the code the minimize and equality suites exercise.
03 A parser test case is a profile with a verdict
simple.pl pipes each simple_tests/**/*.sd into apparmor_parser -S and compares the exit status with #=EXRESULT. A new case is a new file.
Most parser cases need no code. Parser sanity runs simple.pl, a Perl script that walks parser/tst/simple_tests and feeds every .sd file it finds to the parser. It compiles with -S, so nothing is loaded, and a fixed features file stands in for the kernel. Each file carries its verdict in a comment. EXRESULT PASS means the parser must exit with zero; EXRESULT FAIL means it must refuse. DESCRIPTION names the case, and TODO marks a known failure. To add a case, drop one file into the right directory, next to more than two thousand others.
Details and sources
Details
simple.plcollects every*.sdfile undersimple_tests/, recursively, sorted (simple.pl:148-176).- For each file it runs
$APPARMOR_PARSER --config-file=./parser.conf -M features_files/features.all -S -I simple_tests/with stdout and stderr discarded, and writes the file's lines to the parser's stdin (simple.pl:79-116).-Swrites the binary to stdout instead of loading it;-Mtakes the kernel features from a file. - Directives:
#=DESCRIPTION text,#=EXRESULT PASS|FAIL(default PASS),#=TODO(an expected failure; a TODO that passes is reported as a failure),#=DISABLED(not run, reported as a TODO). A core dump always fails (simple.pl:99-145). Each case has a 480 second timeout (simple.pl:13). - The make target is
parser_sanity, which first runsgen_tests(gen-xtrans.py,gen-dbus.py,gen-conditionals.py) to write more cases intosimple_tests/generated_*(Makefile:35-55). - The tree holds 2,250
.sdfiles underparser/tst/simple_tests/(counted withgit ls-tree), before the generated ones. The slide's example is capability/bad_1.sd. - The README for this suite describes the directives and how to add a case (parser/tst/README).
- The utilities parse the same files:
utils/test/test-parser-simple-tests.pykeeps lists of files to skip or to expect the opposite result for (test-parser-simple-tests.py:19-60). A new case the utilities handle differently from the parser must be listed there.
Read more
- parser/tst/.
- Kernel features and the policy ABI: the features files that stand in for a kernel.
04 When two spellings must compile the same
equality.sh compiles with -QKSq, hashes the binary with sha256sum and compares. errors.py checks the exact text on stderr.
Pass or fail isn't always enough. Say you taught the parser a shorter spelling for a D-Bus permission. It should compile to exactly the policy that the long spelling does. That's what equality.sh checks. It pipes each profile to the parser, which compiles it to standard output without loading it or touching the cache, and it hashes the result. verify_binary_equality takes a description, a known good profile and the profiles that must hash the same. verify_binary_inequality asserts the opposite. And when the exact wording of an error matters, errors.py compares the parser's message character for character.
Details and sources
Details
hash_binary_policy()pipes the profile to$APPARMOR_PARSER --features-file features_files/FILE -QKSqand runssha256sumon the output (equality.sh:92-125).-Qskips the kernel load,-Kthe cache,-Swrites to stdout,-qis quiet.verify_binary MODE DESCRIPTION KNOWN-GOOD PROFILE...hashes the known-good profile, then each profile; modes areequality,inequality,xequalityandxinequality(the x modes are expected failures) (equality.sh:144-200, equality.sh:247-268).- A failed equality prints
FAIL: Hash values do not matchwith the parser command and both profiles (equality.sh:184-193). - The slide's example is the first
verify_setcase,dbus sendagainstdbus writeanddbus w, run for several pairs of qualifiers ($p1, $p2) (equality.sh:305-320). errors.pyrunsapparmor_parser --config-file=./parser.conf -S -I errors FILEand asserts the exit status and the exact stderr, after removing two messages about the kernel (errors.py:29-76).
Read more
- The backend pipeline: what two spellings go through before they can hash the same.
05 libapparmor, a log line in and a record out
make -C libraries/libapparmor check runs six tst_* programs and DejaGnu. Each testsuite/test_multi/NAME.in is parsed and compared with NAME.out and NAME.err.
Make check in libapparmor is automake's. It builds six test programs in the source directory, for the log parser, the features API and the kernel interface, and runs them. Then DejaGnu takes over in the testsuite directory. Its biggest part is test_multi: a directory of cases, each an audit line in a .in file. A small program parses the line with libapparmor and prints every field it found. DejaGnu compares that with the matching .out file, and standard error with the .err file. To add a case, you put a log line and the fields you expect side by side.
Details and sources
Details
check_PROGRAMS = tst_aalogmisc tst_aalogparse_cpp tst_aalogparse_reentrancy tst_aalogparse_oldname tst_features tst_kernelandTESTS = $(check_PROGRAMS), linked statically against.libs/libapparmor.a(Makefile.am:71-92).check-localalso checks the shared library's file name (Makefile.am:94-96).testsuite/Makefile.amsetsAUTOMAKE_OPTIONS = dejagnu, buildstest_multi.multifromtest_multi.c, and failscheckwhenlibaalogparse.logis missing or containsERROR(Makefile.am:1-25).multi_test.expcallsmulti-runfor every*.multiprogram (multi_test.exp:1-23).multi-runruns the program on eachtest_multi/*.in, writesout/NAME.outandout/NAME.err, and diffs them withNAME.outandNAME.err(libaalogparse_init.exp:86-160).test_multi.creads the line, callsparse_record()and prints each field of theaa_log_record(test_multi.c:16-79).- The slide's case is
avc_audit_01(.in, .out). The tree has 179.infiles intest_multi/; 142 of them also have a.profilefile, the rules the utilities should derive from the line. - DejaGnu must be installed; without it
checkstops with*** libaalogparse.log not found - is dejagnu installed? ***.
Read more
- Reading a denial: the fields of the log lines these cases parse.
06 The utilities, a table of cases and one method
make -C utils check runs every test/test-*.py against the in-tree library and parser. An AATest lists tests pairs; setup_all_loops() makes one test per pair.
The Python utilities use unittest, with one file per area, like test-capability.py. Make check in utils runs pyflakes, then every test file against the library, the parser and the profiles of your tree. Most test classes derive from AATest and hold a tuple called tests, made of pairs: an input and what it should give. One method, _run_test, checks a pair, and setup_all_loops turns every pair into a test of its own. So a new case is often one more line. Two files reach across: one parses every parser simple test, the other turns libapparmor's log lines into profiles.
Details and sources
Details
utils/Makefile:check: check_severity_db check_pod_files check_lint, thenmake -C test checkandmake -C vim check;check_lintruns$(PYFLAKES)on the tools, theapparmormodule andtest/*.py(Makefile:91-104).utils/test/Makefileruns eachtest-*.pywith every interpreter inPYTHON_VERSIONS(by defaultpython3) and setsPYTHONPATH,LD_LIBRARY_PATH,__AA_CONFDIR,__AA_BASEDIR=../../profiles/apparmor.dand__AA_PARSER=../../parser/apparmor_parser;USE_SYSTEM=1uses the installed ones instead (Makefile:27-42, Makefile:91-92, Make.rules:47-56).- One file:
make -C utils/test check-one-test-NAMErunstest-NAME.py(Makefile:100-102). One test:python3 ./test-FILE.py Class.test_namewith the variables from the README (utils/test/README.md). setup_tests_loop()addstest_0,test_1, ... to a class, one per(test_data, expected)pair intests, each callingself._run_test(test_data, expected)(common_test.py:73-89).test-example.pyis the template (test-example.py:18-57).- The slide's class: test-capability.py:243-254.
test-parser-simple-tests.pyparses every file underparser/tst/simple_testsand keeps lists of exceptions (test-parser-simple-tests.py:19-60).test-libapparmor-test_multi.pychecks the log parser against each.outfile and, for cases with a.profile, the profile the tools derive from the log (test-libapparmor-test_multi.py:178-200, test-libapparmor-test_multi.py:287-315).- CI also runs
make -C utils/test coverage-regression, which fails when a file listed as fully covered drops below 100% or a file listed as incomplete reaches it (Makefile:113-120, .gitlab-ci.yml:93-114). - A new
test-*.pymust also be added toutils/test/task.yamlas a spread variant; theutils/suite's prepare checks it (spread.yaml:283-303).
Read more
- aa-logprof: the tool whose log-to-profile step the
.profilecases check.
07 Tests that need a kernel
A task in tests/profiles/NAME/ loads profiles/apparmor.d/NAME, runs the program and fails on any parser warning or unexpected DENIED line.
Everything so far runs as you, without AppArmor in the kernel. The tests directory is different. Its regression tests confine small C programs and check what the kernel allows. Its profile tests check shipped profiles against the real programs. Take tests/profiles/tar. Before the task, spread loads the tar profile from the tree with --warn=all, and any warning fails it. The task archives and extracts a file, then checks that tar ran under its profile, in enforce mode. Afterwards the profile is removed, and any DENIED line in the kernel log fails the test, unless EXPECT_DENIALS says it was wanted.
Details and sources
Details
- The slide's task: tests/profiles/tar/task.yaml. The profile name is the task's directory name unless
PROFILE_NAMEis set (spread.yaml:210-211). prepare-eachsaves and zeroeskernel.printk_ratelimit, stopsauditdso denials reach the kernel ring buffer, runsdmesg --clear, and loads the profile withapparmor_parser --warn=all --replace --skip-cache --base=profiles/apparmor.d; anything on stderr fails the task (spread.yaml:194-223).restore-eachremoves the profile, restoresauditdand the rate limit, then fails on anyDENIEDline indmesgunless every line matches one of the newline-separated regular expressions inEXPECT_DENIALS, and every expression matched a line (spread.yaml:225-281).tests/bin/actual-profile-ofruns the program under gdb, stops at_startand prints/proc/PID/attr/current, such astar (enforce)(tests/bin/actual-profile-of).tests/regression/apparmor/task.yamlruns one regression script per variantTEST/NAME, withXFAILlisting systems where it is expected to fail; the suite's prepare checks that every test in the Makefile'sTESTShas a variant (task.yaml, spread.yaml:306-338). A script such aschdir.shusesgenprofileandrunchecktestfromprologue.inc(chdir.sh).- Without spread, the README runs the regression suite as root with
sudo make tests(README.md:341-367).
Read more
- tests/regression/apparmor/, apparmor.net: Enforce profile correctness with spread.
- Reading a denial: the
DENIEDlines the suite greps for.
08 One spread.yaml, seven systems
spread.yaml: backend garden, seven systems, prepare builds the tree, five suites. Fedora skips the suites that need AppArmor in the kernel.
To run those tests somewhere safe, AppArmor keeps a spread.yaml at the top of the tree. Spread copies the tree to /tmp/apparmor on each test machine and runs the project's prepare script once, which builds libapparmor, the parser, the utilities and the rest from source. Then it runs each suite's tasks. The machines come from one backend, garden, with seven systems of Debian, Ubuntu, openSUSE and Fedora. Fedora's kernel doesn't enable AppArmor, so the kernel suites skip it, and it runs only the unit tests. Most systems are manual: a plain spread run leaves them alone, unless you name them.
Details and sources
Details
project: apparmor, one backendgardenoftype: adhoc(spread.yaml:1-7);path: /tmp/apparmor(spread.yaml:149).exclude:keeps build output, VM images andbzImageout of the copy (spread.yaml:94-145).preparerunsautogen.shandconfigure --prefix=/usr --with-perl --with-pythononly whenlibraries/libapparmor/Makefileis missing, which keeps repeated-reuseruns fast, then builds libapparmor, the parser and binutils withUSE_INTREE=1, utils,mod_apparmorandpam_apparmor(spread.yaml:151-179).debug-eachprintsuname -a(spread.yaml:181-183).- Systems:
opensuse-cloud-tumbleweed,debian-cloud-12,debian-cloud-13,ubuntu-cloud-22.04,ubuntu-cloud-24.04(allmanual: true),ubuntu-cloud-24.10andfedora-cloud-41(spread.yaml:58-92). The comment above the list says all but one Ubuntu system are manual;fedora-cloud-41has nomanual:line either. - Suites:
tests/profiles/,utils/,tests/unit/,tests/regression/,tests/snapd/. The first, fourth and fifth exclude-fedora-cloud-*with the comment "AppArmor is not enabled in the kernel" (spread.yaml:185-345). tests/unit/wraps the suites of the first half as spread tasks:tests/unit/parser(one variant per unit program),parser-tst-*(one task per parser/tst target),libapparmor,binutils(tests/unit/parser/task.yaml).utils/test/task.yamlhas one variant pertest-*.py.- Spread variant names cannot hold dashes, so
utils/test/task.yamlmapsTEST/aa_decode: aa-decodeand so on; the suite's prepare fails withmissing test variantwhen a file has no entry (spread.yaml:285-303). - The README's example of one task:
spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure;run-spread.shruns six systems in turn (README.md:301-339, run-spread.sh).
Read more
- apparmor.net: Enforce profile correctness with spread.
- Running spread on image-garden and spread names and garden names, on the image-garden map.
09 Allocate and discard, a small VM on purpose
allocate exports QEMU_MEM_OPTION="-m 1536" and QEMU_SMP_OPTION="-smp 2", then runs image-garden allocate SYSTEM.ARCH. discard runs image-garden discard on the address.
The garden backend is spread's adhoc type: two shell scripts. Allocate asks image-garden for a machine, and spread connects to the address it prints. Discard hands that address back. But first, allocate makes the machine small on purpose: one and a half gigabytes of memory and two CPUs, set through QEMU_MEM_OPTION and QEMU_SMP_OPTION. That's just enough to link the parser, because prepare builds with one compiler per CPU, and each CPU more needs memory that small CI machines don't have. Several small workers beat one big one, so most systems ask for four.
Details and sources
Details
allocateresetsPATHtoSPREAD_HOST_PATHwhen it is set: spread adds/snap/bintoPATH, and the image-garden snap cannot run/snap/binentry points (spread.yaml:8-16).discarddoes the same, then runsimage-garden discard "$SPREAD_SYSTEM_ADDRESS"(spread.yaml:52-57).QEMU_MEM_OPTION="-m 1536"andQEMU_SMP_OPTION="-smp 2". The comment: "Use just enough RAM to link the parser on a virtual system with two cores. Using more cores may easily consume more memory, due to make -j$(nproc) ... It is better to have more workers than to have one big worker with lots of resources." (spread.yaml:17-23).ARCH="$(uname -m)", thenexec image-garden allocate "$SPREAD_SYSTEM"."$ARCH" -usb -usbdevice keyboard -usbdevice mouse(spread.yaml:24, spread.yaml:48-51). The hook does not pass--spread.workers: 4onopensuse-cloud-tumbleweed,debian-cloud-12,debian-cloud-13,ubuntu-cloud-22.04andubuntu-cloud-24.10;ubuntu-cloud-24.04andfedora-cloud-41use one (spread.yaml:62-92).run-spread.shis meant for one quad-core machine with 8 GB of RAM (run-spread.sh:1-4).- How image-garden's defaults for these variables compare, and what allocate does without
--spread, are on the image-garden map: The .run script, image-garden allocate.
Read more
- The adhoc backend: allocate and discard, image-garden allocate, The .run script, on the image-garden map.
10 Booting a kernel you built
With ./bzImage present, allocate sets QEMU_STORAGE_OPTION to a plain -drive, empties QEMU_BOOT_FIRMWARE_OPTION and adds -kernel bzImage -append 'root=/dev/sda1 ...'.
Working on the kernel side? Put a bzImage file at the top of the tree, and allocate boots it. First it lets image-garden make the system as usual, so cloud-init's first boot runs on the distribution's own kernel. Then it changes three things. QEMU_STORAGE_OPTION becomes a plain drive, not the default virtio disk, which some images can't boot from. QEMU_BOOT_FIRMWARE_OPTION is set empty, so there's no EFI firmware at all. And QEMU gets your kernel directly, with a command line that names /dev/sda1 as the root, which fits Ubuntu's cloud images. Your kernel must also match the modules installed in the image.
Details and sources
Details
- The whole branch: spread.yaml:25-47. It runs only when a file named
bzImageexists in the directory spread runs from, the top of the tree.bzImageis inexclude:, so it is not copied to the VM (spread.yaml:142-145). The README mentions it in one line (README.md:338-339). image-garden make "$SPREAD_SYSTEM"."$ARCH".run "$SPREAD_SYSTEM"."$ARCH".qcow2 1>&2first, "so that first boot and initialization happen in a standardized manner".QEMU_STORAGE_OPTION="-drive file=$SPREAD_SYSTEM.$ARCH.qcow2,format=qcow2": "Some images may not support virtio enough for booting."QEMU_BOOT_FIRMWARE_OPTION="": "boot a kernel directly and bypass both EFI and BIOS".exec image-garden allocate ... -kernel bzImage -append 'root=/dev/sda1 ro console=tty1 console=ttyS0'; the comment says the command line is tuned to Ubuntu cloud images, whose root file system is the first partition. The kernel must be compatible with the modules in the image's file system.- Known difference between image-garden versions: in image-garden v0.4.2, the version of the CI container, the
.runscript expands${QEMU_BOOT_FIRMWARE_OPTION=...}, which keeps an empty value. From image-garden commit 6aabb92 (first released in v0.5) the.runscript uses${QEMU_BOOT_FIRMWARE_OPTION:=...}, which replaces an empty value with the default EFI drives, so with a newer image-garden the empty setting no longer removes the firmware. Read from both image-garden trees, not run. - The defaults these variables replace: The .run script on the image-garden map.
Read more
- The .run script, image-garden allocate, on the image-garden map.
11 Build dependencies baked into the image
.image-garden.mk defines DEBIAN_, UBUNTU_, OPENSUSE_tumbleweed_ and FEDORA_CLOUD_INIT_USER_DATA_TEMPLATE: packages, package_upgrade and, for openSUSE, security=apparmor.
Every task needs compilers, bison, flex, DejaGnu and a set of Python modules, and installing them in every VM would cost minutes on each run. So .image-garden.mk, next to spread.yaml, extends image-garden's cloud-init user data for each distribution. The Debian template adds a package list, Ubuntu reuses it, and openSUSE and Fedora have their own package names. Each also upgrades every package, so the image boots the newest kernel. openSUSE also switches its boot line from SELinux to AppArmor. All of it happens once, when the image is made. And the CI's container jobs read the Ubuntu list too, so there's one list.
Details and sources
Details
- The file says all package installation happens through cloud-init, and that bumping
unused=1(or touching the file) makes make re-create the images (.image-garden.mk:1-8). DEBIAN_CLOUD_INIT_USER_DATA_TEMPLATEstarts with$(CLOUD_INIT_USER_DATA_TEMPLATE), image-garden's common template, lists 31 packages (build tools,bison,flex,dejagnu,swig, Python modules, and what the regression tests need such asattr,dosfstools,fuse-overlayfs,proftpd-core,tinyproxy) and setspackage_upgrade: trueandpackage_update: true(.image-garden.mk:13-53).UBUNTU_CLOUD_INIT_USER_DATA_TEMPLATE=$(DEBIAN_CLOUD_INIT_USER_DATA_TEMPLATE)(.image-garden.mk:56).OPENSUSE_tumbleweed_CLOUD_INIT_USER_DATA_TEMPLATEappends two commands to the common template'sruncmd::sed -i -e 's/security=selinux/security=apparmor/g' /etc/default/grubandupdate-bootloader, then its own package names (.image-garden.mk:59-101).FEDORA_CLOUD_INIT_USER_DATA_TEMPLATEhas the Fedora names (.image-garden.mk:103-135).- The upgrade comment: image-garden shuts the machine down after the first boot, so the next boot runs the newest kernel (.image-garden.mk:46-52).
- The file also overrides the download URL of
ubuntu-cloud-24.04.x86_64to fetch the image over https like its checksums, because a cached http answer could keep returning an old serial (.image-garden.mk:136-141). .ubuntu-commonin.gitlab-ci.ymlinstallsyqandmake, includes.image-garden.mk, prints$(UBUNTU_CLOUD_INIT_USER_DATA_TEMPLATE)with$(info), and feedsyq '.packages | .[]'toapt-get install --no-install-recommends(.gitlab-ci.yml:26-38).- The README's table of build dependencies per distribution lists the same kinds of packages (README.md:100-115).
- How image-garden picks a template by group and release name, and what the common template contains: Several systems from one base image and Changing what cloud-init does on the image-garden map.
Read more
- Several systems from one base image, Changing what cloud-init does, on the image-garden map.
12 CI makes the image once and caches it
Cache image-garden-base-SYSTEM.ARCH holds dl/ and the EFI files; image-garden-custom-SYSTEM.ARCH- with a key from .image-garden.mk holds the prepared image. git restore-mtime keeps make from rebuilding it.
In GitLab CI the spread stage runs in the image-garden container image, on runners tagged kvm. An image job makes the Ubuntu system, and two caches keep it. The base cache holds the downloaded cloud image and the EFI files, keyed by system and architecture. The custom cache holds the prepared image, and its key is tied to .image-garden.mk, so a new package list starts a new cache. Make judges what's stale by modification times, and a fresh checkout gives every file a new one. So each job first runs git restore-mtime on that file, giving it its last commit's time.
Details and sources
Details
.image-garden-x86_64: stagespread,image: registry.gitlab.com/zygoon/image-garden:v0.4.2, tagslinux,x86_64,kvm,GARDEN_DL_DIR: dl(.gitlab-ci.yml:195-207).before_scriptrunsgit restore-mtime .image-garden.mk, thenimage-garden make --dry-run --debugfor the system's.run,.qcow2,.seed.iso,.user-dataand.meta-data, to show in the log why make would rebuild (.gitlab-ci.yml:208-218).script: whenimage-garden make --questionsays everything is up to date, the job deletes the files it would cache, so GitLab skips the upload of an unchanged cache; otherwise it makes them (.gitlab-ci.yml:219-236).- Caches: key
image-garden-base-${GARDEN_SYSTEM}.${ARCH}with$GARDEN_DL_DIR,efi-code.*.imgandefi-vars.*.img("never mutated so they are safe to share"); key prefiximage-garden-custom-${GARDEN_SYSTEM}.${ARCH}-withfiles: [.image-garden.mk]holding$GARDEN_SYSTEM.*,.seed.iso,.meta-dataand.user-data. Both usepolicy: $CACHE_POLICYandwhen: always(.gitlab-ci.yml:237-259). image-ubuntu-cloud-24.04-x86_64runs for tags, for merge requests that change.image-garden.mkor.gitlab-ci.yml, and for branch pipelines started with the inputbuild-test-imagesset to true (.gitlab-ci.yml:1-6, .gitlab-ci.yml:261-276).git restore-mtimesets the file's time from the last commit that changed it, so it needs that commit in the clone.- The same two ideas on GitHub, in image-garden-action: How the action caches.
Read more
- Running the container in CI, How the action caches, on the image-garden map.
13 Four CI jobs share the spread tasks
spread-ubuntu-cloud-24.04-x86_64: CACHE_POLICY: pull, parallel: 4, tasks split with split --number=r/N/4. The build-all job runs the prepare of spread.yaml in a container.
The spread job runs on Ubuntu 24.04 only, and it pulls the caches but never pushes them. It waits for the image job when the pipeline has one, and otherwise takes the image from the cache. It downloads a pinned build of spread, lists the tasks of three suites, regression, profiles and snapd, and deals them round robin to four parallel jobs. Each runs its share and keeps the logs and artifacts, even when it fails. The unit suites don't need a VM. They run in plain containers, on a build made by the same prepare script, taken from spread.yaml.
Details and sources
Details
.spread-x86_64extends the image job's template and installs spread from the generic package registry of thezygoon/spread-distproject (SPREAD_GITLAB_PROJECT_ID: "65375371") atSPREAD_REV: 413817eda7bec07a3885e0717c178b965f8924e1(.gitlab-ci.yml:278-310).script:./spread -list $SPREAD_ARGS | sort | split --number=r/"${CI_NODE_INDEX:-1}"/"${CI_NODE_TOTAL:-1}" | xargs --verbose ./spread -v -artifacts ./spread-artifacts -v, with the output also written tospread-logs/SYSTEM.log;spread-logsandspread-artifactsare keptwhen: always(.gitlab-ci.yml:311-323).spread-ubuntu-cloud-24.04-x86_64:SPREAD_ARGSnamestests/regression/,tests/profiles/andtests/snapd/ongarden:ubuntu-cloud-24.04,CACHE_POLICY: pull,needs: image-ubuntu-cloud-24.04-x86_64withoptional: true,parallel: 4(.gitlab-ci.yml:325-335).- If neither the image job nor the cache provides the prepared image,
image-garden allocatemakes it before booting. build-allrunsyq -r '.prepare' <spread.yaml | SPREAD_PATH=. bash -xeuinubuntu:latestand keeps the build as an artifact;test-libapparmor,test-parser,test-binutils,test-utils,test-mod-apparmorandtest-profilesrunmake checkon it (.gitlab-ci.yml:44-134).
Read more
- Running the container in CI, Running spread on image-garden, on the image-garden map.
14 Where the tests lead
Roads on this map: the backend pipeline, minimization, denials, aa-logprof, kernel features. On the image-garden map: spread, allocate, the run script, cloud-init templates, CI.
That's how AppArmor tests itself: make check in the parser, libapparmor and the utilities, and spread on image-garden machines for everything that needs a kernel. Where next? On this map, the backend pipeline and minimization decks explain what the equality and minimize tests compare. Reading a denial and aa-logprof show the log lines that test_multi parses. Kernel features covers the files that stand in for a kernel. And on the image-garden map you'll find spread's adhoc backend, allocate and the QEMU options, cloud-init templates, and the image-garden container in CI.
Details and sources
Read more
- On this map: The backend pipeline, Minimisation, Reading a denial, aa-logprof, Kernel features and the policy ABI, From profile text to kernel.
- On the image-garden map: Running spread on image-garden, The adhoc backend: allocate and discard, spread names and garden names, image-garden allocate, The .run script, Several systems from one base image, Changing what cloud-init does, Running the container in CI, How the action caches.
- parser/tst/, tests/regression/apparmor/, apparmor.net: Enforce profile correctness with spread, wiki: Integrating-Test-plan.
Roads from here
RUNNING THE PARSER
- Reading a denial — the log lines that libapparmor's test_multi cases parse
- Kernel features and the policy ABI — the features files that stand in for a kernel in parser tests
TOOLS · utils/
- aa-logprof — turning log lines into profile rules, as the .profile cases check
THE DFA ENGINE · libapparmor_re
- The backend pipeline — the backend stages that equality and minimize tests exercise
- Minimisation — what the minimize tests check
image-garden map ↗
- Running spread on image-garden — on the image-garden map, how spread runs tasks on image-garden VMs
- The adhoc backend: allocate and discard — on the image-garden map, the allocate and discard hooks in general
- image-garden allocate — on the image-garden map, what image-garden allocate does with the system name
- The .run script — on the image-garden map, the QEMU option variables and their defaults
- Several systems from one base image — on the image-garden map, per-distribution cloud-init templates
- Changing what cloud-init does — on the image-garden map, what a project's .image-garden.mk can change
- Running the container in CI — on the image-garden map, running the image-garden container in GitLab CI
- spread names and garden names — on the image-garden map, spread system names and the architecture suffix
- How the action caches — on the image-garden map, the same image caching on GitHub