Testing AppArmor

parser, library, utils for AppArmor developers

make check runs the parser's C unit tests and parser/tst suites, libapparmor's automake and DejaGnu tests and the utilities' unittest files, without root or an AppArmor kernel. The tests that load policy run under spread on image-garden VMs of several distributions, with build dependencies baked into the images by .image-garden.mk and, in GitLab CI, prepared images cached by a key on that file.

On the map: testing

▶ Watch this deck — 14 scenes, ~10 min

Worth seeing first

Part of journeys

01 You changed the parser, did anything break?

TESTING APPARMOR you changed how the parser compiles one rule: did anything else break? suites next to the code parser/ C unit tests, parser/tst suites libraries/libapparmor/ automake tests, DejaGnu utils/ unittest files in utils/test $ make check as you: no root, no AppArmor in the kernel suites that load policy tests/ regression, profiles, snapd a real kernel, a machine you can break debian-cloud-13 ubuntu-cloud-24.04 opensuse-cloud-… fedora-cloud-41 spread + image-garden
A change to the parser raises the question whether anything broke. On the left, three directories with their own suites: parser/ with C unit tests and parser/tst, libraries/libapparmor/ with automake tests and DejaGnu, utils/ with unittest files; make check runs them as you, with no root and no AppArmor in the kernel. On the right, tests/ with the regression, profile and snapd tests loads policy into a kernel, so it runs on four kinds of virtual machine, Debian, Ubuntu, openSUSE and Fedora, through spread and image-garden.

make check in parser/, libraries/libapparmor/ and utils/ needs neither root nor AppArmor in the kernel. The suites under tests/ load policy, so they run in spread on image-garden VMs.

You've changed how the parser compiles one rule, and before you open a merge request you want to know that nothing else broke. AppArmor's tests live next to the code they test. The parser, libapparmor and the Python utilities each have their own suite, and make check runs it as you, with no root and no AppArmor in your kernel. The tests directory holds the rest: tests that load policy into a real kernel and watch what it does. Those need a machine you can break, so AppArmor runs them in virtual machines of several distributions, with spread and image-garden. This deck follows both halves.

Details and sources

Details

  • The README's "AppArmor Testsuites" section lists how to run each suite and where its documentation lives (README.md:290-400). Each component's own make check is listed with its build steps (README.md:141-216).
  • make check in parser/, libraries/libapparmor/ and utils/ drives the in-tree parser and library; none of them writes to the kernel. The parser suites compile with -S or -QKSq and never load policy.
  • tests/regression/apparmor/ needs root: the README runs it with sudo make tests (README.md:341-367). tests/profiles/ and tests/snapd/ exist only as spread tasks.
  • The README also says the test steps gain little from make -j: the harnesses spawn a few long-lived runners that mostly run sequentially (README.md:133-139).
  • Other suites not covered here: binutils (make -C binutils check), changehat/mod_apparmor, profiles (make -C profiles check), tests/stress.

Read more

02 What make check runs in the parser

MAKE CHECK IN PARSER/ $ make -C parser check check_pod_files podchecker *.pod -DUNIT_TEST tst_regex tst_misc tst_symtab tst_variable tst_lib tst_interface tst_network MY_TEST(statement, error) FAIL: error exit status non-zero $ make -C tst tests error_output errors.py caching caching.py minimize minimize.sh equality equality.sh dirtest dirtest.sh parser_sanity simple.pl $ make -C parser -j$(nproc) tst_binaries the step that runs in parallel
make -C parser check runs three things. check_pod_files runs podchecker on the manual pages. Seven unit programs, tst_regex, tst_misc, tst_symtab, tst_variable, tst_lib, tst_interface and tst_network, are built from the parser's sources with -DUNIT_TEST; a failing MY_TEST prints FAIL and makes the program exit non-zero. make -C tst tests runs error_output, caching, minimize, equality, dirtest and parser_sanity against the parser just built. Building tst_binaries with make -j is the part that runs in parallel.

make -C parser check runs check_pod_files, the seven tst_* unit programs and make -C tst tests. Build tst_binaries with -j first to save time.

Make check in the parser does three things. It checks the manual pages. It builds seven small C programs from the parser's own sources, compiled with UNIT_TEST defined, so each source file can carry a main function full of checks. A failed check prints FAIL, and the program exits non-zero. Then it runs make tests in parser/tst, the slower suites that drive the parser binary you just built: the error messages, the cache, minimization, equality, loading a directory, and the big one, parser sanity. Building the seven programs is the step that runs in parallel, so the README builds them first, with make -j.

Details and sources

Details

  • check: check_pod_files tests (Makefile:438-445). check_pod_files runs podchecker -warning -warning *.pod (Make.rules:145-147).
  • TESTS = tst_regex tst_misc tst_symtab tst_variable tst_lib tst_interface tst_network, compiled with TEST_CXXFLAGS = ... -DUNIT_TEST and linked with the parser's objects minus the one under test (Makefile:186-187, Makefile:417-425). tst_lib comes from lib.c and tst_network from network.cc; the others from parser_NAME.c.
  • MY_TEST(statement, error) prints FAIL: error and sets rc = 1 (unit_test.h:49-53). Each file's main() sits under #ifdef UNIT_TEST, as in parser_regex.c (parser_regex.c:1481, parser_regex.c:1884-1910).
  • The tests target runs each binary, then $(MAKE) -s -C tst tests.
  • In parser/tst: tests: error_output caching minimize equality dirtest parser_sanity (Makefile:33). error_output runs errors.py, caching runs pytest -n auto caching.py, minimize, equality and dirtest run their .sh scripts, parser_sanity runs prove on simple.pl after gen_tests (Makefile:49-75).
  • parser_sanity-no-gen skips the generated profiles for faster manual runs; valgrind runs every simple test under valgrind (Makefile:57-60, Makefile:77-78).
  • Without USE_SYSTEM, the parser/tst makefile points LD_LIBRARY_PATH at the in-tree libapparmor (Makefile:11-21); VERBOSE=1 makes prove and pytest verbose.
  • The README builds the unit programs with make -j $(nproc) tst_binaries before make check (README.md:165-173).

Read more

03 A parser test case is a profile with a verdict

PARSER SANITY simple.pl Perl, run by prove parser/tst/simple_tests/capability/bad_1.sd #=DESCRIPTION fail CAP_XXX syntax. #=EXRESULT FAIL /does/not/exist { capability CAP_CHOWN, } apparmor_parser -S nothing is loaded -M features_files/features.all exit status 1 FAIL expected: ok #=TODO a known failure, reported as TODO #=DISABLED skipped, not run a new case is one more .sd file, next to 2,250 others
simple.pl walks parser/tst/simple_tests/ and pipes each .sd file, here capability/bad_1.sd with #=DESCRIPTION fail CAP_XXX syntax and #=EXRESULT FAIL around a profile with capability CAP_CHOWN, into apparmor_parser -S with -M features_files/features.all standing in for the kernel. The parser refuses with exit status 1, which matches FAIL, so the test passes. #=TODO marks a known failure and #=DISABLED skips a case. A new case is one more file next to 2,250 others.

simple.pl pipes each simple_tests/**/*.sd into apparmor_parser -S and compares the exit status with #=EXRESULT. A new case is a new file.

Most parser cases need no code. Parser sanity runs simple.pl, a Perl script that walks parser/tst/simple_tests and feeds every .sd file it finds to the parser. It compiles with -S, so nothing is loaded, and a fixed features file stands in for the kernel. Each file carries its verdict in a comment. EXRESULT PASS means the parser must exit with zero; EXRESULT FAIL means it must refuse. DESCRIPTION names the case, and TODO marks a known failure. To add a case, drop one file into the right directory, next to more than two thousand others.

Details and sources

Details

  • simple.pl collects every *.sd file under simple_tests/, recursively, sorted (simple.pl:148-176).
  • For each file it runs $APPARMOR_PARSER --config-file=./parser.conf -M features_files/features.all -S -I simple_tests/ with stdout and stderr discarded, and writes the file's lines to the parser's stdin (simple.pl:79-116). -S writes the binary to stdout instead of loading it; -M takes the kernel features from a file.
  • Directives: #=DESCRIPTION text, #=EXRESULT PASS|FAIL (default PASS), #=TODO (an expected failure; a TODO that passes is reported as a failure), #=DISABLED (not run, reported as a TODO). A core dump always fails (simple.pl:99-145). Each case has a 480 second timeout (simple.pl:13).
  • The make target is parser_sanity, which first runs gen_tests (gen-xtrans.py, gen-dbus.py, gen-conditionals.py) to write more cases into simple_tests/generated_* (Makefile:35-55).
  • The tree holds 2,250 .sd files under parser/tst/simple_tests/ (counted with git ls-tree), before the generated ones. The slide's example is capability/bad_1.sd.
  • The README for this suite describes the directives and how to add a case (parser/tst/README).
  • The utilities parse the same files: utils/test/test-parser-simple-tests.py keeps lists of files to skip or to expect the opposite result for (test-parser-simple-tests.py:19-60). A new case the utilities handle differently from the parser must be listed there.

Read more

04 When two spellings must compile the same

EQUALITY /t { dbus send, } /t { dbus write, } /t { dbus w, } apparmor_parser -QKSq no load, no cache, quiet sha256sum: A sha256sum: A sha256sum: A verify_binary_equality "dbus send" \ "/t { dbus send, }" \ "/t { dbus write, }" \ "/t { dbus w, }" a description the known-good profile profiles that must hash the same verify_binary_inequality the hashes must differ errors.py: stderr must be exactly AppArmor parser error for errors/single.sd in profile errors/single.sd at line 3: …
Three spellings of one rule, /t { dbus send, }, /t { dbus write, } and /t { dbus w, }, are each piped to apparmor_parser -QKSq, which compiles to stdout without loading, without the cache and quietly; sha256sum gives the same hash A for all three. In equality.sh this is one verify_binary_equality call with a description, the known-good profile and the profiles under test; verify_binary_inequality asserts that hashes differ. errors.py instead compares stderr exactly, for example AppArmor parser error for errors/single.sd in profile errors/single.sd at line 3: Could not open 'failure'.

equality.sh compiles with -QKSq, hashes the binary with sha256sum and compares. errors.py checks the exact text on stderr.

Pass or fail isn't always enough. Say you taught the parser a shorter spelling for a D-Bus permission. It should compile to exactly the policy that the long spelling does. That's what equality.sh checks. It pipes each profile to the parser, which compiles it to standard output without loading it or touching the cache, and it hashes the result. verify_binary_equality takes a description, a known good profile and the profiles that must hash the same. verify_binary_inequality asserts the opposite. And when the exact wording of an error matters, errors.py compares the parser's message character for character.

Details and sources

Details

  • hash_binary_policy() pipes the profile to $APPARMOR_PARSER --features-file features_files/FILE -QKSq and runs sha256sum on the output (equality.sh:92-125). -Q skips the kernel load, -K the cache, -S writes to stdout, -q is quiet.
  • verify_binary MODE DESCRIPTION KNOWN-GOOD PROFILE... hashes the known-good profile, then each profile; modes are equality, inequality, xequality and xinequality (the x modes are expected failures) (equality.sh:144-200, equality.sh:247-268).
  • A failed equality prints FAIL: Hash values do not match with the parser command and both profiles (equality.sh:184-193).
  • The slide's example is the first verify_set case, dbus send against dbus write and dbus w, run for several pairs of qualifiers ($p1, $p2) (equality.sh:305-320).
  • errors.py runs apparmor_parser --config-file=./parser.conf -S -I errors FILE and asserts the exit status and the exact stderr, after removing two messages about the kernel (errors.py:29-76).

Read more

05 libapparmor, a log line in and a record out

LIBAPPARMOR $ make -C libraries/libapparmor check tst_aalogmisc tst_aalogparse_cpp tst_aalogparse_reentrancy tst_aalogparse_oldname tst_features tst_kernel src/: automake's TESTS, run by make check DejaGnu in testsuite/ test_multi/avc_audit_01.in type=AVC msg=audit(1279948288.415:39): apparmor="DENIED" operation="open" … test_multi.multi Event type: AA_RECORD_DENIED Operation: open Profile: /usr/sbin/cupsd Name: /home/user/.ssh/ Denied Mask: r = avc_audit_01.out stderr = avc_audit_01.err a new case: NAME.in NAME.out 179 cases so far
make -C libraries/libapparmor check first runs six automake test programs in src/: tst_aalogmisc, tst_aalogparse_cpp, tst_aalogparse_reentrancy, tst_aalogparse_oldname, tst_features and tst_kernel. Then DejaGnu in testsuite/ runs test_multi.multi on each test_multi/NAME.in, here an audit line type=AVC apparmor=DENIED operation=open, and compares the fields it prints, Event type AA_RECORD_DENIED, Operation open, Profile /usr/sbin/cupsd, Name /home/user/.ssh/, Denied Mask r, with avc_audit_01.out, and stderr with the empty avc_audit_01.err. A new case is a .in file and a .out file side by side.

make -C libraries/libapparmor check runs six tst_* programs and DejaGnu. Each testsuite/test_multi/NAME.in is parsed and compared with NAME.out and NAME.err.

Make check in libapparmor is automake's. It builds six test programs in the source directory, for the log parser, the features API and the kernel interface, and runs them. Then DejaGnu takes over in the testsuite directory. Its biggest part is test_multi: a directory of cases, each an audit line in a .in file. A small program parses the line with libapparmor and prints every field it found. DejaGnu compares that with the matching .out file, and standard error with the .err file. To add a case, you put a log line and the fields you expect side by side.

Details and sources

Details

  • check_PROGRAMS = tst_aalogmisc tst_aalogparse_cpp tst_aalogparse_reentrancy tst_aalogparse_oldname tst_features tst_kernel and TESTS = $(check_PROGRAMS), linked statically against .libs/libapparmor.a (Makefile.am:71-92). check-local also checks the shared library's file name (Makefile.am:94-96).
  • testsuite/Makefile.am sets AUTOMAKE_OPTIONS = dejagnu, builds test_multi.multi from test_multi.c, and fails check when libaalogparse.log is missing or contains ERROR (Makefile.am:1-25).
  • multi_test.exp calls multi-run for every *.multi program (multi_test.exp:1-23). multi-run runs the program on each test_multi/*.in, writes out/NAME.out and out/NAME.err, and diffs them with NAME.out and NAME.err (libaalogparse_init.exp:86-160).
  • test_multi.c reads the line, calls parse_record() and prints each field of the aa_log_record (test_multi.c:16-79).
  • The slide's case is avc_audit_01 (.in, .out). The tree has 179 .in files in test_multi/; 142 of them also have a .profile file, the rules the utilities should derive from the line.
  • DejaGnu must be installed; without it check stops with *** libaalogparse.log not found - is dejagnu installed? ***.

Read more

06 The utilities, a table of cases and one method

UTILS $ make -C utils check pyflakes, then each test/test-*.py libraries/libapparmor parser/apparmor_parser profiles/apparmor.d utils/test/test-capability.py class CapabilityTestParseInvalid(AATest): tests = ( ('capability', (AppArmorException, False)), ('network,', (AppArmorException, False)), ) def _run_test(self, rawrule, expected): self.parseInvalidRule(CapabilityRule, rawrule, expected) setup_all_loops(__name__) test_0 test_1 test_2 test_3 a new case: one more pair in tests two files reach across: test-parser-simple-tests.py test-libapparmor-test_multi.py
make -C utils check runs pyflakes, then every utils/test/test-*.py against the tree's libapparmor, parser/apparmor_parser and profiles/apparmor.d. In test-capability.py the class CapabilityTestParseInvalid derives from AATest and holds a tests tuple of pairs such as ('capability', (AppArmorException, False)); one method, _run_test, checks a pair. setup_all_loops turns each pair into its own test, test_0, test_1 and so on. A new case is one more pair. test-parser-simple-tests.py reads every parser simple test and test-libapparmor-test_multi.py turns libapparmor's log lines into profiles.

make -C utils check runs every test/test-*.py against the in-tree library and parser. An AATest lists tests pairs; setup_all_loops() makes one test per pair.

The Python utilities use unittest, with one file per area, like test-capability.py. Make check in utils runs pyflakes, then every test file against the library, the parser and the profiles of your tree. Most test classes derive from AATest and hold a tuple called tests, made of pairs: an input and what it should give. One method, _run_test, checks a pair, and setup_all_loops turns every pair into a test of its own. So a new case is often one more line. Two files reach across: one parses every parser simple test, the other turns libapparmor's log lines into profiles.

Details and sources

Details

  • utils/Makefile: check: check_severity_db check_pod_files check_lint, then make -C test check and make -C vim check; check_lint runs $(PYFLAKES) on the tools, the apparmor module and test/*.py (Makefile:91-104).
  • utils/test/Makefile runs each test-*.py with every interpreter in PYTHON_VERSIONS (by default python3) and sets PYTHONPATH, LD_LIBRARY_PATH, __AA_CONFDIR, __AA_BASEDIR=../../profiles/apparmor.d and __AA_PARSER=../../parser/apparmor_parser; USE_SYSTEM=1 uses the installed ones instead (Makefile:27-42, Makefile:91-92, Make.rules:47-56).
  • One file: make -C utils/test check-one-test-NAME runs test-NAME.py (Makefile:100-102). One test: python3 ./test-FILE.py Class.test_name with the variables from the README (utils/test/README.md).
  • setup_tests_loop() adds test_0, test_1, ... to a class, one per (test_data, expected) pair in tests, each calling self._run_test(test_data, expected) (common_test.py:73-89). test-example.py is the template (test-example.py:18-57).
  • The slide's class: test-capability.py:243-254.
  • test-parser-simple-tests.py parses every file under parser/tst/simple_tests and keeps lists of exceptions (test-parser-simple-tests.py:19-60). test-libapparmor-test_multi.py checks the log parser against each .out file and, for cases with a .profile, the profile the tools derive from the log (test-libapparmor-test_multi.py:178-200, test-libapparmor-test_multi.py:287-315).
  • CI also runs make -C utils/test coverage-regression, which fails when a file listed as fully covered drops below 100% or a file listed as incomplete reaches it (Makefile:113-120, .gitlab-ci.yml:93-114).
  • A new test-*.py must also be added to utils/test/task.yaml as a spread variant; the utils/ suite's prepare checks it (spread.yaml:283-303).

Read more

  • aa-logprof: the tool whose log-to-profile step the .profile cases check.

07 Tests that need a kernel

TESTS THAT NEED A KERNEL tests/regression/apparmor/ confined C programs, run as root tests/profiles/tar/task.yaml a shipped profile, the real program prepare-each apparmor_parser --replace --warn=all --skip-cache profiles/apparmor.d/tar a warning: fail execute tar -czf archive.tar file.txt tar -xzf archive.tar actual-profile-of tar | MATCH 'tar \(enforce\)' the kernel: profile tar loaded, enforce mode restore-each apparmor_parser --remove profiles/apparmor.d/tar dmesg | grep DENIED a DENIED line: fail unless it matches a regex in EXPECT_DENIALS
tests/ holds suites that need AppArmor in the kernel: tests/regression/apparmor confines small C programs as root, and tests/profiles has one task per shipped profile. For tests/profiles/tar, prepare-each loads profiles/apparmor.d/tar with apparmor_parser --replace --warn=all --skip-cache, and any warning fails the task. The task runs tar -czf and tar -xzf and checks with actual-profile-of that tar runs as tar (enforce). restore-each removes the profile and greps the kernel log for DENIED; a DENIED line fails the test unless it matches EXPECT_DENIALS.

A task in tests/profiles/NAME/ loads profiles/apparmor.d/NAME, runs the program and fails on any parser warning or unexpected DENIED line.

Everything so far runs as you, without AppArmor in the kernel. The tests directory is different. Its regression tests confine small C programs and check what the kernel allows. Its profile tests check shipped profiles against the real programs. Take tests/profiles/tar. Before the task, spread loads the tar profile from the tree with --warn=all, and any warning fails it. The task archives and extracts a file, then checks that tar ran under its profile, in enforce mode. Afterwards the profile is removed, and any DENIED line in the kernel log fails the test, unless EXPECT_DENIALS says it was wanted.

Details and sources

Details

  • The slide's task: tests/profiles/tar/task.yaml. The profile name is the task's directory name unless PROFILE_NAME is set (spread.yaml:210-211).
  • prepare-each saves and zeroes kernel.printk_ratelimit, stops auditd so denials reach the kernel ring buffer, runs dmesg --clear, and loads the profile with apparmor_parser --warn=all --replace --skip-cache --base=profiles/apparmor.d; anything on stderr fails the task (spread.yaml:194-223).
  • restore-each removes the profile, restores auditd and the rate limit, then fails on any DENIED line in dmesg unless every line matches one of the newline-separated regular expressions in EXPECT_DENIALS, and every expression matched a line (spread.yaml:225-281).
  • tests/bin/actual-profile-of runs the program under gdb, stops at _start and prints /proc/PID/attr/current, such as tar (enforce) (tests/bin/actual-profile-of).
  • tests/regression/apparmor/task.yaml runs one regression script per variant TEST/NAME, with XFAIL listing systems where it is expected to fail; the suite's prepare checks that every test in the Makefile's TESTS has a variant (task.yaml, spread.yaml:306-338). A script such as chdir.sh uses genprofile and runchecktest from prologue.inc (chdir.sh).
  • Without spread, the README runs the regression suite as root with sudo make tests (README.md:341-367).

08 One spread.yaml, seven systems

SPREAD.YAML backends: garden: type: adhoc path: /tmp/apparmor prepare: | autogen.sh, configure make: libapparmor, parser, binutils, utils, changehat suites: tests/profiles/ utils/ tests/unit/ tests/regression/ tests/snapd/ opensuse-cloud-tumbleweed manual debian-cloud-12 manual debian-cloud-13 manual ubuntu-cloud-22.04 manual ubuntu-cloud-24.04 manual ubuntu-cloud-24.10 fedora-cloud-41 fedora-cloud-41 no AppArmor in its kernel: only utils/ and tests/unit/ a manual system runs only when you name it $ spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure
An outline of spread.yaml: backend garden of type adhoc, path /tmp/apparmor, a prepare script that runs autogen.sh and configure and builds libapparmor, the parser, binutils, utils and changehat, and five suites: tests/profiles/, utils/, tests/unit/, tests/regression/ and tests/snapd/. The garden backend lists seven systems: opensuse-cloud-tumbleweed, debian-cloud-12, debian-cloud-13, ubuntu-cloud-22.04 and ubuntu-cloud-24.04, all manual, and ubuntu-cloud-24.10 and fedora-cloud-41. Fedora's kernel does not enable AppArmor, so it runs only utils/ and tests/unit/. A manual system runs only when named, as in spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure.

spread.yaml: backend garden, seven systems, prepare builds the tree, five suites. Fedora skips the suites that need AppArmor in the kernel.

To run those tests somewhere safe, AppArmor keeps a spread.yaml at the top of the tree. Spread copies the tree to /tmp/apparmor on each test machine and runs the project's prepare script once, which builds libapparmor, the parser, the utilities and the rest from source. Then it runs each suite's tasks. The machines come from one backend, garden, with seven systems of Debian, Ubuntu, openSUSE and Fedora. Fedora's kernel doesn't enable AppArmor, so the kernel suites skip it, and it runs only the unit tests. Most systems are manual: a plain spread run leaves them alone, unless you name them.

Details and sources

Details

  • project: apparmor, one backend garden of type: adhoc (spread.yaml:1-7); path: /tmp/apparmor (spread.yaml:149). exclude: keeps build output, VM images and bzImage out of the copy (spread.yaml:94-145).
  • prepare runs autogen.sh and configure --prefix=/usr --with-perl --with-python only when libraries/libapparmor/Makefile is missing, which keeps repeated -reuse runs fast, then builds libapparmor, the parser and binutils with USE_INTREE=1, utils, mod_apparmor and pam_apparmor (spread.yaml:151-179). debug-each prints uname -a (spread.yaml:181-183).
  • Systems: opensuse-cloud-tumbleweed, debian-cloud-12, debian-cloud-13, ubuntu-cloud-22.04, ubuntu-cloud-24.04 (all manual: true), ubuntu-cloud-24.10 and fedora-cloud-41 (spread.yaml:58-92). The comment above the list says all but one Ubuntu system are manual; fedora-cloud-41 has no manual: line either.
  • Suites: tests/profiles/, utils/, tests/unit/, tests/regression/, tests/snapd/. The first, fourth and fifth exclude -fedora-cloud-* with the comment "AppArmor is not enabled in the kernel" (spread.yaml:185-345).
  • tests/unit/ wraps the suites of the first half as spread tasks: tests/unit/parser (one variant per unit program), parser-tst-* (one task per parser/tst target), libapparmor, binutils (tests/unit/parser/task.yaml). utils/test/task.yaml has one variant per test-*.py.
  • Spread variant names cannot hold dashes, so utils/test/task.yaml maps TEST/aa_decode: aa-decode and so on; the suite's prepare fails with missing test variant when a file has no entry (spread.yaml:285-303).
  • The README's example of one task: spread -v garden:ubuntu-cloud-24.04:tests/regression/apparmor:at_secure; run-spread.sh runs six systems in turn (README.md:301-339, run-spread.sh).

09 Allocate and discard, a small VM on purpose

ALLOCATE AND DISCARD spread backend garden, type: adhoc allocate: | ARCH="$(uname -m)" exec image-garden allocate \ "$SPREAD_SYSTEM"."$ARCH" \ -usb -usbdevice keyboard -usbdevice mouse ubuntu-cloud-24.04.x86_64 a VM, ssh on localhost localhost:PORT discard: image-garden discard "$SPREAD_SYSTEM_ADDRESS" export QEMU_MEM_OPTION="-m 1536" export QEMU_SMP_OPTION="-smp 2" prepare: make -j"$(nproc)" 2 CPUs: 2 compilers 1536 MB: enough to link VM VM VM VM workers: 4 several small beat one big
spread uses the garden backend of type adhoc. Its allocate script exports QEMU_MEM_OPTION="-m 1536" and QEMU_SMP_OPTION="-smp 2", sets ARCH from uname -m and execs image-garden allocate "$SPREAD_SYSTEM"."$ARCH" with USB keyboard and mouse options. The VM, for example ubuntu-cloud-24.04.x86_64 with 1536 MB and two CPUs, answers with an address spread connects to; discard runs image-garden discard on that address. Prepare builds with make -j$(nproc), so two CPUs mean two compilers, which 1536 MB can link the parser with. Most systems ask for four such workers.

allocate exports QEMU_MEM_OPTION="-m 1536" and QEMU_SMP_OPTION="-smp 2", then runs image-garden allocate SYSTEM.ARCH. discard runs image-garden discard on the address.

The garden backend is spread's adhoc type: two shell scripts. Allocate asks image-garden for a machine, and spread connects to the address it prints. Discard hands that address back. But first, allocate makes the machine small on purpose: one and a half gigabytes of memory and two CPUs, set through QEMU_MEM_OPTION and QEMU_SMP_OPTION. That's just enough to link the parser, because prepare builds with one compiler per CPU, and each CPU more needs memory that small CI machines don't have. Several small workers beat one big one, so most systems ask for four.

Details and sources

Details

  • allocate resets PATH to SPREAD_HOST_PATH when it is set: spread adds /snap/bin to PATH, and the image-garden snap cannot run /snap/bin entry points (spread.yaml:8-16). discard does the same, then runs image-garden discard "$SPREAD_SYSTEM_ADDRESS" (spread.yaml:52-57).
  • QEMU_MEM_OPTION="-m 1536" and QEMU_SMP_OPTION="-smp 2". The comment: "Use just enough RAM to link the parser on a virtual system with two cores. Using more cores may easily consume more memory, due to make -j$(nproc) ... It is better to have more workers than to have one big worker with lots of resources." (spread.yaml:17-23).
  • ARCH="$(uname -m)", then exec image-garden allocate "$SPREAD_SYSTEM"."$ARCH" -usb -usbdevice keyboard -usbdevice mouse (spread.yaml:24, spread.yaml:48-51). The hook does not pass --spread.
  • workers: 4 on opensuse-cloud-tumbleweed, debian-cloud-12, debian-cloud-13, ubuntu-cloud-22.04 and ubuntu-cloud-24.10; ubuntu-cloud-24.04 and fedora-cloud-41 use one (spread.yaml:62-92). run-spread.sh is meant for one quad-core machine with 8 GB of RAM (run-spread.sh:1-4).
  • How image-garden's defaults for these variables compare, and what allocate does without --spread, are on the image-garden map: The .run script, image-garden allocate.

10 Booting a kernel you built

YOUR OWN KERNEL ./bzImage at the top of the tree; spread does not copy it to the VM $ image-garden make "$SPREAD_SYSTEM.$ARCH.run" "$SPREAD_SYSTEM.$ARCH.qcow2" first boot, with cloud-init, on the distribution's own kernel image-garden's default with bzImage QEMU_STORAGE_OPTION -drive file=…,if=none,… -device virtio-blk,… -drive file=SYSTEM.ARCH.qcow2, format=qcow2 QEMU_BOOT_FIRMWARE_OPTION -drive if=pflash: EFI code -drive if=pflash: EFI vars "" no EFI, no BIOS extra QEMU arguments -kernel bzImage -append 'root=/dev/sda1 ro console=tty1 console=ttyS0' the kernel must match the modules installed in the image
With a bzImage file at the top of the tree, allocate first runs image-garden make for the system's .run and .qcow2 files, so cloud-init's first boot runs on the distribution's kernel. Then it changes three things. QEMU_STORAGE_OPTION goes from image-garden's virtio-blk disk to a plain -drive file=SYSTEM.ARCH.qcow2,format=qcow2. QEMU_BOOT_FIRMWARE_OPTION goes from two pflash drives with EFI code and variables to empty, so there is no EFI and no BIOS. QEMU gets -kernel bzImage and -append 'root=/dev/sda1 ro console=tty1 console=ttyS0'. The kernel must match the modules in the image.

With ./bzImage present, allocate sets QEMU_STORAGE_OPTION to a plain -drive, empties QEMU_BOOT_FIRMWARE_OPTION and adds -kernel bzImage -append 'root=/dev/sda1 ...'.

Working on the kernel side? Put a bzImage file at the top of the tree, and allocate boots it. First it lets image-garden make the system as usual, so cloud-init's first boot runs on the distribution's own kernel. Then it changes three things. QEMU_STORAGE_OPTION becomes a plain drive, not the default virtio disk, which some images can't boot from. QEMU_BOOT_FIRMWARE_OPTION is set empty, so there's no EFI firmware at all. And QEMU gets your kernel directly, with a command line that names /dev/sda1 as the root, which fits Ubuntu's cloud images. Your kernel must also match the modules installed in the image.

Details and sources

Details

  • The whole branch: spread.yaml:25-47. It runs only when a file named bzImage exists in the directory spread runs from, the top of the tree. bzImage is in exclude:, so it is not copied to the VM (spread.yaml:142-145). The README mentions it in one line (README.md:338-339).
  • image-garden make "$SPREAD_SYSTEM"."$ARCH".run "$SPREAD_SYSTEM"."$ARCH".qcow2 1>&2 first, "so that first boot and initialization happen in a standardized manner".
  • QEMU_STORAGE_OPTION="-drive file=$SPREAD_SYSTEM.$ARCH.qcow2,format=qcow2": "Some images may not support virtio enough for booting." QEMU_BOOT_FIRMWARE_OPTION="": "boot a kernel directly and bypass both EFI and BIOS".
  • exec image-garden allocate ... -kernel bzImage -append 'root=/dev/sda1 ro console=tty1 console=ttyS0'; the comment says the command line is tuned to Ubuntu cloud images, whose root file system is the first partition. The kernel must be compatible with the modules in the image's file system.
  • Known difference between image-garden versions: in image-garden v0.4.2, the version of the CI container, the .run script expands ${QEMU_BOOT_FIRMWARE_OPTION=...}, which keeps an empty value. From image-garden commit 6aabb92 (first released in v0.5) the .run script uses ${QEMU_BOOT_FIRMWARE_OPTION:=...}, which replaces an empty value with the default EFI drives, so with a newer image-garden the empty setting no longer removes the firmware. Read from both image-garden trees, not run.
  • The defaults these variables replace: The .run script on the image-garden map.

Read more

11 Build dependencies baked into the image

.IMAGE-GARDEN.MK define DEBIAN_CLOUD_INIT_USER_DATA_TEMPLATE $(CLOUD_INIT_USER_DATA_TEMPLATE) packages: - bison - dejagnu - flex … 31 packages in all package_upgrade: true package_update: true endef UBUNTU_CLOUD_INIT_USER_DATA_TEMPLATE= $(DEBIAN_CLOUD_INIT_USER_DATA_TEMPLATE) OPENSUSE_tumbleweed_…TEMPLATE apache2-devel pam-devel … FEDORA_…TEMPLATE httpd-devel pam-devel perl … - sed … s/security=selinux/ security=apparmor/ …grub - update-bootloader installed once, when the image is made include .image-garden.mk $(info $(UBUNTU_CLOUD_INIT_USER_DATA_TEMPLATE)) | yq '.packages | .[]' | xargs apt-get install CI's container jobs: one list
An excerpt of .image-garden.mk: define DEBIAN_CLOUD_INIT_USER_DATA_TEMPLATE starts with image-garden's own $(CLOUD_INIT_USER_DATA_TEMPLATE), adds packages such as bison, dejagnu and flex, and sets package_upgrade and package_update to true; UBUNTU_CLOUD_INIT_USER_DATA_TEMPLATE reuses it. OPENSUSE_tumbleweed_CLOUD_INIT_USER_DATA_TEMPLATE has its own package names and two commands that change security=selinux to security=apparmor in /etc/default/grub and run update-bootloader. FEDORA_CLOUD_INIT_USER_DATA_TEMPLATE has its own package names. The packages are installed once, when image-garden makes the image. The CI's container jobs include .image-garden.mk, print the Ubuntu template and feed its packages through yq to apt-get install.

.image-garden.mk defines DEBIAN_, UBUNTU_, OPENSUSE_tumbleweed_ and FEDORA_CLOUD_INIT_USER_DATA_TEMPLATE: packages, package_upgrade and, for openSUSE, security=apparmor.

Every task needs compilers, bison, flex, DejaGnu and a set of Python modules, and installing them in every VM would cost minutes on each run. So .image-garden.mk, next to spread.yaml, extends image-garden's cloud-init user data for each distribution. The Debian template adds a package list, Ubuntu reuses it, and openSUSE and Fedora have their own package names. Each also upgrades every package, so the image boots the newest kernel. openSUSE also switches its boot line from SELinux to AppArmor. All of it happens once, when the image is made. And the CI's container jobs read the Ubuntu list too, so there's one list.

Details and sources

Details

  • The file says all package installation happens through cloud-init, and that bumping unused=1 (or touching the file) makes make re-create the images (.image-garden.mk:1-8).
  • DEBIAN_CLOUD_INIT_USER_DATA_TEMPLATE starts with $(CLOUD_INIT_USER_DATA_TEMPLATE), image-garden's common template, lists 31 packages (build tools, bison, flex, dejagnu, swig, Python modules, and what the regression tests need such as attr, dosfstools, fuse-overlayfs, proftpd-core, tinyproxy) and sets package_upgrade: true and package_update: true (.image-garden.mk:13-53). UBUNTU_CLOUD_INIT_USER_DATA_TEMPLATE=$(DEBIAN_CLOUD_INIT_USER_DATA_TEMPLATE) (.image-garden.mk:56).
  • OPENSUSE_tumbleweed_CLOUD_INIT_USER_DATA_TEMPLATE appends two commands to the common template's runcmd:: sed -i -e 's/security=selinux/security=apparmor/g' /etc/default/grub and update-bootloader, then its own package names (.image-garden.mk:59-101). FEDORA_CLOUD_INIT_USER_DATA_TEMPLATE has the Fedora names (.image-garden.mk:103-135).
  • The upgrade comment: image-garden shuts the machine down after the first boot, so the next boot runs the newest kernel (.image-garden.mk:46-52).
  • The file also overrides the download URL of ubuntu-cloud-24.04.x86_64 to fetch the image over https like its checksums, because a cached http answer could keep returning an old serial (.image-garden.mk:136-141).
  • .ubuntu-common in .gitlab-ci.yml installs yq and make, includes .image-garden.mk, prints $(UBUNTU_CLOUD_INIT_USER_DATA_TEMPLATE) with $(info), and feeds yq '.packages | .[]' to apt-get install --no-install-recommends (.gitlab-ci.yml:26-38).
  • The README's table of build dependencies per distribution lists the same kinds of packages (README.md:100-115).
  • How image-garden picks a template by group and release name, and what the common template contains: Several systems from one base image and Changing what cloud-init does on the image-garden map.

12 CI makes the image once and caches it

CI: THE IMAGE registry.gitlab.com/zygoon/image-garden:v0.4.2 tags: linux x86_64 kvm job image-ubuntu-cloud-24.04-x86_64 $ image-garden make "$GARDEN_SYSTEM.$ARCH.run" "$GARDEN_SYSTEM.$ARCH.qcow2" … image-garden-base-SYSTEM.ARCH dl/ the cloud image efi-code.*.img efi-vars.*.img never changed image-garden-custom-SYSTEM.ARCH- + a key from .image-garden.mk SYSTEM.* SYSTEM.seed.iso SYSTEM.user-data SYSTEM.meta-data checkout .image-garden.mk is newer than the image: make rebuilds it $ git restore-mtime .image-garden.mk its time is its last commit's: the cached image is fresh
The spread stage of .gitlab-ci.yml runs in registry.gitlab.com/zygoon/image-garden:v0.4.2 on runners tagged linux, x86_64 and kvm. The job image-ubuntu-cloud-24.04-x86_64 runs image-garden make for the .run, .qcow2, seed.iso, user-data and meta-data files. Cache image-garden-base-ubuntu-cloud-24.04.x86_64 holds dl/, the downloaded cloud image, and the EFI code and variable files. Cache image-garden-custom-ubuntu-cloud-24.04.x86_64- with a key from .image-garden.mk holds the prepared system's files. A checkout gives .image-garden.mk the current time, so make would rebuild the image; git restore-mtime .image-garden.mk gives it the time of its last commit, so the cached image stays fresh.

Cache image-garden-base-SYSTEM.ARCH holds dl/ and the EFI files; image-garden-custom-SYSTEM.ARCH- with a key from .image-garden.mk holds the prepared image. git restore-mtime keeps make from rebuilding it.

In GitLab CI the spread stage runs in the image-garden container image, on runners tagged kvm. An image job makes the Ubuntu system, and two caches keep it. The base cache holds the downloaded cloud image and the EFI files, keyed by system and architecture. The custom cache holds the prepared image, and its key is tied to .image-garden.mk, so a new package list starts a new cache. Make judges what's stale by modification times, and a fresh checkout gives every file a new one. So each job first runs git restore-mtime on that file, giving it its last commit's time.

Details and sources

Details

  • .image-garden-x86_64: stage spread, image: registry.gitlab.com/zygoon/image-garden:v0.4.2, tags linux, x86_64, kvm, GARDEN_DL_DIR: dl (.gitlab-ci.yml:195-207).
  • before_script runs git restore-mtime .image-garden.mk, then image-garden make --dry-run --debug for the system's .run, .qcow2, .seed.iso, .user-data and .meta-data, to show in the log why make would rebuild (.gitlab-ci.yml:208-218).
  • script: when image-garden make --question says everything is up to date, the job deletes the files it would cache, so GitLab skips the upload of an unchanged cache; otherwise it makes them (.gitlab-ci.yml:219-236).
  • Caches: key image-garden-base-${GARDEN_SYSTEM}.${ARCH} with $GARDEN_DL_DIR, efi-code.*.img and efi-vars.*.img ("never mutated so they are safe to share"); key prefix image-garden-custom-${GARDEN_SYSTEM}.${ARCH}- with files: [.image-garden.mk] holding $GARDEN_SYSTEM.*, .seed.iso, .meta-data and .user-data. Both use policy: $CACHE_POLICY and when: always (.gitlab-ci.yml:237-259).
  • image-ubuntu-cloud-24.04-x86_64 runs for tags, for merge requests that change .image-garden.mk or .gitlab-ci.yml, and for branch pipelines started with the input build-test-images set to true (.gitlab-ci.yml:1-6, .gitlab-ci.yml:261-276).
  • git restore-mtime sets the file's time from the last commit that changed it, so it needs that commit in the clone.
  • The same two ideas on GitHub, in image-garden-action: How the action caches.

Read more

13 Four CI jobs share the spread tasks

CI: SPREAD spread-ubuntu-cloud-24.04-x86_64 CACHE_POLICY: pull image-ubuntu-cloud-24.04-x86_64 needs: optional: true or the prepared image from the cache $ curl … /packages/generic/spread/$SPREAD_REV/spread.amd64 $ ./spread -list $SPREAD_ARGS | sort | tests/regression/ tests/profiles/ tests/snapd/ split --number=r/"$CI_NODE_INDEX"/"$CI_NODE_TOTAL" | xargs ./spread -v -artifacts ./spread-artifacts -v 1/4 2/4 3/4 4/4 each keeps spread-logs/ and spread-artifacts/, even when it fails build-all: yq -r '.prepare' <spread.yaml | SPREAD_PATH=. bash -xeu then test-libapparmor, test-parser, test-utils … in ubuntu:latest
The job spread-ubuntu-cloud-24.04-x86_64 has CACHE_POLICY pull. It needs the image job image-ubuntu-cloud-24.04-x86_64 only optionally and otherwise takes the prepared image from the cache. It downloads a pinned spread from the spread-dist package registry, lists the tasks of tests/regression/, tests/profiles/ and tests/snapd/, sorts them and splits them round robin with split --number=r/$CI_NODE_INDEX/$CI_NODE_TOTAL across parallel: 4 jobs, each keeping spread-logs and spread-artifacts. The unit suites run in plain ubuntu:latest containers after build-all, which runs the prepare script of spread.yaml through yq and bash.

spread-ubuntu-cloud-24.04-x86_64: CACHE_POLICY: pull, parallel: 4, tasks split with split --number=r/N/4. The build-all job runs the prepare of spread.yaml in a container.

The spread job runs on Ubuntu 24.04 only, and it pulls the caches but never pushes them. It waits for the image job when the pipeline has one, and otherwise takes the image from the cache. It downloads a pinned build of spread, lists the tasks of three suites, regression, profiles and snapd, and deals them round robin to four parallel jobs. Each runs its share and keeps the logs and artifacts, even when it fails. The unit suites don't need a VM. They run in plain containers, on a build made by the same prepare script, taken from spread.yaml.

Details and sources

Details

  • .spread-x86_64 extends the image job's template and installs spread from the generic package registry of the zygoon/spread-dist project (SPREAD_GITLAB_PROJECT_ID: "65375371") at SPREAD_REV: 413817eda7bec07a3885e0717c178b965f8924e1 (.gitlab-ci.yml:278-310).
  • script: ./spread -list $SPREAD_ARGS | sort | split --number=r/"${CI_NODE_INDEX:-1}"/"${CI_NODE_TOTAL:-1}" | xargs --verbose ./spread -v -artifacts ./spread-artifacts -v, with the output also written to spread-logs/SYSTEM.log; spread-logs and spread-artifacts are kept when: always (.gitlab-ci.yml:311-323).
  • spread-ubuntu-cloud-24.04-x86_64: SPREAD_ARGS names tests/regression/, tests/profiles/ and tests/snapd/ on garden:ubuntu-cloud-24.04, CACHE_POLICY: pull, needs: image-ubuntu-cloud-24.04-x86_64 with optional: true, parallel: 4 (.gitlab-ci.yml:325-335).
  • If neither the image job nor the cache provides the prepared image, image-garden allocate makes it before booting.
  • build-all runs yq -r '.prepare' <spread.yaml | SPREAD_PATH=. bash -xeu in ubuntu:latest and keeps the build as an artifact; test-libapparmor, test-parser, test-binutils, test-utils, test-mod-apparmor and test-profiles run make check on it (.gitlab-ci.yml:44-134).

Read more

14 Where the tests lead

ROADS make check, and spread on image-garden on this map re-pipeline re-minimize adm-denials ut-logprof adm-features on the image-garden map sp-overview sp-adhoc rt-allocate rt-run ci-profiles oci-ci
In the centre, make check and spread on image-garden. Roads on this map: re-pipeline and re-minimize for what equality and minimize compare, adm-denials and ut-logprof for the log lines test_multi parses, adm-features for the files that stand in for a kernel. Roads on the image-garden map: sp-overview, sp-adhoc, rt-allocate, rt-run, ci-profiles and oci-ci.

Roads on this map: the backend pipeline, minimization, denials, aa-logprof, kernel features. On the image-garden map: spread, allocate, the run script, cloud-init templates, CI.

That's how AppArmor tests itself: make check in the parser, libapparmor and the utilities, and spread on image-garden machines for everything that needs a kernel. Where next? On this map, the backend pipeline and minimization decks explain what the equality and minimize tests compare. Reading a denial and aa-logprof show the log lines that test_multi parses. Kernel features covers the files that stand in for a kernel. And on the image-garden map you'll find spread's adhoc backend, allocate and the QEMU options, cloud-init templates, and the image-garden container in CI.

Details and sources

Roads from here

RUNNING THE PARSER

TOOLS · utils/

  • aa-logprof — turning log lines into profile rules, as the .profile cases check

THE DFA ENGINE · libapparmor_re

image-garden map ↗