Installing from source

make, make install for users for contributors

make joins image-garden's hand-written makefiles and the rules generated from its list of systems into image-garden.mk; make install puts the script, image-garden.mk, the release notes and the man page under a prefix, /usr/local by default, writing its directories and version into the script, which also runs uninstalled from the tree.

On the map: native

▶ Watch this deck — 9 scenes, ~6 min

Worth seeing first

01 Building it yourself

FROM SOURCE no package an older package a change of your own a checkout of the tree image-garden, a shell script image-garden.mk, a makefile nothing is compiled $ make $ sudo make install image-garden make, with your system's QEMU what make builds where it goes from the tree the checks
No package, an older package or a change of your own lead to a checkout of the tree. make and sudo make install turn it into image-garden, a shell script, and image-garden.mk, a makefile, with nothing compiled; image-garden make then runs with your system's QEMU. The deck covers what make builds, where install puts it, running from the tree, and the checks.

make && sudo make install in a checkout of the tree. Nothing is compiled: the result is a shell script and image-garden.mk.

Maybe your distribution has no package, or its package is older than the release you need, or you're about to change image-garden yourself. Then build it from the tree. There's nothing to compile: image-garden is a shell script and one big makefile, generated as you build it. Two commands do it: make, then sudo make install. After that, image-garden make works as with any other install, with your system's QEMU. This deck shows what make builds, where install puts it, how to run the script straight from the tree, and the checks contributors run.

Details and sources

Details

  • The README: "Follow your distribution's package installation method, or build from source: make && sudo make install" (README.md:114-120).
  • make builds image-garden.mk (all:: image-garden.mk); make install installs it with the script (Makefile:20-50). Nothing in the tree is compiled; image-garden is a /bin/sh script (image-garden:1).
  • The tools a source build needs at run time are in what the host needs.
  • The README lists the tools a source build needs: curl, the six emulators qemu-system-x86_64, -aarch64, -riscv64, -s390x, -ppc64 and -loongarch64 (or those of the architectures you use), qemu-img and xorriso, and for the Ubuntu Core systems with a TPM swtpm and the Secure Boot firmware (README.md:122-126).
  • Every statement in this deck was checked by running make, make install and the script on a git archive of the pinned commit.

02 make builds one file for contributors in place of the general version

ONE FILE image-garden.mk data/ubuntu.ini ... rulegen.py data/050-*.gen.mk data/000-systems.mk every system mk/*.mk hand-written cat 000-systems.mk mk/*.mk 050-*.gen.mk its time: that of its newest input, not of the build needs Python no QEMU yet the same file the snap and the container carry
make runs rulegen.py on each data/*.ini, writing data/050-*.gen.mk, and writes data/000-systems.mk, the list of systems. cat joins 000-systems.mk, the sorted mk/*.mk and the sorted .gen.mk files into image-garden.mk, whose time is set to that of its newest input. Only Python is needed, not QEMU, and the result is the file every install carries.

make runs rulegen.py once per data/*.ini, writes the systems list, and joins it, mk/*.mk and data/050-*.gen.mk into image-garden.mk.

make has one job here: build image-garden.mk. For each data file, such as ubuntu.ini, rulegen writes a makefile of generated rules, and a list of every system is written next to them. Then the hand-written makefiles from the mk directory and the generated ones are joined, in order, into one file. Its time is set to that of its newest input, not to the time of the build. You need Python for rulegen, and that's all; QEMU isn't needed yet. The result is the same file the snap and the container carry, and the one every image-garden make reads.

Details and sources

Details

  • data/050-%.gen.mk: rulegen.py data/%.ini runs ./rulegen.py data/NAME.ini >data/050-NAME.gen.mk; data/000-systems.mk is written by rulegen.py --systems-meta over every .ini (Makefile:57-64).
  • image-garden.mk is cat data/000-systems.mk, the sorted mk/*.mk and the sorted data/050-*.gen.mk; touch --reference then gives it the time of its newest input (Makefile:48-50).
  • Derived: rules in image-garden.mk compare a system's files with $(MAKEFILE_LIST), so a time of the build would make every system's .url look out of date after any rebuild; see how image-garden.mk is put together.
  • rulegen.py runs with Python 3; no QEMU tool is called while building image-garden.mk (run for this map on a host with QEMU; derived for one without).
  • image-garden.mk, data/*.gen.mk and data/000-systems.mk are build output; make clean removes them (Makefile:23-27).

03 What make builds general version

ONE FILE image-garden.mk the list of systems make rules, one set per system rules written by hand joined, in order, into one file its time: that of its newest input, not of the build needs make and Python no QEMU yet the same file the snap and the container carry
make turns the tree's list of systems into rules, one set per system, and joins them in order with the hand-written rules into image-garden.mk, whose time is set to that of its newest input. Only make and Python are needed, not QEMU, and the result is the file the snap and the container carry.

make turns the tree's list of systems into rules and joins them with the hand-written ones into image-garden.mk. It needs Python, not QEMU.

make has one job here: build a single file, image-garden.mk. The tree carries a list of every system image-garden knows. make turns that list into rules, one set per system, and joins them, in order, with the rules written by hand, into that one file. Its time is set to that of its newest input, not to the time of the build. For this step you need make and Python, and that's all; QEMU isn't needed yet. The result is the same file the snap and the container carry, and the one every image-garden make reads.

Details and sources

Details

  • make with no target builds image-garden.mk (Makefile:20-21). The list of systems is the tree's data/*.ini; the build writes one file of generated rules per data file and a list of every system's metadata, and concatenates that list, the hand-written mk/*.mk and the generated rules, each sorted, into image-garden.mk (Makefile:48-64).
  • touch --reference gives image-garden.mk the time of its newest input (Makefile:48-50). Derived: rules in image-garden.mk compare a system's files with $(MAKEFILE_LIST), so a time of the build would make every system's .url look out of date after any rebuild.
  • The generator is a Python 3 program; no QEMU tool is called while building image-garden.mk (run for this map on a host with QEMU; derived for one without).
  • make clean removes image-garden.mk and the intermediate files (Makefile:23-27).

04 What install puts where

UNDER A PREFIX /usr/local/ image-garden image-garden.mk README.md NEWS* image-garden.1 bin/image-garden include/image-garden.mk share/doc/image-garden/ share/man/man1/image-garden.1 sed -e 's,@PREFIX@,$(prefix),' -e 's,@INCLUDEDIR@,$(includedir),' -e 's,@DOCDIR@,$(datarootdir)/doc/…,' -e 's,@VERSION@,$(VERSION),' run on the installed script, so the command reads INCLUDEDIR/image-garden.mk and version prints VERSION $ make install DESTDIR=$PWD/stage prefix=/usr what packagers do
make install copies image-garden to /usr/local/bin, image-garden.mk to /usr/local/include, README.md and the NEWS files to /usr/local/share/doc/image-garden and image-garden.1 to /usr/local/share/man/man1. sed writes the prefix, the include and doc directories and the version into the script, which then reads INCLUDEDIR/image-garden.mk and prints the version. Packagers run make install DESTDIR=… prefix=/usr.

prefix ?= /usr/local: bin/image-garden, include/image-garden.mk, share/doc/image-garden/, share/man/man1/image-garden.1. sed fills in @PREFIX@, @INCLUDEDIR@, @DOCDIR@ and @VERSION@.

sudo make install copies four kinds of file under a prefix, /usr/local unless you say otherwise. The image-garden script goes to its bin directory, image-garden.mk to include, the README and the release notes to the doc directory, and the man page to the manual. On its way in, the script is edited: the prefix, the include and doc directories and the version are written into it. That's how the command finds its makefile, in the include directory, and what image-garden version prints. Packagers add DESTDIR, to stage the files elsewhere, and usually a prefix of /usr.

Details and sources

Details

  • prefix ?= /usr/local, bindir, includedir = $(prefix)/include, datarootdir = $(prefix)/share, man1dir = $(datarootdir)/man/man1, and DESTDIR ?= (Makefile:3-14).
  • install copies image-garden to $(bindir) and runs sed -i -e 's,@PREFIX@,$(prefix),' -e 's,@INCLUDEDIR@,$(includedir),' -e 's,@DOCDIR@,$(datarootdir)/doc/image-garden,' -e 's,@VERSION@,$(VERSION),' on it; image-garden.mk to $(includedir); README.md, NEWS, NEWS.snappy, NEWS.docker and their .license files to $(datarootdir)/doc/image-garden; image-garden.1 to $(man1dir) (Makefile:29-46).
  • The script then has GARDEN_PREFIX=PREFIX, GARDEN_INCLUDE=INCLUDEDIR and GARDEN_DOC=DATAROOTDIR/doc/image-garden, so an includedir or datarootdir set apart from the prefix is followed (image-garden:10-12), and image-garden make runs make -f "$GARDEN_INCLUDE"/image-garden.mk (image-garden:987-993).
  • Run for this map: make install DESTDIR=… prefix=/usr staged usr/bin/image-garden (with GARDEN_PREFIX=/usr), usr/include/image-garden.mk, usr/share/doc/image-garden/ with seven files and usr/share/man/man1/image-garden.1, and the staged script printed image-garden 0.8.0. With includedir=P/inc the script had GARDEN_INCLUDE=P/inc and found its makefile there (run at the pinned commit).
  • Because all three NEWS files are installed, image-garden news from a source install prints the main, the snap and the container notes (run).

05 Running it from the tree

NO INSTALL your checkout, after make image-garden GARDEN_PREFIX=@PREFIX@ nothing was written in image-garden.mk NEWS next to itself $ ln -s $PWD/image-garden ~/bin/ $ cd ~/project; image-garden make ... $ image-garden version image-garden @VERSION@ edit the tree make try the system
In a checkout after make, the script still says GARDEN_PREFIX=@PREFIX@, so it looks for image-garden.mk and NEWS next to its real path, also through a symbolic link in PATH, and image-garden make works in any project directory. image-garden version prints image-garden @VERSION@. The quickest loop for a change of your own: edit the tree, make, try the system.

Uninstalled, the script finds image-garden.mk and NEWS next to its real path. image-garden version prints image-garden @VERSION@.

You don't have to install it at all. Run the script from the checkout, after make, and it notices that nothing was written into it. It then looks for image-garden.mk and the release notes next to itself, following a symbolic link if you put one in your PATH. So image-garden make works straight from the tree, in any project directory. Only one thing gives it away: image-garden version prints the placeholder, @VERSION@, because the version is written in at install time. For a change of your own, that's the quickest loop: edit the tree, run make, try the system.

Details and sources

Details

  • When GARDEN_PREFIX is still the placeholder, the script sets GARDEN_ITSELF=$(readlink -f "$0") and uses its directory as GARDEN_PREFIX, GARDEN_INCLUDE and GARDEN_DOC: "This hacks supports both installed and uninstalled version equally" (image-garden:13-19).
  • image-garden version prints image-garden @VERSION@ from an uninstalled tree; news reads NEWS next to the script (image-garden:909-917). Both run for this map.
  • image-garden make list-systems, run from another directory with the script of the tree, listed the systems; make warned about EDK2 firmware missing on that host, as any install does.
  • make must have built image-garden.mk first: the script runs make -f on it (image-garden:987-993).
  • ~/bin and ~/project on the slide are examples.

06 The project's own checks for contributors

MAKE CHECK $ make check check-sh shellcheck: the script, hooks, stubs, tests check-py mypy --strict: rulegen.py and the test helpers check-rulegen good and bad .ini check-dl-pkgproxy a stand-in pkgproxy check-allocate a fake SSH server check-rebase a moved download check-dl-unpack-lock a shared download check-pkgproxy-guest-shell guest setup none of them boots a VM check-rebase needs qemu-img no shellcheck or mypy: make stops, "program shellcheck is required" the container's build-garden stage runs the same make check
make check runs check-sh with shellcheck over the shell scripts and check-py with mypy --strict over the Python, then six tests: check-rulegen, check-dl-pkgproxy, check-allocate, check-rebase, check-dl-unpack-lock and check-pkgproxy-guest-shell. None boots a VM; check-rebase needs qemu-img. A missing shellcheck or mypy stops make; the container build runs make check too.

make check: check-sh (shellcheck), check-py (mypy --strict), check-rulegen, check-dl-pkgproxy, check-allocate, check-rebase, check-dl-unpack-lock, check-pkgproxy-guest-shell.

Before you send a change, run make check. It runs shellcheck over the shell scripts, from image-garden itself to the snap's hooks and stubs, and mypy in strict mode over the Python. Then six tests: rulegen accepting good definitions and rejecting bad ones, image downloads through a stand-in pkgproxy, allocate against a fake SSH server, rebase onto a moved download, the lock around unpacking a shared download, and the pkgproxy setup that guests run. None of them boots a VM, though the rebase test needs qemu-img. Both checkers must be installed, or make stops and names the missing one. The container build runs the same make check.

Details and sources

Details

  • check depends on check-sh check-py check-rulegen check-dl-pkgproxy check-allocate check-rebase check-dl-unpack-lock check-pkgproxy-guest-shell (Makefile:66-67).
  • check-sh runs shellcheck over FILES_SH: the script, spread-chooser, the snap's hooks, qemu-stubs/, docker-stubs/docker and the test scripts; check-py runs mypy --strict over rulegen.py and three test helpers (Makefile:69-97); the list also holds qemu-stubs/virtiofsd and tests/ci/run-systems.sh. A missing checker is $(error program shellcheck is required) or … mypy is required (Makefile:90, Makefile:94).
  • The six tests: tests/check-rulegen.sh (valid and invalid image definitions, and no Python traceback for a data mistake), check-dl-pkgproxy.sh (downloads through a stand-in pkgproxy, tests/dl-pkgproxy-server.py), check-allocate.sh (ports and SSH, with tests/fake-sshd.py), check-rebase.sh (rebase of a prepared image whose download moved to another GARDEN_DL_DIR, named with and without .qcow2; it runs qemu-img, tests/check-rebase.sh:1-39), check-dl-unpack-lock.sh (concurrent unpacking of one download, "No network, no KVM"), check-pkgproxy-guest-shell.sh (the guests' pkgproxy setup under dash and busybox sh) (Makefile:99-115).
  • The container's build-garden stage installs qemu-utils among its tools and runs make check before make install (Dockerfile:58-71).
  • make fmt formats with shfmt and black (Makefile:117-126).

07 When the makefile is missing

A MAKEFILE NOT FOUND $ make install prefix=/opt/ig includedir=/opt/ig/inc GARDEN_INCLUDE=/opt/ig/inc written into the installed script /opt/ig/bin/image-garden /opt/ig/inc/image-garden.mk found the script copied alone to another machine: /opt/ig/bin/image-garden /opt/ig/inc/image-garden.mk missing make: /opt/ig/inc/image-garden.mk: No such file or directory make: *** No rule to make target '/opt/ig/inc/image-garden.mk'. Stop. your project is fine: this is the script's own makefile $ sudo make install prefix=/opt/ig includedir=/opt/ig/inc
make install prefix=/opt/ig includedir=/opt/ig/inc writes GARDEN_INCLUDE=/opt/ig/inc into the installed script and puts image-garden.mk there, so the script finds it. Copied alone to another machine, the script still looks in /opt/ig/inc, where there is no makefile, and image-garden make prints make: /opt/ig/inc/image-garden.mk: No such file or directory and No rule to make target for it. The project is not missing anything; run make install on that machine.

make: INCLUDEDIR/image-garden.mk: No such file or directory means the script cannot find its makefile where make install put it. Install both files together, with the same prefix and includedir.

One mistake gives a message that's easy to misread. The installed script looks for image-garden.mk in one place only: the include directory that make install wrote into it. That's the prefix's include directory, or the includedir you gave. Copy the script alone to another machine, or remove the makefile, and image-garden make fails before doing anything: make says the makefile at that path doesn't exist, and that there's no rule to make it. Your project isn't missing a file; image-garden is. Run make install again on that machine, so that the script and its makefile arrive together.

Details and sources

Details

  • The installed script looks in @INCLUDEDIR@, which make install replaces with $(includedir), the directory it copies image-garden.mk to; @DOCDIR@ becomes $(datarootdir)/doc/image-garden, where the release notes go (image-garden:10-12, Makefile:12, Makefile:29-40). An uninstalled script looks next to itself instead (image-garden:13-19).
  • Run for this map: after make install prefix=P includedir=P/inc the script had GARDEN_INCLUDE=P/inc, and image-garden make list-systems listed the systems. With P/inc/image-garden.mk removed, and LC_ALL=C, it printed make: P/inc/image-garden.mk: No such file or directory and make: *** No rule to make target 'P/inc/image-garden.mk'. Stop. The slide uses /opt/ig for P.
  • A script copied without its makefile behaves the same (derived). image-garden make runs make -f "$GARDEN_INCLUDE"/image-garden.mk (image-garden:987-993).

08 Where to go from here general version

ROADS From source What the host needs tools and emulators Distribution packages built by others The container image the same tree, built again image-garden make what to run next
Four roads from Installing from source: what the host needs, distribution packages, the container image and image-garden make

Next: what the host needs, distribution packages, the container image and image-garden make.

So a source build is two commands: make builds one file from the tree, and install puts it and the script under a prefix, with the version written in. Next, the host tools deck lists what you need to install alongside: the emulators, curl, xorriso and the rest. The packages deck covers builds others make for you, and the container deck one more build of the same tree. And the command line deck shows what to run once it's installed.

Details and sources

09 Where to go from here for contributors in place of the general version

ROADS From source What the host needs tools and emulators Assembling the .mk file by file rulegen what it writes for a system Distribution packages built by others The container image the same tree, built again image-garden make what to run next
Six roads from Installing from source: what the host needs, how image-garden.mk is assembled, rulegen, distribution packages, the container image and image-garden make

Next: what the host needs, how image-garden.mk is assembled, rulegen, distribution packages, the container image and image-garden make.

So a source build is two commands: make joins the rules into one file, and install puts it and the script under a prefix, with the version written in. Next, the host tools deck lists what you need to install alongside. The assembly deck shows how image-garden.mk is put together, file by file, and the rulegen deck what it writes for a system. The packages deck covers builds others make for you, the container deck one more build of the same tree, and the command line deck what to run next.

Details and sources

Read more

Basis of the statements on this page

Read in the Makefile, image-garden and README of image-garden at the pinned commit ("code"), and checked by running make, make install (with DESTDIR, prefix and includedir) and the uninstalled script on a git archive of that commit ("run").

Targets

TargetWhat it does
make (all)builds image-garden.mk from data/*.ini and mk/*.mk
make installinstalls the script, image-garden.mk, the docs and the man page
make cleanremoves image-garden.mk, data/050-*.gen.mk and data/000-systems.mk
make checkshellcheck, mypy --strict and six tests (one needs qemu-img)
make fmtshfmt and black over the scripts
make distimage-garden-VERSION.tar.gz of the sources
make docker-test-smokebuilds the container image and runs three smoke tests

Install locations

VariableDefaultHolds
prefix/usr/localeverything below
bindir$(prefix)/binimage-garden
includedir$(prefix)/includeimage-garden.mk; written into the script, which looks there
datarootdir$(prefix)/sharedoc/image-garden/: README.md, NEWS, NEWS.snappy, NEWS.docker and their licences; written into the script for image-garden news
man1dir$(datarootdir)/man/man1image-garden.1
DESTDIRemptya staging root, for packaging

Examples

make && sudo make install # /usr/local make install DESTDIR=$PWD/stage prefix=/usr # stage a package's files ./image-garden version # uninstalled: image-garden @VERSION@ make check # needs shellcheck and mypy

Messages

MessageCause
program shellcheck is requiredmake check without shellcheck (or mypy)
make: INCLUDEDIR/image-garden.mk: No such file or directorythe script's makefile is not where make install put it
image-garden @VERSION@the script runs uninstalled, from the tree

Roads from here

RUNTIME · make rules and the CLI

CONTAINER · OCI image

NATIVE · packages

BUILD TIME · data and rules