The lexer

start conditions, keywords for AppArmor developers for security researchers for application developers

The scanner cuts profile text into tokens for the grammar. Words are looked up in one keyword table. Twenty-nine start conditions on a stack give each rule class its own vocabulary. Includes are a stack of input buffers that the grammar does not see. The -p option prints what the scanner read.

On the map: inside the parser

▶ Watch this deck — 9 scenes, ~7 min

Worth seeing first

Part of journeys

01 The grammar never sees your characters

THE LEXER abi <abi/5.0>,include <tunables/global>profile demo /usr/bin/demo {/etc/demo/** r,audit deny @{HOME}/.ssh/** rw,/usr/bin/[a-z]* ix,} the scanner · parser_lex.l /etc/demo/** r, TOK_ID TOK_MODE TOK_END_OF_RULE the grammar · parser_yacc.y keywords · scanner states · quoting · includes for people who work on the parser, and for anyone reading one of its errors
The demo profile on the left; a scanner box reads it and emits the tokens TOK_ID, TOK_MODE and TOK_END_OF_RULE for one rule, which a grammar box receives

The scanner (parser/parser_lex.l, flex) turns profile text into tokens; the grammar (parser/parser_yacc.y, bison) asks for them one at a time.

Here is demo's profile, cut down to three rules, an include and an ABI line. Before the parser can know what it means, it has to cut the text into pieces. That's the scanner's job, the first stage of the run. It reads characters and hands the grammar one token at a time: a name, a permission string, the comma that ends a rule. The grammar never sees your characters. This deck shows how that cutting works: keywords, scanner states, quoting and includes. It's mostly for people who work on the parser's code. Two scenes are for anyone who has stared at a parser error and wondered what it meant.

Details and sources

Details

  • The profile followed through this district and through the DFA engine decks (file demo3.sd):
abi <abi/5.0>,
include <tunables/global>
profile demo /usr/bin/demo {
  /etc/demo/** r,
  audit deny @{HOME}/.ssh/** rw,
  /usr/bin/[a-z]* ix,
}
  • The scanner is generated by flex from parser/parser_lex.l; the grammar by bison from parser/parser_yacc.y. yyparse() is called once per policy file (parser_main.c:1353) and calls the scanner for each token.
  • A token is a number (TOK_ID, TOK_MODE, ...; the list is parser_yacc.y:78-193) and, for some tokens, a value: the text of a name or of a permission string.
  • Output shown here is from the parser built from the AppArmor tree at 836052d21, run with -Q -K (nothing loaded into a kernel, no cache) and the tree's abi/5.0 feature set.

Read more

02 One rule, five tokens for AppArmor developers for security researchers

ONE RULE, FIVE TOKENS audit deny @{HOME}/.ssh/** rw, auditTOK_AUDIT denyTOK_DENY words, looked up @{HOME}/.ssh/**TOK_ID longest match: variable and tail are one name rwTOK_MODE a mode only before a blank or a comma ,TOK_END_OF_RULE ends the rule yylval.id = "@{HOME}/.ssh/**" yylval.mode = "rw" the text travels with the token; the keyword tokens carry none
The rule audit deny @{HOME}/.ssh/** rw, cut into five pieces, each with its token: TOK_AUDIT, TOK_DENY, TOK_ID, TOK_MODE and TOK_END_OF_RULE

@{HOME}/.ssh/** is one TOK_ID because {LABEL} matches more text than {SET_VARIABLE}; rw is a TOK_MODE only when a blank or a comma follows.

Take the second rule and watch the scanner cut it. Audit and deny are plain words; each is looked up and comes back as a keyword token. Next, the @{HOME} variable. On its own a variable is a token of its own kind, but here it has a tail, and flex always takes the longest match, so variable and tail become one name. Then the two permission letters. They count as a mode only because a comma or a blank follows them; otherwise they'd be a name. Last, the comma, which ends the rule. Five tokens, and for two of them the text travels along as the token's value.

Details and sources

Details

  • Flex chooses the pattern that matches the most text and, between equal lengths, the one written first.
  • audit and deny match {VARIABLE_NAME} and are found in the keyword table (parser_lex.l:856-909).
  • @{HOME}/.ssh/** matches {SET_VARIABLE} (seven characters, TOK_SET_VAR, parser_lex.l:714-717) and {LABEL}, which is a /, a variable, a : or an & followed by name characters (parser_lex.l:301, parser_lex.l:838-841). {LABEL} is longer, so the result is one TOK_ID. A variable followed directly by a blank is a TOK_SET_VAR; the grammar accepts either through id_or_var (parser_yacc.y:1037-1038).
  • Permission letters: MODE_CHARS is ([RrWwaLlMmkXx])|(([Pp]|[Cc])[Xx])|(([Pp]|[Cc])?([IiUu])[Xx]) (parser_lex.l:257); the rule is ({MODES})/([[:space:],]), with the blank or comma as trailing context (parser_lex.l:843-846). The trailing character counts for the match length, which is how rw, wins over the one-word pattern.
  • The comma is TOK_END_OF_RULE in the initial state and in the rule-class states (parser_lex.l:911-917).
  • The scanner has no token dump unless it is built with DEBUG (parser_lex.l:55-61); the cut shown here is derived from the patterns.

03 Keywords are a table, not patterns for AppArmor developers for security researchers

KEYWORDS ARE A TABLE, NOT PATTERNS signal {VARIABLE_NAME}[[:alpha:]][[:alnum:]_]* keyword_table 55 entries TOK_SIGNAL push SIGNAL_MODE a token, and for some a state demo not found: -1 TOK_ID a name, not an error user = owner rewrite = alias unmount = umount three synonyms in the table
The word signal matches the pattern VARIABLE_NAME, is found in the keyword table and becomes TOK_SIGNAL with the scanner state SIGNAL_MODE; the word demo is not found and becomes TOK_ID; three synonyms are listed

{VARIABLE_NAME} matches the word, get_keyword_token() looks it up in keyword_table (55 entries, parser/parser_misc.c:81); no entry means TOK_ID.

You might expect one scanner rule per keyword. There are almost none. A word made of letters matches one general pattern, and its action looks the word up in a table of fifty-five keywords. A hit gives the token. For some of them, signal, network, mount and the like, it also switches the scanner into a state of its own, because what follows has its own vocabulary. A miss isn't an error: the word is simply a name. That's why a profile can be called demo without any quoting. The table also holds three synonyms: user for owner, rewrite for alias, and unmount for umount. Adding a keyword starts here.

Details and sources

Details

  • VARIABLE_NAME is [[:alpha:]][[:alnum:]_]* (parser_lex.l:296).
  • The action (parser_lex.l:856-909) calls get_keyword_token(yytext) (parser_misc.c:187-191), a lookup in keyword_table (parser_misc.c:81-139).
  • -1: the word is returned as TOK_ID through processunquoted (parser_lex.l:862-866).
  • Tokens that select a scanner state: TOK_RLIMIT, TOK_NETWORK, TOK_CHANGE_PROFILE, TOK_MOUNT, TOK_REMOUNT, TOK_UMOUNT, TOK_DBUS, TOK_SIGNAL, TOK_PTRACE, TOK_UNIX, TOK_ABI, TOK_USERNS, TOK_MQUEUE and TOK_IF (parser_lex.l:867-907). Every other keyword pushes the initial state again (parser_lex.l:908).
  • Synonyms: user and owner are both TOK_OWNER, rewrite and alias both TOK_ALIAS, unmount and umount both TOK_UMOUNT (parser_misc.c:95-112).
  • Words that are keywords only inside one rule class are ordinary patterns of that state and are not in the table: listen, accept, connect, getopt, setopt and shutdown (parser_lex.l:627-634), create (parser_lex.l:636-638), getattr and setattr (parser_lex.l:645-648). Anywhere else they are names.
  • Some words are both, a pattern of their own state and an entry of the table: send, receive, bind, read, write, eavesdrop, trace, readby, tracedby (parser_lex.l:650-679), safe and unsafe (parser_lex.l:593-596), open and delete. In the initial state they are keywords: /etc/demo/** send, gives syntax error, unexpected TOK_SEND, expecting TOK_MODE, where /etc/demo/** listen, gives unexpected TOK_ID (both run with the tree's parser).
  • A few words are matched before the table because the scanner must look ahead: priority, flags, peer, xattrs and identities in front of = (parser_lex.l:408-436), all (parser_lex.l:811-813), profile and hat (parser_lex.l:848-850), include (parser_lex.l:781-809).
  • Known discrepancy, by title: "Scanner state IOURING_MODE can never be entered" (the table has io_uring, the action has no case for it).

04 Twenty-nine states on a stack for AppArmor developers for security researchers

29 STATES ON A STACK signal send set=(term) peer=demo, INITIAL signal SIGNAL_MODE set = EXTCOND_MODE the scanner looks ahead for = itself ( LIST_VAL_MODE ) SIGNAL_MODE peer = EXTCOND / SUB_VALUE demo SIGNAL_MODE , INITIAL %option stack · 29 exclusive states declared with %x · PUSH, POP and BEGIN macros
A signal rule and the scanner states it passes through: INITIAL, SIGNAL_MODE, EXTCOND_MODE, LIST_VAL_MODE, back to SIGNAL_MODE, EXTCOND_MODE, SUB_VALUE, SIGNAL_MODE and INITIAL, each move labelled with the text that causes it

29 exclusive start conditions (%x, parser/parser_lex.l:331-359) and %option stack; a condition name is recognised by the = behind it, inside the scanner.

Why states? Because one word can mean different things. At the start of a rule, audit is a keyword; inside a signal rule it's just a name. So the scanner keeps a stack of states, twenty-nine besides the initial one. Follow this rule. The keyword pushes the signal state. Set, before an equals sign, is a condition name; the scanner checks for that equals sign itself, because by the time the grammar could ask, the next token would already be cut. The parenthesis opens a list state and the closing one returns. Peer and its value work the same way. And the comma pops back to where we started.

Details and sources

Details

PurposeStates
a name followsSUB_ID (after profile, hat, ^), SUB_ID_WS (after ->), SUB_VALUE
a rule classNETWORK_MODE, RLIMIT_MODE, MOUNT_MODE, DBUS_MODE, SIGNAL_MODE, PTRACE_MODE, UNIX_MODE, CHANGE_PROFILE_MODE, USERNS_MODE, MQUEUE_MODE, IOURING_MODE, ABI_MODE
conditions and listsEXTCOND_MODE, EXTCONDLIST_MODE, LIST_VAL_MODE, LIST_COND_MODE, LIST_COND_VAL, LIST_COND_PAREN_VAL, IDENTITY_VALUE, INTEGER_MODE
assignments and ifASSIGN_MODE, ASSIGN_EXPR_START, ASSIGN_EXPR_MODE, EXPR_MODE
includesINCLUDE, INCLUDE_EXISTS
  • The look-ahead: {VARIABLE_NAME}/{WS}*= returns TOK_CONDID and pushes EXTCOND_MODE in one action. The comment there: "we match to the = in the lexer so that we can switch scanner state. By the time the parser see the = it may be too late as bison may have requested the next token from the scanner" (parser_lex.l:437-444).
  • In the rule on the slide: ( moves from EXTCOND_MODE to LIST_VAL_MODE with BEGIN, not a push, so ) pops straight back to the signal state (parser_lex.l:479-503); = followed by a value moves to SUB_VALUE, and the value pops (parser_lex.l:464-470, parser_lex.l:491). The sequence on the slide is derived from these rules.
  • The keyword table is consulted only in the initial state. In a rule-class state a word that is not one of the state's own patterns is a TOK_ID (parser_lex.l:690-695): signal audit, gives syntax error, unexpected TOK_ID, expecting TOK_CONDID or TOK_END_OF_RULE (run with the tree's parser).
  • After profile, hat or ^ the scanner is in SUB_ID, where only a name or a quoted name matches (parser_lex.l:456-462, parser_lex.l:820, parser_lex.l:848-850). That is why a profile may be named like a keyword or like permission letters.
  • The comma ends a rule in the initial state and in the rule-class states and pops the state (parser_lex.l:911-917). A new line is counted in most states (parser_lex.l:919-924); in the two assignment states it also ends the assignment (parser_lex.l:584-589, parser_lex.l:743-748).
  • Derived: a keyword without a state of its own pushes the initial state and nothing pops it, so the stack grows by one entry for each such keyword; flex enlarges the stack as needed.
  • Known discrepancy, by title: "\"flex scanner jammed\" instead of an error message, after priority= and after +" (two states are missing from the last rule of the file).

Read more

05 The first decoding of a name for AppArmor developers for security researchers

THE FIRST DECODING as written in the token "/etc/demo/my file"->/etc/demo/my filequotes stripped /etc/demo/\x41->/etc/demo/Aescape decoded * ? [ ] { } ^ , \decoded into one of these: keeps a backslash /etc/demo/a\052b->/etc/demo/a\*boctal 052 is a star: still literal /etc/demo/a\*b->/etc/demo/a\*bunchanged processid → processquoted → processunquoted the remaining backslashes are read by the glob conversion
Four names as written and as carried by the token: a quoted path loses its quotes, backslash x41 becomes A, an octal escape for a star becomes backslash star, and backslash star is unchanged

processid() strips quotes and decodes escapes; a decoded * ? [ ] { } ^ , \ keeps a backslash (parser/parser_misc.c:488) for convert_aaregex_to_pcre.

A name token doesn't carry the text exactly as you wrote it. The scanner decodes it first. Quotes are stripped, so a path with a blank arrives as plain text. A numeric escape, such as \x41, turns into its character, a capital A. But there is one exception. If the decoded character is one that globbing treats specially, a star, a question mark, a bracket, a brace, a caret, a comma or a backslash, it keeps a backslash in front. So an octal escape for a star still arrives as a literal star, not as a wildcard. Those backslashes are for the second decoding, the glob conversion.

Details and sources

Details

  • A name is {IDS}: any characters except blank, tab, carriage return, new line, ", ! and a , that is not followed by a name character; a backslash in front of a blank, a tab, ", ! or , lets that character in (parser_lex.l:262-267), or {QUOTED_ID}: text between double quotes, with \" allowed inside (parser_lex.l:287-288). In the initial state a path is matched by {LABEL}, which begins with /, a variable, : or & and continues with the same characters (parser_lex.l:301).
  • processid (parser_misc.c:527-535) calls processquoted for a string that starts with " (parser_misc.c:511-525), which removes the two quotes and calls processunquoted (parser_misc.c:468-509).
  • processunquoted decodes an escape with strn_escseq (lib.c:158): up to three octal digits, \d and up to three decimal digits, \x and up to two hex digits, \\, \", and the letters a e f n r t.
  • A decoded character from the set *?[]{}^,\ is written with a backslash in front (parser_misc.c:488-490); any other decoded character is written as it is; a sequence that is not a valid escape is copied unchanged, backslash included (parser_misc.c:495-502).
  • The right-hand column is what -D rule-exprs prints after aare: for these rules; the dump is made later, so for other names it also shows expanded variables.
  • Output shown here is from the parser built from the AppArmor tree at 836052d21, run with -Q -K (nothing loaded into a kernel, no cache) and the tree's abi/5.0 feature set.
  • Known discrepancies, by title: "A path ending in an escaped backslash cannot be written: quoted it runs to the next quote, unquoted it swallows the blank"; "New lines inside a quoted string are not counted"; "ID_FIRST_CHARS is defined and never used, so a name may start with #".

Read more

06 include is not in the grammar for AppArmor developers for security researchers

INCLUDE IS NOT IN THE GRAMMAR include <tunables/global> demo3.sd tunables/global saved: demo3.sd, line 2 tunables/home saved: tunables/global, line 15 end of file: close, pop, go on after the include line the grammar tokens one stream include cache: names seen in this body a second include of the same file is skipped
A stack of input buffers: demo3.sd at the bottom, tunables/global pushed on top of it, tunables/home on top of that; at end of file a buffer is popped; one token stream leaves the stack; a box for the include cache

include_filename() pushes a flex buffer (parser/parser_lex.l:207); <<EOF>> pops it (400-406). An include cache per profile body skips a file already included there.

The grammar has no rule for include. The scanner handles it alone. When it meets the word, it reads the file name, opens the file, and pushes a new input buffer. The old file name and line number are saved, so messages can still say where you are. Tunables/global includes more files, so the stack grows. At the end of a file the scanner closes it, pops, and carries on after the include line. The grammar just sees one long stream of tokens. And a set of names remembers what has already been included in this profile body, so a second include of the same file is skipped.

Details and sources

Details

  • The patterns include/{WS} and include{WS}+if{WS}+exists/{WS} push the states INCLUDE and INCLUDE_EXISTS (parser_lex.l:789-794, parser_lex.l:804-809); the two #include forms do the same after the warning deprecated use of '#include' (parser_lex.l:781-787, parser_lex.l:796-802).
  • In those states one pattern matches the file name as <name>, <"name">, "name" or a bare name and calls include_filename(filename, lt, exists) (parser_lex.l:376-398).
  • include_filename (parser_lex.l:162-225): a name in angle brackets is searched in the -I directories and then the base directory by search_path (parser_include.c:154-185); any other name is opened as written. A regular file is pushed with push_include_stack(fullpath) and yypush_buffer_state(yy_create_buffer(yyin, YY_BUF_SIZE)) (parser_lex.l:203-207). A directory is read with include_dir_cb, which pushes every regular file in it whose name is not blacklisted (parser_lex.l:128-160, parser_lex.l:208-217).
  • End of file: fclose(yyin); pop_include_stack(); yypop_buffer_state(); and yyterminate() when no buffer is left (parser_lex.l:400-406).
  • The include stack keeps current_filename and current_lineno of each including file and restarts the count at 1 (parser_include.c:187-246). The line number on the slide for tunables/global (15) is where that file includes tunables/home.
  • The include cache is a set of path names (file_cache.h:27-49). Each profile body starts with an empty one and the previous one is restored at the closing brace (parser_yacc.y:393-398, parser_yacc.y:453-455); each policy file starts with a new one (parser_main.c:1267-1268).
  • The token stream is continuous. With an included file that holds only /etc/demo/** and the line r, after the include, the parser compiles the rule /etc/demo/** r, (run with the tree's parser).
  • A missing file is the error Could not open 'NAME' (parser_lex.l:192-196); with if exists the scanner returns without a message.
  • Known discrepancy: a profile file that includes itself is read again and again until no file descriptor is left; the message is then Could not open 'abi/5.0': Too many open files. The include inside the body is not found in the include cache, because every profile head starts an empty cache, and MAX_NEST_LEVEL (parser_include.c:58) is not checked anywhere. Run with the tree's parser.
  • Known discrepancy, by title: "A comment or #include on the last line needs a final new line".

07 Ask the parser what it read

ASK THE PARSER WHAT IT READ: -p $ apparmor_parser -p demo3.sdabi <abi/5.0>,##included <tunables/global># ------------------------------------------------------------------ 7 lines in, 516 out ##included <tunables/home>##failed include <tunables/global.d> an include if exists; no such file profile demo /usr/bin/demo {/etc/demo/** r,audit deny @{HOME}/.ssh/** rw, not expanded yet a profile body with include <abstractions/base> twice: ##skipped duplicate include <abstractions/base>
Output of apparmor_parser -p on the demo profile: the abi line, a line ##included tunables/global, further ##included and ##failed include lines, the profile with its variable unexpanded, and a ##skipped duplicate include line from a second profile

apparmor_parser -p FILE prints the flattened profile: ##included, ##failed include and ##skipped duplicate include mark what the scanner did with each include.

The scanner can show you its own work. With the -p option it prints back each piece of text as it matches it, and the parser stops there. Demo's seven lines become 516. Each included file is announced by a line that starts with two hash marks. A failed include is not an error here: it marks an include if exists whose file wasn't there. Your own rules come out at the end, unchanged; variables are still variables, because nothing has been interpreted yet. And if a profile body includes the same abstraction twice, the second one is marked as a skipped duplicate.

Details and sources

Details

  • -p, --preprocess: the parser prints the profile with its includes flattened and stops after parsing; nothing is compiled, cached or loaded (parser_main.c:1378-1379).
  • The scanner itself does the printing: each of its actions echoes the matched text when the option is set (DUMP_PREPROCESS, parser_lex.l:64). The include lines are not echoed; in their place the scanner prints a mark (parser_lex.l:170-184, parser_lex.l:223):
MarkMeaning
##included <NAME>the file was found on the search path and its text follows
##included "NAME"the same for a name written in quotes or bare
##failed include <NAME>the file was not found; printed for include if exists (a plain include of a missing file ends the run with an error)
##skipped duplicate include <NAME>the file was already included in this profile body, or in the preamble
  • For the seven-line demo3.sd the output has 516 lines: 14 ##included marks and 8 ##failed include marks, all from tunables/global and the files it includes. The slide shows lines 1 to 5, two of the marks and the first lines of the profile. The last panel is from a second profile whose body has include <abstractions/base> twice.
  • Comments of the included files are kept. Variables are not expanded and conditionals are not evaluated in this output.
  • The output is a profile again, except for the ## lines, which are comments.
  • Output shown here is from the parser built from the AppArmor tree at 836052d21, run with -Q -K (nothing loaded into a kernel, no cache) and the tree's abi/5.0 feature set.

for AppArmor developers for security researchers

In the code

  • -D variables stops one step later and prints the symbol table (parser_main.c:1386-1389); -d runs post-processing and prints the built profiles.

08 Three messages from the scanner

WHEN THE SCANNER STOPS /etc/demo/a!b r,AppArmor parser error for /work/lex1.sd in profile /work/lex1.sd at line 3: Lexer found unexpected character: '!' (0x21) in state: INITIAL quote the path /etc/demo/** rq,AppArmor parser error for /work/lex8.sd in profile /work/lex8.sd at line 4: syntax error, unexpected TOK_ID, expecting TOK_MODE rq is not a mode, so it is a name include "/work/inc1"AppArmor parser error for /work/lex5.sd in profile /work/inc1 at line 2: syntax error, unexpected TOK_END_OF_RULE, expecting TOK_MODE line 2 of the included file for FILE the file given to apparmor_parser in profile FILE the file being read: the same one, or an include the first error ends the run
Three profile lines with the error each produces: an exclamation mark in a path gives Lexer found unexpected character; the permission string rq gives unexpected TOK_ID, expecting TOK_MODE; an error inside an included file names both the policy file and the included file

Lexer found unexpected character names the character and the scanner state. In the header, "for FILE" is the policy file and "in profile FILE" is the file being read, which may be an include.

First: lexer found unexpected character. The scanner met a character no pattern accepts, here an exclamation mark in a path. Quote the path and it's fine. Second, a typo in a permission string. There's no message about permissions. The scanner didn't recognise the letters as a mode, so it called them a name, and the grammar complains that it wanted a mode and got a name. Third, read the header carefully. After the word for comes the file you gave the parser. After in profile comes the file being scanned, which may be an include, and the line number counts in that file. And the first error ends the run.

Details and sources

Details

  • Lexer found unexpected character: 'C' (0xNN) in state: STATE: the last rule of the scanner matches any character that no other rule of the current state accepts. In a path the characters that need quotes are a blank, a tab, ", ! and a comma at the end; write the path in double quotes.
  • A permission string is recognised only as a whole: letters from r w a l m k x with the exec forms (ix, px, cx, ux, pix and so on, either case) and then a blank or a comma. rq does not qualify, is taken as a word, is not a keyword, and reaches the grammar as a name: syntax error, unexpected TOK_ID, expecting TOK_MODE.
  • The header is AppArmor parser error for FILE in profile CURRENT at line N: MESSAGE. FILE is the policy file named on the command line. CURRENT is the file the scanner is reading; inside an include it is the included file, and N is a line of that file. Despite the words, it is not a profile name.
  • The third example: /work/lex5.sd includes /work/inc1, whose second line is /etc/demo/extra, r, (a comma inside the path ends the rule early).
  • Other messages of the scanner: Could not open 'NAME' for an include that is missing; Variable declarations do not accept trailing commas for @{CONF}=/etc/demo,; the warning deprecated use of '#include' (shown with --warn=all or --warn=pound-include; --warn=include is refused as Invalid --warn option include; run with the tree's parser).
  • The parser stops at the first error with exit status 1.
  • Output shown here is from the parser built from the AppArmor tree at 836052d21, run with -Q -K (nothing loaded into a kernel, no cache) and the tree's abi/5.0 feature set. The paths start with /work/ because that is where the files were when the parser ran.
  • Known discrepancy, by title: "\"flex scanner jammed\" instead of an error message, after priority= and after +": priority=x /etc/demo/** r, prints only flex scanner jammed and exits with status 2 (run with the tree's parser).

Read more

for AppArmor developers for security researchers

In the code

09 Where to go from here

ROADS The lexer The grammar tokens become objects From AARE to regular expressions the second decoding Quoting and escaping the author's side include and comments Running apparmor_parser -p and friends From profile text to kernel the whole run
Roads from the lexer: the grammar, from AARE to regular expressions, quoting and escaping, include, running apparmor_parser and from profile text to kernel

Next: the grammar. Related: from AARE to regular expressions, quoting and escaping, include, comments and line structure, running apparmor_parser.

So that's the scanner: one table of keywords, a stack of states, a stack of input buffers, and names that arrive already decoded. From here the tokens go to the grammar, where the first objects are built. The backslashes the scanner left in place are picked up again by the glob conversion. If you'd rather see these rules from the profile author's side, the decks on quoting, on include and on comments cover them. The -p option, and the others that stop the parser early, are in the deck on running the parser. And the pipeline deck shows where this stage sits in the whole run.

Details and sources

Details

Read more

Basis of the statements on this page: "tree" means the parser built from the AppArmor commit 836052d21 (6.0.0~alpha2) was run (-Q -K, or -p, with the tree's abi/5.0 feature set), with nothing loaded into a kernel. "code" means the parser source was read and not run; "derived" means the result was reasoned from that code.

What the scanner returns for common text

TextTokenNotes
a path, :ns:name, &name, @{VAR}tailTOK_ID{LABEL}; quotes stripped, escapes decoded (code)
@{VAR} aloneTOK_SET_VAR
${VAR}TOK_BOOL_VAR
permission letters before a blank or ,TOK_MODE
a word in the keyword tableits tokenmay select a state
any other wordTOK_ID
,TOK_END_OF_RULEin the initial and rule-class states
{ }TOK_OPEN TOK_CLOSE
( )TOK_OPENPAREN TOK_CLOSEPAREN( opens a list state
->TOK_ARROWa name follows
= += ?= :=TOK_EQUALS TOK_ADD_ASSIGN TOK_COND_ASSIGN TOK_OVERRIDE_ASSIGNan assignment follows
^TOK_CARETa hat name follows
name before =TOK_CONDIDpriority, flags, peer, xattrs, identities have tokens of their own
# to the end of the linenonecomment
include NAMEnonethe file becomes the input

The keyword table

network unix capability if else not defined change_profile unsafe safe link owner user other subset audit deny allow prompt set rlimit alias rewrite ptrace file mount remount umount unmount pivot_root in dbus signal send receive bind read write eavesdrop peer trace tracedby readby abi userns mqueue delete open io_uring override_creds sqpoll all priority and or: 55 words (parser_misc.c:81-139) (code).

Messages

MessageCauseBasis
Lexer found unexpected character: 'C' (0xNN) in state: STATEa character no rule of the current state acceptstree
Could not open 'NAME'an include whose file is missingtree
Could not open 'PATH' in 'NAME'a file in an included directory cannot be openedcode
Could not process include directory 'PATH' in 'NAME'reading an included directory failedcode
Variable declarations do not accept trailing commasa comma at the end of an assignmenttree
deprecated use of '#include' (warning, class pound-include)#include instead of includetree
flex scanner jammedsee the known discrepancy in the details of the scenes on start conditions and on the scanner's messagestree

for AppArmor developers for security researchers

Functions

FunctionWhereWhat
get_keyword_tokenparser_misc.c:187-191word to token, or -1
processid, processquoted, processunquotedparser_misc.c:468-535the value of a name token
include_filenameparser_lex.l:162-225open an include and push a buffer
include_dir_cbparser_lex.l:128-160one file of an included directory
search_pathparser_include.c:154-185look a name up in the -I directories and the base directory
push_include_stack, pop_include_stack, reset_include_stackparser_include.c:205-246file name and line for messages
yyerrorparser_yacc.y:1748-1757print the header and the message, exit(1)

Roads from here

START HERE

PREAMBLE AND REUSE

PROFILE STRUCTURE

RUNNING THE PARSER

INSIDE THE PARSER