From AARE to regular expressions
globs become patterns for AppArmor developers for security researchers for application developers
The last step of the front end rewrites each AppArmor glob as the regular expression string the backend receives. By then quotes and numeric escapes are decoded and variables are expanded and repeated slashes are removed. One loop with three flags does the rewriting. Its errors come after parsing and have no line number.
On the map: inside the parser
▶ Watch this deck — 8 scenes, ~6 min
Worth seeing first
- The grammar — the entry whose name is converted here
- Globbing (AARE) — what each wildcard means in a profile
Part of journeys
- From profile text to kernel tables, step 4 of 13 · previous: The grammar · next: How the kernel checks an access
01 After parsing, the name is still a glob
convert_aaregex_to_pcre() (parser/parser_regex.c:118) rewrites one AppArmor glob as a regular expression string. It runs last in the front end, once per pattern.
After parsing, demo's first rule is an entry in memory, and its name is still what you typed: a path that ends in two stars. The backend can't use that. It builds state machines from regular expressions, and a glob isn't one. So the last thing the front end does is translate. One function does it, and its output is the exact string the backend receives. What each wildcard means is in the globbing deck. This deck is the function itself: what reaches it, how it walks a pattern, what it escapes, and what it reports. One scene is for anyone who has met a regex grouping error.
Details and sources
Details
pattern_t convert_aaregex_to_pcre(const char *aare, int anchor, int glob, std::string& pcre, int *first_re_pos)(parser_regex.c:118-119). "AARE" is the AppArmor regular expression syntax of profiles: the globs*,**,?,[ ]and{ }. The name says PCRE; the output is the smaller regular expression syntax that the backend's own parser reads.- Order of work for one profile:
post_process_profileadds implied rules, expands variables, applies aliases, merges rules and then callsprocess_profile_rules(parser_policy.c:127-165), which converts the attachment and every rule and hands the strings to the backend (parser_regex.c:1215-1283). - For a file rule the call is in
process_dfa_entry(parser_regex.c:648-688). - The profile followed in this deck (file
demo3.sd):
abi <abi/5.0>,
include <tunables/global>
profile demo /usr/bin/demo {
/etc/demo/** r,
audit deny @{HOME}/.ssh/** rw,
/usr/bin/[a-z]* ix,
}
- Dumps are from the parser built from the AppArmor tree at 836052d21, run with
-Q -K -D rule-exprs(nothing loaded into a kernel, no cache) and the tree'sabi/5.0feature set. Eachaare:line is printed by the conversion itself.
Read more
- Globbing (AARE): what the wildcards match. The expression tree: what the backend does with the string.
- apparmor.d(5) Globbing AARE.
02 Three things are already done for AppArmor developers for security researchers
Before the conversion: processunquoted (scanner), process_variables (post-processing) and filter_slashes (parser/parser_regex.c:61-96). A leading // survives.
The function doesn't see your profile text. Three things have already happened to a pattern. The scanner stripped quotes and decoded escapes, keeping a backslash only in front of glob characters. Post-processing expanded variables: the @{HOME} variable is no longer a variable, it's a pair of braces with two alternatives inside, so the conversion needs no idea what a variable is. And just before the call, filter_slashes squeezes every run of slashes into one. Expansion often leaves doubles behind, like the one after home here. One exception: exactly two slashes at the very start are kept; the code comment calls that a namespace of its own.
Details and sources
Details
- The scanner:
processunquoteddecodes escapes and keeps a backslash before a decoded* ? [ ] { } ^ , \(parser_misc.c:468-509); see The lexer. - Variables:
block_rules::process_variablesexpands the name and the link name of every entry and callsexpand_variablesof every rule object (block_rules.cc:393-416). A variable with several values becomes{a,b}. The-ddump shows the entry at that point:({/home//*,/root}/.ssh/**);tunables/homedefines@{HOME}as@{HOMEDIRS}/*/ /root/and@{HOMEDIRS}as/home/. filter_slashes(parser_regex.c:61-96) rewrites the string in place: a/that follows a/is dropped; when the string starts with exactly two slashes they are both kept ("special case for linux // namespace", lines 70-74). A single trailing slash is kept, although the comment above the function says trailing slashes are filtered.- For a file rule:
filter_slashes(entry->name)unless the entry is a change_profile rule, then the conversion (parser_regex.c:658-660); the same for the link target (parser_regex.c:699-701). filter_slashesis also applied to the attachment (parser_variable.c:76), mount points and devices (mount.cc:649-653), D-Bus paths (dbus.cc:190) and unix socket addresses (af_unix.cc:174-178).- The right-hand column of the slide is what
-D rule-exprsprints afteraare:;//etc/xand/etc//demo///confare from a profile written for this deck. - Dumps are from the parser built from the AppArmor tree at 836052d21, run with
-Q -K -D rule-exprs(nothing loaded into a kernel, no cache) and the tree'sabi/5.0feature set. Eachaare:line is printed by the conversion itself. - Known discrepancy, by title: "A variable after a prefix of only slashes keeps its leading
/, so the rule starts with//and never matches".
Read more
03 One loop and three flags for AppArmor developers for security researchers
bEscape, ingrouping and incharclass (parser/parser_regex.c:132-134) are all the state there is. {/home/*,/root}/.ssh/** becomes (/home/[^/\x00]*|/root)/\.ssh/[^/\x00][^\x00]*.
The function is a single loop over the characters with three pieces of memory: was the last character a backslash, how deep are we in braces, are we inside brackets. Watch the deny rule go through. The open brace becomes a parenthesis and the depth goes to one. The star becomes any run without a slash; it gets no mandatory first character, because a comma follows it, not a slash. The comma, being inside braces, becomes a bar. The close brace closes the group. The dot is a regex character, so it's escaped. And the final two stars, a whole path component, get the mandatory character and then anything.
Details and sources
Details
- State:
bool bEscape,int ingrouping,int incharclassandint grouping_count[MAX_ALT_DEPTH](parser_regex.c:132-135). The loop iswhile (error == error_type::no_error && *sptr)with oneswitch (*sptr)(parser_regex.c:149-405);bEscapeis cleared at the end of every pass except the one that reads the backslash. - The cases:
\(parser_regex.c:152-168),*(parser_regex.c:169-229),?(parser_regex.c:231-254),[and](parser_regex.c:256-280),{and}(parser_regex.c:282-330),,(parser_regex.c:332-355),^and$(parser_regex.c:358-366),.,+,|,(,)and every other character (parser_regex.c:374-402). - The mandatory character: when the output so far ends in
/and the run of stars is followed by/or the end of the pattern,[^/\x00]is written first (parser_regex.c:178-205). The test looks only for/or the end. A star that is followed by,or}inside a group does not get it, so{/home/*,/root}/.ssh/**also describes/home//.ssh/...; a path with an empty component does not reach the matcher from the kernel, so nothing more is matched in practice (derived). - The first line of the dump for this rule:
aare: {/home/*,/root}/.ssh/** -> (/home/[^/\x00]*|/root)/\.ssh/[^/\x00][^\x00]*. - The output string is appended to the caller's buffer; the function never clears it.
- Dumps are from the parser built from the AppArmor tree at 836052d21, run with
-Q -K -D rule-exprs(nothing loaded into a kernel, no cache) and the tree'sabi/5.0feature set. Eachaare:line is printed by the conversion itself.
Read more
- Globbing (AARE): the slash rule as an author meets it. The expression tree: the same three patterns after the next parser.
- parser/parser_regex.c.
04 Escaping runs in two directions for AppArmor developers for security researchers
. + | ( ) are always escaped, ^ $ outside a class. \* \[ \] \{ \} stay escaped; \? and \, become ? and ,.
Escaping runs in two directions. Some characters mean nothing in a profile but a lot in a regular expression: dot, plus, bar, parentheses, and outside brackets, caret and dollar. The function always escapes those, so a dot in a file name stays a dot. Then the reverse case: you escaped a glob character. A \* must still be a literal star for the next parser, so it stays escaped; so do escaped brackets and braces. An escaped question mark or comma can drop its backslash, because the regex parser gives those no meaning. Any other backslash that isn't a valid escape is dropped too, with a warning.
Details and sources
Details
- Always escaped:
.,+,|,(,)(parser_regex.c:374-380). Escaped outside a character class and copied inside one:^and$(parser_regex.c:358-366). - An escaped glob character is written with its backslash:
\*(parser_regex.c:170-177),\[and\](parser_regex.c:257-260, parser_regex.c:269-272),\{and\}(parser_regex.c:283-286, parser_regex.c:307-310).\\is written as\\(parser_regex.c:152-168). \?is written as?(parser_regex.c:232-238);\,as,outside a class and as\,inside one (parser_regex.c:333-345).- A backslash before any other character (parser_regex.c:382-402): when
str_escseqaccepts what follows as an escape sequence (\x00,\000,\d000and the letter escapes), the sequence is copied with its backslash for the backend to decode; otherwise the backslash is dropped and, with warnings of classformatenabled, the parser printsCharacter q was quotedunnecessarily, dropped preceding quote ('\') character. - The scanner has already decoded most escapes, so what reaches this function with a backslash is mainly the set
* ? [ ] { } ^ , \(see The lexer). - The unit tests of the function list the expected output for each of these inputs (parser_regex.c:1566-1699).
- Dumps are from the parser built from the AppArmor tree at 836052d21, run with
-Q -K -D rule-exprs(nothing loaded into a kernel, no cache) and the tree'sabi/5.0feature set. Eachaare:line is printed by the conversion itself. - Known discrepancies, by title: "Parser messages with wrong text or wrong line" (the missing blank in "quotedunnecessarily", and the line number of that warning, which is the last line of the file); "Character classes:
\^loses its escape, and*or?inside[...]are expanded".
Read more
- Quoting and escaping: which characters an author has to escape.
- The expression tree: the grammar of the parser that reads these strings.
05 Braces are counted, brackets are copied for AppArmor developers for security researchers
{a,b} becomes (a|b), nested fewer than 50 deep, with at least one comma per group. [...] is copied; a negated class written in a profile can match /.
Braces and brackets are where the function must remember its place. For braces it keeps a depth, which must stay below fifty, and for each level a count of commas. A group with no comma is refused, so braces around a single word are an error. But a comma with nothing after it is fine: an empty alternative, which means optional. Brackets are simpler. The function marks that it's inside a class and copies the characters through, leaving it to the backend's regex parser. Inside, braces and commas are just characters. And a negated class is copied as written, so unlike a question mark it can match a slash.
Details and sources
Details
{:ingrouping++; at 50 (MAX_ALT_DEPTH, parser_regex.c:122) the errorRegex grouping error: Exceeded maximum nesting of {}; otherwise the comma count of the new level is cleared and(is written. So 49 levels are the most that convert: a pattern nested 49 deep compiles and one nested 50 deep is refused (both run with the tree's parser) (parser_regex.c:282-304).,inside a group and outside a class: the count of the current level goes up and|is written; elsewhere the comma is copied (parser_regex.c:346-353).}: a count of zero is the errorRegex grouping error: Invalid number of items between {}; theningrouping--and)(parser_regex.c:306-330). So{conf}is refused and{c,}gives(c|).- At the end of the pattern an open group or class is the error
Regex grouping error: Unclosed grouping or character class, expecting close }(parser_regex.c:411-416). [:incharclass = 1and the character is copied;]clears it (parser_regex.c:256-280). Inside a class{,}and,are copied (parser_regex.c:288-290, parser_regex.c:312-314). The class is parsed later by the regular expression grammar of the backend (parse.y:109-150).- The globs
*,**and?are written as classes that exclude/(and NUL); a class written by the author has no such exclusion, so[^a]matches/as well. - Dumps are from the parser built from the AppArmor tree at 836052d21, run with
-Q -K -D rule-exprs(nothing loaded into a kernel, no cache) and the tree'sabi/5.0feature set. Eachaare:line is printed by the conversion itself. - Known discrepancy, by title: "Character classes:
\^loses its escape, and*or?inside[...]are expanded". With the tree's parser/etc/demo/[a*]zis converted to/etc/demo/[a[^/\x00]*]z. - Known discrepancy: a
}without an opening{prints two messages, "Invalid number of items between {}" and then "Invalid close }, no matching open { detected", because the comma count is tested before the depth (parser_regex.c:316-324). Run with the tree's parser. - Known discrepancy, by title: "
@{u16}has an unbalanced bracket and also matches[and 65536 to 69999".
Read more
- Globbing (AARE): braces and character sets in a profile. The expression tree: how the class is read.
- apparmor.d(5) Globbing AARE.
06 The function returns more than a string for AppArmor developers for security researchers
pattern_t is Basic, TailGlob, Regex or Invalid; *first_re_pos is the offset of the first glob. process_profile_name_xmatch uses both.
The function returns more than a string: it also classifies the pattern. Basic, with no glob; tail glob, plain text that ends in two stars; regex, for everything else; and invalid. And it reports where the first glob was. Most callers only ask: invalid or not? The interesting user is the attachment. If a profile's name is basic, and there's no separate attachment, no alias name and no xattr condition, the parser builds no attachment machine; the kernel compares names. Otherwise the position of the first glob seeds the match length that ranks competing profiles. The other rule classes use the same function: D-Bus names, signal peers, mount points.
Details and sources
Details
enum class pattern_t { Basic, TailGlob, Regex, Invalid }(immunix.h:139-144). The function starts withBasic; a**that ends the pattern while the type is stillBasicmakes itTailGlob; every other glob, class or group makes itRegex(parser_regex.c:206-221); an error makes itInvalid(parser_regex.c:437-438).*first_re_posis set once, to the offset of the first glob in the input, and stays 0 when there is none (update_re_pos, parser_regex.c:121).- The attachment, in
process_profile_name_xmatch(parser_regex.c:478-518): the attachment, or the profile name when there is none, is converted with&prof->xmatch_lenas the position; for aBasicpattern the length isstrlen(name). ABasicpattern in a profile without alias names, separate attachment or xattrs getsprof->xmatch = NULL; anything else goes to the backend as a small DFA. - Kernel statements are read from the kernel source and were not run: without an xmatch DFA the kernel attaches by comparing the executable's name with the profile name (domain.c:485).
- File rules store the type in
entry->pattern_type(parser_regex.c:664); no other code inparser/reads it, and nothing outside the unit tests distinguishesTailGlobfromRegex. - Other callers:
convert_entryappends the conversion of a value, or the default "match anything" pattern when the value is absent (parser_regex.c:1316-1336);build_list_val_exprjoins a list of values with|(parser_regex.c:1286-1314). Direct calls: dbus.cc:229-281, signal.cc:330, ptrace.cc:141, af_unix.cc:275-299, network.cc:434, mount.cc:769. - The
globargument:glob_defaultwrites[^/\x00]*,[^\x00]*and[^/\x00];glob_nullwrites[^/]*,.*and[^/], which also match a NUL byte (parser_regex.c:98-114).glob_nullis used for xattr values (parser_regex.c:582-584) and abstract unix socket addresses (af_unix.cc:275). - Known discrepancy, by title: "
NotCharSetNode::contains_oob()is inverted, soxmatch_lenstops at the first wildcard".
Read more
- Profile attachment: how the match length decides between profiles. Extended attribute conditions.
- The backend pipeline: the three DFAs, of which the attachment one is the first.
07 A pattern error has no line number
Regex grouping error: ... says what is wrong; Unable to parse input line 'PATTERN' quotes the pattern with variables expanded. Neither has a file name or a line number.
A pattern mistake looks like this, and it's unlike a syntax error. No file name, no line number. That's because the conversion runs after the whole file has been parsed, when the parser no longer knows which line a pattern came from. So read it like this. The first line says what's wrong: here, braces with nothing to choose between. The second quotes the pattern after variables were expanded; search your profile for it, and remember the problem may hide in a variable's value. The name in front is the parser program, not your file. The remaining three lines are each caller giving up; two of them name the profile.
Details and sources
Details
- The complete output for the profile
demowith the rule/etc/demo/{conf} r,. The path at the start of three of the lines is where the parser program was when it ran:
/src/apparmor/parser/apparmor_parser: Regex grouping error: Invalid number of items between {}
/src/apparmor/parser/apparmor_parser: Unable to parse input line '/etc/demo/{conf}'
ERROR processing regexs for profile demo, failed to load
ERROR generating policy re for profile demo, failed to load
/src/apparmor/parser/apparmor_parser: Errors found in file. Aborting.
- The first lines for other mistakes:
| Pattern | First line |
|---|---|
/etc/demo/{conf} | Regex grouping error: Invalid number of items between {} |
/etc/demo/{a,b | Regex grouping error: Unclosed grouping or character class, expecting close } |
/etc/demo/[ab | the same text, also with "expecting close }" |
/etc/demo/a] | Regex grouping error: Invalid close ], no matching open [ detected |
/etc/demo/a} | Regex grouping error: Invalid number of items between {} and then Regex grouping error: Invalid close }, no matching open { detected |
- A mistake in the attachment or in a pattern used as the profile name ends differently: after the first two lines come
Invalid profile name 'PATTERN' - bad regular expressionandERROR processing xmatch for profile NAME, failed to load. - The quoted pattern is the one the conversion received. A rule written as
@{CONF}/xis quoted with the value of the variable in place of@{CONF}: the value itself for one value,{a,b}for several. - With
--warn=allthe conversion can also print a warning,Character C was quotedunnecessarily, dropped preceding quote ('\') character. It does carry a file and a line, but the line is the last line of the file. - The exit status of these runs was 255; a syntax error gives 1.
- Output shown here is from the parser built from the AppArmor tree at 836052d21, run with
-Q -K(nothing loaded into a kernel, no cache) and the tree'sabi/5.0feature set. - Known discrepancies, by title: "Parser messages with wrong text or wrong line"; "An undefined variable is reported without file name or line".
Read more
- Globbing (AARE): the same mistakes with their fixes. Variables.
- The grammar: errors that do have a line. Running apparmor_parser.
for AppArmor developers for security researchers
In the code
- The messages of the conversion: parser_regex.c:275, parser_regex.c:295, parser_regex.c:317, parser_regex.c:323, parser_regex.c:414, parser_regex.c:421 and
Unable to parse input line '%s'at parser_regex.c:430. They usePERRORwithprogname, notyyerror. - Line 3 is printed by
block::gen_policy_re(block.cc:90), line 4 byprocess_profile_rules(parser_regex.c:1252), line 5 byprocess_profile(parser_main.c:1393). - The attachment messages: parser_regex.c:502 and parser_regex.c:1274.
Regex error: trailing '\' escape character(parser_regex.c:421) was not reached from a profile: a pattern that ends in a backslash is stopped earlier, by the scanner (known discrepancy, by title: "A path ending in an escaped backslash cannot be written: quoted it runs to the next quote, unquoted it swallows the blank").
08 Where to go from here
Next: the backend pipeline, then the expression tree. Related: globbing, quoting and escaping, profile attachment, the grammar.
So that's the bridge: decoded text in, one loop over it, backslashes added and removed, braces counted, classes copied, and out comes a string with a type and a position. That string is passed to the backend's add_rule function together with the permissions, and with that the front end's work is done. The backend pipeline deck picks up from exactly this point, with the same three rules. The tree deck shows the next parser, the one that reads these strings. For what the wildcards mean when you write a profile, go to globbing and to quoting. And the attachment deck explains what the match length is for.
Details and sources
Details
- The stage in four lines: the pattern arrives decoded, expanded and with single slashes; one loop rewrites it; regex-only characters are escaped and glob characters translated; the result is a string plus a pattern type and the position of the first glob.
- The hand-off:
dfarules->add_rule(tbuf.c_str(), entry->priority, entry->rule_mode, entry->perms, audit, parseopts)(parser_regex.c:685-688);aare_rules::add_ruleis at aare_rules.cc:46.-D rule-exprsprints both sides, theaare:line of the front end and therule:line of the backend:
aare: /etc/demo/** -> /etc/demo/[^/\x00][^\x00]* rule: /etc/demo/[^/\x00][^\x00]* -> /etc/demo/[^\x0/]([^\x0])* priority=0 (0x10004/0)< 0x10004>
- The backend pipeline: the nine stages that start with these strings. The expression tree: string to nodes.
- From the author's side: Globbing (AARE), Quoting and escaping, Variables, Profile attachment.
- Earlier stages: The lexer, The grammar, Post-processing.
- Read for this deck: AppArmor 836052d21 and, for one statement about attachment, Linux 97d73c825.
Read more
Read more
Basis of the statements on this page: "tree" means the parser built from the AppArmor commit 836052d21 (6.0.0~alpha2) was run (-Q -K -D rule-exprs, with the tree's abi/5.0 feature set), with nothing loaded into a kernel. "code" means the parser source was read and not run; "derived" means the result was reasoned from that code.
Character by character
| Read | Written | Notes | Basis |
|---|---|---|---|
* | [^/\x00]* | after / and before / or the end, [^/\x00] first | tree |
** | [^\x00]* | the same mandatory character | tree |
? | [^/\x00] | tree | |
[ ... ] | copied | {, }, ,, ^, $ are copied inside | tree |
{ , } | (, a bar, ) | at least one comma; at most 49 levels | tree |
. + ( ) and the bar | with a backslash | always | tree |
^ $ | \^ \$ | outside a class | tree |
\* \[ \] \{ \} \\ | unchanged | tree (\*, \{, \}), code | |
\? \, | ? , | \, stays inside a class | tree |
\x00 \000 \d000 and letter escapes | unchanged | decoded by the backend | code |
\ + any other character | the character | warning of class format | tree |
| anything else | copied | tree |
With glob_null (xattr values, abstract unix socket addresses) the three globs are [^/]*, .* and [^/] (code).
What was done to the pattern earlier
| Step | Where | Effect |
|---|---|---|
| quotes and escapes | scanner, processid | quotes removed; escapes decoded; a decoded * ? [ ] { } ^ , \ keeps a backslash |
| variables | post-processing, process_variables | @{VAR} replaced by its value, or by {a,b} for several values |
| aliases | post-processing | an entry whose name starts with an alias source is duplicated with the target |
| slashes | filter_slashes, just before the conversion | runs of / collapsed; exactly two leading slashes kept |
Return value
pattern_t | Meaning | Example |
|---|---|---|
Basic | no glob | /usr/bin/demo |
TailGlob | plain text and one final ** | /etc/demo/** |
Regex | any other glob, class or group | /usr/bin/[a-z]* |
Invalid | an error was printed | /etc/demo/{conf} |
*first_re_pos: offset of the first glob in the input; 0 without one (code).
Messages
| Message | Cause | Basis |
|---|---|---|
Regex grouping error: Invalid number of items between {} | a group without a comma | tree |
Regex grouping error: Unclosed grouping or character class, expecting close } | { or [ not closed | tree |
Regex grouping error: Invalid close ], no matching open [ detected | ] without [ | tree |
Regex grouping error: Invalid close }, no matching open { detected | } without { | tree |
Regex grouping error: Exceeded maximum nesting of {} | 50 or more levels | tree |
Regex error: trailing '\' escape character | pattern ends in a backslash | code |
Unable to parse input line 'PATTERN' | follows each of the above | tree |
Invalid profile name 'PATTERN' - bad regular expression | the pattern is an attachment or profile name | tree |
Character C was quotedunnecessarily, dropped preceding quote ('\') character | warning, class format | tree |
for AppArmor developers for security researchers
Functions and callers
| Function | Where | What |
|---|---|---|
convert_aaregex_to_pcre | parser_regex.c:118-445 | the conversion |
filter_slashes | parser_regex.c:61-96 | collapse repeated slashes in place |
process_profile_name_xmatch | parser_regex.c:478-628 | attachment, alias names, xattr values |
process_dfa_entry | parser_regex.c:648-787 | file, link and change_profile entries |
convert_entry, clear_and_convert_entry, build_list_val_expr | parser_regex.c:1286-1336 | helpers for the other rule classes |
aare_rules::add_rule | aare_rules.cc:46 | where the string enters the backend |
Roads from here
ATTACHMENT, NAMESPACES, STACKING
- Profile attachment — how the match length ranks competing profiles
FILE ACCESS
- Globbing (AARE) — the wildcard rules from the author's side
PREAMBLE AND REUSE
- Variables — where the braces of an expanded variable come from
PROFILE STRUCTURE
- Quoting and escaping — escapes from the author's side
INSIDE THE PARSER
- The lexer — the first decoding of a name
- The grammar — where the entry was built
- Post-processing (no slides yet) — the steps that run just before the conversion
THE DFA ENGINE · libapparmor_re
- The backend pipeline — the backend that receives the converted strings
- The expression tree — the next parser that turns each string into nodes