Skip to content

Command-line tools

One line: grep, sed and awk are POSIX engines — leftmost-longest, no lazy quantifiers, no lookaround, BRE by default — and the same pattern typed into a shell and into a program means something different three times over: the syntax family, the match rule, and the quoting the shell does before the tool ever sees it.

Level: 201 · you are about to type a pattern after grep

Three flavours in three flags

Flag Flavour ( + ? { | mean Has
grep, sed (default) BRE — basic literals; the operators are \( \+ \? \{ | \1 backreferences (required by POSIX in BRE)
grep -E, sed -E, awk, egrep ERE — extended operators, as in every language backreferences only as a common extension
grep -P, pcre2grep PCRE2 as in Perl everything on the Keywords shelf — lookaround, \K, \d, lazy quantifiers

BRE's inverted escaping is the first thing that bites: a+ in grep is a literal a+, and a\+ is one-or-more. ERE is close enough to the language engines to be pasted from them — until the match rule differs.

Measured

The match-semantics lesson runs the POSIX tools on the command line, by absolute path so the grep on PATH cannot be something else:

Verified output of leftmost_tools_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

tool and pattern                        subject    result
grep -E -o 'a|ab'                       ab         ab
grep -E -o '[0-9]+|[0-9]+\.[0-9]+'      v 3.14 w   3.14
sed -E 's/a|ab/[&]/'                    ab         [ab]
sed -E 's/[0-9]+|[0-9]+\.[0-9]+/[&]/'   v 3.14 w   v [3.14] w
awk match($0, /a|ab/)                   ab         ab at 1 len 2
awk match($0, /[0-9]+|[0-9]+\.[0-9]+/)  v 3.14 w   3.14 at 3 len 4

A badly ordered alternation — [0-9]+|[0-9]+\.[0-9]+, integer first — comes back right from grep -o, sed and awk, because POSIX takes the longest alternative. Every language engine takes the first and returns 3. A pattern that works on the command line can therefore be wrong in the program it is pasted into, and the tool will not have warned.

And the oldest spelling of a backreference, from the backreferences chapter:

Verified output of doubled_word_grep_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

BRE  (grep):
the the cat
fox fox
ERE  (grep -E):
the the cat
fox fox

The three things that differ

  1. Match rule. Leftmost-longest. Two definitions of "the match" has the C program and the two libcs that disagree about submatches.
  2. No lazy quantifiers. a+? in ERE is (a+)?. GNU grep -E and sed -E and both awks read <.+?> as (<.+>)? and take the whole line; macOS BSD grep is lazy; macOS BSD sed refuses. Greedy and lazy has the six runs.
  3. The shell gets the pattern first. Inside double quotes, $ and ` and \ are the shell's; inside single quotes nothing is. \d is not POSIX — [[:digit:]] or [0-9] — and grep -P is the only grep that understands it.

GNU and BSD

The Mac ships BSD tools; Linux ships GNU. They agree on POSIX and disagree past it: grep -o and -P are GNU extensions BSD grep partly has, sed -i takes an argument on BSD and not on GNU, sed -E works on both but \+ in BRE is GNU-only, and BSD sed refuses a+?. A shell script that must run on both stays inside POSIX ERE and uses [[:class:]] names, or calls perl -pe, which is the same everywhere.

Keywords on this page

POSIX classes · Alternation · Lazy quantifier · Backreference · \K — grep -oP 'key=\K\w+' is the idiom · \Q…\E — grep -F is the tool's version.

See also