`awk` is three programs¶

Level: 201 · for anyone whose report script has to survive a move between machines

One line: awk on your Mac and awk on your Linux box are different programs written by different people, and on non-ASCII text they give different answers — one of them disagreeing with itself, counting bytes in length() and characters in the regex engine at the same moment.

Which `awk` is this?¶

There is no such program as "awk". There are three in common use, and two of them install as awk:

Implementation	Ships as `awk` on	Notes
BWK awk (the one-true-awk / `nawk`)	macOS — `awk version 20200816`	Kernighan's own; the book's awk
mawk	Ubuntu / Debian — `mawk 1.3.4`	fast, small, and byte-only by design
gawk	neither, by default	GNU's; the only one with real multibyte support

So a script that runs on your laptop and on CI has already run under two different awks, and nothing announced the switch. awk --version says which — mawk needs -W version — and it is worth the two seconds before trusting a count.

In the terminal¶

Everything below runs in the C locale, and is byte-identical under all three implementations — BWK awk, mawk and gawk — which is why it can be a recorded answer key at all.

Verified output of awk_counts_bytes_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

1. WHICH awk IS THIS? THE ANSWER IS NOT THE SAME ON TWO MACHINES
   macOS ships the one-true-awk (BWK awk); Ubuntu ships mawk; gawk is
   what many people mean by 'awk' and is on neither by default. This
   script prints nothing about which one it is, because the identity is
   exactly what differs — and in THIS locale all three agree anyway.

2. length() COUNTS BYTES HERE
$ cat cafe.txt
café
$ awk "{print length(\$0)}" cafe.txt
5
   Four characters on screen, five from length(). In the C locale that
   is every awk's answer. In a UTF-8 locale it is still mawk's and still
   BWK awk's, and gawk says 4 — see the page.

3. substr() CUTS WHERE length() COUNTS, SO IT CUTS INSIDE A CHARACTER
$ awk "{printf \"%s\", substr(\$0,1,4)}" cafe.txt | xxd -p
636166c3
   63 61 66 c3 — 'caf' and the FIRST HALF of é. The output is no longer
   valid UTF-8, and awk neither warned nor failed. This is the same shape
   as cut -b and as a fixed-width field: an offset is a byte offset.

4. gsub(/./) AGREES WITH length() — IN THIS LOCALE
$ awk "{n=gsub(/./,\"X\"); print n, \$0}" cafe.txt
5 XXXXX
   Five. Both the regex engine and length() are counting bytes, so they
   agree. On one of the three awks, in a UTF-8 locale, they stop agreeing
   with EACH OTHER inside a single program. That is the page's point.

5. toupper() IS AN ASCII PROMISE HERE
$ awk "{print toupper(\$0)}" cafe.txt
CAFé
$ awk "{print tolower(\$0)}" shout.txt
ŻÓŁw
   café became CAFé and ŻÓŁW became ŻÓŁw: the ASCII letters changed and
   nothing else did. Half a word case-mapped is worse than none, because
   it reads as a typo rather than as an encoding problem.

6. WHAT DOES WORK PERFECTLY: FIELDS WITH AN ASCII SEPARATOR
$ cat rows.csv
Ada,café,3
Ben,naïve,7
$ awk -F, "{print \$1 \" wants \" \$2}" rows.csv
Ada wants café
Ben wants naïve
   Splitting on a comma and moving whole fields around never looks inside
   a character, so it is safe at any width in any locale. Most awk in the
   world is this, which is why the trap above stays hidden for years.

7. COUNTING CHARACTERS PORTABLY, WITHOUT A LOCALE
$ awk "{n=\$0; c=gsub(/[\\200-\\277]/,\"\",n); print length(\$0)-c}" cafe.txt
4
   Four — the right answer, from an awk that thinks in bytes. UTF-8's
   design is what makes it possible: continuation bytes are exactly
   0x80-0xBF and nothing else uses that range, so bytes minus
   continuation bytes is the character count. It works on all three awks
   in any locale, which no built-in can claim.

Where the three part company¶

Change the locale and the agreement ends. One file, café, four characters and five bytes:

Measured 2026-09-06 — BWK awk 20200816 (macOS 26.6), mawk 1.3.4 and GNU Awk 5.2.1 (ubuntu:24.04), LC_ALL=en_US.UTF-8. Not machine-checked: CI has one awk and cannot see the other two.

                        mawk        BWK awk      gawk 5.2.1
length($0)              5 bytes     5 bytes      4 characters
substr($0,4,1)          c3          c3           é
gsub(/./,"X")           5           4            4
toupper("café")         CAFé        CAFÉ         CAFÉ
tolower("ŻÓŁW")         ŻÓŁw        żółw         żółw

Read the BWK awk column down. length() says five and gsub(/./) says four — in the same program, on the same string, in the same locale. The regex engine was taught about characters and the string functions were not, so length($0) and gsub(/./,"X") count different things and neither is documented as the odd one out. A loop that walks for (i=1; i<=length($0); i++) substr($0,i,1) will therefore step byte by byte through a string whose regexes think in characters.

mawk is the honest one here: it ignores the locale entirely and is bytes everywhere, so at least it cannot contradict itself. gawk is the correct one: characters everywhere, and gawk -b (--characters-as-bytes) turns it back into a byte tool when that is what you want.

The rule, and the portable trick¶

Do not use length() or substr() to count or slice text you did not generate, unless you have pinned the implementation and the locale. Splitting on a delimiter is always safe — -F, never looks inside a field — and that is most of what awk is used for, which is why this stays hidden for years.

When you genuinely need a character count and cannot pin anything, count it out of the bytes. UTF-8 makes this exact: continuation bytes are precisely 0x80–0xBF, and nothing else uses that range, so

{ n = $0; c = gsub(/[\200-\277]/, "", n); print length($0) - c }

prints 4 for café on all three awks, in any locale — bytes minus continuation bytes is the character count. It is in the recorded example above, so it is checked. That the trick works at all is a property of the encoding rather than of awk: it is the same self-synchronisation that lets grep find a substring in UTF-8 without decoding, and that a fixed-width format has to give up.

What it does with bytes that are not text¶

BWK awk is the only tool in this chapter that fails usefully on undecodable input — it names the record:

Measured 2026-09-06 — macOS 26.6, LC_ALL=en_US.UTF-8. Three lines, all containing 'line'; the middle one holds ff fe.

$ awk '/line/' invalid.txt
good line
awk: towc: multibyte conversion failure on: '?? line'
 input record number 2, file invalid.txt
 source line number 1
$ echo $?
2

One line of output, an error naming input record number 2 and the file, and exit 2. Compare grep on the identical file: two of three lines, no message, exit 0. END { print NR } still reports 3, so awk read the record and could not match on it — the failure is in the conversion, not the input. On Ubuntu, mawk processes all three lines without comment, because it never converts anything.

If you are coming from Python or ABAP¶

Python: len(s) on a str is gawk's length() and never mawk's — Python decided the byte/character question at the type boundary, so there is no locale and no implementation to check. The awk trap has no Python equivalent because there is no configuration under which len() and re disagree about the same string. What does transfer is the discipline the trick teaches: when you must count characters out of bytes, sum(1 for b in data if b & 0xC0 != 0x80) is the same arithmetic, and it is worth being able to write.

ABAP (Not machine-checked — CI cannot run ABAP.) strlen( ) is gawk's length(), xstrlen( ) is mawk's, and ABAP makes you choose by choosing a type — the ambiguity awk resolves per implementation, ABAP resolves in the declaration. The awk situation is closest to reading a file with OPEN DATASET … IN BINARY MODE and then applying strlen habits to the xstring: the numbers look plausible and count the wrong thing. And as always, name the code page at the boundary rather than trusting a default.

Try it¶

awk --version (or awk -W version) on every machine your scripts run on. Write down what you find.
printf 'café\n' | awk '{print length($0)}' under LC_ALL=C and under your own locale, on a Mac and on a Linux box. Four answers, two of them from the same command.
Run the continuation-byte trick on a file of Polish or Japanese text and check it against wc -m in a UTF-8 locale.
awk '{for(i=1;i<=length($0);i++) printf "%s|", substr($0,i,1)}' on an accented word, then look at the output with xxd.

Practice¶

Whose awk is this? On a file containing café, predict length($0) and gsub(/./,"X") in the C locale. Then name the three awks in common use, say which one is the default on macOS and which on Ubuntu, and give the command that tells you which you have.

Then the failure the page is named for: describe how one awk can disagree with itself about a single string in a single run, and which two functions to distrust.

Answers

Verified output of awk_kata_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

THE FILE: 4 characters, 5 bytes
   café   636166c3a90a

IN THE C LOCALE
   awk length($0)          5
   awk gsub(/./,"X")       5
   Five and five. A character is a byte here, so both halves of awk agree
   -- and the answer they agree on is the BYTE count, not the four
   characters a person would count.

THERE ARE THREE awks, AND YOU HAVE ONE OF THEM
   BWK awk (the one true awk)  the default on macOS and the BSDs
   mawk                        the default /usr/bin/awk on Debian/Ubuntu
   gawk                        GNU awk, the default on Fedora, and
                               installable everywhere
   They are different programs by different authors with different
   Unicode support, and none of them is 'awk' in a portable sense. Ask:
       awk --version    (or awk -W version for older mawk)

WHAT SEPARATES THEM IS THE UTF-8 LOCALE, NOT THE C ONE
   Measured 2026-09-07: in the C locale all three count bytes in both
   length() and the regex engine, so all three agree with each other and
   with themselves. Put a UTF-8 locale in front of them and the
   guarantees come apart -- some versions of BWK awk count BYTES in
   length() while the regex engine matches CHARACTERS, so length(awk_kata_sh.sh) and
   gsub(/./,"X") disagree about one string in one run of one program.

THE PRACTICAL RULE
   If an awk script measures or slices text, pin the locale and say which
   awk it needs -- or move the job to a tool with one implementation.
   substr() and length() are the two functions to distrust; field
   splitting on a delimiter is safe, because a delimiter is a byte
   sequence and every awk finds it the same way.
   awk -F" " "{print \$1}"  -> café

awk is three programs¶

Which awk is this?¶