grep on text that is not ASCII¶
Level: 201 · for anyone who greps a file they did not write
One line: grep has no idea what encoding your file is in, takes its definition of "a character" from the locale, and — on one of the two greps in common use — will silently drop a line it cannot decode and still exit 0.
Three decisions grep makes before it prints anything¶
grep is a byte-matching program wearing a text interface. Three things happen between your pattern and its answer, and none of them is announced:
- The locale decides what a character is. In the C locale
.matches one byte; in a UTF-8 locale it matches one character. Same file, same pattern, different count. - A single NUL byte reclassifies the file as binary, after which grep will not show you the matches it found. High bytes that are valid UTF-8 do not do this; one
00does — and so, for GNU grep in a UTF-8 locale, does a byte that is not valid UTF-8, measured below. - The bytes are matched as bytes. Your pattern is
caf; a UTF-16 file spells that63 00 61 00 66 00; there is no match and no error.
The first is a nuisance. The second is visible. The third is the one that quietly loses data, and it has a fourth sibling that is worse, at the bottom of this page.
In the terminal¶
Everything below is identical on macOS and Ubuntu — it is the recorded answer key CI checks on both.
Verified output of grep_on_non_ascii_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
1. IN THE C LOCALE, A 'CHARACTER' IS A BYTE
$ cat cafe.txt
café
$ xxd -p cafe.txt
636166c3a90a
$ grep -o . cafe.txt | wc -l
5
Four characters on the screen. Five answers from grep, because in this
locale '.' means one byte and 'é' is two of them. Nothing warned you.
2. THE PORTABLE WAY TO FIND NON-ASCII LINES
$ grep -n "[^ -~]" mixed.txt
2:café here
The class is 'any byte outside space through tilde' — the printable
half of ASCII. It needs no -P, so it works on BSD grep too, where -P
does not exist at all. It also flags tabs, which are outside that range
and are usually worth seeing anyway.
3. -c COUNTS LINES, NOT MATCHES
$ grep -c a mixed.txt
3
$ grep -o a mixed.txt | wc -l
4
Three lines contain an 'a'; there are four of them. -c answers the
first question. If you wanted the second, -o and wc are the pair.
4. ONE NUL BYTE, AND THE FILE STOPS BEING TEXT
$ xxd withnul.txt
00000000: 6865 6c6c 6f00 776f 726c 640a 6865 6c6c hello.world.hell
00000010: 6f20 6167 6169 6e0a o again.
$ grep -c hello withnul.txt
2
$ grep -a hello withnul.txt | cat -v
hello^@world
hello again
grep found both lines — -c proves it — and without -a it will not show
you either one. What it prints instead is a notice, and the wording and
even the STREAM of that notice differ between the two greps; the page
has both. -a (--binary-files=text) is the flag that says 'show me anyway'.
One NUL is enough. High bytes alone are not: a UTF-8 file of accented
letters is still text to grep.
5. UTF-16: THE WORD IS RIGHT THERE AND GREP CANNOT SEE IT
$ xxd u16.txt
00000000: fffe 6300 6100 6600 e900 0a00 ..c.a.f.....
$ grep -a caf u16.txt # -a, so this is not the binary rule
(no match — exit 1)
'caf' is in that dump: 63 00 61 00 66 00. It is not in the FILE as the
three bytes 63 61 66, and grep searches bytes, so there is no match and
no error. A clean exit 1 that means 'not present' when the truth is
'present, spelled differently'.
6. THE FIX IS TO DECODE FIRST, NOT TO SEARCH HARDER
$ iconv -f UTF-16LE -t UTF-8 u16.txt | grep -a caf
café
$ iconv -f UTF-16LE -t UTF-8 u16.txt | grep -a caf | xxd -p
efbbbf636166c3a90a
It matches. Name the encoding, convert, then search — grep is a byte
tool, so hand it bytes it can answer about.
The hex shows what the text line hides: the first three bytes are
ef bb bf, the byte-order mark, converted along with everything else and
now sitting invisibly at the front of your search result. Naming
UTF-16LE told iconv the order, so it treated the mark as content.
In Python¶
The shell above runs in the C locale, so . means one byte there. Python has no such setting: a str pattern searches characters and a bytes pattern searches bytes, and you pick by which kind of object you pass. Reading the two blocks side by side is the point — the answers are the same numbers, and only one of the two languages made you choose on purpose.
Verified output of grep_on_non_ascii_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. THE DOT: THE CHOICE IS A TYPE, NOT A LOCALE
len('café') 4
len(b'caf\xc3\xa9') 5
len(re.findall('.', 'café')) 4
len(re.findall(b'.', ...utf-8)) 5
Four and five, from one file, and nothing in between decided it for
you: the str pattern searched characters, the bytes pattern searched
bytes. grep makes the same choice from the locale, silently.
2. FINDING THE NON-ASCII LINES
line 1: ascii only plain ascii
line 2: NON-ASCII café here
line 3: ascii only more ascii
The class is the same idea as the shell's [^ -~], written the way a
language with escapes lets you write it. grep's BSD build has no -P,
so it cannot take \x escapes at all — hence the printable-range trick.
3. A NUL IS JUST A CHARACTER
b'...' contains a NUL True
lines matching 'hello' 2
Python refused to show them? False
grep calls this file binary and prints a notice instead of the lines.
Python has no such rule — the NUL is character U+0000 and the search
works. A rule that protects a terminal from control bytes is not a
rule about what the text is.
4. UTF-16 — THE SEARCH THAT NEEDS A DECODE, NOT A BETTER PATTERN
the bytes fffe630061006600e9000a00
b'caf' in the bytes False
'caf' in decoded utf-16 True
The first answer is False and the second is True, for one file. This is
exactly grep's failure on the same file, and the fix is the same: name
the encoding and decode, then search. Note that decoding as 'utf-16'
with no LE/BE suffix consumed the mark; 'utf-16-le' would have handed
it back as a leading U+FEFF, which is the shell example's last section.
5. THE BYTES THAT ARE NOT VALID TEXT
strict decode UnicodeDecodeError at byte 14
strict decode raised True
surrogateescape: lines kept 3
...lines containing 'line' 3
re-encodes to the same bytes True
Three lines in, three lines out, and the file round-trips byte for
byte. Python's three answers to invalid input are: raise (strict),
replace (lose the bytes), or surrogateescape (keep them). One of the
two greps has a fourth answer — skip the line and say nothing — and
the page names which.
The part that is not the same on both machines¶
Two of grep's behaviours differ between BSD grep, which ships with macOS, and GNU grep, which ships with Ubuntu — both of them entries in the list of platform splits this library records — so no answer key can cover them and nothing above tries to. Both were measured on one file, on the two machines named in the caption.
1. On invalid UTF-8, BSD grep drops the line¶
This is the one to remember. The file has three lines, and all three contain the word line. The middle one also contains two bytes that are not valid UTF-8.
$ printf 'good line\nbad \377\376 line\nlast line\n' > invalid.txt
# macOS, BSD grep
$ LC_ALL=C grep -an line invalid.txt $ LC_ALL=en_US.UTF-8 grep -an line invalid.txt
1:good line 1:good line
2:bad ?? line 3:last line
3:last line
# Ubuntu, GNU grep 3.11
$ LC_ALL=C grep -an line invalid.txt $ LC_ALL=en_US.UTF-8 grep -an line invalid.txt
1:good line 1:good line
2:bad ?? line 2:bad ?? line
3:last line 3:last line
Four runs, one file. Three of them find three lines. The fourth — BSD grep in a UTF-8 locale — finds two, and reports nothing at all about the third: no warning, no diagnostic, exit status 0. The line did not fail to match; it failed to be considered, because grep could not decode it into characters and moved on.
All four of those runs used -a, and the default is not the same picture. Re-measured on 2026-09-10 on the same two builds and the same file, without -a, GNU grep in a UTF-8 locale also leaves line 2 out — but it counts it, and it tells you:
LC_ALL=en_US.UTF-8 lines printed on stderr grep -c
BSD grep line 1 and 3 nothing 2
BSD grep -a line 1 and 3 nothing 2
GNU grep line 1 and 3 grep: invalid.txt: binary file matches 3
GNU grep -a line 1, 2 and 3 nothing 3
So the split is not that one grep hides the line — by default both do — but that GNU found it, withheld it as binary and said so, while BSD never counted it and says nothing, with or without -a. And what BSD loses depends on the pattern as well as the line: grep bad, whose match is line 2's first three characters, finds it, while grep line, grep ' ' and grep 'bad.*line' do not. Binary is a verdict, not a property has the same file beside the other readers that call bytes binary.
That is the worst failure shape in this whole library, and it is worth naming why: iconv refuses and file guesses, and both of those are answers you can act on. A silent skip is not an answer at all.
It is also unique to grep. Handed the identical file in the identical locale, BSD sed stops and exits 1, BSD awk stops and names the input record, and cut -c stops and exits 74 — every one of them a failure a script can catch. grep alone loses a line, says nothing and reports success. The chapter's family table puts all four side by side. If you are grepping a file whose encoding you do not know — an export, a log, anything that crossed a system boundary — run the search in LC_ALL=C, where grep is a pure byte matcher and every line is considered:
You lose Unicode-aware character classes, which you almost never needed, and you gain the guarantee that no line was skipped.
2. The "binary file" notice has two wordings, on two different streams¶
# macOS, BSD grep — the notice goes to STDOUT
$ grep hello withnul.txt
Binary file withnul.txt matches
# Ubuntu, GNU grep 3.11 — the notice goes to STDERR
$ grep hello withnul.txt
grep: withnul.txt: binary file matches
The stream is the part that matters, because it decides what happens when you redirect:
| macOS (BSD) | Ubuntu (GNU 3.11) | |
|---|---|---|
grep p f on a terminal |
a notice | a notice |
grep p f > out.txt |
the notice is in out.txt |
out.txt is empty, no visible complaint |
| exit status | 0 — it matched | 0 — it matched |
So the same pipeline gives you a junk line on one machine and an empty file on the other, and reports success on both. grep -a (long form --binary-files=text) is the fix and behaves identically on both; -I goes the other way and skips such files entirely.
Finding the non-ASCII lines: three spellings, one portable¶
This is the search worth keeping in your fingers — "show me every line in this file that is not plain ASCII" — and the obvious spelling is the one that does not travel:
| Spelling | Works on | Note |
|---|---|---|
grep -n '[^ -~]' f |
both | any byte outside space–tilde; also flags tabs, which is usually welcome |
grep -nP '[^\x00-\x7F]' f |
GNU only | BSD grep has no -P at all — it exits with a usage error |
grep -n "$(printf '[\200-\377]')" f |
both | only the high bytes; misses control characters |
The first is the one to use. Run it under LC_ALL=C and it is a byte test, which is what you wanted.
If you are coming from Python or ABAP¶
Python: the choice grep takes from the locale, Python takes from the type — re on a str matches characters, re on a bytes matches bytes, and mixing them raises TypeError rather than guessing. The habit that transfers is the opposite direction: where you would reach for LC_ALL=C grep, in Python you open(path, 'rb') and search bytes, or open(path, errors='surrogateescape') and keep the undecodable bytes alive through the search — the option the shell has no equivalent of.
ABAP (Not machine-checked — CI cannot run ABAP.) FIND and REPLACE on a string work in characters, always, because ABAP's string is already decoded — there is no locale to consult and no byte mode to fall into. The analogue of LC_ALL=C grep is doing the search on an xstring with FIND ... IN BYTE MODE, and the analogue of grep's silent skip does not exist: an undecodable byte fails at cl_abap_conv_in_ce, at the boundary, before any search. The trap runs the other way — an ABAP developer expects the character answer everywhere and gets the byte answer the moment the data is on a file or an RFC.
Try it¶
- Take any file on your machine and run
grep -c ''on it, thenwc -l. When do they disagree? (The trailing newline is why.) - Run
grep -n '[^ -~]' *.csvin a folder of exports. Every hit is a place where the encoding matters and you did not know it. - Reproduce the silent skip:
printf 'good\nbad \377\376 keep\nlast\n' > f, then rungrep -c keep funderLC_ALL=Cand under a UTF-8 locale. If you are on Linux you will get 1 both times; borrow a Mac to see the 0. - Now do the same search with
rgand see which of the two answers it gives.
Practice¶
What does . match? On a file containing café (5 bytes, 4 characters), predict the result of grep -c '^caf.$' and grep -c '^caf..$' — first in the C locale, then in a UTF-8 one. Two of those four answers are 1 and two are 0; say which.
Then: which of grep, rg, find and Python asks the locale what a character is, and which ask nobody? Four tools in one pipeline can hold four different definitions at the same moment — say why none of them is misconfigured.
Answers
Verified output of grep_kata_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
THE FILE
café 636166c3a90a (6 bytes, 4 characters)
1. HOW MANY CHARACTERS DOES caf. MATCH?
grep -c "^caf.$" 0
grep -c "^caf..$" 1
In the C locale a character IS a byte, so the four-character word
needs FIVE dots' worth of pattern -- and caf. does not reach the end
of the line. In a UTF-8 locale the first pattern matches and the
second does not. Same file, same grep, opposite answers, and the
only thing that changed was an environment variable.
2. WHICH IS RIGHT?
Both. The question 'does caf. match café' has no answer until
somebody says what a character is, and grep asks the locale rather
than the file. The file has no opinion: it is five bytes either way.
3. THE OTHER TOOLS ANSWER THE SAME QUESTION DIFFERENTLY
grep asks LC_CTYPE
rg never asks the locale; it reads the file's first bytes for a
BOM and otherwise works in bytes
find asks nobody and compares bytes
python asks nothing since 3.7 -- str is code points, always
So four tools in one pipeline can hold four different definitions of
'a character' at the same moment, and none of them is misconfigured.
4. THE ESCAPE, AND ITS PRICE
LC_ALL=C grep -c caf 1 -- a literal string is unaffected
For a FIXED string the locale changes nothing, which is why LC_ALL=C
is a safe default for searching: no interpretation, no undecodable
input, the same answer on every machine. It only bites when the
pattern contains ., [[:alpha:]], a range, or a case-insensitive flag
-- everything that needs to know how wide a character is.
5. AND THE FAILURE SHAPE WORTH REMEMBERING
Of the tools in this chapter, grep is the one that can find FEWER
lines than are there and still exit 0 -- a short answer with a
success status. sed reports an illegal byte sequence, awk names the
record, cut refuses. Only grep can say nothing at all.
See also¶
ripgrep— the Rust grep — one implementation, no locale, and it reads UTF-16trandsortwork a byte at a time — the same locale question, with no flag to escape it- Validation is a boundary — where the "is this valid?" check belongs, and what each tool does when it fails
- Locale and
LC_CTYPE— the setting all of this hangs on - Byte order and the BOM — why the converted line has three extra bytes at the front