`sed` matches patterns, not bytes¶

Level: 201 · for anyone repairing a text file with a one-liner

One line: sed 's/é//' removes the é and leaves naïve alone, where tr -d 'é' damages both words — because sed was given a two-byte sequence to match and tr was given a set of bytes to delete, and that difference survives even in the C locale where neither knows what a character is.

The distinction that makes `sed` the right tool¶

tr takes a set: every byte in it goes, wherever it is found. sed takes a pattern: a sequence, matched in order, as a unit. For ASCII the two look the same, because every character is one byte and a set of one byte is a sequence of one byte. Above U+007F they part company completely, and only one of them is still doing what you asked.

That is why the rule on the previous page — tr is for ASCII — has a second half: for anything above U+007F, the tool is sed.

In the terminal¶

Verified output of sed_matches_patterns_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

1. THE ONE THING tr CANNOT DO
$ cat two.txt
café naïve
$ xxd -p two.txt
636166c3a9206e61c3af76650a
$ tr -d "é" < two.txt | xxd -p          # the byte-set tool
636166206e61af76650a
$ sed "s/é//" < two.txt | xxd -p        # the pattern tool
636166206e61c3af76650a
   Same request, two results. tr was given the byte SET {c3, a9} and
   deleted both bytes wherever it met them, so it took the c3 out of the
   middle of ï and left an orphan af. sed was given the two-byte SEQUENCE
   c3 a9 and matched it as a unit, so naïve is untouched and both words
   are still valid UTF-8. That is the whole reason to reach for sed.
   Note this works even here, in the C locale, where sed has no idea what
   a character is: a pattern is a sequence of bytes and so is é.

2. BUT '.' IS STILL A BYTE IN THIS LOCALE
$ sed "s/./X/g" cafe.txt
XXXXX
   Four characters on screen, five X. The dot means one byte here. In a
   UTF-8 locale the same command prints four — both seds agree about that,
   so this one is the locale talking, not the implementation.

3. y/// IS tr INSIDE sed, AND IT INHERITS tr'S PROBLEM
$ sed "y/é/e/" cafe.txt
   (refused — exit 1)
   y transliterates character by character, which in this locale means
   byte by byte: 'é' is two bytes and 'e' is one, so the two sides are
   different lengths and sed will not guess. Both seds refuse; they word
   the complaint differently, which is why only the exit status is shown.
   The refusal is the good outcome — compare tr, which just did it.

4. THE CRLF REPAIR, WHICH IS THE COMMONEST REAL USE
$ cat -vet crlf.txt
dos^M$
unix$
$ sed "s/\r$//" crlf.txt | cat -vet
dos$
unix$
   ^M$ is a CRLF line and $ alone is an LF line. The pattern is anchored
   to the end on purpose: an unanchored s/\r// would also delete a CR
   sitting legitimately inside a quoted CSV field.

5. sed DOES NOT ADD A TRAILING NEWLINE
$ xxd -p nonl.txt
6e6f20747261696c696e67206e65776c696e65
$ sed "s/trailing/final/" nonl.txt | xxd -p
6e6f2066696e616c206e65776c696e65
   Neither dump ends in 0a. sed passed the missing newline through rather
   than tidying it up, and both seds agree. That matters because a lot of
   tools do not — and a file that gains a byte in a pipeline is a file
   whose checksum has changed.

6. THE TWO SPELLINGS OF EDIT-IN-PLACE
   BSD sed  : sed -i '' 's/x/y/' file      # the '' is a required argument
   GNU sed  : sed -i    's/x/y/' file      # and this FAILS on BSD
   portable : sed 's/x/y/' file > tmp && mv tmp file
   -i is the one sed flag with no portable spelling. On BSD the argument
   is the backup suffix and it is mandatory; on GNU it is optional and
   attached (-i.bak). A script that uses bare -i is a script that works
   on Linux and silently writes a file called 's/x/y/' on a Mac.

What the locale changes, and what it does not¶

Two things on this page depend on the locale and both seds agree about both, which is worth saying plainly because most of this chapter is about the two disagreeing:

	C locale	UTF-8 locale
`s/./X/g` on `café`	`XXXXX` — five bytes	`XXXX` — four characters
`s/é//` on `café naïve`	correct	correct
`y/é/e/`	refused — different lengths	works

The middle row is the important one. Matching a literal non-ASCII string works in every locale, because the pattern and the text are the same bytes either way. You do not need a UTF-8 locale to search and replace accented text with sed; you need one only when the pattern itself has to reason about characters — ., a range, a quantifier on a multi-byte character.

y/// is the exception, and it is instructive: y is tr living inside sed, character by character. In the C locale that means byte by byte, so é (two bytes) and e (one) are different lengths and sed refuses. It refuses rather than corrupting, which is the whole difference between it and tr — but the fix is not a flag, it is s///.

The part that is not the same on both machines¶

sed inherits the same multibyte machinery grep does, so on BSD it fails on undecodable bytes in a UTF-8 locale. It fails differently, and better:

Measured 2026-09-06 — macOS 26.6 (BSD) and ubuntu:24.04 (GNU sed 4.9), file = three lines, all containing 'line', the middle one holding the invalid bytes ff fe. Not machine-checked: no key can match both.

                             lines out   exit   said
macOS, LC_ALL=en_US.UTF-8       1 of 3     1    sed: RE error: illegal byte sequence
macOS, LC_ALL=C                 3 of 3     0    —
Ubuntu, either locale           3 of 3     0    —

sed stops at the bad line and tells you, with a non-zero exit. That is a failure you can catch in a script. Compare grep on the identical file, which finds two of the three lines, says nothing, and exits 0 — the reason that page calls it the worst failure shape in the library. Same operating system, same locale, same undecodable byte; one tool raises and one lies. The chapter's family table puts all four side by side.

The same defence works for both: LC_ALL=C, where sed is a byte matcher, considers every line, and cannot fail this way.

`-i` is the one flag with no portable spelling¶

sed -i ''  's/x/y/' file    # BSD / macOS — the '' is a REQUIRED argument
sed -i     's/x/y/' file    # GNU / Linux — and this fails on BSD
sed 's/x/y/' file > tmp && mv tmp file    # works everywhere

On BSD the argument to -i is the backup suffix and it is mandatory; on GNU it is optional and must be attached (-i.bak). So sed -i 's/x/y/' file on a Mac reads s/x/y/ as the backup suffix and file as the script — which fails, but the failure mode of the reverse mistake is worse: sed -i '' … on GNU treats '' as the script and edits nothing while reporting success. There is no spelling that does the right thing on both. Write the redirect, or test for the platform once at the top of the script.

If you are coming from Python or ABAP¶

Python: re.sub(r'é', '', text) is s/é// and has no byte/character ambiguity — the pattern and the subject are both str, so the match is over characters, always. The tr/sed distinction does not exist because str.translate takes a mapping keyed by code point, not a set of bytes, so it cannot cut a character in half. What does transfer is the -i lesson in a different form: re.sub returns a new string and never edits a file, so the read–modify–write is yours to write, and yours to get the encoding right on both ends.

ABAP (Not machine-checked — CI cannot run ABAP.) REPLACE ALL OCCURRENCES OF 'é' IN text WITH '' is the s/// behaviour, over characters, with no locale involved — ABAP's string is already decoded. The y/// trap has a direct analogue in TRANSLATE text USING '…', which pairs characters positionally and is fine on a string for exactly the reason sed's y is not fine in the C locale: ABAP is counting characters and the C locale is counting bytes. The place to be careful is REPLACE ... IN BYTE MODE on an xstring, which is tr's world and has tr's hazards.

Try it¶

printf 'café naïve\n' | tr -d 'é' | iconv -f UTF-8 -t UTF-8; echo $? then the same with sed 's/é//'. The exit status is the difference.
sed 's/./X/g' on an accented file under LC_ALL=C and under your own locale. Count the X.
Try sed -i 's/a/b/' f on a Mac and read the error. Then look for -i in your own scripts.
Run sed -n '/x/p' over a file with a broken byte, under both locales, and check $? each time.

Practice¶

Why does sed get right what tr gets wrong? Predict sed 's/é/e/' and tr 'é' 'e' on café naïve. Then run the sed again with LC_ALL=C and explain why the answer does not change — and then predict LC_ALL=C sed 's/f./X/' on the same file, as bytes, where it very much does.

Finish with the portability question: write the in-place edit that works on both GNU and BSD sed. There is no -i spelling that does.

Answers

Verified output of sed_kata_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

1. THE EDIT THAT WORKS
   sed 's/é/e/'  -> cafe naïve
   tr 'é' 'e'    -> cafee nae\xafve
   sed matched a two-byte sequence and replaced it with one byte. tr was
   given two source bytes and one replacement byte, so it maps BOTH c3
   and a9 to 'e' -- and applies that to ï as well.

2. WHY sed IS RIGHT WITHOUT KNOWING ANYTHING
   in the C locale: cafe naïve
   Identical. sed did not consult the locale to get this right; the
   pattern is a byte sequence and the file contains that byte sequence.
   Where the locale WOULD matter is a pattern like . or [[:alpha:]],
   which have to know how wide a character is.
   LC_ALL=C sed "s/f./X/" -> 636158a9206e61c3af76650a
   In the C locale . is ONE BYTE, so f. matched f and the first half of
   é -- and the a9 left behind is now an orphan, which is why the output
   above is hex rather than text: it is not valid UTF-8 any more. LC_ALL=C
   is the flag to reach for when you want no interpretation, and the trap
   when you wanted some.

3. THE -i SPLIT, WHICH IS A REAL PORTABILITY PROBLEM
   GNU sed:  sed -i    's/a/b/' f        (no argument)
   BSD sed:  sed -i '' 's/a/b/' f        (empty argument required)
   Neither accepts the other's spelling, so there is NO -i form that runs
   on both. The portable answer is not a clever quoting trick:
       sed 's/a/b/' f > tmp && mv tmp f
   which is also the only form that cannot half-write the file if the
   command fails.
   after the portable form: cafe naïve

4. AND THE ONE THING BOTH seds AGREE ON
   s/x/y/ with no match  exit=0
   Zero. sed does not report whether it substituted anything, so a script
   cannot tell 'edited' from 'found nothing to edit' by status alone.
   That is what /q and grep -q are for, run first.

sed matches patterns, not bytes¶

The distinction that makes sed the right tool¶