Skip to content

tr and sort work a byte at a time

Level: 201 · for anyone cleaning up a text file with a pipeline

One line: tr -d 'é' on café naïve deletes a byte out of the middle of naïve too, because tr was never given a character — it was given the byte set {c3, a9} — and sort, uniq and cut -c have the same shape of problem with no flag to escape it.

The four tools and the one question

These are the tools you reach for after grep has found the lines: delete something, order them, count the distinct ones, slice a column. All four are byte-oriented at heart, and each has a different amount of honesty about it:

Tool What it says it does What it does
tr translate or delete characters bytes, always — no locale, no flag, no exception
sort sort lines byte order in the C locale, collation order in any other
uniq drop duplicate lines compares bytes, so two spellings of one word are two values
cut -c cut by character bytes, unless the locale is a UTF-8 one and the platform is BSD — its own page has the measurement; cut -b is the honest spelling

tr is the one that damages data, so it goes first.

In the terminal

Verified output of tr_and_sort_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

1. tr -d DELETES BYTES, AND THE BYTES ARE SHARED
$ cat two_words.txt
café naïve
$ xxd -p two_words.txt
636166c3a9206e61c3af76650a
$ tr -d "é" < two_words.txt
caf na\xafve
$ tr -d "é" < two_words.txt | xxd -p
636166206e61af76650a
   You asked it to delete é from café. It also damaged naïve, which
   contains no é at all. The hex says why: é is c3 a9 and ï is c3 af, and
   tr does not take a character — it takes a SET OF BYTES, here {c3, a9}.
   In café both bytes were in the set, so the letter went. In naïve only
   the c3 was, so half of ï went and the orphaned af stayed. One word is
   short a letter; the other is no longer valid UTF-8 at all.

2. WHICH IS NOT A BUG, IT IS THE DOCUMENTED CONTRACT
$ printf "abc" | tr "abc" "xyz"
xyz
   tr maps a set of single-byte values to another set of single-byte
   values. There is no multi-byte character in that description anywhere.
   For anything above U+007F the tool you want is sed, which matches a
   whole pattern, not a byte set:
$ sed "s/é//" < two_words.txt | xxd -p
636166206e61c3af76650a
   café lost its é; naïve is untouched, and both are still valid UTF-8.

3. sort IN THE C LOCALE IS BYTE ORDER, NOT ALPHABETICAL ORDER
$ cat words.txt
zebra
żaba
aaa
éclair
Echo
$ sort words.txt
Echo
aaa
zebra
éclair
żaba
   Echo before aaa, because 'E' is 0x45 and 'a' is 0x61 — every capital
   sorts before every lowercase. And éclair and żaba are at the bottom,
   after every ASCII word, because their first byte is above 0x7f. This
   is a correct byte ordering and nobody's idea of alphabetical.

4. uniq COMPARES BYTES, SO TWO SPELLINGS ARE TWO VALUES
$ cat turtles.txt      # the same word twice, NFC then NFD
żółw
żółw
$ sort turtles.txt | uniq -c | sed "s/^ *//"
1 żółw
1 żółw
   (the leading spaces are stripped — BSD and GNU uniq pad the count to
    different widths, which is the kind of thing this library records)
   Two groups of one. On the screen they are the same word; uniq is
   comparing seven bytes against nine. Normalize before you deduplicate,
   or the count is fiction.

5. cut -c AND cut -b, AND WHY THE HONEST ONE IS -b
$ cut -c1-4 < two_words.txt | xxd -p
636166c30a
$ cut -b1-4 < two_words.txt | xxd -p
636166c30a
   Identical here, because in the C locale a character IS a byte, and both
   cut through the middle of é: 63 61 66 c3 is three letters and half of a
   fourth. -b promises bytes and keeps the promise everywhere; -c promises
   characters and delivers bytes whenever the locale is not a UTF-8 one.
   For fixed-width records, -b is the one that means what you meant.

6. tr AND CASE: [:upper:] IS AN ASCII PROMISE HERE
$ cat shout.txt
ŻÓŁW
$ tr "[:upper:]" "[:lower:]" < shout.txt
ŻÓŁw
   ŻÓŁw. Exactly one letter changed — the ASCII W — because in the C
   locale [:upper:] is A-Z and none of Ż, Ó, Ł is in it. That is the worst
   possible outcome: not a refusal, not a conversion, a word that is now
   half lowercased and looks like a typo rather than an encoding bug.
   In a UTF-8 locale the two machines this library runs on give two
   different answers to this same command — the page has both.

Why tr is the dangerous one

Every other tool on this page gives you a wrong answer. tr gives you a wrong file, and it does it quietly:

café naïve      63 61 66 c3 a9 20 6e 61 c3 af 76 65
tr -d 'é'   ->  63 61 66       20 6e 61    af 76 65
caf na?ve

Read the two hex rows. tr deleted c3 and a9 wherever it found them. In café that was the whole letter. In naïve the c3 belonged to ï, so what is left is a lone af — a byte that starts nothing, in a file that no longer decodes. If that file is now written back to disk you have created the kind of damage the rest of this library is about diagnosing, in one command, with no error and exit status 0.

The rule is simple and absolute: tr is for ASCII. Deleting or replacing anything above U+007F is sed, which matches a whole pattern, or Python, or iconv if the job is actually a re-encoding. tr -d '\r' is fine. tr -d 'é' is not.

tr -cd '\11\12\40-\176' — keep tab, newline and printable ASCII, delete everything else — is the one non-ASCII use that is honest, because it is stated in bytes and deletes whole multi-byte characters only by deleting all of their bytes.

The locale, and where the two machines part company

Everything above is the C locale, where all four tools are byte tools and both platforms agree. Change the locale and one of them changes behaviour on only one of the machines:

Measured 2026-09-06 — macOS 26.6 (BSD) and ubuntu:24.04 (GNU coreutils 9.4), same file, same commands. Not machine-checked: no key can match both.
$ printf '\305\273\303\223\305\201W\n' > shout.txt        # ŻÓŁW

# both machines, C locale
$ LC_ALL=C tr '[:upper:]' '[:lower:]' < shout.txt
ŻÓŁw                       # only the ASCII W changed

# macOS, UTF-8 locale                  # Ubuntu, UTF-8 locale
$ LC_ALL=en_US.UTF-8 tr … < shout.txt  $ LC_ALL=en_US.UTF-8 tr … < shout.txt
żółw                                   ŻÓŁw

BSD tr case-folds the Polish letters in a UTF-8 locale; GNU tr does not, and its documentation is explicit that it handles single-byte characters only. So a pipeline that lowercases names works on your Mac and silently half-works in production. Neither is wrong about its own contract; there simply is no portable tr answer for non-ASCII case, and the fix is not a flag — it is a different tool.

sort is the happier case: the two machines agree, and the locale changes the answer on both in the same way.

Measured 2026-09-06 — macOS 26.6 and ubuntu:24.04, byte-identical output on both.
$ cat words.txt        $ LC_ALL=C sort         $ LC_ALL=en_US.UTF-8 sort
zebra                  Echo                    aaa
żaba                   aaa                     Echo
aaa                    zebra                   éclair
éclair                 éclair                  żaba
Echo                   żaba                    zebra

The middle column is byte order: every capital before every lowercase, and everything non-ASCII dumped at the end. The right column is a collation: case-insensitive at the first level, accents treated as variants of their base letter, so éclair files under E and żaba under Z, next to zebra. That is what a human means by alphabetical, and it is only available in a locale.

Which means the practical advice splits:

  • Sorting for a human to read — a report, a list of names — needs a UTF-8 locale. LC_ALL=C sort will put Zoë before alice and file every accented name in a heap at the bottom.
  • Sorting for a machine — comm, join, sort -u, a checksum over sorted lines, anything two programs must agree on — needs LC_ALL=C, because byte order is the only order every machine, version and locale computes identically. A sort that depends on the environment is a diff that changes on someone else's laptop.

The habits

  1. tr for ASCII only. tr -d '\r', tr 'a-z' 'A-Z' on identifiers, and nothing above U+007F.
  2. Pin the locale on sort, in both directions. LC_ALL=C sort when two programs must agree; a UTF-8 locale when a person will read it. Never leave it to whatever the shell happened to inherit.
  3. Normalize before uniq or sort -u. Otherwise the count is over spellings, not words. The find page is the same bug wearing a filename.
  4. cut -b, not cut -c, for fixed-width records — and remember that a byte offset can land inside a character. cut has its own page now, because the same command returns four bytes on Ubuntu and five on a Mac; the interface version is the fixed-width field problem.
  5. When a pipeline mangles text, xxd each stage. The stage where the hex stops being valid UTF-8 is the culprit, and it is usually a tr.

If you are coming from Python or ABAP

Python: str.translate and str.replace work on characters and cannot produce the tr damage above — the closest equivalent is bytes.translate, and you have to reach for bytes on purpose. sorted() is the C-locale case: it orders by code point, always, with no locale involved, which is the deterministic behaviour you want for machines. For the human ordering there is no batteries-included answer — locale.strxfrm exists and is process-global and awkward, and the real answer is PyICU. Being told "there is no correct sort in the standard library" is itself the lesson: collation is a data problem, not an algorithm problem.

ABAP (Not machine-checked — CI cannot run ABAP.) TRANSLATE … TO UPPER CASE on a string is character-based and handles the Polish letters, so the tr trap does not exist — but its xstring sibling, and any SORT on an internal table, take you straight back to the same question: SORT itab BY field uses the code-page binary order, which is byte order, and SORT … AS TEXT uses the locale collation. Those are exactly the C-locale and UTF-8-locale columns above, spelled as one keyword, and choosing between them has the same rule — AS TEXT for a report a human reads, plain SORT for anything two systems must agree on.

Try it

  1. printf 'café naïve\n' | tr -d 'é' | iconv -f UTF-8 -t UTF-8. The exit status is the damage report.
  2. Sort a list of names with LC_ALL=C and then with your own locale. Count how many move.
  3. sort -u a file containing the same word in NFC and NFD. Then normalize it first and do it again.
  4. Take a fixed-width export and cut -b1-20 a line with an accented character near byte 20. Then xxd the result and find the half-character.

Practice

Delete one letter, damage two words. Predict the output of tr -d 'é' on café naïve — as bytes — then the output of sed 's/é//' on the same file. One of them damages naïve; say which and exactly why.

Then generalise it: sort sort, cut -b and sed into tools that take a set of bytes and tools that take a sequence, and say which of the two can ever see a multi-byte character.

Answers

Verified output of tr_and_sort_kata_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

THE FILE
   text  café naïve
   bytes 636166c3a9206e61c3af76650a
   é is c3a9 and ï is c3af

1. tr -d 'é'
   result caf na\xafve      bytes 636166206e61af7665
   naïve lost a byte too. tr was never handed a character: it was handed
   the byte SET {c3, a9}, and it deletes every occurrence of either. ï is
   c3 af, so its c3 matched and vanished -- leaving af, which is not a
   valid UTF-8 sequence on its own. The output is no longer text.

2. sed 's/é//' ON THE SAME FILE
   result caf naïve     bytes 636166206e61c3af7665
   Correct, and it is not because sed knows about characters. sed was
   given a two-byte SEQUENCE to match -- c3 then a9, in that order -- and
   c3 af does not match it. A sequence has an order; a set does not.
   That difference survives the C locale, where neither tool knows what a
   character is.

3. THE SAME SHAPE IN sort AND uniq
   sort:  B a b é 
   In the C locale sort compares BYTES, so uppercase sorts before
   lowercase and é lands after everything -- e9 and c3 are just numbers.
   Change the locale and the order changes, because collation is the one
   thing sort does ask the environment about.

4. AND cut -c, WHICH HAS NO ESCAPE AT ALL
   cut -b1-4: 636166c30a
   Four bytes: 63 61 66 c3. The last one is half a character, so the
   output is not text -- and -b is the HONEST flag here, because it says
   bytes. -c claims characters and the two cuts disagree about whether it
   delivers them.

THE RULE
   Ask whether a tool takes a SET of bytes or a SEQUENCE. tr, cut -b and
   sort take sets or offsets and cannot see a multi-byte character. sed,
   grep and awk take patterns, which have order, and get this right even
   when they have no idea what a character is.

See also