The five worth installing¶
Level: reference · for anyone who has hit the limits of the base toolbox
One line: hexyl, uchardet, recode, dos2unix and GNU coreutils each answer one question the tools already on your machine answer badly — and each one is measured here against exactly that baseline, so you can see what you are buying.
Nothing in this library depends on any of them, and CI has none of them, so no answer key on this page comes from an optional tool. Every session below is dated and names the machine it ran on. The one machine-checked block is the baseline: what you can already do.
First, the baseline¶
Four files, and everything the tools you already have can say about them. Read this before the install list — half the reason to install something is knowing precisely where the free answer stops.
Verified output of base_toolbox_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
1. WHAT IS IN THESE FILES
utf8.txt 636166c3a92031e282ac0a
latin1.txt 636166e90a
cp1252.txt 507265697320313030800a
mixed.txt 646f73206c696e650d0a756e6978206c696e650a6c6173740d0a
2. file --mime-encoding — THE ANSWER YOU ALREADY HAVE
$ file --mime-encoding utf8.txt latin1.txt cp1252.txt
utf8.txt: utf-8
latin1.txt: iso-8859-1
cp1252.txt: unknown-8bit
Three files, three qualities of answer. utf-8 is an inference from
valid structure and is nearly always right. iso-8859-1 is a proof of a
NEGATIVE — 'not valid UTF-8' — with a plausible 8-bit table named, and
file cannot tell 8859-1 from 8859-2 or 8859-15 because on this byte
they agree. unknown-8bit is the honest surrender: 0x80 is unassigned in
every ISO 8859 table, so file will not name one. uchardet names it.
3. xxd's TEXT COLUMN — ONE GLYPH FOR EVERY PROBLEM
$ printf "caf\\303\\251 1\\342\\202\\254\\n\\000A" | xxd
00000000: 6361 66c3 a920 31e2 82ac 0a00 41 caf.. 1.....A
Look at the right-hand column: caf.. 1.....A. Five dots, standing in
for two continuation bytes of é, three of €, a newline and a NUL. They
are four completely different kinds of byte and xxd draws them all the
same. That is what hexyl fixes — see the page.
4. iconv AS A YES/NO VALIDATOR — THE ONE PORTABLE USE
utf8.txt valid UTF-8
latin1.txt NOT valid UTF-8
cp1252.txt NOT valid UTF-8
That is the whole of what the base toolbox can prove. Which table the
invalid two are in is not a question iconv answers — it is the question
uchardet exists for, and the answer is still a guess.
5. LINE ENDINGS WITHOUT dos2unix
$ cat -vet mixed.txt
dos line^M$
unix line$
last^M$
lines ending CRLF : 2
lines in total : 3
$ sed "s/\r$//" mixed.txt | cat -vet
dos line$
unix line$
last$
^M$ is a CRLF line and $ alone is an LF line, so cat -vet is the free
diagnosis and sed is the free repair. dos2unix -i gives you the same
counts in one line, and dos2unix does the repair without a regex you
have to get right — which matters on the day the file is UTF-16.
1. hexyl — a hex dump with categories¶
xxd's text column draws one glyph, ., for a continuation byte, a newline, a NUL and a control character alike. hexyl gives each category its own glyph and its own colour, so the shape of a file is visible instead of counted.
$ printf 'caf\303\251 1\342\202\254\n\000A' | hexyl --color never
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
│00000000│ 63 61 66 c3 a9 20 31 e2 ┊ 82 ac 0a 00 41 │caf×× 1×┊××_⋄A │
└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘
$ printf 'caf\303\251 1\342\202\254\n\000A' | xxd
00000000: 6361 66c3 a920 31e2 82ac 0a00 41 caf.. 1.....A
Compare the two text columns. caf.. 1.....A against caf×× 1×┊××_⋄A: where xxd draws one ., hexyl draws a glyph per category and paints it.
byte glyph colour
41 'A' A cyan printable ASCII, drawn as itself
00 NUL ⋄ dim
09 TAB _ green
0a LF _ green
20 SPACE (space) green
1b ESC • green
7f DEL • green
c3 a9 × yellow any byte above 127
Five glyphs, four colours — and note that they do not line up one to one: whitespace and the other ASCII controls share green and are told apart only by _ against •. That is still four more distinctions than xxd offers, and on a real terminal the colour is the part you read without looking.
Use it for looking; keep xxd for working, because xxd -p and xxd -r -p are a round trip and hexyl has no reverse gear.
2. uchardet — a detector, where file only has a validator¶
This is the one that earns its place on a bad day. file can prove a file is not valid UTF-8; naming which 8-bit table it actually is, is a different and harder job, and file mostly declines it.
file --mime-encoding uchardet
utf8.txt café 1€ utf-8 UTF-8
latin1.txt café iso-8859-1 ISO-8859-2
cp1252.txt Preis…€ unknown-8bit WINDOWS-1252
The third row is why you install it. 0x80 is unassigned in every ISO 8859 table — it is in the C1 control range — so file correctly refuses to name one and tells you nothing you can act on. uchardet says WINDOWS-1252, which is right, and is the single most common answer for a file that came off a Windows desktop in Europe.
The second row is why you do not trust it. uchardet says ISO-8859-2 where file said ISO-8859-1, and neither of them is checkable: the file's only non-ASCII byte is e9, which is é in both tables. There is no evidence in that file to decide, so a detector's job there is to pick the likeliest table given the language it thinks it sees — a guess, better informed than file's and still a guess. It becomes wrong on the first character where the two tables differ, which for Polish or Czech text is soon.
So: uchardet narrows the field; it does not settle it. The only thing that settles it is asking whoever wrote the file.
3. recode — the converter that refuses by default¶
iconv is on every machine and does the same job. recode is worth having for two things: a surface syntax you can actually type, and a safer default.
$ recode utf8..latin1 file.txt
recode: file.txt failed: Untranslatable input in step `ISO-10646-UCS-2..ISO-8859-1'
$ xxd -p file.txt
636166c3a92031e282ac0a # unchanged — it refused and did not touch the file
$ recode -f utf8..latin1 file.txt # -f: force
$ xxd -p file.txt
636166e920310a # café 1 — the € is simply gone
The refusal is the feature. Latin-1 has no euro sign, so there is no honest conversion, and recode says so and stops with the file intact. -f converts anyway and silently deletes the character it could not carry — one byte of output where three went in, no warning, exit 0.
That matters because iconv's behaviour here is one of this library's platform splits: GNU iconv refuses like recode does, and macOS/BSD iconv silently transliterates. So on a Mac, recode is the tool that behaves the way you assumed iconv did.
recode -l lists the tables, and there are far more of them than iconv -l has for the same job, including the surface encodings — recode ../b64, recode ../qp — which is occasionally exactly what a mail header needs.
4. dos2unix — the line-ending kit, with a report mode¶
Everything dos2unix converts, sed 's/\r$//' also converts, as the baseline block shows. What you are buying is -i, which changes nothing and tells you what you have:
$ dos2unix -i mixed.txt bom.txt
2 1 0 no_bom text mixed.txt
1 0 0 UTF-8 text bom.txt
The columns are DOS lines, Unix lines, Mac (bare CR) lines, BOM, file type, name. Three of those you would otherwise get from three separate commands, and the BOM column is the one people forget to check — it is the difference between a CSV that opens correctly in Excel and one that opens with a stray character in the first header.
unix2dos goes the other way, and -ic lists only the files that would actually change, which is what you want in a loop over a directory.
The real argument for it over sed is the day the file is not what you assumed: sed 's/\r$//' on a UTF-16 file will happily edit the byte 0d wherever it appears, including inside a character. dos2unix knows about encodings and refuses.
5. coreutils — the GNU tools on a Mac¶
Not a new capability; a second opinion. Installing GNU coreutils on macOS gives you god, gwc, gtr, gsort and the rest alongside the BSD originals, which turns every platform split this library documents from something CI reports into something you can run side by side on one machine.
That is worth the disk space for one reason in particular. This repo's sharpest disagreement is the named-character row of od -a — GNU masks the high bit off and prints a letter that is nowhere in the file, BSD asks isprint() in your locale and emits the raw byte — and RESOURCES.md runs both on the same file in the same second to show it. Before coreutils, that demonstration needed two machines.
The same trick works for the other splits on these pages: gtr '[:upper:]' '[:lower:]' against BSD tr reproduces the half-lowercased Polish word without leaving your desk, and ggrep against BSD grep reproduces the silently dropped line.
What is deliberately not here¶
bat— a syntax-highlightingcat. Excellent, and genuinely nothing to do with encodings.icu4c— bringsuconv, which does normalization (uconv -x nfc) as well as transcoding, and is the right answer to the NFC/NFD comparison problem. It is left off the five because it is keg-only, needs aPATHedit, and the same job is one line of Python.piconv— Perl'siconv, and you already have it: Perl ships with macOS and Ubuntu both, so it belongs in the baseline rather than the install list. Its value is Perl's much larger set of encoding aliases, so it accepts the name your colleague's email used.
See also¶
uni— the character's name — the sixth one, which earns a page of its own- Inspecting a file — the base tools these are measured against, in depth
fileguesses — why the free detector's answers come in four shapes and only two are evidence- RESOURCES.md — the wider list, and the
od -ademonstrationcoreutilsmakes possible