iconv¶
Level: 101 → 201 · for anyone with a terminal
One line: iconv -f FROM -t TO re-encodes bytes from one table to another, and every hard part of it is a claim it cannot check — -f is your assertion about the input, //TRANSLIT is a table that ships with the implementation, and its yes/no validator accepts a whole family of sequences that Python and Rust refuse.
iconv is the terminal's decode and encode, fused into one command. It reads bytes under the table you name with -f, and writes them under the table you name with -t. That is the whole tool, and it is genuinely simple. What is not simple is that three of its four inputs are unverifiable: the source table is something you promise, the transliteration table is something the implementation promises, and the answer to "were these bytes valid?" turns out to be something iconv gets wrong in one direction on both platforms at once.
The one job¶
The demonstration is one character wide. café in ISO-8859-1 is 63 61 66 e9 — four bytes, four characters, one byte each, because that is what an 8-bit table means. In UTF-8 the same word is 63 61 66 c3 a9. Nothing about the text changed; the agreement about how to write it did, and the file grew by a byte.
That is also the shape of the failure. If you tell iconv the input is Latin-1 when it is really UTF-8, it will not object — it cannot — and each byte of c3 a9 becomes its own Latin-1 character, so é comes out as é. This is mojibake produced by a command that exited 0.
In the terminal¶
Verified output of iconv_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
1. THE ONE JOB: BYTES IN ONE TABLE, OUT IN ANOTHER
the file (ISO-8859-1) 63 61 66 e9 20 a4 20 31 30 30 0a
-t UTF-8 63 61 66 c3 a9 20 c2 a4 20 31 30 30 0a
and back again 63 61 66 e9 20 a4 20 31 30 30 0a
e9 became c3 a9 and a4 became c2 a4: same two characters, four bytes
instead of two. The text did not change; the agreement did.
Round trip identical: yes
2. THE NAME IS NOT THE TABLE — six spellings, one conversion
-f ISO-8859-1 -> 63 61 66 c3 a9 20 c2 a4 20 31 30 30 0a
-f ISO8859-1 -> 63 61 66 c3 a9 20 c2 a4 20 31 30 30 0a
-f LATIN1 -> 63 61 66 c3 a9 20 c2 a4 20 31 30 30 0a
-f latin1 -> 63 61 66 c3 a9 20 c2 a4 20 31 30 30 0a
-f CP819 -> 63 61 66 c3 a9 20 c2 a4 20 31 30 30 0a
-f L1 -> 63 61 66 c3 a9 20 c2 a4 20 31 30 30 0a
Aliases, all of them, for one table. 'iconv -l' lists what your machine
knows — and prints a different SHAPE on the two platforms, so it is on
the page and not here.
3. WHAT ICONV CANNOT DO IS DETECT
UTF-8 read as Latin-1 63 61 66 c3 83 c2 a9 20 c3 82 c2 a4 20 31 30 30 0a
exit=0. Every byte 00-FF is a character in Latin-1, so decoding UTF-8
as Latin-1 cannot fail — it can only be wrong. c3 a9 came back as
c3 83 c2 a9, which is 'é'. That is mojibake, produced silently, by a
successful command. -f is a claim YOU make; iconv never checks it.
4. THE REFUSAL, AND THE ONLY PART OF IT WORTH READING
invalid input exit=1
valid input exit=0
unknown table exit=1
The exit status is the same on both platforms; the MESSAGE is not, and
on macOS it does not describe the problem at all. Test the status.
5. THE THREE POLICIES FOR A CHARACTER THE TARGET CANNOT HOLD
the file (UTF-8) 61 e2 82 ac 62 0a
-t ASCII exit=1 out=61
-t ASCII//IGNORE exit=1 out=61 62 0a
-t ASCII//TRANSLIT out is pure ASCII: yes, and its bytes are NOT recordable
Plain stops at the character it cannot write and keeps what came before.
//IGNORE drops it — and still exits 1, on both platforms, so a script
under 'set -e' dies on the line that did exactly what it was asked.
//TRANSLIT substitutes: 'EUR' here, but WHAT it substitutes is a table
that ships with the implementation. See the page.
6. THE VALIDATOR, AND THE ONE FAMILY IT WAVES THROUGH
'iconv -f UTF-8 -t UTF-8' is the portable yes/no test for 'are these
bytes valid UTF-8'. Judged on exit status, it agrees on both platforms:
plain ASCII 61 62 63 exit=0
cafe + U+00E9 63 61 66 c3 a9 exit=0
lone high byte e9 exit=1
truncated 2-byte c3 exit=1
stray continuation 80 exit=1
surrogate U+D800 ed a0 80 exit=1
overlong slash c0 af exit=1
overlong NUL c0 80 exit=1
byte fe fe exit=1
U+10FFFF, the last f4 8f bf bf exit=0
U+110000, one past f4 90 80 80 exit=0
f5: no code point f5 90 80 80 exit=0
five-byte sequence fb bf bf bf bf exit=0
Read the last three rows. Those bytes are not valid UTF-8 by any
edition of the standard since 2003 — and iconv accepts them, on BOTH
platforms, exit 0. Python and Rust refuse all three. The validator is
right about every classic malformation and wrong about the range.
Section 3 is the one to read twice. Decoding UTF-8 as Latin-1 cannot fail, because every one of the 256 byte values is a character in Latin-1 — the table is total. So the only signal you will ever get that -f was wrong is that the text looks wrong afterwards. iconv is not a detector and has no opinion about your claim. Nothing in the terminal detects encodings; file guesses, and says so.
Section 5's exit status is the practical trap. //IGNORE did exactly what it was asked — it dropped the € and wrote the rest — and still exited 1, on both platforms. A script running under set -e therefore dies on the line that worked. If you mean "drop what does not fit and carry on", you have to write iconv -f UTF-8 -t ASCII//IGNORE in > out || true, and then you have thrown away the status that would have told you about a real failure. There is no spelling that gives you both.
Section 6 is the finding. iconv -f UTF-8 -t UTF-8 is the portable way to ask are these bytes valid UTF-8? — Validation is a boundary records it as agreeing on both platforms, exit status and all, and it does. It is right about the lone high byte, the truncated sequence, the stray continuation byte, the surrogate, both overlongs and fe. And it accepts f4 90 80 80, f5 90 80 80 and a five-byte sequence — all three of which name code points above U+10FFFF, which has not been valid UTF-8 since Unicode capped the code space in 2003 ↗. The two implementations agree with each other and disagree with every language in this library.
In Python¶
Verified output of iconv_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. THE SAME THREE PARTS
the file (ISO-8859-1) 63 61 66 e9 20 a4 20 31 30 30 0a
.decode('iso-8859-1') 'café ¤ 100\n'
.encode('utf-8') 63 61 66 c3 a9 20 c2 a4 20 31 30 30 0a
iconv -f X -t Y is one expression: decode with the table you claim
the bytes are in, encode with the table you want. The str in the
middle is what iconv never gives you a name for.
2. THE SAME SILENT WRONG ANSWER
UTF-8 bytes .decode('iso-8859-1') -> 'café ¤ 100\n'
.encode('utf-8') -> 63 61 66 c3 83 c2 a9 20 c3 82 c2 a4 20 31 30 30 0a
No exception, because Latin-1 has all 256 bytes. Python is exactly as
unable to detect an encoding as iconv is; the difference is that the
claim is an argument here and a flag there.
3. //IGNORE AND //TRANSLIT, SPELLED AS errors=
errors=strict UnicodeEncodeError at byte 1: ordinal not in range(128)
errors=ignore 61 62 0a
errors=replace 61 3f 62 0a
errors=xmlcharrefreplace 61 26 23 38 33 36 34 3b 62 0a
errors=backslashreplace 61 5c 75 32 30 61 63 62 0a
errors=namereplace 61 5c 4e 7b 45 55 52 4f 20 53 49 47 4e 7d 62 0a
'strict' is plain iconv, 'ignore' is //IGNORE. There is no //TRANSLIT:
the closest is 'replace', which writes 3f — a question mark — and
never invents 'EUR'. Nothing in the standard library transliterates,
and that is a decision rather than a gap: what € becomes in ASCII is
a question about language, not about encoding.
4. WHERE PYTHON REFUSES AND ICONV DOES NOT
plain ASCII 61 62 63 accepted
cafe + U+00E9 63 61 66 c3 a9 accepted
lone high byte e9 refused: unexpected end of data
surrogate U+D800 ed a0 80 refused: invalid continuation byte
overlong slash c0 af refused: invalid start byte
U+10FFFF, the last f4 8f bf bf accepted
U+110000, one past f4 90 80 80 refused: invalid continuation byte
f5: no code point f5 90 80 80 refused: invalid start byte
five-byte sequence fb bf bf bf bf refused: invalid start byte
Compare the shell run. iconv agrees with Python on every classic
malformation and disagrees on the last three: it accepts them and
exits 0. The top of the code space is U+10FFFF — the highest number
UTF-16's surrogate pairs can express — and RFC 3629 restricted UTF-8
to the same ceiling in 2003. Those bytes name numbers above it, so
they are not code points at all; Python refuses them and Rust does
too (str::from_utf8 returns Utf8Error with valid_up_to 0 on all
three, and char::from_u32(0x110000) is None).
The middle of the expression is the thing iconv has no name for. data.decode(X).encode(Y) has a str in the middle — text, held as code points, belonging to no table. iconv goes from bytes to bytes and never surfaces it, which is why its errors are about positions in a file rather than about characters. Everything on Encode, decode and errors applies here; iconv is that page with the str hidden.
Section 3 is the honest comparison, and Python loses one round. errors='strict' is plain iconv, errors='ignore' is //IGNORE, and there is no //TRANSLIT — the nearest is errors='replace', which writes a question mark and never invents EUR. That is a deliberate absence: what € should become in ASCII is a question about language and house style, not about encoding, and the answer differs by country. iconv answers it anyway, differently on each platform, which is the next section.
Section 4 is the same table as the shell run, from the other side. Read the two together: every row where iconv says exit=1, Python raises; every row where iconv says exit=0, Python accepts — except the last three.
The three things no answer key can hold¶
Everything above is byte-identical on macOS and Ubuntu. These are not, so they are here, dated, rather than in a program's recorded output.
//TRANSLIT is a per-implementation table¶
character macOS 26.6.2 (Apple iconv) ubuntu:24.04 (glibc 2.39)
é c3 a9 27 65 'e 3f ?
ż c5 bc 7a z 3f ?
€ e2 82 ac 45 55 52 EUR 45 55 52 EUR
ß c3 9f 73 73 ss 73 73 ss
— e2 80 94 2d - 2d 2d --
😀 f0 9f 98 80 (nothing) 3f ?
exit status 1 0
Two of the six agree. The rest do not, and note how they disagree: macOS transliterates é to 'e and ż to z, which is a real attempt at the letter; glibc under the C locale gives up and writes ?, which is what errors='replace' does in Python. The em dash becomes one hyphen on one platform and two on the other, so a fixed-width report changes width depending on the machine that generated it. And the exit status differs as well, so even a script that ignores the output cannot branch on it portably.
The practical rule: //TRANSLIT is fine for a human reading the result once, and unusable for anything a second program parses. If you need a specific romanization, do it in a language with a table you chose.
The refusal message describes the problem on only one of them¶
macOS 26.6.2 iconv: iconv(): Inappropriate ioctl for device
ubuntu:24.04 iconv: illegal input sequence at position 1
Both exit 1. Only one of them says what happened: macOS reports ENOTTY, an errno left over from somewhere else entirely, on a file that is not a terminal and was never asked to be. Grepping iconv's stderr for the word illegal is a script that works on your CI runner and reports nothing on your laptop. Test the exit status.
The unknown-table message splits the same way — iconv: iconv_open(UTF-8, NOPE): Invalid argument against iconv: conversion from 'NOPE' is not supported — and both of those exit 1 too.
-l prints a different shape, and -c loses a byte¶
iconv -l lists the tables your machine knows. On macOS it prints 215 lines, each one a whitespace-separated group of aliases for a single table. On Ubuntu's glibc it prints 1180 lines, one name per line, each with a trailing //. Neither is wrong and no script can read both. To ask whether a specific table exists, do not grep the list — try the conversion on an empty file and read the status:
And -c, which asks iconv to drop invalid input rather than stop, has a narrower split than it looks. Measured 2026-09-07 across nine inputs, the two agree everywhere except one shape: when the byte following the skipped one is the last byte in the file, macOS discards it.
in macOS 26.6.2 ubuntu:24.04
61 e9 61 61 agree (nothing follows)
61 e9 62 61 61 62 DIFFER — the 62 is gone
61 e9 0a 61 61 0a DIFFER — the newline is gone
61 e9 62 63 61 62 63 61 62 63 agree
61 e9 c3 a9 61 c3 a9 61 c3 a9 agree
A one-byte tail vanishes; a two-byte tail survives. On a real file the lost byte is almost always the final newline, which is quiet enough that nobody notices — and is exactly the byte the trailing newline is about. Use -c to look at damaged input, never to repair a file you intend to keep.
If you are coming from Python or ABAP¶
Python. iconv -f X -t Y is data.decode(X).encode(Y) and nothing more, so the whole of Encode, decode and errors transfers — including the part that matters most, which is that decode is where a wrong guess becomes silent damage. Two differences worth carrying: Python's error policies have names and are the same on every machine, where //TRANSLIT is not; and Python has no transliteration at all, so code that needs é → e is code you write and can test.
ABAP. The direct equivalent is cl_abap_conv_codepage, whose create_in( codepage = 'UTF-8' ) / create_out( ) pair is -f and -t with the same asymmetry — the code page you name on the way in is an assertion nobody checks, and getting it wrong produces a perfectly valid string full of the wrong characters. The older cl_abap_conv_in_ce / cl_abap_conv_out_ce classes do the same job and are still everywhere in older code. What ABAP does not give you is //IGNORE or //TRANSLIT: a character the target code page cannot hold raises CX_SY_CONVERSION_CODEPAGE, and the substitution policy is a parameter on some of the older APIs rather than a suffix on the encoding name. Verify any SAP code-page number against the system — SCP and the TCP00 tables — rather than trusting one written down anywhere, including here. (Not machine-checked — CI cannot run ABAP.)
Try it¶
- Take the worst file you have — the CSV from a supplier, the export that came out with
éin it — and runiconv -f UTF-8 -t UTF-8 thatfile >/dev/null; echo $?. A1means it is not UTF-8 and you now know that much for certain. A0means it is probably UTF-8, with the exception section 6 found. - Convert something of yours from UTF-8 to
ASCII//TRANSLITand read what came out. Then run the same command on the other operating system you have access to — a container, a server, a colleague's laptop — and diff the two. iconv -l | wc -lon every machine you can reach. The number is a fact about that machine's C library, and it is why "just useiconv" is not a portable instruction for an unusual code page.- Find a script of yours that pipes
iconvinto something else. Check whether it reads the exit status, and whether it would survive//IGNOREreturning 1.
Practice¶
Four commands, one file. The file f holds the six bytes 61 e9 62 c3 a9 0a. Before running anything, write down what each of these prints and what it exits with:
iconv -f UTF-8 -t UTF-8 f
iconv -f ISO-8859-1 -t UTF-8 f
iconv -c -f UTF-8 -t UTF-8 f
iconv -f UTF-8 -t ASCII//IGNORE f
Then say which of the four answers would be the same on the other platform, and which single byte is the one to watch.
Answers
Verified output of iconv_kata_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
THE FILE
f = 61 e9 62 c3 a9 0a
'a', then the LATIN-1 byte for e-acute, then 'b', then e-acute
written properly in UTF-8, then a newline. One file, two spellings
of the same letter — which is what makes it worth four commands.
THE FOUR COMMANDS
iconv -f UTF-8 -t UTF-8 f exit=1 out=61
iconv -f ISO-8859-1 -t UTF-8 f exit=0 out=61 c3 a9 62 c3 83 c2 a9 0a
iconv -c -f UTF-8 -t UTF-8 f exit=0 out=61 62 c3 a9 0a
iconv -f UTF-8 -t ASCII//IGNORE f exit=1 out=61 62 0a
WHAT EACH ONE DID
1 The validator. e9 is not a legal UTF-8 start byte, so iconv stops
there and exits 1 — after writing the 61 it had already converted.
A refusal is not an empty file; partial output is normal.
2 No failure is possible: Latin-1 has all 256 bytes. e9 became the
correct c3 a9, and the ALREADY-correct c3 a9 became c3 83 c2 a9.
Half the file was repaired and half was broken, in one pass,
exit 0. This is what a wrong -f looks like from the outside.
3 -c skips the byte it cannot read and keeps going: e9 is gone and
everything else survives, exit 0.
4 //IGNORE drops what ASCII cannot hold — the invalid byte AND the
valid e-acute, because neither is an ASCII character — and still
exits 1.
WHICH ANSWERS TRAVEL, AND THE BYTE TO WATCH
Rows 1, 2 and 4 are identical on macOS and Ubuntu. Row 3 is identical
HERE and is the fragile one: after -c skips e9 there are four bytes
left, and macOS only differs when exactly one byte follows the skip,
in which case it discards it. Delete 'b' and the e-acute from f and
the two platforms print different answers for row 3.
The byte to watch is e9. It is a perfectly good e-acute in one table
and not a character at all in the other, and every line above is a
different policy for that one disagreement.
See also¶
- Inspecting a file —
iconvto UTF-16, dumped before and after, and the byte order it picks when you do not name one - Validation is a boundary — the validator idiom in full, and the boundary it belongs at
- Encode, decode and errors — the same two operations with the
strin the middle visible - Mojibake — what a wrong
-fproduces, and how to reverse it fileguesses — the tool that will guess-ffor you, and what kind of claim that is- Code pages — what a table like ISO-8859-1 actually is