The codecs registry¶
Level: 201 · for Python programmers
One line: str.encode() and bytes.decode() are two doors into a registry with more in it than they can reach — and the room they cannot reach is where the stream bugs live, because a codec is an object with state, so decoding a stream one chunk at a time is not the same operation as decoding its bytes.
Most of the codecs ↗ module is a second spelling for things you already have. codecs.encode(s, "utf-8") is s.encode("utf-8"), codecs.open has been redundant since open() grew an encoding= argument in Python 3, and a page that toured all of it would be the docs with worse formatting. Three things in it are not a second spelling of anything, and this page is those three.
The first is a matter of typing. .encode() is declared str → bytes and .decode() is bytes → str, but the registry also holds transforms that are str → str (rot13) and bytes → bytes (base64, zlib, hex). Neither shape fits through either door, so both methods refuse — and the mechanism is not a missing codec but a flag on the CodecInfo object saying whether the methods are allowed to use it.
The second is state, and it is the one with a bug attached. "…".encode() takes a whole string and returns whole bytes; it has no memory, because it never needs one. A stream does need one. A UTF-8 character is one to four bytes, and a chunk boundary is a fact about your buffer size, not about the text — so a read that arrives in pieces will eventually cut a character in half, and the loop everybody writes first (chunk.decode() per chunk) is wrong. codecs.getincrementaldecoder returns a class whose instances hold the leftover bytes between calls, and that instance is the correct answer.
The third is that the eight named error handlers are a registry too, opened by PEP 293 ↗, and adding a ninth takes about ten lines.
Verified output of the_codecs_registry_py.py — regenerated by tools/run_examples.py, never hand-typed.
THE CODECS REGISTRY
========================================================================
str.encode / bytes.decode are two doors into a registry that has
more rooms than they can reach. This is a tour of the other rooms.
1. NOT EVERY CODEC IN THE REGISTRY IS A TEXT CODEC
------------------------------------------------------------------------
A CodecInfo carries a flag saying whether .encode()/.decode() may
use it. It is the flag, not a missing codec, that decides.
name asked canonical text codec? direction
utf-8 utf-8 yes str <-> bytes
latin-1 iso8859-1 yes str <-> bytes
rot13 rot-13 no not str <-> bytes
base64 base64 no not str <-> bytes
zlib zlib no not str <-> bytes
hex hex no not str <-> bytes
So the two methods refuse them -- by name, and the message names
the door that does work:
'abc'.encode('rot13')
LookupError: 'rot13' is not a text encoding; use codecs.encode() to handle arbitrary codecs
b'YWJj'.decode('base64')
LookupError: 'base64' is not a text encoding; use codecs.decode() to handle arbitrary codecs
Through codecs.encode / codecs.decode the same names work, and
each one has its own pair of types:
codecs.encode('abc', 'rot13') -> 'nop'
codecs.decode('nop', 'rot13') -> 'abc'
codecs.encode(b'abc', 'base64') -> b'YWJj\n'
codecs.encode(b'abc', 'hex') -> b'616263'
codecs.encode('abc', 'base64') !! TypeError
rot13 is str->str. base64, hex and zlib are bytes->bytes.
Neither shape fits through a door typed str->bytes.
2. A CHUNK BOUNDARY IS NOT A CHARACTER BOUNDARY
------------------------------------------------------------------------
'Zażółć' is 6 characters and 10 UTF-8 bytes:
5a 61 c5 bc c3 b3 c5 82 c4 87
Read it in two chunks with the cut at byte 3, which lands in
the middle of the two bytes that spell 'ż':
chunk 0 5a 61 c5
chunk 1 bc c3 b3 c5 82 c4 87
Decoding each chunk on its own -- the obvious loop -- is wrong:
[c.decode('utf-8') for c in chunks]
UnicodeDecodeError, reason: unexpected end of data
And the usual reflex for that exception makes it worse, because
it stops raising and starts lying:
with errors='replace' -> 'Za��ółć'
characters: 7, expected 6
equal to the original text? False
no exception was raised. One character became two U+FFFD.
3. A CODEC IS AN OBJECT, AND THE OBJECT REMEMBERS
------------------------------------------------------------------------
codecs.getincrementaldecoder returns a class. An instance of it
keeps whatever it could not finish and uses it on the next call.
fresh decoder, getstate() = (b'', 0)
.decode(chunk 0) -> 'Za' <- 'ż' is NOT here yet
getstate() = (b'\xc5', 0) <- it is here, held back
.decode(chunk 1) -> 'żółć'
joined -> 'Zażółć' equal to the original? True
The state is one incomplete sequence, so the chunk size does not
matter at all. One byte at a time is the worst case and it works:
10 calls of one byte each -> 'Zażółć'
equal to the original? True
4. final=True IS THE OTHER HALF OF THE CONTRACT
------------------------------------------------------------------------
Holding bytes back is right in the middle of a stream and wrong at
the end of one. The decoder cannot tell which it is in, so you say.
a truncated stream: 5a 61 c5 bc c3 b3 c5 82 c4
.decode(truncated) -> 'Zażół'
no error: a held byte is normal, more input may be coming
.decode(b'', final=True) !! UnicodeDecodeError
reason: unexpected end of data
Forget final=True and a truncated file decodes clean, one
character short, with nothing anywhere reporting it.
5. THE ENCODER IS STATEFUL TOO, AND ITS BUG IS SILENT
------------------------------------------------------------------------
The same split on the way out has no exception to warn you at all.
utf-16 begins with a byte order mark. One per stream -- but the
one-shot encoder does not know it is in a stream.
join of per-part .encode() ff fe 5a 00 61 00 ff fe 7c 01 f3 00 42 01 07 01
incremental encoder ff fe 5a 00 61 00 7c 01 f3 00 42 01 07 01
one-shot, whole string ff fe 5a 00 61 00 7c 01 f3 00 42 01 07 01
incremental == one-shot? True
naive == one-shot? False
The naive one is 2 bytes longer -- exactly one more mark --
and it decodes without error. What it decodes to is the problem:
'Za\ufeffżółć'
index 2 is U+FEFF ZERO WIDTH NO-BREAK SPACE
equal to the original text? False
A second BOM in mid-stream is not a BOM. It is a zero-width
character in your data, and it prints as nothing.
6. THE SAME SHAPE IS NOT THE SAME PROMISE
------------------------------------------------------------------------
Every codec in the registry offers the incremental interface, non-
text ones included. Offering it and honouring it are different.
base64 of b'abcdefgh' in one call b'YWJjZGVmZ2g=\n'
the same bytes as 5 + 3 b'YWJjZGU=\nZmdo\n'
identical? False
does it round-trip? False
Each chunk was padded as if it were the end of the stream, so
the result is malformed base64. What a decoder then does with
it is not even stable across Python versions: 3.11 and 3.12
return b'abcde' and report success, dropping three bytes;
3.13 and later raise binascii.Error. Only the line above is
true on all of them, which is why it is the one printed.
zlib, same treatment: identical? True
does it round-trip? True
zlib streams. base64 does not. The type signature is identical.
7. THE NINTH ERROR HANDLER IS THE ONE YOU WRITE
------------------------------------------------------------------------
Python ships eight named policies. The list is a registry too, and
PEP 293 made it open. A handler is a function of one argument.
built in: 8
strict strict_errors
ignore ignore_errors
replace replace_errors
xmlcharrefreplace xmlcharrefreplace_errors
backslashreplace backslashreplace_errors
namereplace namereplace_errors
surrogateescape surrogateescape
surrogatepass surrogatepass
Ten lines, and it is now a name like any other:
codecs.lookup_error('audit') -> audit
b'caf\xe9 \xff au lait'.decode('utf-8', 'audit')
-> 'caf<e9> <ff> au lait'
'Zażółć'.encode('latin-1', 'audit')
-> b'Za[U+017C]\xf3[U+0142+U+0107]'
One name, both directions, and the argument is a different
exception class each way. A handler that assumes one of them
raises AttributeError the first time it meets the other.
the log the built-in handlers cannot give you:
decode at 3 b'\xe9'
decode at 5 b'\xff'
encode at 2 'ż'
encode at 4 'łć'
8. TWO THINGS THE CONTRACT REQUIRES THAT NOTHING ANNOUNCES
------------------------------------------------------------------------
A handler returns (replacement, resume position). Both halves have
a rule, and neither is checked until it is broken.
First: it is called once per RUN, not once per character.
'Zażółć'.encode('latin-1', 'counted') -> b'Za?\xf3?'
unencodable characters: 3, handler calls: 2
[2:3] 'ż'
[4:6] 'łć'
Returning one character for a run of two shortens the string,
which is how a 'replacement' loses a character count.
Second: the position must move forward, or nothing ends.
a handler returning exc.start instead of exc.end:
RuntimeError: handler called 4 times on one bad byte
Uncaught, that is not an exception. It is a hang.
========================================================================
Two methods reach the text codecs, one shot at a time, with the eight
policies. Everything else in the registry -- the non-text codecs, the
stateful objects, the ninth handler -- is reached through the module.
What the run shows¶
The refusal is a flag, not an absence. codecs.lookup("rot13") succeeds; the codec is there, canonically named rot-13, sitting in the encodings package next to utf_8. What str.encode consults is _is_text_encoding, and the LookupError it raises names the door that does work. That flag is the cleanest demonstration in the language that "encoding" means two unrelated things — a character-to-byte table, and any reversible transform at all — and the standard library files both under one word and then needs a boolean to tell them apart. The theory half is the sibling library's: rotation is not encryption ↗.
The naive chunked decode fails, and the usual fix for it is worse than the failure. Cutting "Zażółć" at byte 3 lands between the two bytes of ż, and per-chunk .decode() raises UnicodeDecodeError with reason unexpected end of data. Reach for errors="replace" — the reflex that exception trains — and the program stops raising and starts lying: 7 characters where there were 6, one real character rendered as two U+FFFD, and no exception anywhere. That is the shape of the bug worth remembering. It is not that chunked decoding crashes; it is that the obvious way to stop it crashing turns a loud fault into silent corruption, at a position that depends on your buffer size and so moves when the file does.
The decoder's state is visible, and it is one incomplete sequence. After chunk 0, getstate() returns (b'\xc5', 0) — the lead byte of ż, held back because the decoder cannot yet know what it will become. That is the whole trick. Because the held-back part is bounded by the longest sequence the encoding has, the chunk size is irrelevant: feeding the same bytes one at a time, ten calls, reconstructs the string exactly.
final=True is the other half of the contract, and forgetting it is a second silent bug. Holding bytes back is correct mid-stream and wrong at the end of one, and the decoder cannot tell which it is in — so you tell it. A truncated stream decodes without complaint until the last call says the stream is over; only then does the held byte become an error. A loop that never passes final=True reads a truncated file as clean text, one character short, with nothing reporting it.
The encoder is stateful too, and its version of the bug never raises at all. utf-16 opens with a byte order mark, one per stream — but a one-shot encoder does not know it is in a stream, so joining per-chunk .encode("utf-16") writes the mark once per chunk. The result is two bytes longer than the one-shot encoding — exactly one more mark — decodes without error, and contains U+FEFF ZERO WIDTH NO-BREAK SPACE sitting in the middle of the text as ordinary data. It prints as nothing. It compares unequal. A second BOM is not a BOM — it is an invisible character in your string, which is the same trap byte order and the BOM ↗ is about, arriving from the encoder side.
Offering the incremental interface and honouring it are different things. Every codec in the registry has an IncrementalEncoder, non-text ones included, and the type signatures are identical — so this is a distinction no annotation can carry. zlib genuinely streams: chunked output is byte-identical to one-shot. base64 does not. It pads every chunk as though it were the end of the stream, so encoding eight bytes as 5 + 3 gives b'YWJjZGU=\nZmdo\n' where one call gives b'YWJjZGVmZ2g=\n'. What happens next is not even stable across Python versions: 3.11 and 3.12 decode that malformed input to b'abcde' and report success, throwing three bytes away silently, while 3.13 and later raise binascii.Error. The example prints only the round-trip check, which is False on all four.
The ninth handler is ten lines, and it is called once per run. The backlog entry that prompted this page guessed a sixth handler; Python ships eight (strict, ignore, replace, xmlcharrefreplace, backslashreplace, namereplace, surrogateescape, surrogatepass), so anything you register is the ninth — the count is measured in section 7 rather than repeated. Two details the signature does not advertise. One name serves both directions, and the argument is a different exception class each way: a handler written against UnicodeDecodeError alone dies on AttributeError the first time somebody encodes with it. And the handler fires once per contiguous run of bad input, not once per character — "Zażółć" to latin-1 is three unencodable characters and two calls, [2:3] and [4:6]. Return one replacement character for a two-character run and the string quietly gets shorter, which is exactly how replace loses a character count. The second element of the return is where to resume, and it must move forward; returning exc.start is not an exception, it is a hang.
If you are coming from Rust or ABAP¶
Rust has no codec registry, and the absence is a decision rather than a gap. std ships exactly one encoding — str is UTF-8 by definition — so the whole conversion story is str::from_utf8, which returns Result<&str, Utf8Error>, and anything else is the encoding_rs ↗ crate. The interesting part for this page is that Utf8Error::error_len() returns None for exactly the case section 2 is about: input that is valid so far but incomplete, distinguished in the type system from input that is wrong. Python needs final=True to make that distinction because its decoder cannot see the difference; Rust makes the caller match on it. The chunked-read problem itself is the subject of reading lines efficiently ↗.
ABAP (Not machine-checked — CI cannot run ABAP.) The nearest thing is cl_abap_conv_codepage, and the shape of the difference is the point: it is a class you instantiate, not a registry you can add to. You pick a code page and get a converter; there is no register_error equivalent, so the policy set is closed — a replacement character or an exception, and nothing you can write yourself. The stateful problem is real there too and is usually dodged rather than solved: the idiom for a file arriving in pieces is OPEN DATASET … IN BINARY MODE into an xstring, accumulate the bytes, and convert once at the end. That works, and it is worth knowing that it works because you have avoided the incremental case, not because ABAP handles it — buffering the whole payload is the same fix as buffering it in Python, with the same memory cost. Any code-page number you find in a document should be verified against the system that will run the job.
Try it¶
- Take any UTF-8 file with non-ASCII text in it and read it in a loop with
f.read(3), decoding each chunk. Then do it again with an incremental decoder. The first one's behaviour depends on where the characters happen to fall relative to 3. codecs.getincrementaldecoder("utf-16")()fed one byte at a time. Watchgetstate()— UTF-16 holds back for a different reason than UTF-8, and a surrogate pair makes it hold back twice as far.- Register a handler that returns
("", exc.end)and one that returns("", exc.start). Run the second with a timeout you are ready to hit. - Write a handler that raises instead of replacing, but only after logging the offset. That is
strictplus evidence, which is the policy nobody ships. codecs.lookup("rot13")._is_text_encoding, then the same for"utf-8". One underscore is the whole reason"abc".encode("rot13")raises.
See also¶
- Encode and decode — the two doors themselves, and what each
errors=policy costs stris notbytes— the type boundary the_is_text_encodingflag is defending- Filenames are not text —
surrogateescape, the built-in handler that is closest to what you would write yourself - Opening a file — where that held-back lead byte becomes visible without your asking:
open()runs an incremental decoder for you, andtell()in text mode returns its state packed into an integer, which is why a 6-byte file can report position 340282367000166625996085689099021713410 - Encode, decode and errors ↗ — the eight built-in handlers as a matrix, including the two that raise
TypeErroron decode - Rotation is not encryption ↗ — why
rot13being in the codec registry is the right filing and still not cryptography - Byte order and the BOM ↗ — the mark section 5 writes twice, and what reads it