Skip to content

Encode and decode

Level: 101 → 201 · for anyone starting from zero

One line: .encode() and .decode() are directions, not conversions — and the errors= argument decides, in advance, what your program is willing to lose.

There are only two doors between str and bytes, and remembering which is which is easier if you picture the process boundary. Text lives inside Python. Bytes are what travel on a disk, a socket, or a pipe. So .encode() is the way out and .decode() is the way in — and the method you are allowed to call tells you which type you are already holding.

The second argument is the one that matters in production. Every codec eventually meets a character it cannot write, or a byte sequence it cannot read, and errors= is where you say what should happen then. The default is strict, which raises. Every other policy is a decision to destroy information quietly.

Verified output of encode_and_decode_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. THE TWO DIRECTIONS
     str  --encode()-->  bytes      leaving Python
     bytes --decode()-->  str       entering Python

     'Zażółć'.encode('utf-8')   = b'Za\xc5\xbc\xc3\xb3\xc5\x82\xc4\x87'
     b'Za\xc5\xbc\xc3\xb3\xc5\x82\xc4\x87'.decode('utf-8') = 'Zażółć'

2. THE SAME TEXT IN FOUR ENCODINGS
   Not four representations of one thing -- four different files.

     utf-8      10 bytes  5A 61 C5 BC C3 B3 C5 82 C4 87
     utf-16-le  12 bytes  5A 00 61 00 7C 01 F3 00 42 01 07 01
     utf-32-le  24 bytes  5A 00 00 00 61 00 00 00 7C 01 00 00 F3 00 00 00 42 01 00 00 07 01 00 00
     cp1250      6 bytes  5A 61 BF F3 B3 E6

3. WHAT A TABLE CANNOT HOLD
   cp1250 is the Windows Central European table. It has Polish.
   latin-1 does not, and says so rather than guessing:

     cp1250     ok    b'Za\xbf\xf3\xb3\xe6'
     latin-1    raises UnicodeEncodeError on 'ż' at index 2

4. THE errors= POLICIES, AND WHAT EACH ONE COSTS
   Encoding 'Zażółć' to latin-1, which cannot represent it:

     strict               raises -- the default, and the only one that loses nothing
     ignore               b'Za\xf3'
     replace              b'Za?\xf3??'
     xmlcharrefreplace    b'Zaż\xf3łć'
     backslashreplace     b'Za\\u017c\xf3\\u0142\\u0107'
     namereplace          b'Za\\N{LATIN SMALL LETTER Z WITH DOT ABOVE}\xf3\\N{LATIN SMALL LETTER L WITH STROKE}\\N{LATIN SMALL LETTER C WITH ACUTE}'

5. THE ASYMMETRY WORTH KNOWING
   'replace' on the way OUT writes '?'. On the way IN it writes U+FFFD.
   Neither is reversible: the original is gone, and the file now says so.

     b'Za\xff\xfe\xc3\xb3'
       .decode('utf-8', 'replace')          -> 'Za��ó'
       .decode('utf-8', 'ignore')           -> 'Zaó'
       .decode('utf-8', 'backslashreplace') -> 'Za\\xff\\xfeó'
       .decode('latin-1')                   -> 'Zaÿþó'

     latin-1 never raises: every byte 00-FF is a character in it. That
     makes it useful as a byte-preserving codec and lethal as a guess.

6. THE ONE POLICY THAT IS REVERSIBLE
   surrogateescape hides undecodable bytes in a private range and puts
   them back byte-for-byte on the way out. This is how Python survives
   a filename that is not valid UTF-8.

     decoded  -> 'Za\udcff\udcfeó'
     re-encoded -> b'Za\xff\xfe\xc3\xb3'
     identical to the original: True

Four observations from that run.

"The same text in four encodings" is a lie the phrase itself tells. They are four different files: 10 bytes, 12, 24, and 6. Nothing about a str says how long it will be on disk. A fixed-width database column sized in bytes and filled from a str is the bug this produces, and it is the one that actually bites SAP data.

cp1250 holds Polish and latin-1 does not, and the failure is loud. UnicodeEncodeError carries .start, .end, .object and .reason, so the exception can tell you exactly which character broke and where — worth catching and reporting rather than wrapping in a bare except.

replace is not symmetric. Going out it writes ?; coming in it writes U+FFFD (�). Both are irreversible: the original is gone and the file now contains a permanent record that something was lost. That is a feature — � is supposed to be visible — but it means "just add errors='replace'" is a decision to corrupt data, not a way to avoid an error.

latin-1 never raises on decode. Every one of the 256 byte values is a character in it, so bytes.decode("latin-1") always succeeds. That makes it genuinely useful as a byte-preserving round trip, and lethal as a guess: it turns "I don't know this encoding" into "here is some text" with no error anywhere. If you have ever seen ó where ó belonged, this is the mechanism.

surrogateescape is the one reversible policy. It parks undecodable bytes in a reserved range and puts them back byte-for-byte on the way out. This is not a curiosity — it is how Python survives a filename that is not valid UTF-8, which is why it reappears in filenames are not text.

If you are coming from ABAP

The two verbs are cl_abap_conv_codepage=>create_out( )->convert( ) and create_in( )->convert( ), and the shape is the same: you name a code page and you get an exception when the target cannot hold the source. The difference is what happens when you don't name one. ABAP falls back to the system code page, so the same program gives different bytes on two systems; Python has no fallback for .encode() at all — the default is UTF-8, fixed, everywhere. (Not machine-checked — CI cannot run ABAP. Any specific SAP code-page number should be verified against the system.)

Try it

  1. Encode "€" to cp1252, latin-1 and ascii. Two of the three raise. Read exc.reason on each.
  2. Take a UTF-8 file, decode it as latin-1, re-encode it as UTF-8, and compare to the original. This is mojibake, manufactured deliberately — and the round trip back is the subject of mojibake round trip ↗.
  3. Write bytes that are invalid UTF-8 to a file, read it with errors="surrogateescape", write it back out, and diff. Then do the same with errors="replace" and diff again.

Practice

Seven crossings of the boundary. For each, write down the result — or the name of the exception — and mark the direction: is this text leaving Python, or bytes arriving?

  1. "€".encode("cp1252")
  2. "€".encode("latin-1")
  3. "€".encode("ascii", "replace")
  4. b"\xff".decode("latin-1")
  5. b"\xff".decode("utf-8")
  6. b"\xff".decode("utf-8", "replace")
  7. b"\xff".decode("utf-8", "ignore")

Then two questions. Which of the seven destroyed information? And which one neither raised nor gave you the right answer — that one is the mechanism behind every ó you have ever seen where an ó belonged.

Answers

Verified output of encode_and_decode_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

     expression                           dir  result                    note
     -------------------------------------------------------------------------------------------------
     '€'.encode('cp1252')                 out  b'\x80'                   cp1252 has the euro, at one byte
     '€'.encode('latin-1')                out  raises UnicodeEncodeError latin-1 predates it, and says so
     '€'.encode('ascii', 'replace')       out  b'?'                      '?' -- the euro is gone
     b'\xff'.decode('latin-1')            in   'ÿ'                       latin-1 NEVER raises: 256 of 256
     b'\xff'.decode('utf-8')              in   raises UnicodeDecodeError 0xff cannot start a UTF-8 sequence
     b'\xff'.decode('utf-8', 'replace')   in   '�'                       U+FFFD -- a permanent record
     b'\xff'.decode('utf-8', 'ignore')    in   ''                        the byte is simply gone

     The 'dir' column is the half people get backwards, and there is
     nothing to memorise: .encode() is only on str and .decode() is
     only on bytes, so the method you can REACH tells you which type
     you are already holding. Text lives inside Python; bytes are
     what travel. Out is encode, in is decode.

     WHICH ONES LOSE INFORMATION?
     '€'.encode('cp1252')             -> b'\x80'  -> '€'   same: True
     '€'.encode('ascii', 'replace')   -> b'?'     -> '?'   same: False

     Only one policy on the list is reversible, and it is not on the
     list -- because it is the one nobody reaches for:
     b'Za\xff\xfe\xc3\xb3'.decode('utf-8', 'surrogateescape')
       -> 'Za\udcff\udcfeó'
       -> re-encoded b'Za\xff\xfe\xc3\xb3'
       -> identical to the original: True

     'replace' and 'ignore' are decisions to destroy data quietly.
     'strict' is the default and the only one that loses nothing by
     raising. 'surrogateescape' is the only one that loses nothing
     by keeping -- which is how Python survives a filename that is
     not valid UTF-8.

     THE TRAP IN LINE 4
     b'\xff'.decode('latin-1') did not fail, and it was not right.
     Every one of the 256 byte values is a character in latin-1, so
     the call always succeeds -- which makes it genuinely useful as
     a byte-preserving round trip and lethal as a guess. It turns
     'I do not know this encoding' into 'here is some text' with no
     error anywhere. If you have ever seen 'ó' where 'ó' belonged,
     this is the mechanism:
     'ó'.encode('utf-8').decode('latin-1')   'ó'
     and back again                         'ó'

See also