Bytes that are not text

Level: 201 · for Python programmers

One line: A POSIX filename is bytes, not text, so some of them cannot be decoded at all — and surrogateescape is the error handler that carries an undecodable byte through a str and puts it back unchanged, which is how os.listdir() can hand you a name nobody can read and open() can still open it.

name = raw.decode("utf-8", "surrogateescape")   # bytes in
raw  = name.encode("utf-8", "surrogateescape")  # the same bytes out

Those two lines are the whole mechanism, and you have been running them for years without naming them: on POSIX, os.listdir(), open(), os.environ and sys.argv all use exactly that pair. PEP 383 ↗ (Martin von Löwis, Python 3.1) is where it was decided.

The problem it solves

A Unix filename is any sequence of bytes except NUL and /. Nothing requires it to be UTF-8, nothing records what it was meant to be, and on a disk that has been through a decade and three sysadmins some of it will not be anything in particular. Python 3 hands you filenames as str, so somebody has to decode those bytes, and for a name written in Latin-1 in 2009 the correct answer is that it cannot be done.

That answer is useless. The file exists. It has a name, an inode, and a size, and a program that raises UnicodeDecodeError while listing a directory cannot read past it, let alone rename the thing. Refusing to decode is right and it leaves you with nothing.

So PEP 383 widens what a str may contain, just enough:

With this PEP, non-decodable bytes >= 128 will be represented as lone surrogate codes U+DC80..U+DCFF.

One byte, one code point, U+DC00 + byte. The name is now a str with a hole in it — twelve characters, one of which is not a character — and every string operation works on it, and encoding it back with the same handler returns the original bytes exactly.

Why a surrogate, and why only 128 of them

The escape needs a range that correctly decoded text can never contain, or it would collide with somebody's real data the first time it met it. Unpaired low surrogates are that range: Unicode reserves D800–DFFF for UTF-16's pairing arithmetic, never assigns a character there, and no UTF can encode one. chr(0xDCE9).encode('utf-8') raises. That is the property being borrowed — a str holding one of these has been through this handler, because nothing else can produce one.

Only bytes 0x80–0xFF get an escape, which is 128 code points and is why the range is DC80–DCFF rather than the whole low half. Bytes below 128 are refused, and the PEP says why: smuggling them would be a security risk "if the bytes are security-critical when interpreted as characters on a target system, such as path name separators". / is 0x2F. A scheme that could carry one through a str would turn a filename into a directory traversal, so the escape has no spelling for the bytes that mean something to the kernel.

The idea is not Python's, and it is not only Python's. PEP 383 credits Markus Kuhn's UTF-8b ↗; the web platform arrived at the same trick independently with a different reserved range — x-user-defined maps 0x80–0xFF into the private use area rather than into surrogates — which is worked out beside Python's on "Handles Unicode" is four questions. Two committees, two reserved ranges, one idea: to carry a byte you cannot read, spend a code point nobody can legitimately mean.

PEP 383 is also honest about the part it does not solve:

Data obtained from other sources may conflict with data produced by this PEP. Dealing with such conflicts is out of scope of the PEP.

Section 9 of the program below is what that sentence looks like in running code.

In Python

Verified output of surrogateescape_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. THE BYTE THAT IS NOT TEXT
------------------------------------------------------------------------
   A POSIX filename is any bytes except NUL and '/'. It is not
   required to be UTF-8, and on a disk that has been around a while
   some of it will not be. Here is one, twelve bytes:

     b'caf\xe9 au lait'
     63 61 66 e9 20 61 75 20 6c 61 69 74

   Python 3 hands you filenames as str, so somebody has to decode it,
   and the honest answer is that it cannot be done:

     RAW.decode('utf-8')
       UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in
         position 3: invalid continuation byte

   That is correct and it is useless. The file exists, it has a name,
   and a program that raises here cannot rename it, delete it, or
   even finish listing the directory it sits in.

2. THE MAPPING
------------------------------------------------------------------------
     RAW.decode('utf-8','surrogateescape')         'caf\udce9 au lait'

   len(escaped) = 12, the same as the byte count, and the fourth
   position now holds one code point that is not a character:

     U+0063 U+0061 U+0066 U+DCE9 U+0020 U+0061 U+0075 U+0020 U+006C U+0061 U+0069 U+0074

   PEP 383 puts every undecodable byte at U+DC00 + byte:

     byte  code point
     0x80  U+DC80
     0xC3  U+DCC3
     0xE9  U+DCE9
     0xFF  U+DCFF
     ...
     every byte 0x80..0xFF lands on U+DC00+byte:  True

   So the escape occupies exactly 128 code points, U+DC80..U+DCFF,
   one for each byte that could ever need one.

3. AND BYTES BELOW 0x80 ARE REFUSED, ON PURPOSE
------------------------------------------------------------------------
   The handler is an ordinary object, so you can ask it directly.
   Hand it a failure over an ASCII byte and it declines to escape it,
   re-raising the error it was called to fix:

     handler(error over byte 0x41)
       UnicodeDecodeError: 'utf-8' codec can't decode byte 0x41 in
         position 0: invalid start byte
     handler(error over byte 0x7F)
       UnicodeDecodeError: 'utf-8' codec can't decode byte 0x7f in
         position 0: invalid start byte
     handler(error over byte 0x80)                 ('\udc80', 1)
     handler(error over byte 0xFF)                 ('\udcff', 1)

   PEP 383 states the reason: smuggled bytes would be a security risk
   when the target system reads them as characters, 'such as path name
   separators', so the PEP rejects smuggling bytes below 128. '/' is
   0x2F and NUL is 0x00; the escape deliberately has no spelling for
   the bytes that mean something to the kernel.

4. WHY A SURROGATE
------------------------------------------------------------------------
   The range has to be one that correctly decoded text can never
   contain, or the escape would collide with somebody's real data.
   U+DC80..U+DCFF are unpaired low surrogates: reserved by Unicode for
   UTF-16's pairing arithmetic, never assigned to a character, and not
   encodable by any UTF at all --

     chr(0xDCE9).encode('utf-8')
       UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9'
         in position 0: surrogates not allowed
     chr(0xDCE9).encode('utf-16-le')
       UnicodeEncodeError: 'utf-16-le' codec can't encode character
         '\udce9' in position 0: surrogates not allowed

   -- which is exactly the property being borrowed. No decode of valid
   input can produce one, so a str holding one has been through this
   handler. The idea is Markus Kuhn's, who called it UTF-8b; PEP 383
   credits him and is candid about what the argument does not cover:
   'Data obtained from other sources may conflict with data produced
   by this PEP. Dealing with such conflicts is out of scope of the
   PEP.' Section 9 is what that sentence looks like in code.

5. THE ROUND TRIP, PROVED BY EXHAUSTION
------------------------------------------------------------------------
   The claim is not 'it usually works'. Every one of the 65,536
   two-byte strings, valid UTF-8 or not, decodes and re-encodes to
   itself:

     round-trip failures over all 65,536 two-byte strings:  0

   And through the two functions that apply the policy for you, over
   every single byte a filename may contain:

     os.fsencode(os.fsdecode(b)) != b, for b in 0..255:     0

     sys.getfilesystemencoding()       'utf-8'
     sys.getfilesystemencodeerrors()   'surrogateescape'

   Those two are what os.listdir(), open(), os.environ and sys.argv
   already use on this system. You have been running this handler for
   years without naming it.

6. surrogateescape IS NOT surrogatepass
------------------------------------------------------------------------
   Two handlers one letter apart, one input, and the confusion is
   expensive because both of them succeed:

     escaped.encode('utf-8','surrogateescape')     b'caf\xe9 au lait'
     escaped.encode('utf-8','surrogatepass')       b'caf\xed\xb3\xa9 au lait'

   surrogateescape puts the byte back: 12 bytes in, 12 bytes out.
   surrogatepass encodes the surrogate ITSELF, as the three bytes
   ED B3 A9 -- the spelling the WTF-8 specification gives an unpaired
   surrogate, and not valid UTF-8 by any reading.

   Mix them and the length changes with no error anywhere:

     surrogatepass out, surrogateescape back in:
       14 bytes  ->  12 bytes
       and the two str values in between are equal:  True

   That second line is the finding. The str keeps no record of which
   handler made it, so one byte and three bytes become the same object
   the moment they are inside.

7. WHERE IT LEAKS
------------------------------------------------------------------------
   The escaped str is an ordinary str until something turns it back
   into bytes without being told how:

     escaped.encode('utf-8')
       UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9'
         in position 3: surrogates not allowed
     json.dumps(escaped).encode('utf-8')
       UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9'
         in position 4: surrogates not allowed

   And it does not travel to every codec even when you do say the
   handler's name. PEP 383: 'Encodings that are not compatible with
   ASCII are not supported by this specification.'

     escaped.encode('ascii','surrogateescape')     b'caf\xe9 au lait'
     escaped.encode('latin-1','surrogateescape')   b'caf\xe9 au lait'
     escaped.encode('cp1252','surrogateescape')    b'caf\xe9 au lait'
     escaped.encode('utf-8','surrogateescape')     b'caf\xe9 au lait'
     escaped.encode('utf-16-le','surrogateescape')
       UnicodeEncodeError: 'utf-16-le' codec can't encode character
         '\udce9' in position 3: surrogates not allowed
     escaped.encode('utf-32-le','surrogateescape')
       UnicodeEncodeError: 'utf-32-le' codec can't encode character
         '\udce9' in position 3: surrogates not allowed

   The four that work include 'ascii', which cannot represent the
   character at all -- the handler returns the raw byte and the codec
   passes it through. The two that fail are the ones whose own output
   is code units, where a lone surrogate is illegal on the way out for
   the same reason it is on the way in.

   Meanwhile every operation that stays inside str succeeds silently:

     len(escaped)                                  12
     escaped.upper()                               'CAF\udce9 AU LAIT'
     escaped.startswith('caf')                     True
     escaped in {escaped: 1}                       True
     json.dumps(escaped)                           '"caf\\udce9 au lait"'

   That last one is a JSON document containing \udce9 -- syntactically
   fine, accepted by json.loads, and impossible to write to a UTF-8
   file. json.dumps did not fail; it deferred.

8. AND PRINTING IT DEPENDS ON THE LOCALE
------------------------------------------------------------------------
   sys.stdout is a codec too, with an errors policy chosen at startup
   from the environment. In this process:

     sys.stdout.encoding  'utf-8'
     sys.stdout.errors    'surrogateescape'

   Both configurations modelled over a buffer, which is all a text
   stream is -- same string, same encoding, two error policies:

     errors=strict
       UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9'
         in position 3: surrogates not allowed
     errors=surrogateescape  wrote 12 bytes, positions 3-4 are e9 20

   So the same print() raises on a machine in a UTF-8 locale and puts
   a raw 0xE9 on the terminal in the C locale, where Python switches
   UTF-8 mode on and hands stdout this very handler. Neither is a bug
   and there is no portable answer, so a program that has to SHOW one
   of these names should ask for the shape it wants:

     escaped.encode('utf-8','backslashreplace')    b'caf\\udce9 au lait'

9. THE ESCAPE IS NOT TAGGED
------------------------------------------------------------------------
   Section 6 said the str keeps no record of where its surrogate came
   from. Here is the consequence, in four lines with no error in them:

     JSON on the wire            "caf\udce9 au lait"
     json.loads gives            'caf\udce9 au lait'
     equal to our filename?      True
     os.fsencode(loaded)         b'caf\xe9 au lait'

   A JSON string may hold any \uXXXX escape, unpaired surrogates
   included, and Python's decoder accepts them. So anything that
   parses JSON and then builds a path can write one arbitrary byte per
   escape into a filename -- a byte that never appeared in the
   document. This is the conflict PEP 383 put out of scope, and it is
   yours: check the name you parsed, not the one you print.

In Rust

Rust has no surrogateescape and cannot have one: char is a Unicode scalar value, so char::from_u32(0xDCE9) returns None and the code points PEP 383 needs do not exist as values. The problem is the same, and the answer is a second type rather than a second encoding — OsStr, which on Unix simply is the bytes, so it round-trips by never converting. What Rust will not do is let those bytes into a String in disguise.

Verified output of surrogateescape_rs.rs — regenerated by tools/run_examples.py, never hand-typed.

1. THE SAME TWELVE BYTES
------------------------------------------------------------------------
     63 61 66 e9 20 61 75 20 6c 61 69 74

     str::from_utf8      Err(invalid utf-8 sequence of 1 bytes from index 3)
     valid_up_to()       3
     error_len()         Some(1)

   Same verdict as Python's strict decode, with the offset in a
   method rather than in a sentence.

2. THE ESCAPE HAS NOWHERE TO LIVE
------------------------------------------------------------------------
   PEP 383 parks the byte at U+DCE9. In Rust that value is not a
   char, and the constructor says so rather than producing one:

     char::from_u32(0xDCE9)   None
     char::from_u32(0x00E9)   Some('é')

   A char is a Unicode scalar value, and the surrogate range is
   the part of the code space that definition removes. So the
   trick is not merely absent from std -- there is no value for
   it to use. String is bytes that promise UTF-8, and a promise
   with an escape hatch in it is not one.

3. WHAT std OFFERS INSTEAD: LOSSY, AND HONEST ABOUT IT
------------------------------------------------------------------------
     from_utf8_lossy          "caf� au lait"
     its bytes                63 61 66 ef bf bd 20 61 75 20 6c 61 69 74
     same as we started with? false

   One byte in, three bytes out: U+FFFD is EF BF BD. This is
   Python's errors='replace', and like it, it is a one-way door.
   Nothing downstream can tell which byte was lost, or how many
   there were.

   The one piece of information std does hand back is whether
   anything was replaced at all, and it is in the return type:
   Cow::Borrowed means the bytes were already UTF-8 and nothing
   was allocated, Cow::Owned means a replacement was made.

     caf\xe9 au lait  Cow::Owned     (a byte was replaced)
     caf au lait      Cow::Borrowed  (nothing was replaced)

4. THE ANSWER IS A SECOND TYPE, NOT A SECOND ENCODING
------------------------------------------------------------------------
     OsStr::from_bytes(RAW)
     name.len()               12
     name.to_str()            None
     name.as_bytes()          63 61 66 e9 20 61 75 20 6c 61 69 74
     byte-identical to RAW?   true

   That last line is the whole design. OsStr round-trips because
   it never converted: on Unix it IS the bytes, and to_str()
   returns Option rather than guessing. Python carries the byte
   through a str by widening what a str may contain; Rust carries
   it by refusing to call it a str at all.

   Path is the same bytes with path methods on it, so a name you
   cannot print is still one you can open, split and rename:

     path.extension()         None
     path.file_name()         true
     path.display()           caf� au lait

   display() is the show-it-anyway door, and it is lossy by
   contract -- the U+FFFD above, not the bytes. It is for humans,
   the way Python's backslashreplace is; neither is a value to
   pass on to anything.

5. THE TRADE, IN ONE TABLE
------------------------------------------------------------------------
     question                     Python 3            Rust
     one type or two?             str                 String + OsString
     undecodable byte becomes     U+DC80..U+DCFF      nothing; it stays a byte
     reversible?                  yes, same handler   yes, no conversion happened
     can it leak into text?       yes -- section 9    no; the types differ
     cost                         a str may hold a    every use site must say
                                  non-character       which of the two it wants

Where it leaks

The escaped str is an ordinary str — len(), .upper(), slicing, sorting and dict keys all work, silently — right up to the moment something turns it back into bytes without being told how. Then:

what you do	what happens
name.encode('utf-8')	UnicodeEncodeError: surrogates not allowed
name.encode('utf-8', 'surrogateescape')	the original bytes
name.encode('latin-1', 'surrogateescape')	the original bytes — the handler is codec-independent on the way out, and works even on ascii, which cannot represent the character at all
name.encode('utf-16-le', 'surrogateescape')	still raises. PEP 383: "Encodings that are not compatible with ASCII are not supported by this specification"
json.dumps(name)	succeeds, producing a document containing \udce9 that no UTF-8 encoder will write
print(name)	depends on the machine — see below

That last row is the one that surprises people, and it is a question about the environment, not about Python. sys.stdout is a codec too, with an errors policy chosen at interpreter startup — and it is not the same policy the filesystem gets:

Measured 2026-09-07 — CPython 3.14 on macOS 26.6 (arm64) and on python:3.14-slim under Docker (Debian, x86-64). Identical on both, which is why the environment is the variable. Not machine-checked: no answer key can hold a value that is chosen from the environment.

LC_ALL          utf8_mode   sys.stdout          filesystem
C               1           utf-8/surrogateescape   utf-8/surrogateescape
C.UTF-8         0           utf-8/surrogateescape   utf-8/surrogateescape
en_US.UTF-8     0           utf-8/strict            utf-8/surrogateescape

Two things to take from that. The filesystem column never moves — os.listdir() decodes with this handler whatever your locale says, so the reading half of this page is not locale-dependent at all; only showing the result is. And the stdout column does not simply track UTF-8 Mode ↗: C.UTF-8 has the mode off and gets the handler anyway, so "C-family locale" is the pattern rather than "UTF-8 Mode", and reaching for sys.flags.utf8_mode to predict it would be wrong.

The practical consequence is that the same print(name) raises on a developer's laptop in en_US.UTF-8 and writes a raw 0xE9 to the log in a container that never set a locale. Neither is a bug and there is no portable answer, so a program that has to show one of these names should ask for the shape it wants — name.encode('utf-8', 'backslashreplace') — rather than hope. This is the same rule as Rust's Path::display(): a rendering for humans, never a value to pass on.

surrogateescape is not surrogatepass

Two handlers one letter apart, both of which succeed, and they do opposite things.

handler	'caf\udce9 au lait' encodes to	what it means
surrogateescape	caf e9 au lait — 12 bytes	put the byte back
surrogatepass	caf ed b3 a9 au lait — 14 bytes	encode the surrogate itself, the spelling WTF-8 ↗ gives an unpaired surrogate

surrogatepass exists for Windows, where filenames are UTF-16 units that may include unpaired surrogates and so are not representable in strict UTF-8 at all; since PEP 529 ↗ the Windows filesystem encoding is utf-8 with surrogatepass. Two platforms, two handlers, one sys.getfilesystemencodeerrors() to ask which you are on.

Mix them and a length changes with no error anywhere: 14 bytes in through surrogatepass, 12 bytes out through surrogateescape. The reason is the finding worth carrying off this page — a str keeps no record of which handler made it. One byte and three bytes become the same object the moment they are inside, and the object is what everything downstream sees.

The escape is not tagged

That is not only a footgun, it is an input. A JSON string may hold any \uXXXX escape, unpaired surrogates included, and Python's json.loads accepts them:

>>> loaded = json.loads('"caf\\udce9 au lait"')
>>> os.fsencode(loaded)
b'caf\xe9 au lait'

Four lines, no error. Anything that parses JSON and then builds a path can write one arbitrary byte per escape into a filename — a byte that never appeared anywhere in the document. It is the shape Canonicalize, then check is about, arriving through the error handler rather than through normalization: the check ran on the text you parsed, and the bytes that reached the kernel were somebody else's. Validate the name you parsed, not the one you print.

If you are coming from Python or ABAP

Python. Say the handler's name at the boundary and nowhere else. os.fsdecode / os.fsencode apply the platform's own pair for you and are what you want for anything path-shaped; sys.getfilesystemencoding() and sys.getfilesystemencodeerrors() report which pair that is. codecs.lookup_error('surrogateescape') returns the handler as an object you can call, which is how the program above shows the sub-128 refusal without a filesystem. And open(path, errors='surrogateescape') is the read that keeps undecodable bytes alive through a whole pipeline — the option grep has no equivalent of. If you only need to look at the data, errors='replace' is fine and errors='ignore' never is: it deletes evidence and reports nothing.

ABAP. (Not machine-checked — CI cannot run ABAP.) There is no equivalent, and the reason is structural rather than an omission: ABAP's string is the UCS-2 subset of UTF-16, so a lone surrogate has no home there either — the same wall Rust's char puts up, reached from the other side. The reversible path is therefore Rust's, not Python's: do not convert. Read the file with OPEN DATASET … IN BINARY MODE into an xstring and keep it as bytes for as long as the job allows, converting only the parts you have a code page for; cl_abap_conv_in_ce with a replacement character is errors='replace', a one-way door, and there is no errors='surrogateescape' behind it. Treat any code-page number you find in a document as something to verify against the system that will run the job.

Try it

• python3 -c "import sys; print(sys.stdout.encoding, sys.stdout.errors)", then again with LC_ALL=C in front of it. That one word is the difference between print() raising and print() writing a byte your terminal cannot draw.
• On Linux: touch $'caf\xe9', then python3 -c "import os; print(ascii(os.listdir('.')))". On macOS the touch fails with Errno 92 — APFS refuses a filename that is not valid UTF-8, so the file this page is about cannot be created on a Mac at all.
• Take a str from os.listdir() and run any(0xDC80 <= ord(c) <= 0xDCFF for c in name). That is the test for "this name is not text", and it is three characters cheaper than a try.
• Feed json.loads('"\\udce9"') into anything in your own code that builds a path from parsed input, and see how far it gets.

Practice

Predict, then run. One str, made the way this page makes them:

name = b"caf\xe9".decode("utf-8", "surrogateescape")   # 'caf\udce9'

For each of the nine ways out below, write down which of three things happens — it raises, it gives the bytes back (63 61 66 e9), or it succeeds and produces something else — and for the ones that succeed, how many bytes:

name.encode('utf-8')                          name.encode('utf-8', 'surrogatepass')
name.encode('utf-8', 'surrogateescape')       name.encode('utf-8', 'backslashreplace')
name.encode('latin-1', 'surrogateescape')     name.encode('utf-8', 'replace')
name.encode('ascii', 'surrogateescape')       name.encode('utf-8', 'ignore')
name.encode('utf-16-le', 'surrogateescape')

Two of the nine raise, and one of those two is the interesting refusal. Two more succeed and are wrong, and one of those two is the same length as the right answer — say which, and say what check in your own code would have caught it.

Then three questions the table does not answer:

1. The escape is U+DC00 + byte, so does every one of the 256 bytes have a spelling? Predict the result for chr(0xDC00), chr(0xDC2F), chr(0xDC7F) and chr(0xDC80) — and if any of them are refused, say what those particular bytes have in common.
2. Given only a str containing U+DCE9, can you tell whether it came from surrogateescape, from a source literal, or from json.loads? Answer before you run anything.
3. Finally, the one to work out on paper: '\udce9'.encode('utf-8', 'surrogatepass').decode('utf-8', 'surrogateescape'). Every step succeeds. How long is the result?

python3 04_Python/surrogateescape/examples/surrogateescape_kata_py.py

Answers

Nothing in this key reads the filesystem, the locale or the clock — deliberately, because the page's own print(name) row is chosen from the environment and is exactly what an answer key may not hold. Every value is shown as hex or ascii(), and the key was diffed byte for byte against CPython 3.11, 3.12, 3.13 and 3.14 before it was recorded.

Verified output of surrogateescape_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

the bytes      63 61 66 e9
the str        'caf\udce9'   len 4

PART ONE -- NINE WAYS OUT

   encode('utf-8')                          UnicodeEncodeError
   encode('utf-8', 'surrogateescape')       4 bytes  63 61 66 e9                the bytes back
   encode('latin-1', 'surrogateescape')     4 bytes  63 61 66 e9                the bytes back
   encode('ascii', 'surrogateescape')       4 bytes  63 61 66 e9                the bytes back
   encode('utf-16-le', 'surrogateescape')   UnicodeEncodeError
   encode('utf-8', 'surrogatepass')         6 bytes  63 61 66 ed b3 a9          something else
   encode('utf-8', 'backslashreplace')      9 bytes  63 61 66 5c 75 64 63 65 39 something else
   encode('utf-8', 'replace')               4 bytes  63 61 66 3f                something else
   encode('utf-8', 'ignore')                3 bytes  63 61 66                   something else

   Three give the bytes back and they use three different codecs, which
   is the first answer: on the way OUT the handler does the work, not
   the codec. `ascii` cannot represent U+DCE9 and returns 0xE9 anyway.

   utf-16-le is the one that breaks the pattern, and PEP 383 says why in
   a sentence -- "encodings that are not compatible with ASCII are not
   supported by this specification". The escape is defined over bytes,
   and UTF-16's unit is two of them.

   Then the two that succeed and lie. `surrogatepass` writes SIX bytes,
   encoding the surrogate itself rather than putting the byte back, and
   `replace` writes FOUR -- the right length, the wrong file. 63 61 66
   3f is `caf?`, which is the same size as the answer, printable, and
   past any check that counts. `ignore` at least gets shorter.

PART TWO -- WHICH BYTES CAN THE ESCAPE CARRY?

   U+DC00  would be byte 0x00   -> UnicodeEncodeError
   U+DC2F  would be byte 0x2f   -> UnicodeEncodeError
   U+DC7F  would be byte 0x7f   -> UnicodeEncodeError
   U+DC80  would be byte 0x80   -> 80
   U+DCE9  would be byte 0xe9   -> e9
   U+DCFF  would be byte 0xff   -> ff

   The arithmetic is U+DC00 + byte, so the obvious guess is that all 256
   have a spelling. Half of them do not: the escape starts at DC80, and
   everything below is refused.

   Bytes under 128 are ASCII: they decode correctly, so they never need
   an escape, and giving them one would only build a way to smuggle
   them. Two of the refusals show what that would cost. Byte 0x2f is `/`
   and byte 0x00 is NUL -- the only two bytes a POSIX filename may not
   contain -- so a handler with a spelling for them would let a decoded
   name become a path the caller never wrote.

PART THREE -- WHERE DID THIS STR COME FROM?

   made by the handler   'caf\udce9'
   typed as a literal    'caf\udce9'
   parsed out of JSON    'caf\udce9'
   all three equal       True
   fsencode(the JSON one) == the original bytes   True

   One object, three provenances, no way to tell them apart -- a str
   keeps no record of which handler made it, and `==` was never going to
   say otherwise. That is the page's finding turned into the question a
   reviewer should ask: the last line is a byte reaching the kernel that
   appeared nowhere in the document it was parsed from.

   So the answer to "which of the three is safe" is none of them, and
   the question is the wrong one. What is safe is validating the name
   you parsed rather than the one you print.

PART FOUR -- ONE CHARACTER IN, THREE OUT, AND NOTHING RAISED

   start                                '\udce9'                   1 character
   .encode(utf-8, surrogatepass)        ed b3 a9                   3 bytes
   .decode(utf-8, surrogateescape)      '\udced\udcb3\udca9'       3 characters
   .encode(utf-8, surrogateescape)      ed b3 a9                   back to the same bytes: True

   Every step succeeded and the bytes still reverse, which is why this
   one is hard to catch: neither handler is wrong on its own. But they
   were asked opposite questions. `surrogatepass` wrote the surrogate
   AS a character, three bytes of UTF-8; `surrogateescape` read those
   three bytes as three undecodable bytes and escaped each one. Length
   1 became length 3, with no exception and no warning anywhere.

   The rule this leaves you with is the page's, stated as an operation
   rather than a fact: the handler is part of the format. Write down
   which one made the bytes, because the str will not remember, and
   `sys.getfilesystemencodeerrors()` only tells you which one the
   PLATFORM would have used.

See also

• Encode, decode and errors — all eight errors= policies in both directions; this page is the one of them that is reversible
• Validation is a boundary — where the UTF-8 check runs, and the same four handlers on the same bad byte
• UTF-16 and surrogates — what the D800–DFFF range is reserved for
• find, and filenames that are bytes — the filesystem half, measured: APFS refuses invalid UTF-8, Linux takes any bytes
• Python text in practice — the same rule as one line of a checklist
• "Handles Unicode" is four questions — the same round trip scored against six languages, and the web's private-use-area version of it
• String is bytes that promise UTF-8 — why the promise has no escape hatch
• OsStr, Path, and WTF-8 — OsStr and Path as an API, where this page uses them only as the contrast: the three doors out, and the Windows side where WTF-8 spells what surrogatepass spells
• Rust strings in practice — the same rules as one line of a checklist
• Canonicalize, then check — the same laundering, one transformation along
• PEP 383 ↗ — the specification, and short enough to read in full
• PEP 540 — UTF-8 Mode ↗ — why the C locale changes what print() does
• PEP 529 ↗ — the Windows half, and why it needed the other handler
• The WTF-8 specification ↗ — what surrogatepass writes