Skip to content

Filenames are not text

Level: 301 · deep dive

One line: A POSIX filename is a bag of bytes with exactly two forbidden values, which is not the same thing as a string — surrogateescape is the trick that lets Python pretend otherwise without losing data, and which bags of bytes your filesystem will actually accept turns out to be the least portable thing in this chapter.

Everywhere else in this chapter, the encoding is a decision your program makes: you choose UTF-8, you call .encode(), and the bytes are the ones you asked for. A filename is the opposite. The bytes already exist, somebody else chose them, and your program's only job is to not damage them on the way through. Python still has to hand you a str, because open() takes a str and pathlib is built on one — so it decodes a name it may have no right to decode, using a scheme designed to be reversible rather than correct.

Verified output of filenames_are_not_text_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. WHAT IS ACTUALLY FORBIDDEN IN A NAME
     byte values a POSIX name component may not contain: 2 of 256
     they are: [0, 47]  =  '/' and NUL
     Everything else is legal. Not 'legal text' -- legal.

2. SO A NAME NEED NOT BE TEXT AT ALL
     raw name from the kernel   b'caf\xe9'
     sys.getfilesystemencoding()  'utf-8'  errors='surrogateescape'
     os.fsdecode(raw)           'caf\udce9'
     U+DCE9 is a lone surrogate -- a code point with no character,
     parked there to hold byte 0xE9 until someone asks for it back.

3. THE ROUND TRIP THAT IS GUARANTEED
     fsencode(fsdecode(b)) == b   for 256 of 256 possible bytes
     That is the whole promise: no name is lost by being decoded.

4. BUT THAT STR IS NOT A STRING YOU CAN USE
     name.encode('utf-8')       raises UnicodeEncodeError
     Path(name)                 'caf\udce9'
     pathlib holds a str, so it holds the surrogate too. open() and
     os.stat() take it happily -- print(), json.dumps() and a socket
     do not. The name is safe to *use* and unsafe to *display*.

5. AND 'THE SAME NAME' IS A FILESYSTEM'S OPINION
     nfc  'café'       4 code points  636166c3a9
     nfd  'café'      5 code points  63616665cc81
     nfc == nfd                 False
     Two different byte strings. Whether they name two different files
     is not Python's decision and not POSIX's -- see the table on the
     page: one filesystem says yes, another says no.

Two byte values, out of 256. That is the entire POSIX rule for a name component: not /, because that separates components, and not NUL, because C strings end there. There is no rule that a name be valid UTF-8, no rule that it be valid in any encoding, and no rule that two people looking at the same directory agree about what it says. A name is a key, and the kernel compares keys as bytes.

surrogateescape is a smuggling scheme, not a decoding. Each byte that cannot be UTF-8 is parked in a lone surrogate — 0xE9 becomes U+DCE9 — in a block of code points that can never appear in real decoded text, so the mapping is unambiguous and reversible. All 256 single bytes survive the round trip, which is the only property it promises. It does not promise you a readable name, and the moment you treat the result as text you find out: .encode("utf-8") raises, print() raises, json.dumps() raises. The str is safe to use and unsafe to display — pass it to open(), and if you have to show it, name.encode(errors="replace") or repr() it.

This is why os.fsencode() and os.fsdecode() exist as a pair. Not str(path) and not bytes(path): those two. They are the round trip the standard library guarantees, and using anything else at that boundary is where a name gets mangled. If your program only ever moves a name from listdir to open, you can leave it as the str Python gave you and never think about this — the whole design is to make the lazy path correct.

What a filesystem actually accepts

The stub this page replaces said a filename may hold "bytes that are not valid UTF-8 in any encoding." That is POSIX, and it is true on Linux. It is false on a Mac, which is worth knowing before you write a test that only passes on one of them. Both halves measured 2026-09-06, same script, one machine:

Measured 2026-09-06 — not verbatim engine output; two runs of the same probe, side by side
                                      macOS 26.6 / APFS        Linux / ext4 (python:3.12-slim)
  sys.getfilesystemencoding()         utf-8, surrogateescape   utf-8, surrogateescape
  create a name holding byte 0xE9     OSError 92               OK
                                      "Illegal byte sequence"  listdir -> 636166e9
  write "café" as NFC (636166c3a9),
    then listdir                      636166c3a9               636166c3a9
  ...then stat() the NFD spelling     OK                       FileNotFoundError

Two findings there, and the second is the one that corrects the folklore.

APFS enforces valid UTF-8. The classic surrogateescape demonstration — make a file whose name is not text, list it, open it — cannot be run on a Mac at all. The kernel refuses at creation with EILSEQ. So the machinery on this page is, on macOS, defensive coding against names that arrive from somewhere else: a mounted ext4 volume, a tarball, an SMB share, a zip written on Linux. It is not dead code, but you cannot reproduce the case locally without a Linux container, and a colleague testing your handler on their Mac will report that your test is broken.

APFS is normalization-insensitive, not normalizing. This is the opposite of what almost every page on the subject says, this library's own Counting characters included until today. HFS+ really did normalize: it decomposed your name to NFD on the way in, and handed NFD back. APFS does not — the bytes come back exactly as written, 636166c3a9 in and 636166c3a9 out. What it does instead is match both spellings, so a name you never wrote also opens the file. The practical difference matters: under HFS+ the bug was "the name I stored changed", and under APFS it is "two byte strings I can prove are unequal both find the same file, and only one of them is in my database." Compare filenames by normalizing both sides, not by trusting either.

The name Finder shows you is not the name on disk

A related surprise from the same family, and this one is unchanged since the HFS+ days even though its usual explanation is not. Create a file from the shell with a colon in it and ask Finder what it is called:

Measured 2026-09-06 on macOS 26.6, APFS — abridged session
$ touch 'a:b'
$ ls
a:b
$ osascript -e 'tell application "Finder" to get name of (POSIX file "…/a:b" as alias)'
a/b

The shell sees a:b and Finder sees a/b, of the same file. The swap is done for display, in the Cocoa file-manager layer, because / is the shell's separator and : was the classic Mac OS one — the reason usually given is "HFS+ uses the colon as a separator", which is history rather than mechanism, since the volume here is APFS. What it means for a program is small but sharp: a filename a user reads to you over the phone may not be the filename in the directory entry, and the byte that differs is one of the two POSIX forbids.

If you are coming from ABAP

There is no equivalent, and the absence is the lesson. OPEN DATASET takes a string, the name is whatever the application server's code page can express, and the platform underneath is a single well-known one per system — so the question "what if the name is not text?" does not arise in ABAP the way it does in a Python job that reads a directory somebody else filled. Where it does bite is the boundary: a file dropped by an external system onto the application server, or read from a share, whose name holds a character the system code page cannot represent. That is the same failure as this page's, arriving through the only door ABAP leaves open for it. Check the file's name against the system's code page before you assume OPEN DATASET will find it — and verify that code page on the system rather than trusting a number from documentation. (Not machine-checked — CI cannot run ABAP.)

Try it

  1. Run the probe yourself on both platforms: create a file named b"caf\xe9" in a temporary directory, list it, and open it again by the name listdir gave back. On a Mac, catch the OSError and print errno. Then run the same script under docker run --rm -v "$PWD":/w -w /w python:3.12-slim, and watch it succeed.
  2. Write the two-line function that makes a name safe to print without ever using it to open anything. Which of errors="replace", errors="backslashreplace" and repr() would you want in a log file you might later have to paste back into a shell?
  3. os.listdir(".") returns str; os.listdir(b".") returns bytes. Find a case where the second is the only correct call. (Hint: you are writing something that copies names verbatim from one directory to another and must not normalize anything.)

See also