`OsStr`, `Path`, and WTF-8¶

Level: 301 · deep dive

One line: A filename is whatever the operating system will accept, which on Windows can include a code unit no UTF-8 can represent — so Path is not a String, to_string_lossy is a decision rather than a convenience, and into_string is the only door out that hands the name back when it fails.

Everything else in this chapter is about a promise Rust makes and keeps. This page is about the one place it declines to make one, on purpose, because the promise would be false: the operating system did not agree to hand you text.

Four pairs, and only one of them has promised anything¶

owned	borrowed	what has been promised about the bytes
`Vec<u8>`	`&[u8]`	nothing at all
`OsString`	`&OsStr`	whatever this OS accepts as a name
`PathBuf`	`&Path`	the same, and separators mean something
`String`	`&str`	valid UTF-8

Read down the last column and the design falls out. &Path is not an inconvenient &str; it is a different point on the same axis, one promise weaker, and it exists because the kernel's guarantee is weaker than UTF-8. On Unix a filename is any bytes except NUL and /. On Windows it is any sequence of 16-bit code units, and Windows does not require surrogates to be paired ↗ — so a name can hold half of a character, and no valid UTF-8 exists for it.

The API consequence is one line: take &Path in a signature the way you take &str, and let the caller pass a &str, a String, a PathBuf or an OsString — AsRef<Path> covers all four, and a function that takes &str has quietly excluded every name it cannot spell.

In Rust¶

Verified output of osstr_path_and_wtf8_rs.rs — regenerated by tools/run_examples.py, never hand-typed.

1. FOUR PAIRS, ONE SHAPE — OWNED, AND BORROWED
   owned      borrowed     owned   borrow   what has been promised about the bytes
   Vec<u8>    &[u8]           24       16   nothing at all
   OsString   &OsStr          24       16   whatever the OS accepts as a name
   PathBuf    &Path           24       16   the same, plus separators mean something
   String     &str            24       16   valid UTF-8
   The same two-type pattern four times over, and the same two sizes: an owned
   one is a pointer, length and capacity; a borrowed one is a pointer and a
   length. Only the last row has made a promise, and only it can be printed
   without a decision. Take &Path in a signature the way you take &str.

2. THREE DOORS OUT OF AN OsStr, AND ONLY ONE HANDS IT BACK
   the name: 63 61 66 e9 2e 74 78 74   (8 bytes)
   to_str()          None
      -> Option<&str>. None means 'not UTF-8'. You have LOST the name unless
         you still hold the OsStr — the None carries nothing.
   to_string_lossy() "caf�.txt"
      -> Cow<str>. Always succeeds. The e9 has become ef bf bd:
         63 61 66 ef bf bd 2e 74 78 74
         You have CORRUPTED the name. Open that and you get ENOENT.
   into_string()     Err(OsString) — 8 bytes, identical to the input? true
      -> Result<String, OsString>. The failure case RETURNS THE VALUE. That is
         the door to use when you need a String and might not get one, because
         it is the only one that leaves you able to carry on with the name.
   Same three contracts as from_utf8 / from_utf8_lossy / the unsafe one, one
   type along — check, replace, or (here) check and hand it back.

3. THE Cow IS A DECISION, AND IT TELLS YOU WHICH ONE IT MADE
   name                        Cow          what that means for you
   70 6c 61 69 6e 2e 74 78 74  Borrowed     already UTF-8; the &str points at the same bytes
   63 61 66 c3 a9 2e 74 78 74  Borrowed     already UTF-8; the &str points at the same bytes
   63 61 66 e9 2e 74 78 74     Owned        a new String was built, and it is NOT this name
   ff fe                       Owned        a new String was built, and it is NOT this name
   So `matches!(c, Cow::Owned(_))` is 'this name cannot be represented', asked
   without a second pass. Calling .to_string() or .into_owned() on the result
   throws that answer away — which is why to_string_lossy() is a decision and
   not a convenience: the convenient spelling is the one that loses the flag.

4. Path IS THOSE BYTES WITH PATH METHODS ON THEM
   path              2f 76 61 72 2f 6c 6f 67 2f 63 61 66 e9 2e 74 78 74
   .parent()         Some("/var/log")
   .file_name()      8 bytes
   .file_stem()      63 61 66 e9
   .extension()      Some("txt")   <- a &OsStr that happens to be ASCII
   .is_absolute()    true
   PathBuf::push + set_extension -> 2f 76 61 72 2f 6c 6f 67 2f 63 61 66 e9 2e 67 7a
   Every one of those worked on a name no String can hold. Splitting, joining
   and opening never look inside a component at all; only DISPLAYING needs
   the name to be text, which is why display() is the method with the
   disclaimer attached. Comparing is the one with a wrinkle — next section.

5. AND EQUALITY TIDIES THE SEPARATORS AND NOTHING ELSE
   Path::eq compares components(), not raw bytes, so a little SYNTAX is
   normalized on the way — and only syntax:
     "a/b"     == "a//b"      true
     "a/b"     == "a/b/"      true
     "a/b"     == "a/./b"     true
     "a/b"     == "./a/b"     false
     "a/c"     == "a/b/../c"  false
     "a.txt"   == "A.TXT"     false
   Repeated separators, a trailing one and an inner '.' all collapse; a
   LEADING './' does not, and neither does '..' — because the thing before it
   might be a symlink, so std refuses to guess. Case is untouched too, even
   though the volume this ran on may not care.

   And then each component is compared as BYTES, which is where it bites:
     composed    11 bytes   c5 bc c3 b3 c5 82 77 2e 74 78 74
     decomposed  13 bytes   7a cc 87 6f cc 81 c5 82 77 2e 74 78 74
     identical on screen, and nfc == nfd is false
   Nothing in std will tell you those are the same word. On macOS APFS they
   are the SAME FILE — the filesystem is normalization-insensitive — while on
   Linux they are two files. So a program that compares paths itself and a
   program that opens them can reach opposite conclusions on one machine.
   The rule that survives all of it: a Path comparison answers a question
   about NAMES. If you meant to ask about files, ask the OS — canonicalize.

6. as_encoded_bytes(), AND WHY IT IS NOT CALLED as_bytes()
   as_encoded_bytes()  63 61 66 e9 2e 74 78 74
   Portable — it exists on every platform, where as_bytes() is Unix-only. But
   read its contract before you use the result for anything: the standard
   library calls the encoding 'an unspecified, platform-specific,
   self-synchronizing superset of UTF-8', and says any sub-slice that is not
   valid UTF-8 should be treated as OPAQUE and 'only comparable within the same
   Rust version built for the same target platform'.
   So: fine for searching, splitting on ASCII, hashing in memory. Not a value
   to write to a file, put in a database or send over a network — and that is
   why from_encoded_bytes_unchecked is unsafe, and why the method's name says
   'encoded' rather than pretending these are simply the bytes.

7. WHAT THAT SUPERSET IS, ON THE PLATFORM THIS PROGRAM CANNOT RUN ON
   A Windows filename is a sequence of UTF-16 code units, and NTFS does not
   check that surrogates are paired — so a name may contain a LONE surrogate,
   which no valid UTF-8 can represent. Rust stores those names as WTF-8.

   WTF-8 is UTF-8's arithmetic with the surrogate ban lifted. Encoding U+D800
   through the ordinary three-byte template, by hand:
     U+D800  ->  ed a0 80
     and UTF-8 refuses those same three bytes: from_utf8 is Ok? false
   That is the whole of it — a real encoding with a real spelling, which every
   UTF-8 decoder is required to reject. Python spells the same three bytes
   errors='surrogatepass'. Rust never lets them into a String at all, and
   keeps them in OsString instead, which is the reason the type exists.

   The Windows-only API is a different shape for the same reason: there is no
   as_bytes()/from_bytes() there, only encode_wide() and from_wide(), which
   work in u16 code units — because on that platform the bytes were never the
   thing the OS handed you. See the lesson for the facts that cannot be run
   on the two machines CI has.

The three doors, and which one to walk through¶

Section 2 is the part worth memorising, because the three methods look interchangeable and are not:

to_str() -> Option<&str> — the cheap check. The None carries nothing, so if you have dropped the OsStr the name is gone. Right when you are filtering — "process only the files I can name".
to_string_lossy() -> Cow<'_, str> — always succeeds, and the returned string is not the name any more: the bad bytes are U+FFFD now, and opening it gives ENOENT. Right for a log line, an error message, a progress display. Never right for a value you pass on.
into_string() -> Result<String, OsString> — the one that hands it back. The Err variant is your OsString, unharmed, so you can report the failure and continue working with the name.

That is check / replace / check-and-return, which is from_utf8 / from_utf8_lossy one type along — except that the third door here is strictly better than from_utf8's, because Result<String, OsString> returns something you can keep using rather than a byte vector you have to re-wrap.

`Cow` is the decision, and `.to_string()` throws it away¶

to_string_lossy returns Cow::Borrowed when the name was already UTF-8 and Cow::Owned when it had to substitute — so the return type is also the answer to "could this name be represented?", computed for free by the work that was already being done. matches!(c, Cow::Owned(_)) is that question.

Which is why the method is a decision and not a convenience: the convenient spelling, path.to_string_lossy().to_string(), discards the only place your program could have noticed. If a name is going into a database, a report or a shell command, the Owned branch is the one where you have to decide what a corrupted name means — and the borrow-or-not is telling you which names those are before you look at any of them.

And `Path` equality tidies the separators, and nothing else¶

Section 5 is the one that catches people who have done everything else right, and it is more interesting than "it compares bytes" — which was this page's first draft, and which the program above disproved in its own output. Path::eq compares components(), and components() does a small, documented amount of normalisation ↗: repeated separators are ignored, a trailing one is dropped, and an inner . collapses. So a/b, a//b, a/b/ and a/./b are all one path.

That is the whole of it. A leading ./ survives, because it is a real CurDir component; .. survives, because a/b/.. need not be a if b is a symlink and std will not guess; case survives; and each surviving component is then compared as bytes. So żółw.txt composed (11 bytes) and decomposed (13 bytes) are unequal, on every platform, and nothing in std will tell you they are the same word.

The trap is that the filesystem may disagree with your program on the same machine. macOS's APFS is normalization-insensitive — creating both spellings leaves one file — while Linux leaves two. So a program that compares paths as data and a program that opens them can reach opposite conclusions in the same directory, and only on one of the two platforms. The rule that survives both: a Path comparison answers a question about names. If the question was about files, ask the OS — fs::canonicalize, or just open them.

The Windows half, which CI cannot run¶

This library runs every example on Ubuntu and macOS (CONTRIBUTING.md), so the platform this page's title comes from is the one platform nothing here executes on. The program above can show the WTF-8 arithmetic — U+D800 encoded through UTF-8's own three-byte template is ED A0 80, and str::from_utf8 rejects exactly those bytes — because that is arithmetic and a refusal, both of which are the same everywhere. Everything below is read off the standard library rather than run, and is marked accordingly.

Read 2026-09-07 from the std source and rustdoc shipped with rustc 1.98.0 (88d9e12ae) — library/std/src/os/windows/ffi.rs, library/std/src/sys/os_str/mod.rs, library/std/src/ffi/os_str.rs. NOT machine-checked: no runner in this repo is a Windows machine, and no answer key here holds a claim about a platform CI does not execute.

OsStr's internal encoding, by target
    windows, uefi     WTF-8            sys/os_str/mod.rs selects mod wtf8
    everything else   the bytes         ...selects mod bytes

Why WTF-8 is needed there
    "the 16-bit code units in Windows strings may contain isolated surrogate
     code points which are not paired together ... For compatibility with code
     that does not enforce these pairings, Windows does not enforce them either."
                                        — std::os::windows::ffi module docs

The Windows-only API is u16, not u8
    OsStringExt::from_wide(&[u16]) -> OsString      stable since 1.0
    OsStrExt::encode_wide()        -> EncodeWide    stable since 1.0
    from_wide is documented lossless: encode_wide on the result "will always
    return the original code units". There is no from_bytes/as_bytes there.

The portable byte view, and its actual contract
    OsStr::as_encoded_bytes()      stable since 1.74
    "The byte encoding is an unspecified, platform-specific, self-synchronizing
     superset of UTF-8 ... any sub-slice of bytes that is not valid UTF-8 should
     be treated as opaque and only comparable within the same Rust version built
     for the same target platform."

And WTF-8 itself says not to send it anywhere
    "Since WTF-8 must not be used for interchange, this library deliberately
     does not provide access to the underlying bytes of WTF-8 strings"
                                        — the std wtf8 module's own doc comment

Three things follow, and the last is the useful one.

WTF-8 is a real encoding with a specification — Simon Sapin's ↗ — and not a joke about the name. It is UTF-8 with the surrogate prohibition lifted, so every valid UTF-8 string is valid WTF-8 and the extra three-byte sequences (ED A0 80 through ED BF BF) are exactly the ones UTF-8 forbids. Python writes the same bytes under errors='surrogatepass', which is what Bytes that are not text records; the two languages agree on the spelling and disagree completely about which type may hold it.

as_encoded_bytes() is not as_bytes(), and the name is the warning. Its contract does not say "UTF-8 on Unix, WTF-8 on Windows" — it says unspecified, and explicitly limits comparability to one Rust version on one target. So the bytes are fine to search, split on ASCII and hash in memory, and are not a value to write to a file, store in a database, or send over a network. That is also why from_encoded_bytes_unchecked is unsafe: the safety condition is not "is this valid UTF-8" but "did these bytes come out of as_encoded_bytes on this build".

And the practical rule for a cross-platform program is to never look. Path will join, split, compare, open, read and rename a name you cannot spell, on both platforms, without ever converting it. The conversion is only needed to show the name to a person — and for that, display() and to_string_lossy() are exactly right, precisely because their output is not a name.

If you are coming from Python or ABAP¶

Python. os.path takes and returns str, so Python's answer to this problem is not a second type but a second codec: undecodable filename bytes are parked at U+DC80..U+DCFF by surrogateescape on Unix, and on Windows PEP 529 ↗ sets the filesystem encoding to utf-8 with surrogatepass — the same three-byte WTF-8 spelling Rust keeps in OsString. The comparison is on Bytes that are not text; what matters here is the API shape it produces. Python's str means you can concatenate, os.path.join, sort and print a filename with no ceremony, and the cost is that a name and a piece of text are the same type, so nothing marks the moment one becomes the other — print(name) is the failure, and it depends on the machine's locale. Rust's cost is the mirror image: Path::new("a").join(x) needs x: AsRef<Path> rather than a &str, and printing needs .display(). If you are porting a script, the useful translation is not method-by-method: it is that every place Python's str silently did both jobs is a place where Rust will make you say which one you meant, and roughly one of those places per script is a real bug.

ABAP. (Not machine-checked — CI cannot run ABAP.) There is no OsStr and no need for one, because ABAP does not hand you the operating system's namespace. OPEN DATASET takes a string for the physical filename, and on an application server that name goes through the platform's own conversion — so a name the system cannot represent is not a case the language gives you a type for; it is an I/O error at open time. The nearest thing to this page's discipline in ABAP is the logical-file-name machinery (FILE_GET_NAME / transaction FILE), which exists so a program never spells a physical path itself, and that is worth reading as the same advice arrived at from the other end: do not construct names as data. Where this page does transfer is the content rather than the name — an xstring read IN BINARY MODE is Vec<u8>, and converting it with cl_abap_conv_in_ce is the decision into_string() makes explicit. Verify any specific code-page number against the system that will run the job.

Try it¶

Run the example. It is a Unix program — std::os::unix::ffi is how an OsStr is built from arbitrary bytes, and Windows has no such constructor, which is section 7's whole point. On Windows it will not compile, and the compile error is the lesson.
Walk a directory you own with fs::read_dir and print, for each entry, whether to_string_lossy() came back Borrowed or Owned. On a tidy disk every one is Borrowed; if any is not, you have found a name your other tools have been guessing at.
Take a function of yours that takes &str and means "a path". Change it to impl AsRef<Path> and see how many call sites get shorter.
Create the same filename twice on your machine — once as you would type it, and once with its accents written as combining marks, which python3 -c "import unicodedata as u; print(u.normalize('NFD', 'żółw'))" will print for you (the ó comes back as an o followed by U+0301). Count the files. Then run the same experiment in a Linux container and count again.

Practice¶

Three doors, one name. A directory entry comes back as the eight bytes 63 61 66 e9 2e 74 78 74 — caf, a Latin-1 é, and .txt. For each of to_str(), to_string_lossy() and into_string(), write down the return type, what you get for this name, and — the part that matters — what you are still holding afterwards.

Then: which of the three would you use to write the name into an error message, and which to keep processing the file? And say what Path::new(a) == Path::new(b) actually compares — first for two names that draw the same word on screen, then for "a/b" against "a//b", "./a/b" and "a/b/../b".

Answers

Verified output of osstr_path_and_wtf8_kata_rs.rs — regenerated by tools/run_examples.py, never hand-typed.

THE NAME: 63 61 66 e9 2e 74 78 74   'caf', a Latin-1 e-acute, and '.txt'

1. to_str()  ->  Option<&str>
   value          None
   still holding  the OsStr, if you kept it. The None itself carries nothing —
                  it does not tell you which byte was wrong, or what the name
                  was. Use it to FILTER, where dropping the name is the point.

2. to_string_lossy()  ->  Cow<'_, str>
   value          "caf�.txt"
   its bytes      63 61 66 ef bf bd 2e 74 78 74
   Cow variant    Owned — so a substitution happened
   still holding  a String that is NOT this file. e9 became ef bf bd, so the
                  name is three bytes longer and opening it gives ENOENT.

3. into_string()  ->  Result<String, OsString>
   value          Err(OsString)
   the Err holds  63 61 66 e9 2e 74 78 74 — byte-identical to the input? true
   still holding  THE NAME. This is the only one of the three whose
                  failure path leaves you able to carry on.

4. SO: WHICH ONE FOR WHICH JOB
   into the error message   to_string_lossy(). Its output is for a person to
                            read, and a person can read caf<?>.txt. Never pass
                            that string to anything that will open a file.
   to keep processing       neither — keep the OsStr or the Path and never
                            convert at all. Path will join, split, compare,
                            open and rename it as it stands. If a String is
                            genuinely required, into_string(), because its Err
                            gives the name back so the fallback still has it.

5. Path EQUALITY COMPARES COMPONENTS, AND EACH COMPONENT AS BYTES
   a  11 bytes  c5 bc c3 b3 c5 82 77 2e 74 78 74
   b  13 bytes  7a cc 87 6f cc 81 c5 82 77 2e 74 78 74
   a == b   false
   Both draw the word zolw with its Polish diacritics, and Path calls them
   different — because Path::eq compares components() and each component is
   compared as BYTES. No Unicode normalization, and no case folding.

   What components() DOES normalize is separator syntax, and only that:
     "a/b"     == "a//b"      true
     "a/b"     == "a/b/"      true
     "a/b"     == "a/./b"     true
     "a/b"     == "./a/b"     false
     "a/c"     == "a/b/../c"  false
     "a.txt"   == "A.TXT"     false
   Repeated and trailing separators collapse, and so does an inner '.'; a
   leading './' does not, and '..' is left alone on purpose — 'b' might be a
   symbolic link, so 'a/b/..' need not be 'a' and std will not pretend it is.

   Every false above except the case one names two spellings that may well be
   the same file, and the case one is the same file on a default macOS or
   Windows volume. So the honest summary is that Path equality answers a
   question about NAMES, tidying only the syntax it can be certain about.
   If you meant to ask about FILES, ask the OS: fs::canonicalize.

OsStr, Path, and WTF-8¶