From UTF-8, and lossy

Level: 201 · working knowledge

One line: String::from_utf8 returns a Result because bytes might not be text; String::from_utf8_lossy replaces what it cannot read with U+FFFD; and from_utf8_unchecked is you signing the promise yourself, with unsafe as the signature.

Three functions, one input, one output type — and three completely different things done about failure. Choosing between them is not a style question. It is the decision about what happens to a byte your program cannot read, and it has to be made somewhere; these three are simply the three places std will let you put it.

Check, replace, or promise

	returns	on bad bytes	what you still have afterwards
String::from_utf8(v)	Result<String, FromUtf8Error>	fails	the bytes — the error owns them, into_bytes() gives them back
String::from_utf8_lossy(&v)	Cow<'_, str>	replaces with U+FFFD	a string, and a Cow::Owned flag saying something was replaced
unsafe { String::from_utf8_unchecked(v) }	String	nothing — it does not look	undefined behaviour, if you were wrong

The first row is the one to reach for by default, and the reason is in its last column rather than its second. FromUtf8Error does not merely tell you the conversion failed; it is still holding your Vec<u8>, so nothing has been consumed and every other option is still open — including deciding, three frames up, that this file was Latin-1 all along. from_utf8_lossy has already thrown that away by the time it returns.

The error is a measurement

Utf8Error carries two numbers and no advice.

valid_up_to() is a byte offset: everything before it is well-formed UTF-8, guaranteed, so &bytes[..e.valid_up_to()] can be turned into a &str with no further check. That is the number a tool prints when it tells you which byte of a file stopped making sense.

error_len() is the verdict, and it is the half that gets missed. Some(n) means these n bytes are definitely not text — skip them and carry on. None means the input ended in the middle of a character, and more bytes might well fix it. Those are not two flavours of the same failure; a program that treats them alike is either corrupting valid files or accepting invalid ones.

That distinction is why the two functions in the table are enough to build the ones std leaves out. errors='ignore' is "advance past error_len() bytes"; errors='backslashreplace' is the same loop with the skipped bytes printed. Both are a short loop over those two numbers, and the Rust example below writes one of them out in full.

In Rust

Verified output of from_utf8_and_lossy_rs.rs — regenerated by tools/run_examples.py, never hand-typed.

1. THREE FUNCTIONS, THREE CONTRACTS
   String::from_utf8(v)             -> Result<String, FromUtf8Error>
       CHECK. Returns the bytes to you on failure; nothing is lost or changed.
   String::from_utf8_lossy(&v)      -> Cow<str>
       REPLACE. Always succeeds, and the bad bytes are gone for good.
   String::from_utf8_unchecked(v)   -> String            (unsafe)
       PROMISE. No check at all. You are the proof, and a wrong one is
       undefined behaviour rather than a panic.
   Same input, same output type, three different things done about failure.

2. THE ERROR IS TWO NUMBERS, NOT A SENTENCE
   case                  bytes                  valid_up_to  error_len
   valid                 63 61 66 c3 a9                  Ok          -
   Latin-1 é             63 61 66 e9 20 61 75             3          1
   cut mid-character     e0 b2                            0       None
   bad continuation      e0 b2 28                         0          2
   surrogate, as UTF-8   ed a0 80                         0          1
   three stray bytes     80 80 80                         0          1
   valid_up_to() is a byte offset: everything before it IS text, and slicing
   there is safe without another check. That is the number a tool reports when
   it says which byte of your file stopped making sense.

3. AND error_len() IS A VERDICT, WHICH IS THE HALF PEOPLE MISS
   Some(n)   these n bytes are DEFINITELY not text. Skip them and carry on.
   None      the input just ENDED mid-character. More bytes might fix it.

   Same three bytes, arriving whole and arriving in two reads:
     all 4      f0 9f 98 80  -> from_utf8 is Ok
     first 1    f0           -> valid_up_to 0  error_len None
     first 2    f0 9f        -> valid_up_to 0  error_len None
     first 3    f0 9f 98     -> valid_up_to 0  error_len None
   Every prefix says None. A chunked reader that treats None as an error
   corrupts every character that straddles a read boundary; one that treats
   it as 'hold these bytes and ask again' is correct. That is the whole
   difference between a decoder and an incremental decoder.

4. LOSSY, AND HOW MANY U+FFFD IT SPENDS
   case                  bytes                  U+FFFD   result
   valid                 63 61 66 c3 a9              0   "café"
   Latin-1 é             63 61 66 e9 20 61 75        1   "caf� au"
   cut mid-character     e0 b2                       1   "�"
   bad continuation      e0 b2 28                    1   "�("
   surrogate, as UTF-8   ed a0 80                    3   "���"
   three stray bytes     80 80 80                    3   "���"
   The count is not one per bad byte, and not one per attempt. It is one per
   MAXIMAL SUBPART: the longest prefix of the bad run that could still have
   become a character. 'e0 b2' is one such prefix, so one mark; three stray
   continuation bytes are three failed starts, so three. The Unicode Standard
   makes this a RECOMMENDATION and not a rule, so it is worth checking that
   your other language agrees rather than assuming it does.

5. Cow: THE RETURN TYPE TELLS YOU WHETHER ANYTHING HAPPENED
   valid                 Borrowed  — the bytes were already text, nothing allocated
   Latin-1 é             Owned     — a replacement was made, so a new String exists
   cut mid-character     Owned     — a replacement was made, so a new String exists
   bad continuation      Owned     — a replacement was made, so a new String exists
   surrogate, as UTF-8   Owned     — a replacement was made, so a new String exists
   three stray bytes     Owned     — a replacement was made, so a new String exists
   So `matches!(s, Cow::Owned(_))` is the question 'was my input damaged?',
   asked after the fact and for free. from_utf8_lossy is the only repair in
   std that hands back that flag; the Result version hands back the bytes.

6. THE HANDLER std DOES NOT SHIP, BUILT FROM THOSE TWO NUMBERS
   valid                 café
   Latin-1 é             caf\xe9 au
   cut mid-character     \xe0\xb2
   bad continuation      \xe0\xb2(
   surrogate, as UTF-8   \xed\xa0\x80
   three stray bytes     \x80\x80\x80
   That loop is short, and it is reversible where lossy is not: every byte is
   still named. Python spells this errors='backslashreplace' and ships eight
   such handlers; Rust ships one, and the two numbers you need to write the
   others yourself. That is the trade, and it is the same one the whole
   language makes: fewer defaults, and the parts in reach.

7. AND THE unsafe ONE
   from_utf8_unchecked on bytes that ARE valid  -> "café"   (sound, and free)
   from_utf8_unchecked on bytes that are NOT     -> undefined behaviour.
   Not a panic, not a wrong answer, not an error you can catch: a &str whose
   promise is false, handed to code that is allowed to assume it. The right
   use is a hot path where something upstream already checked and you can
   point at the check. 'It has always been UTF-8 so far' is not the check.

Section 3 is the one that costs money in production. Every prefix of a four-byte character reports error_len() == None, so a reader that pulls a fixed number of bytes at a time and calls from_utf8 on each block fails whenever a block boundary lands inside a character. How often that is depends entirely on the text: never for pure ASCII, which is why the bug survives every test written in English, and routinely for Polish, Greek or Japanese, where most characters are more than one byte wide. The fix is not a bigger buffer — a bigger buffer changes the rate and not the outcome. It is to hold the valid_up_to().. tail and prepend it to the next read, which is what an incremental decoder is.

Section 4 is a number worth knowing rather than guessing: from_utf8_lossy spends one U+FFFD per maximal subpart, not one per bad byte. e0 b2 is a single truncated three-byte sequence, so one mark; 80 80 80 is three separate failed starts, so three. The rule comes from the Unicode Standard's Best Practices for Using U+FFFD ↗ — and it is a recommendation, not a conformance requirement, so two decoders may legitimately disagree about how many marks one broken run deserves. The next section is that recommendation checked against a second implementation rather than assumed.

Section 5 is Cow doing something more useful than saving an allocation. from_utf8_lossy returns Cow::Borrowed when the bytes were already text and Cow::Owned when a replacement was made — so matches!(s, Cow::Owned(_)) answers "was my input damaged?" after the fact and for free. Nothing else in std hands you that flag. It is the reason to bind the result before using it rather than writing String::from_utf8_lossy(&v).to_string(), which discards the one piece of information the repair produced.

In Python

Verified output of from_utf8_and_lossy_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. THE SAME SIX INPUTS, AND THE SAME OFFSET
   case                  bytes                   start   end  end-start
   valid                 63 61 66 c3 a9             ok     -          -
   Latin-1 é             63 61 66 e9 20 61 75        3     4          1
   cut mid-character     e0 b2                       0     2          2
   bad continuation      e0 b2 28                    0     2          2
   surrogate, as UTF-8   ed a0 80                    0     1          1
   three stray bytes     80 80 80                    0     1          1
   `start` is Rust's valid_up_to(), byte for byte, on every row. `end - start`
   is Rust's error_len() on every row where Rust reports a number. Two languages,
   two spellings, one measurement — because the measurement is a property of the
   BYTES and not of whoever is reading them.

2. EXCEPT ON THE ROW WHERE RUST HAS A THIRD ANSWER
   case                  bytes                 valid prefix?   what that means
   valid                 63 61 66 c3 a9                    -   already text
   Latin-1 é             63 61 66 e9 20 61 75             no   invalid — nothing appended makes this text
   cut mid-character     e0 b2                           yes   INCOMPLETE — more bytes could still fix it
   bad continuation      e0 b2 28                         no   invalid — nothing appended makes this text
   surrogate, as UTF-8   ed a0 80                         no   invalid — nothing appended makes this text
   three stray bytes     80 80 80                         no   invalid — nothing appended makes this text
   Rust puts that column in the return type: error_len() is None for the rows
   marked yes and Some(n) for the rows marked no. Python's exception has no field
   for it — `end` just runs to the end of the input — so the question has to be
   asked of a different object, the incremental decoder, with final=False.

3. WHICH IS THE BUG A CHUNKED READ MAKES
   '\U0001F600'.encode() = f0 9f 98 80   — four bytes, one character
   split     chunk 1      chunk 2      decode each      incremental
   after 1   f0           9f 98 80     raises / raises  '😀'
   after 2   f0 9f        98 80        raises / raises  '😀'
   after 3   f0 9f 98     80           raises / raises  '😀'
   BOTH halves raise, at every cut: one is a start byte with nothing after it,
   the other is continuation bytes with nothing before them. That is what
   `for chunk in f: chunk.decode()` meets at a read boundary. The last column is
   an IncrementalDecoder holding the partial sequence until the rest arrives —
   same bytes, and only one of the two columns is the file.

4. THE REPLACEMENT COUNT, WHICH BOTH LANGUAGES GET FROM THE SAME RULE
   case                  bytes                  U+FFFD   result
   valid                 63 61 66 c3 a9              0   'café'
   Latin-1 é             63 61 66 e9 20 61 75        1   'caf� au'
   cut mid-character     e0 b2                       1   '�'
   bad continuation      e0 b2 28                    1   '�('
   surrogate, as UTF-8   ed a0 80                    3   '���'
   three stray bytes     80 80 80                    3   '���'
   Compare that column against the Rust program's: identical on all six. Both
   follow the Unicode Standard's recommendation of one U+FFFD per maximal
   subpart, which is a recommendation and not a requirement — so this is a
   measurement of two implementations agreeing, not a guarantee that any two do.

5. AND THE HANDLERS RUST DOES NOT SHIP
   errors=strict             UnicodeDecodeError at byte 3
   errors=replace            'caf� au'
   errors=ignore             'caf au'
   errors=backslashreplace   'caf\\xe9 au'
   errors=surrogateescape    'caf\udce9 au'
   (Those are reprs, so backslashreplace's single backslash is shown doubled and
   surrogateescape's parked byte is shown as the escape \udce9 — printing that
   string to a UTF-8 stream would raise — see 'Bytes that are not text'.)
   Rust's std has the first two, under the names from_utf8 and from_utf8_lossy.
   `ignore` and `backslashreplace` are a short loop over valid_up_to/error_len;
   `surrogateescape` cannot be written at all, because the code points it parks
   bytes in are the ones a Rust char is defined not to hold.

Section 1 is the point of putting these two pages side by side. Utf8Error::valid_up_to() and UnicodeDecodeError.start are the same number on every row, and error_len() is end - start on every row where Rust reports one. That is not a coincidence and not an API convention either language chose: it is a property of the bytes, and both languages are reporting the same measurement of the same thing in different words. Learn the number once and you can read either error.

Section 2 is where they part, and it is a genuine gap rather than a spelling difference. Rust distinguishes incomplete from invalid in the return type; Python's UnicodeDecodeError has no field for it, and end simply runs to the end of the input either way. The question is answerable in Python — codecs.getincrementaldecoder('utf-8')() with final=False succeeds on a valid prefix and raises on a genuinely bad one — but you have to know to ask a different object, which is why so much Python treats a chunk boundary as a corrupt file.

Section 4 is the maximal-subpart rule holding on both sides: identical U+FFFD counts across all six inputs. Since the standard only recommends the behaviour, that is a measurement of two implementations agreeing rather than a guarantee that any two will, and it is worth re-running against a third language before relying on the count.

If you are coming from Python or ABAP

Python. bytes.decode('utf-8') is String::from_utf8, and the mapping of the errors= argument onto Rust is short because Rust's std ships one handler: strict is from_utf8, replace is from_utf8_lossy, and there is no third. ignore and backslashreplace are a loop over valid_up_to() / error_len() — the Rust example writes one out — and surrogateescape cannot be written at all, because it works by parking each undecodable byte at U+DC80..U+DCFF and a Rust char is defined not to hold those. That is not an omission; it is the same refusal the type system makes everywhere else, and Bytes that are not text is the page about what Rust does instead. The other real difference is where the failure lands: Python raises, so an un-caught bad byte kills the call stack from wherever it happened; Rust returns, so the failure is a value your function signature has to admit to. Porting a try/except UnicodeDecodeError usually means deciding, for the first time, which layer actually owns the decision.

ABAP. (Not machine-checked — CI cannot run ABAP.) The conversion is cl_abap_conv_in_ce (bytes → text) or the newer cl_abap_codepage=>convert_from( ), and the shape is Python's rather than Rust's: a bad byte raises cx_sy_conversion_codepage, so the choice is between a TRY/CATCH and setting a replacement character up front — which is errors='strict' and errors='replace', and there is nothing corresponding to from_utf8_lossy's Cow flag telling you afterwards that a replacement happened. There is also no ABAP equivalent of valid_up_to(): the exception says the conversion failed, not which byte offset it failed at, so locating the bad record in a large xstring means bisecting it yourself. Given that, the ABAP habit worth importing from this page is Rust's first row — keep the xstring until the conversion has succeeded, so a failure still leaves you holding the bytes. And treat any code-page number you find in a document as something to verify against the system that will run the job.

Try it

1. Take a file of yours that a tool once refused — a CSV, a log, an export. Read it with std::fs::read (not read_to_string), call std::str::from_utf8, and print valid_up_to(). Then xxd the four bytes either side of that offset and see what is actually there.
2. Run the same file through from_utf8_lossy and count the U+FFFDs. If the count is small, you have a handful of bad bytes; if it is enormous, the file is probably not broken UTF-8 at all but a different encoding read as UTF-8 — which is a mojibake problem and needs the bytes back, not a repair.
3. Write the four-line errors='ignore' version — same loop as the example's backslash_replace, with the bad bytes dropped instead of printed — and run it beside from_utf8_lossy on the same input. The lengths will differ, and the difference is exactly the number of replacement characters.
4. Find a from_utf8_lossy(...).to_string() or a .into_owned() in code you own. Ask whether the Cow it threw away was the only place that run could have noticed its input was damaged.

Practice

Three contracts, one input. For the four bytes 63 61 66 e9 — caf followed by a Latin-1 é — write down what each of String::from_utf8, String::from_utf8_lossy and from_utf8_unchecked gives you, and what you are still holding afterwards in each case.

Then the two numbers. Give valid_up_to() and error_len() for 63 61 66 e9, and for 63 61 66 e9 20. They differ, and the reason is the whole lesson — say what changed. Finally: from_utf8_lossy(b"\x80\x80\x80") produces how many replacement characters, and why is that not one?

Answers

Verified output of from_utf8_and_lossy_kata_rs.rs — regenerated by tools/run_examples.py, never hand-typed.

THE INPUT: 63 61 66 e9   'caf' and a Latin-1 e-acute

1. from_utf8 — CHECK
   Err(FromUtf8Error). Nothing was converted and nothing was lost:
   into_bytes() returns 63 61 66 e9 — identical to the input? true
   You still hold the bytes, so you can still decide this file was Latin-1.

2. from_utf8_lossy — REPLACE
   "caf�"   6 bytes out of 4 in
   Cow::Owned — so something WAS replaced
   One byte in, three out: U+FFFD is ef bf bd. The e9 is gone, and no
   later code can tell it was ever there or what it was.

3. from_utf8_unchecked — PROMISE
   A String whose bytes are not UTF-8. Not a panic and not a wrong answer:
   undefined behaviour, because every &str method is allowed to assume the
   promise. Correct only where something upstream already checked and you
   can point at the check. Nothing is run here, deliberately.

4. THE TWO NUMBERS, AND WHY ONE BYTE CHANGES THEM
   63 61 66 e9     valid_up_to 3   error_len None
   63 61 66 e9 20  valid_up_to 3   error_len Some(1)
   valid_up_to is 3 both times: 'caf' is text either way.
   error_len changes because e9 is a legal START byte — it opens a three-byte
   sequence. On its own at the end of input, the decoder cannot yet say the
   input is wrong, only that it is INCOMPLETE, so it answers None. Append any
   byte that is not a continuation (0x20 is a space) and the sequence can
   never be completed, so the verdict becomes Some(1): one byte, definitely
   not text. None means 'read more'; Some(n) means 'skip n and carry on'.

5. THREE STRAY CONTINUATION BYTES
   from_utf8_lossy(80 80 80) -> "���"
   3 replacement characters, not one.
   The rule is one U+FFFD per MAXIMAL SUBPART — the longest prefix that could
   still have become a character. 0x80 is a continuation byte, so it cannot
   begin anything; each one is a failed start of its own and gets its own
   mark. Compare a truncated sequence, where the bytes DO belong together:
     e0 b2        -> 1 mark
     f0 9f 98     -> 1 mark
   Two and three bytes, one mark each, because each is one interrupted
   character. The count is about structure, not about how many bytes went bad.

See also

• String is bytes that promise UTF-8 — the promise these three functions make, keep or skip
• char is four bytes — why surrogateescape has no Rust spelling
• Slicing by byte — the other place a byte offset has to land on a boundary
• Encode, decode and errors — all eight Python handlers, and what each one costs
• Bytes that are not text — the handler Rust answers with a second type instead
• Validation is a boundary — the same check, in four languages and one shell command
• Rust strings in practice — where these three sit in a checklist
• String::from_utf8 ↗ — the standard library's own words, including what unsafe is promising