Validation is a boundary¶
Level: 201 → 301 · for anyone who has decoded a sequence by hand
One line: Every language decodes UTF-8 by the same rules and gets the same answer; what they disagree about is where the check happens and what is left of it afterwards — C never checks, Python checks at the door and forgets, Rust checks at the door and the type remembers.
The slide this page started from¶
Bob Steagall's CppCon 2018 talk Fast Conversion From UTF-8 with C++, DFAs, and SSE Intrinsics ↗ has a slide called Valid Sequence Example. It shows three byte sequences in binary, with the payload bits highlighted, and their code points underneath:
1: 0111.1101 U+007D 0x7D closing brace }
2: 1100.0010 1010.1001 U+00A9 0xC2 0xA9 copyright sign ©
3: 1110.0010 1000.1001 1010.0000 U+2260 0xE2 0x89 0xA0 not equal to ≠
Nothing on that slide is about C++. It is RFC 3629 ↗, and every example on this page decodes those same three sequences and prints those same three answers. The lead byte announces the length in its high bits, each continuation byte contributes its low six, and the payload bits concatenate. That is the whole encoding, and UTF-8 by hand is the page that drills it.
So if the decoding is identical everywhere, why is there a whole conference talk about it? Because the slide is titled Valid Sequence Example, and something has to decide.
What "valid" excludes, and why it is not obvious¶
A byte sequence can fail to be UTF-8 in six ways, and only the first two are the kind of damage you would guess at:
| The bytes | What is wrong |
|---|---|
89 |
a continuation byte with no lead byte in front of it |
E2 89 |
a lead byte announcing three bytes, with only two present |
C0 AF |
2F — an ASCII / — written the two-byte way. An overlong form |
E0 80 AF |
the same /, written the three-byte way |
ED A0 80 |
U+D800, a UTF-16 surrogate, which is not a character |
F5 80 80 80 |
a number above U+10FFFF, the top of Unicode |
The last four are the interesting ones, because the bits decode perfectly well. C0 AF follows the two-byte template exactly and yields 47. It is forbidden anyway — one character must have exactly one encoding, or a security check that rejects a byte sequence can be walked straight past by an overlong spelling of it that some later layer decodes back into the real thing. That is not hypothetical, and the RFC says so itself: section 10 ↗ gives a parser that blocks 2F 2E 2E 2F (/../) and lets 2F C0 AE 2E 2F through, and notes that exactly this was used by a widespread web-server worm in 2001. Which is why the rule is reject, and not canonicalise and continue.
So "is this valid UTF-8?" is a real question with a real table behind it, and every program that reads bytes from outside itself has to answer it exactly once. Where you put that once is the whole design.
| Where the check runs | What is left afterwards | |
|---|---|---|
| C | nowhere, unless you write it | a char *. strlen counts bytes and has no opinion |
| Python | bytes.decode('utf-8') |
a str of code points. The UTF-8 is gone, and a surrogate can still get in |
| Rust | str::from_utf8 |
a &str, whose type is the proof. Everything after skips the check |
| the shell | iconv -f UTF-8 -t UTF-8 |
an exit status, and a byte offset |
In Python¶
Verified output of validation_is_a_boundary_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. THE THREE SEQUENCES FROM THE SLIDE, DECODED
1: 0111.1101 U+007D: 0x7D (})
2: 1100.0010 1010.1001 U+00A9: 0xC2 0xA9 (©)
3: 1110.0010 1000.1001 1010.0000 U+2260: 0xE2 0x89 0xA0 (≠)
1 lead byte + 0/1/2 continuation bytes; the payload bits concatenate, nothing else moves.
2. SIX WAYS TO BE INVALID, AND WHAT THE EXCEPTION KNOWS
89 start=0 end=1 invalid start byte lone continuation byte
e2 89 start=0 end=2 unexpected end of data truncated 3-byte sequence
c0 af start=0 end=1 invalid start byte overlong '/' — two bytes for U+002F
e0 80 af start=0 end=1 invalid continuation byte overlong '/' — the three-byte way
ed a0 80 start=0 end=1 invalid continuation byte UTF-16 surrogate U+D800
f5 80 80 80 start=0 end=1 invalid start byte above U+10FFFF
start is how far the bytes were text; the reason names which rule broke.
Two of the reasons are the rule table talking. C0 is an 'invalid start byte' because
C0 and C1 can only ever begin an overlong form, so no second byte is needed to reject it.
ED is a fine start byte, so ED A0 fails one byte later: after ED the only legal
second bytes are 80..9F, and A0 is where the surrogates begin.
3. THE SAME BAD BYTES, FIVE ERROR HANDLERS
replace -> 'caf� au lait'
ignore -> 'caf au lait'
backslashreplace -> 'caf\\xe9 au lait'
surrogateescape -> 'caf\udce9 au lait'
strict (default) -> raises at byte 3
Only 'strict' preserves the fact that something was wrong. The rest are decisions.
4. AFTER THE BOUNDARY, PYTHON DOES NOT HOLD UTF-8 AT ALL
text = '≠'
len(text) = 1 (code points)
len(text.encode()) = 3 (UTF-8 bytes, rebuilt on demand)
A str is a sequence of code points. The UTF-8 was consumed at decode() and is gone;
every .encode() builds it again. Nothing in the object records that a check ever ran.
5. AND A str CAN HOLD WHAT UTF-8 CANNOT ENCODE
chr(0xD800) = '\ud800' <- built without complaint
.encode('utf-8') -> UnicodeEncodeError: surrogates not allowed
chr(0x10FFFF) = '\U0010ffff' (the top of the range; chr(0x110000) is a ValueError)
So Python checks on the way IN and again on the way OUT, because between the two
the type allows a value no UTF-8 file can contain.
Section 2 is the rule table talking. C0 is reported as an invalid start byte without Python even looking at the next byte, because C0 and C1 can only ever begin an overlong form — they are excluded at the lead. ED is a perfectly good start byte, so ED A0 fails one byte later, as an invalid continuation byte: after ED the only legal second bytes are 80–9F, and A0 is exactly where the surrogates begin. Those two facts are one range table, and you will meet it again in Rust and in the C example, unchanged.
Sections 4 and 5 are the Python half of this page's point. A str is a sequence of code points, not UTF-8 — the encoding was consumed at decode() and every .encode() builds it again — so nothing in the object records that a check ever ran. And the type is wider than UTF-8: chr(0xD800) builds a lone surrogate without complaint, which is why Python has to check a second time, on the way out, and why UnicodeEncodeError: surrogates not allowed exists at all.
In the terminal¶
Verified output of validation_is_a_boundary_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
1. THE THREE SEQUENCES FROM THE SLIDE, AS BYTES ON A PIPE
$ printf '\x7d' | xxd
00000000: 7d }
$ printf '\xc2\xa9' | xxd
00000000: c2a9 ..
$ printf '\xe2\x89\xa0' | xxd
00000000: e289 a0 ...
2. iconv FROM UTF-8 TO UTF-8 IS A VALIDATOR: EXIT 0 MEANS YES
\x7d exit 0 U+007D
\xc2\xa9 exit 0 U+00A9
\xe2\x89\xa0 exit 0 U+2260
\x89 exit 1 lone continuation byte
\xe2\x89 exit 1 truncated 3-byte sequence
\xc0\xaf exit 1 overlong '/'
\xe0\x80\xaf exit 1 overlong '/' the 3-byte way
\xed\xa0\x80 exit 1 UTF-16 surrogate U+D800
\xfe exit 1 a byte that can never appear in UTF-8
3. AND THE PLACE WHERE iconv IS NOT THE SAME VALIDATOR
\xf4\x90\x80\x80 exit 0 U+110000 - one past the top of Unicode
\xf5\x80\x80\x80 exit 0 U+140000
\xf7\xbf\xbf\xbf exit 0 U+1FFFFF
Exit 0, on macOS and on Linux both: iconv accepts 4-byte sequences that encode
a number above U+10FFFF. Python and Rust reject all three. Neither side is
confused - iconv is checking the older, wider UTF-8 that ran to 31 bits, and
RFC 3629 (2003) capped the encoding at U+10FFFF to match what UTF-16 can name.
"Valid UTF-8" is not one question. Say which validator you asked.
4. WHAT IT WROTE BEFORE IT STOPPED IS valid_up_to
$ printf 'caf\xc3\xa9 \xe9 oops' | iconv -f UTF-8 -t UTF-8 | xxd
00000000: 6361 66c3 a920 caf..
The six bytes of "café " came through; the stream stopped at the bare \xe9.
5. SO THE OFFSET OF THE FIRST BAD BYTE IS ONE PIPELINE
caf\xc3\xa9 \xe9 oops valid up to byte 6
all \xe2\x89\xa0 good valid up to byte 12
\xed\xa0\x80 at the front valid up to byte 0
(wc -c on whatever iconv managed to emit. On a real file: iconv -f UTF-8 -t UTF-8 < f | wc -c)
6. WHAT NO VALIDATOR CAN TELL YOU
iconv answers "are these bytes UTF-8?" and nothing else. It cannot tell you what
they ARE. Every byte sequence in existence is valid Latin-1, so a file that is
valid UTF-8 is usually valid under three other encodings too, meaning something
different in each. `file --mime-encoding` guesses; iconv verifies one guess at a
time. Validity is a property of the bytes. The encoding is a fact about where
they came from, and it is not in the file.
iconv -f UTF-8 -t UTF-8 is the one-liner worth keeping: converting an encoding to itself does nothing except run the decoder, so the exit status is a verdict on the file, and the bytes it managed to emit before stopping are the offset of the first bad one.
Section 3 is the surprise, and it is not a bug. On both macOS and Ubuntu, iconv accepts F4 90 80 80, F5 80 80 80 and F7 BF BF BF — all of which encode numbers above U+10FFFF — while Python and Rust reject all three. iconv is checking the original UTF-8, which ran to 31 bits in up to six bytes; RFC 3629 capped it at U+10FFFF in 2003 so that everything expressible in UTF-8 is also expressible in UTF-16. Both validators are correct about the specification they implement. "Valid UTF-8" is not one question, and a page that says "we validate the input" without saying which validator has not said much.
One thing this script deliberately does not do: iconv -c, the flag that drops invalid bytes. On bad input macOS iconv -c stops at the first bad byte while GNU iconv -c skips it and keeps going, so the same repair command produces two different files. Use it interactively if you like; do not put it in a pipeline that runs on more than one machine.
In Rust¶
Verified output of validation_is_a_boundary_rs.rs — regenerated by tools/run_examples.py, never hand-typed.
1. THE THREE SEQUENCES FROM THE SLIDE, DECODED
1: 0111.1101 U+007D: 0x7D (})
2: 1100.0010 1010.1001 U+00A9: 0xC2 0xA9 (©)
3: 1110.0010 1000.1001 1010.0000 U+2260: 0xE2 0x89 0xA0 (≠)
Same three sequences, same bits. UTF-8 is UTF-8; only the checking differs.
2. SIX WAYS TO BE INVALID, AND WHAT Utf8Error KNOWS
89 valid_up_to=0 error_len=Some(1) lone continuation byte
e2 89 valid_up_to=0 error_len=None truncated 3-byte sequence
c0 af valid_up_to=0 error_len=Some(1) overlong '/' - two bytes for U+002F
e0 80 af valid_up_to=0 error_len=Some(1) overlong '/' - the three-byte way
ed a0 80 valid_up_to=0 error_len=Some(1) UTF-16 surrogate U+D800
f5 80 80 80 valid_up_to=0 error_len=Some(1) above U+10FFFF
error_len = Some(n): definitely wrong, skip n bytes and resynchronise.
error_len = None: the bytes ran out mid-sequence - a valid PREFIX, not an error yet.
3. WHY THAT None MATTERS: THE SAME BYTES, ONE BYTE LATER
e2 89 -> Err(None)
e2 89 a0 -> Ok("≠")
A reader on a socket keeps the tail and waits. A reader that saw Some(n) throws it away.
4. PAST THE BOUNDARY, THE TYPE REMEMBERS - SO NOTHING CHECKS AGAIN
s = "café ≠"
s.len() = 9 (bytes - it is still a byte buffer)
s.chars().count() = 6 (a full decode, but no validation)
s.as_bytes() = [63 61 66 c3 a9 20 e2 89 a0]
`&str` MEANS valid UTF-8. chars() bottoms out in an unsafe fn whose safety comment
reads "bytes must produce a valid UTF-8-like string" - it trusts the type and skips
every range check. That saved scan is what the invariant buys.
5. AND A char CANNOT HOLD WHAT UTF-8 CANNOT ENCODE
size_of::<char>() = 4 (a Unicode scalar value, not a code unit)
char::from_u32(0xD800) = None <- the surrogate Python built happily
char::from_u32(0x110000) = None
char::from_u32(0x10FFFF) = Some('\u{10ffff}')
The gap Python leaves between decode and encode does not exist here: there is no
value of type char that encode_utf8 could fail on. It is not checked, it is unrepresentable.
6. WHEN YOU WANT THE BYTES ANYWAY: lossy, AND THE unsafe DOOR
89 lossy -> "�" 1 replacement char(s)
e2 89 lossy -> "�" 1 replacement char(s)
c0 af lossy -> "��" 2 replacement char(s)
e0 80 af lossy -> "���" 3 replacement char(s)
ed a0 80 lossy -> "���" 3 replacement char(s)
f5 80 80 80 lossy -> "����" 4 replacement char(s)
How many U+FFFD you get is error_len, applied repeatedly - the Unicode 'maximal subpart' rule.
String::from_utf8(vec![E2,89]).unwrap_err().into_bytes() = [226, 137]
The failed conversion hands the bytes back rather than dropping them.
And str::from_utf8_unchecked is the same conversion with the check removed - `unsafe`,
because a wrong promise here is undefined behaviour, not a panic.
Section 2 is the same six inputs as the Python example, and Utf8Error carries the same two facts Python's exception does — how far the bytes were text, and what went wrong — but it splits the second one differently, and the split is the useful part. error_len: Some(n) means definitely not UTF-8; skip n bytes and resynchronise. error_len: None means the bytes ran out mid-sequence — a valid prefix, not an error yet. Section 3 shows why that distinction has to exist: E2 89 is None, and one byte later the very same buffer is Ok("≠"). A decoder reading from a socket keeps the tail and waits; a decoder that saw Some(n) throws it away. A validator that returns a bare bool cannot tell those apart, which is the first thing to notice about the hand-written C one further down.
Section 4 is the part with no counterpart in the other three languages. &str means valid UTF-8 — the type is the proof — so chars() does not re-check. It bottoms out in an unsafe fn in core whose safety comment reads "bytes must produce a valid UTF-8-like string": it trusts the type and skips every range check, on every character, for the rest of the program. That saved scan is what the invariant is for. Python re-validates at each decode() because it has to; Rust validates once because it can prove it already did.
Section 5 is the same idea one level down. char is a Unicode scalar value — four bytes, and the surrogate range and everything above U+10FFFF are simply not inhabitants of the type. Where Python leaves a gap between decode and encode that a lone surrogate can sit in, Rust has no such gap: there is no char that encode_utf8 could fail on. Not checked — unrepresentable.
Section 6 is worth comparing with the Python run above, because the two agree exactly: C0 AF becomes two U+FFFD, ED A0 80 becomes three, F5 80 80 80 becomes four. That is not a coincidence or a shared implementation. Both follow the Unicode Standard's maximal subpart recommendation (chapter 3 ↗), which says how much to consume per replacement character — which is exactly error_len, applied in a loop. Plenty of decoders emit one U+FFFD per bad sequence instead, so this is a place where two implementations agreeing tells you both read the spec.
What Rust's core actually does¶
The interesting thing about the validator behind from_utf8 is how ordinary it is. It lives in core/src/str/validations.rs ↗ — about 280 lines, most of them comments — and it is built from two pieces. First a 256-entry table mapping a first byte to how many bytes the sequence has, where 0 means this byte cannot lead: 80–BF, C0, C1, and F5–FF. Then, for the multi-byte cases, explicit ranges on the second byte, quoted here from the 1.98.0 source:
// UTF8-3 = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2( UTF8-tail ) /
// %xED %x80-9F UTF8-tail / %xEE-EF 2( UTF8-tail )
match (first, next!()) {
(0xE0, 0xA0..=0xBF)
| (0xE1..=0xEC, 0x80..=0xBF)
| (0xED, 0x80..=0x9F)
| (0xEE..=0xEF, 0x80..=0xBF) => {}
_ => err!(Some(1)),
}
Four arms, and they are the four rules from the table above: (0xE0, 0xA0..=0xBF) is the no-overlong rule, (0xED, 0x80..=0x9F) is the no-surrogate rule. The 4-byte case has (0xF0, 0x90..=0xBF) and (0xF4, 0x80..=0x8F) for the same two reasons at the top of the range. This is a hand-unrolled DFA: Steagall's talk writes the transitions as a table and drives them with a loop, core writes them as match arms, and the two accept exactly the same language.
Two things worth knowing about it in practice. There is a fast path, but it is for ASCII only: when the pointer is aligned, the loop skips two usize words at a time as long as no byte has its top bit set, which is why validating an English log file is nearly free. And there is no SIMD — core's validator is scalar, one branch per non-ASCII byte. That is the gap Steagall's SSE version and the simdutf8 ↗ crate fill, both processing 16 or 32 bytes per iteration; the crate is a port of the algorithm behind simdutf ↗, and on non-ASCII text it is several times faster than std. Same verdicts, different throughput — which is what the talk is about, and the reason it is a talk about speed rather than about rules.
The C view¶
C is where none of this is done for you, so it is the place to see both halves at once: nothing objects to bytes Unicode forbids, and the check you have to write instead is about thirty lines.
validation_is_a_boundary_c.c in full — pasted here by tools/run_examples.py from the file CI runs.
/* The C view: nothing checks, so here is the check you have to write.
*
* Build & run: cc -std=c11 -Wall -Wextra validation_is_a_boundary_c.c -o v && ./v
*/
#include <stdio.h>
#include <string.h>
#include <stddef.h>
typedef enum { UTF8_OK, UTF8_BAD, UTF8_INCOMPLETE } utf8_status;
/* The whole of RFC 3629, as a range table. This is the function Rust's core
* library runs once at the boundary, and the function Bob Steagall's CppCon
* talk rewrites as a DFA and then as SSE intrinsics. It is not big. It is
* just never written for you. */
static utf8_status utf8_validate(const unsigned char *v, size_t len, size_t *valid_up_to)
{
size_t i = 0;
while (i < len) {
const size_t start = i;
const unsigned char c = v[i];
if (c < 0x80) { i++; continue; } /* plain ASCII */
size_t need; /* continuation bytes required */
if (c >= 0xC2 && c <= 0xDF) need = 1;
else if (c >= 0xE0 && c <= 0xEF) need = 2;
else if (c >= 0xF0 && c <= 0xF4) need = 3;
else { *valid_up_to = start; return UTF8_BAD; } /* 80..C1 and F5..FF cannot lead */
const size_t avail = len - i - 1;
for (size_t k = 1; k <= need && k <= avail; k++) {
const unsigned char b = v[i + k];
int ok;
if (k > 1) ok = (b >= 0x80 && b <= 0xBF);
else if (c == 0xE0) ok = (b >= 0xA0 && b <= 0xBF); /* no overlong 3-byte */
else if (c == 0xED) ok = (b >= 0x80 && b <= 0x9F); /* no surrogates */
else if (c == 0xF0) ok = (b >= 0x90 && b <= 0xBF); /* no overlong 4-byte */
else if (c == 0xF4) ok = (b >= 0x80 && b <= 0x8F); /* nothing above U+10FFFF */
else ok = (b >= 0x80 && b <= 0xBF);
if (!ok) { *valid_up_to = start; return UTF8_BAD; }
}
if (avail < need) { *valid_up_to = start; return UTF8_INCOMPLETE; }
i += need + 1;
}
*valid_up_to = len;
return UTF8_OK;
}
static void show_bytes(const unsigned char *v, size_t len)
{
for (size_t i = 0; i < len; i++) printf("%s%02x", i ? " " : "", v[i]);
}
int main(void)
{
/* U+D800 as UTF-8 would be ED A0 80 - a sequence Unicode forbids outright. */
const char surrogate[] = "\xed\xa0\x80";
const char overlong[] = "\xc0\xaf";
printf("1. C HOLDS WHAT UNICODE FORBIDS, AND NOTHING OBJECTS\n");
printf(" char surrogate[] = \"\\xed\\xa0\\x80\"; strlen = %zu, sizeof = %zu\n",
strlen(surrogate), sizeof surrogate);
printf(" char overlong[] = \"\\xc0\\xaf\"; strlen = %zu, sizeof = %zu\n",
strlen(overlong), sizeof overlong);
printf(" Both compiled. Both have a length. printf(\"%%s\") would write those bytes to your\n");
printf(" terminal unchanged. No function in the C library has an opinion about UTF-8,\n");
printf(" because a char* is a run of bytes and that is the entire type.\n\n");
printf("2. SO YOU WRITE THE CHECK YOURSELF - THE SAME INPUTS AS THE OTHER TWO EXAMPLES\n");
struct { const unsigned char *v; size_t len; const char *why; } cases[] = {
{ (const unsigned char *)"\x7d", 1, "U+007D, the slide's line 1" },
{ (const unsigned char *)"\xc2\xa9", 2, "U+00A9, the slide's line 2" },
{ (const unsigned char *)"\xe2\x89\xa0", 3, "U+2260, the slide's line 3" },
{ (const unsigned char *)"\x89", 1, "lone continuation byte" },
{ (const unsigned char *)"\xe2\x89", 2, "truncated 3-byte sequence" },
{ (const unsigned char *)"\xc0\xaf", 2, "overlong '/'" },
{ (const unsigned char *)"\xed\xa0\x80", 3, "UTF-16 surrogate U+D800" },
{ (const unsigned char *)"\xf5\x80\x80\x80", 4, "above U+10FFFF" },
{ (const unsigned char *)"\xe0\x80\xaf", 3, "overlong '/' the 3-byte way" },
};
const char *names[] = { "OK", "BAD", "INCOMPLETE" };
for (size_t i = 0; i < sizeof cases / sizeof cases[0]; i++) {
size_t up_to = 0;
const utf8_status st = utf8_validate(cases[i].v, cases[i].len, &up_to);
printf(" ");
show_bytes(cases[i].v, cases[i].len);
printf("%*s%-11s valid_up_to=%zu %s\n",
(int)(14 - 3 * cases[i].len), "", names[st], up_to, cases[i].why);
}
printf("\n");
printf("3. THE TWO ANSWERS THAT ARE NOT THE SAME ANSWER\n");
printf(" BAD = these bytes are not UTF-8 and never will be. Resynchronise.\n");
printf(" INCOMPLETE = a valid prefix that ran out. Keep it and read more.\n");
printf(" That is Rust's Some(n) and None, and the distinction a 'return false' validator loses.\n\n");
printf("4. WHAT THE TALK IS ACTUALLY ABOUT\n");
printf(" The loop above is ~30 lines and branches once per byte. A DFA replaces the branches\n");
printf(" with a table lookup per byte; SSE intrinsics check 16 bytes per iteration.\n");
printf(" Same rules, same verdicts - only the throughput changes.\n");
return 0;
}
Verified output of validation_is_a_boundary_c.c — regenerated by tools/run_examples.py, never hand-typed.
1. C HOLDS WHAT UNICODE FORBIDS, AND NOTHING OBJECTS
char surrogate[] = "\xed\xa0\x80"; strlen = 3, sizeof = 4
char overlong[] = "\xc0\xaf"; strlen = 2, sizeof = 3
Both compiled. Both have a length. printf("%s") would write those bytes to your
terminal unchanged. No function in the C library has an opinion about UTF-8,
because a char* is a run of bytes and that is the entire type.
2. SO YOU WRITE THE CHECK YOURSELF - THE SAME INPUTS AS THE OTHER TWO EXAMPLES
7d OK valid_up_to=1 U+007D, the slide's line 1
c2 a9 OK valid_up_to=2 U+00A9, the slide's line 2
e2 89 a0 OK valid_up_to=3 U+2260, the slide's line 3
89 BAD valid_up_to=0 lone continuation byte
e2 89 INCOMPLETE valid_up_to=0 truncated 3-byte sequence
c0 af BAD valid_up_to=0 overlong '/'
ed a0 80 BAD valid_up_to=0 UTF-16 surrogate U+D800
f5 80 80 80 BAD valid_up_to=0 above U+10FFFF
e0 80 af BAD valid_up_to=0 overlong '/' the 3-byte way
3. THE TWO ANSWERS THAT ARE NOT THE SAME ANSWER
BAD = these bytes are not UTF-8 and never will be. Resynchronise.
INCOMPLETE = a valid prefix that ran out. Keep it and read more.
That is Rust's Some(n) and None, and the distinction a 'return false' validator loses.
4. WHAT THE TALK IS ACTUALLY ABOUT
The loop above is ~30 lines and branches once per byte. A DFA replaces the branches
with a table lookup per byte; SSE intrinsics check 16 bytes per iteration.
Same rules, same verdicts - only the throughput changes.
Section 1 is the whole reason the talk exists. char surrogate[] = "\xed\xa0\x80"; compiles, has a strlen, and printf("%s") will write those three bytes to your terminal — because a char * is a run of bytes, and that is the entire type. Nothing in the C library has an opinion about UTF-8. Neither does std::string; C++ added char8_t and u8"" literals in C++20, which give UTF-8 text a distinct type but still no guarantee, and the standard's own conversion facility (std::wstring_convert, std::codecvt_utf8) was deprecated in C++17 and removed in C++26. So you write it yourself, which is what Steagall did.
Section 2 runs the hand-written validator over the same inputs as the Python and Rust examples, and the three agree on every one — including that E2 89 is a different answer from the rest. UTF8_BAD and UTF8_INCOMPLETE are Rust's Some(n) and None under other names, and the fact that a validator wants three return values rather than two is the design detail that a bool quietly loses.
If you are coming from Python or ABAP¶
Python. You already do this — bytes.decode() at the incoming boundary, str.encode() at the outgoing one — and the thing to take from Rust is not a technique but a guarantee. Python's str is wider than what UTF-8 can express (section 5 of the Python run: chr(0xD800) is legal), so the check on the way out is not redundant; it is catching a value that the type permitted and the encoding does not. Rust closes that gap by making the value unrepresentable. The other half transfers directly: Utf8Error::valid_up_to() is UnicodeDecodeError.start, errors='replace' is String::from_utf8_lossy, errors='strict' is String::from_utf8, and errors='surrogateescape' has no Rust equivalent at all, because it works by storing exactly the surrogates Rust's char cannot hold.
ABAP. ABAP splits the two states the way Python does, into separate types rather than one type with a promise: xstring is bytes, string is characters, and the conversion between them is the boundary. cl_abap_codepage=>convert_to( ) and convert_from( ) are the modern pair, and a bad conversion raises cx_sy_conversion_codepage — so ABAP does check, at the same place Python does, and forgets in the same way afterwards. Two differences worth holding onto. A Unicode ABAP system stores string in a fixed-width UTF-16 form internally, not UTF-8, so strlen( ) counts neither bytes nor quite the same units a Rust chars().count() would — verify the actual code page on your own system rather than trusting a number from a document. And the boundary in ABAP is usually a file or an RFC rather than a function call: OPEN DATASET … IN LEGACY TEXT MODE CODE PAGE … is where the encoding is named, and getting that addition wrong is the ABAP form of every mojibake on the Mojibake page. (Not machine-checked — CI cannot run ABAP.)
Try it¶
cd 03_Encodings/validation_is_a_boundary/examples
python3 validation_is_a_boundary_py.py
bash validation_is_a_boundary_sh.sh
rustc --edition 2024 validation_is_a_boundary_rs.rs -o /tmp/vb && /tmp/vb
cc -std=c11 -Wall -Wextra validation_is_a_boundary_c.c -o /tmp/vc && /tmp/vc
Then on something real. Point the shell recipe at a file you did not write — iconv -f UTF-8 -t UTF-8 < some.csv > /dev/null; echo $? — and if it fails, get the offset with iconv -f UTF-8 -t UTF-8 < some.csv | wc -c and look at the byte with xxd -s <offset> -l 16 some.csv. Then decide which of the six rows in the table at the top you are looking at.
And one to work out without the machine: F0 82 82 AC is four bytes that decode, by the templates alone, to U+20AC — the euro sign, whose real encoding is the three bytes E2 82 AC. Which row of the table is it, and which of the four match arms in core rejects it? (Check yourself: add it to the bad array in the Rust example and re-run.)
Practice¶
One bad byte, three languages. The bytes caf\xe9.txt are a Latin-1 é loose in a stream that was declared UTF-8.
Say what Python does with .decode('utf-8') under strict, replace and surrogateescape — and which of the three is reversible. Then the question the page is about: after the decode returns, what can a function downstream tell about where that string came from, in Python, in Rust, and in C? Sort the three languages by how much of the check survives the function call.
Answers
Verified output of validation_is_a_boundary_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.
the bytes 63 61 66 e9 2e 74 78 74 -- e9 is not a legal UTF-8 lead byte
PYTHON: THE CHECK IS AT THE DOOR, AND THEN IT IS OVER
strict UnicodeDecodeError at byte 3
errors='replace' 'caf�.txt' lossy, and cannot be undone
errors='surrogateescape' 'caf\udce9.txt'
...and back True
Once decode() returns, the value is a str and the str type makes no
promise about where it came from. Nothing downstream can tell a
checked string from a repaired one -- which is why the error handler
is a POLICY DECISION and not a convenience.
RUST: THE CHECK IS AT THE DOOR AND THE TYPE REMEMBERS
String::from_utf8(v) -> Result<String, FromUtf8Error>
String::from_utf8_lossy(v) -> Cow<str>, U+FFFD substituted
str::from_utf8_unchecked -> unsafe, and the word is the point
A &str is a PROOF that the bytes were checked. The check happens
once, at the same boundary Python checks at, and then it is carried
in the type -- so a function taking &str cannot be handed unchecked
bytes by accident, and no later code re-validates 'to be safe'.
C: THERE IS NO DOOR
char* is bytes. Nothing decodes, so nothing can refuse, and a
sequence like this travels to the far end of the program unexamined
-- where strlen counts 8, printf emits whatever the terminal makes of
it, and the first thing that notices is a person looking at output.
WHAT THE THREE HAVE IN COMMON, WHICH IS THE ACTUAL LESSON
All three agree these bytes are not UTF-8. The decoders are the same
algorithm. What differs is how much of that knowledge survives the
function call:
C nothing was ever asked
Python it was asked and the answer was discarded
Rust it was asked and the answer is in the type
So 'is this string valid' is a question you can only ask at a
boundary. One byte further in, the honest answer in two of the three
languages is: it depends who checked, and you cannot find out.
See also¶
- Overlong sequences — the four rows of the table above that decode perfectly and are forbidden anyway, and the boundary bytes that fall out
- UTF-8 by hand — the templates this page assumes, with a pencil
- UTF-16 and surrogates — where
U+D800comes from, and why UTF-8 has to forbid it - Mojibake — the failure that happens when the bytes are valid and the table was still wrong
- From UTF-8, and lossy —
Utf8Error,Cow<str>andOsStr, as an API rather than as a comparison Stringis bytes that promise UTF-8 — the promise this page leans on, from the type's side- Strings, in the Rust learning library ↗ — where the Rust half of this page goes next:
Stringvs&str, slices,char, and the byte-boundary panic, taught as a language rather than as an encoding