Where std stops¶
Level: 201 · for anyone who has typed cargo add for a text job
One line: std validates UTF-8, converts UTF-16, maps case and reads a char's own properties — and stops at nine jobs, each with a crate standing where it stopped: a legacy encoding, a guessed one, normalization, what a person calls one character, terminal columns, bytes that are mostly text, validation at memory bandwidth, a pattern, and the locale. Every one is measured here against what bare rustc can do, so you can see what a cargo add buys.
Nothing in this page's answer key came from a crate. The examples here compile with bare rustc and no Cargo (CONTRIBUTING.md), so the one machine-checked block is the baseline — how far std gets on each job — and every crate result is in a fence dated and labelled with the version it ran under, built in a scratch Cargo project on one machine. The sibling Rust library keeps the reading list of string crates ↗: fifteen crates, described and deliberately not run. This page is the ones that sit on the encoding boundary, run.
The same page exists for Python — Where the standard library stops — and the two gaps are in different places: Python's standard library has the legacy encodings and the normalization that are crates here, and the crates here carry a Unicode table a year newer than that interpreter's.
First, the baseline¶
Nine jobs, std only. Two of them std finishes; on the rest it gets exactly as far as the type system reaches, which is the pattern of the whole chapter.
Verified output of where_std_stops_rs.rs — regenerated by tools/run_examples.py, never hand-typed.
1. LEGACY ENCODINGS -- std decodes UTF-8 and UTF-16, plus one table by accident
b as char over 63 61 66 e9 -> "café" Latin-1 IS the first 256 code points
the same cast over cp1252 bytes -> "Preis\u{85} 100\u{80}"
0x80 became U+0080, a C1 control, not the euro sign. windows-1252 needs a
32-entry table for 80..9f, windows-1250 a 128-entry one, Shift_JIS a state
machine. std ships none of them: from_utf8, from_utf16, and that is all.
from_utf8(5a 61 bf f3 b3 e6) -> Err((2, Some(1)))
2. DETECTION -- validation is the only test std has, and it only names UTF-8
utf8 63 61 66 c3 a9 valid UTF-8: true
latin1 63 61 66 e9 valid UTF-8: false
cp1250 5a 61 bf f3 b3 e6 valid UTF-8: false
ascii 70 6c 61 69 6e valid UTF-8: true
'Not UTF-8' is where std stops. Which 8-bit table the bytes are in instead
is a guess from letter statistics, and the page measures Firefox's guesser.
3. NORMALIZATION -- two spellings of one word, and == has one opinion
"café" == "cafe\u{301}" -> false
bytes 5 vs 6, chars 4 vs 5
std has no nfc(). Python's unicodedata.normalize is a standard-library
call; in Rust it is a crate, and the page measures it.
4. WHAT A PERSON CALLS ONE CHARACTER -- three rulers in std, and none is that one
e + acute bytes 3 chars 2 utf16 2 a person sees 1
flag PL bytes 8 chars 2 utf16 4 a person sees 1
family bytes 25 chars 7 utf16 11 a person sees 1
Devanagari kshi bytes 12 chars 4 utf16 4 a person sees 1
thumbs up + tone bytes 8 chars 2 utf16 4 a person sees 1
split_whitespace: ["can't", "stop,", "won't", "stop;", "3.14", "isn't", "3,14.", "東京都"]
Whitespace is the only word boundary std knows. UAX #29's boundaries, which
keep can't and 3.14 whole and drop the punctuation, are a crate.
5. COLUMNS -- {:<8} pads by chars().count(), which is the wrong ruler
|café | chars 4
|日本語 | chars 3
|👨👩👧👦 | chars 7
|🇵🇱 | chars 2
The bars are meant to line up. A terminal draws 日本語 six columns wide and
the family two. std has no width function and no East_Asian_Width property;
both are a crate.
6. BYTES THAT ARE MOSTLY TEXT -- std has the pieces, and each one is a line
lines: 3
find "au": Some(5)
utf8_chunks(): [("caf", "e9"), (" au lait\nZa", "bf"), ("", "f3 b3"), ("", "e6"), (" g", "ea 9c"), ("l", "b9"), ("\r\nplain", "")]
from_utf8_lossy: "caf� au lait\nZa��� g�l�\r\nplain"
No lines(), no find(), no chars() on a [u8] -- but since Rust 1.79
utf8_chunks() is std, and it is the loop that bstr and from_utf8_lossy share.
7. VALIDATION -- std checks every byte, and what that costs is not in this key
from_utf8 over 28 MB of Polish text -> Ok
How long it took, and how long simdutf8 takes over the same bytes, is on
the page in a dated fence: a speed is a fact about one machine.
8. REGEX -- there is none in std; char predicates cover the small cases
split on !is_alphabetic: ["Zażółć", "gęślą", "jaźń", "can", "t", "stop"]
is_alphabetic('ż') = true is_numeric('٣') = true
That is \p{L}+ and \p{N} by hand. Anything with structure -- alternation,
repetition, a class named by script -- is the regex crate, measured on the page.
9. SORTING -- sort() is code point order, and std has no locale to ask
sort(): Zawadzki Zebra cmentarz lód zebra ćma Łukasiewicz łódź żaba
sort_by_key(lowercase): cmentarz lód Zawadzki Zebra zebra ćma Łukasiewicz łódź żaba
ł, ć and ż land after z in both. A Polish dictionary puts ć after c and ł
after l, and the rules that know that are locale data -- which is why std
does not carry them and ICU4X does. Measured on the page.
Three things in that block matter more than any crate below. Section 1's cast — b as char — is a one-line Latin-1 decoder, and it is correct for the same reason char is four bytes warned that char as u8 is wrong: Latin-1 is the first 256 code points, so the cast is the table. Run it over Windows-1252 bytes and 0x80 becomes a control character rather than a euro sign, which is the whole difference between the two tables and a page of its own. Section 6's utf8_chunks() is std closing a gap: since Rust 1.79 the loop that From UTF-8, and lossy hand-rolls from valid_up_to() is a method on [u8], and it is what bstr and from_utf8_lossy both run. And section 4's split_whitespace is the only word boundary std has; the sentence it splits keeps every comma and semicolon glued to its word, which is what the crate for the job is for.
1. A legacy encoding — encoding_rs¶
std decodes UTF-8 and UTF-16 and nothing else. encoding_rs is Firefox's implementation of the WHATWG Encoding Standard ↗ — the document that says what a browser must do with charset=latin1 — and that pedigree decides everything below, starting with what a label means:
Encoding::for_label(b"latin1") -> windows-1252 every Latin-1 label means Windows-1252: the Standard's rule
Encoding::for_label(b"iso-8859-1") -> windows-1252
Encoding::for_label(b"ascii") -> windows-1252 and so does "us-ascii"
Encoding::for_label(b"utf-16") -> UTF-16LE and "unicode"
Encoding::for_label(b"gb2312") -> GBK
Encoding::for_label(b"sjis") -> Shift_JIS and "ms932", "windows-31j"
Encoding::for_label(b"utf-7") -> None not in the Standard, on purpose
Encoding::for_label(b"utf-32") -> None
Encoding::for_label(b"cp437") -> None no DOS code pages, no EBCDIC, and of the Mac tables only macintosh
WINDOWS_1250.encode("Zażółć gęślą jaźń") -> 5a 61 bf f3 b3 e6 20 67 ea 9c 6c b9 20 6a 61 9f f1 had_errors false
WINDOWS_1250.decode(those bytes) -> "Zażółć gęślą jaźń" had_errors false
WINDOWS_1252.decode(the same bytes) -> "Za¿ó³æ gêœl¹ jaŸñ" had_errors false mojibake, and no error: every byte is defined
ISO_8859_2.decode(the same bytes) -> "Zażółć gę\u{9c}lš ja\u{9f}ń" had_errors false two C1 controls where cp1250 had ś and ź
ISO_8859_2.encode("café 1€ 😀") -> "café 1€ 😀" had_errors true an HTML character reference, not a ?
WINDOWS_1252.decode(b"\x81") -> "\u{81}" had_errors false the five unassigned bytes decode to C1 controls
WINDOWS_1252.decode(ef bb bf 63 61 66 c3 a9) -> used UTF-8, "café" decode() sniffs the BOM whatever encoding you called it on
WINDOWS_1252.decode(ff fe 63 00 61 00) -> used UTF-16LE, "ca"
WINDOWS_1252.decode(ff fe 00 00 63 00 00 00) -> used UTF-16LE, "\0c\0" a UTF-32LE BOM is a UTF-16LE BOM and two NULs, to a Standard with no UTF-32
UTF_8.decode_without_bom_handling(ef bb bf 78) -> "\u{feff}x" 2 chars
UTF_8.new_decoder() chunk "caf" f0 9f last=false -> "caf" had_errors false two bytes held back
chunk 98 80 "!" last=true -> "caf😀!" had_errors false
the first chunk alone, last=true -> "caf�" had_errors true
Read it as a browser would. A label is a lookup in a closed table with the web's history baked in: latin1 means Windows-1252 because that is what every page labelled Latin-1 actually was; utf-7 is refused because a browser that accepted it was an XSS vector and utf-32 because no page used it; cp437 was never a web encoding at all. Off the web that is a limitation — there is no DOS or EBCDIC table here — and on it, it is the specification. Encoding what the table cannot hold writes €, the HTML form-submission rule, rather than a ? or a refusal, and had_errors is the flag that says it happened. The five unassigned Windows-1252 bytes decode to C1 controls, without an error — the Standard chose that over failure, which is ftfy's sloppy codec on the Python page written into a standard. And decode() reads the byte-order mark before anything else, so the encoding you named is a default rather than an instruction; decode_without_bom_handling is the spelling that means what it says. Byte order and the BOM has the marks; the UTF-32 row is the one that looks like a bug and is the specification.
The last three lines are the reason the crate is worth having even for UTF-8. new_decoder() is an incremental decoder: handed f0 9f at the end of a chunk with last=false it holds the two bytes and waits, and the same bytes with last=true are an error — the distinction From UTF-8, and lossy found in error_len()'s None, as an API that keeps the state for you.
2. A guessed one — chardetng¶
The baseline's section 2 is all std can say: not UTF-8. chardetng is the other half of Firefox — the detector it runs on a page with no label at all — and its API is shaped by that job:
file guess(None, Allow) guess(Some(b"pl"), Allow) guess(None, Deny)
utf8.txt UTF-8 UTF-8 windows-1252
latin1.txt windows-1252 ISO-8859-2 windows-1252
cp1252.txt windows-1252 windows-1250 windows-1252
latin2_short ISO-8859-2 ISO-8859-2 ISO-8859-2
latin2_long ISO-8859-2 ISO-8859-2 ISO-8859-2
cp1250_long windows-1250 windows-1250 windows-1250
utf8_pl_long UTF-8 UTF-8 windows-1252
sjis Shift_JIS Shift_JIS Shift_JIS
utf16le_nobom windows-1252 ISO-8859-2 windows-1252
one_byte windows-1252 ISO-8859-2 windows-1252
Three columns, three findings. It tells ISO-8859-2 from Windows-1250 on the Polish sample, in both directions, with or without a hint — the six letters the Python baseline found are enough, and charset-normalizer could not do it. The second argument is a top-level domain, and it is what a browser has when it has nothing else: .pl turns every ambiguous row Polish, including a Windows-1252 file that becomes Windows-1250 and one byte e9 that becomes ISO-8859-2. And the Deny column is what Firefox actually passes: the crate's own documentation says a browser must not allow UTF-8 as a result, to avoid creating a situation where Web content starts depending on unlabeled detection of UTF-8 — so a valid UTF-8 page with no label is decoded as Windows-1252 by design, and the mojibake is the cost of not rewarding pages that omit the label. Off the web, pass Allow.
Two things it does not do, also by design. There is no confidence number — the guess is an &'static Encoding and nothing else — and there is no UTF-16 detection, because a UTF-16 page carries a BOM or a label, so the utf16le_nobom row comes out as Windows-1252 garbage in all three columns. chardet in Python finds it; a browser never needed to.
3. Normalization — unicode-normalization¶
"café" == "cafe" + U+0301 is false in std, will stay false, and there is no method that makes the two equal. In Python this is unicodedata.normalize, a standard-library call; here it is the crate that The check that ran too early found every Rust codebase reaching for:
"café" == "cafe\u{301}" -> false
"cafe\u{301}".nfc().collect::<String>() == "café" -> true is_nfc("café") true, is_nfc("cafe\u{301}") false
"Łódź".nfd() -> "Ło\u{301}dz\u{301}" drop the combining marks -> "Łodz" ł has no decomposition
"fi".nfkc() -> "fi" "①".nfkc() -> "1" "10²".nfkc() -> "102" "ｱ".nfkc() -> "ア"
"Å".nfd() -> "A\u{30a}"
The API is an iterator adaptor — s.nfc() yields chars — so a normalization costs one allocation when you collect it and none when you compare as you go, and is_nfc is the cheap check to run before either. The four forms and what each is for are on Normalization, and the Łodz row is that page's finding: the recipe for stripping accents is a normalization, and normalization does not know that ł is an l.
4. What a person calls one character — unicode-segmentation¶
Baseline section 4: bytes, code points and UTF-16 units are the three rulers std has, and none of them is the one a cursor moves by. A code point is not a character has all five; this crate is UAX #29 ↗, the rules for the fourth, and for words and sentences besides:
string bytes chars graphemes(true)
e + U+0301 3 2 1
🇵🇱 two regional indicators 8 2 1
family: four people, three ZWJ 25 7 1
🧑🧑🧒 family, Emoji 15.1 (2023) 18 5 1
👍🏽 thumbs up + skin tone 8 2 1
क्षि Devanagari kshi 12 4 1
ก + U+0E33 Thai sara am 6 2 1
ab CR LF c 5 5 4
"Zażółć gęślą jaźń — can't stop, won't stop; 3.14 isn't 3,14. 東京都"
split_whitespace ["Zażółć", "gęślą", "jaźń", "—", "can't", "stop,", "won't", "stop;", "3.14", "isn't", "3,14.", "東京都"]
unicode_words ["Zażółć", "gęślą", "jaźń", "can't", "stop", "won't", "stop", "3.14", "isn't", "3,14", "東", "京", "都"]
Every grapheme row is 1, including the Devanagari conjunct that a 2020 Python library gets wrong, because this crate's table is current: UNICODE_VERSION is a public constant, and it read 17.0.0 on a day Python 3.14 read 16.0. The (true) is extended grapheme clusters, the ones every modern platform means; false is the legacy definition and exists for compatibility.
unicode_words is the other half, and its last three entries are the honest edge of the algorithm: it keeps can't and 3.14 whole and drops the punctuation, then splits 東京都 into three, because UAX #29 without a dictionary has no way to know where a Japanese word ends. Section 9's segmenter has the dictionary.
5. Columns — unicode-width¶
Baseline section 5 is the table whose bars do not line up. unicode-width is UAX #11 ↗ with the emoji sequences added, as two traits — one on char, one on str — and the two do not always agree, on purpose:
string chars str::width() width_cjk() char::width() per code point
café 4 4 4 1 1 1 1
cafe + U+0301 5 4 4 1 1 1 1 0
日本語 3 6 6 2 2 2
family: four people, three ZWJ 7 2 2 2 0 2 0 2 0 2
🇵🇱 2 2 2 1 1
❤ and ❤ + U+FE0F 1 / 2 1 / 2 1 / 2 1 and 1 0
① 1 1 2 1
≈ 1 1 2 1
a TAB b 3 3 3 1 None 1
ESC [ 3 1 m r e d ESC [ 0 m 12 12 12 None 1 1 1 1 1 1 1 None 1 1 1
NUL 1 1 1 None
The str width is the one to call: it knows that a ZWJ sequence draws as one glyph and that two regional indicators make one flag, where summing char::width gives 8 and 2. width_cjk is the other reading of East Asian ambiguous width — ① and ≈ are one column in a Western terminal and two in a CJK one — and the crate makes you choose rather than choosing for you.
The control rows are the decision to know about. char::width says None for a tab, an escape and a NUL; str::width counts each as one column, so an ANSI-coloured string measures twelve. Python's wcwidth returns -1 for the same input. Neither is wrong — there is no width for a byte that moves the cursor — but a program that pads by this number without stripping the escapes first gets a different wrong answer in each language.
6. Bytes that are mostly text — bstr¶
Baseline section 6: std can do everything bstr does, one line at a time — split for lines, windows().position() for a search, utf8_chunks() for the decode. bstr is those lines as methods on [u8], for the file that is text apart from the bytes that are not:
BStr::new(raw) -> "caf\xe9 au lait\nZa\xbf\xf3\xb3\xe6 g\xea\x9cl\xb9\r\nplain" Debug prints the text and escapes the rest
raw.lines() -> "caf\xe9 au lait" · "Za\xbf\xf3\xb3\xe6 g\xea\x9cl\xb9" · "plain"
line 1 .words() -> ["Za", "g", "l"] a word ends at a byte that is not text
raw.to_str() -> Err: valid_up_to 3, error_len Some(1) the same two numbers as std
raw.find("au") -> Some(5) raw.find_byte(0xe6) -> Some(18) raw.is_utf8() -> false
raw.chars() -> 'c' 'a' 'f' U+FFFD ' ' 'a' 'u' ' ' ... one U+FFFD per maximal subpart, as from_utf8_lossy
raw.graphemes() -> the same count as chars() on every line here
std, for comparison:
raw.utf8_chunks() -> ("caf", e9) (" au lait\nZa", bf) ("", f3 b3) ("", e6) (" g", ea 9c) ("l", b9) ("\r\nplain", -)
The Debug line is worth the dependency on its own: a Vec<u8> prints as a list of numbers, and a BStr prints as the text it mostly is with the bytes it is not shown as escapes — which is what you want in a log line and what OsStr, Path, and WTF-8 wanted for a filename. Everything else is std's own semantics reached through a &[u8]: chars() spends one U+FFFD per maximal subpart, exactly as from_utf8_lossy does, and words() stops at a bad byte rather than guessing through it. The type does not promise UTF-8, and nothing on it pretends otherwise.
7. Validation at memory bandwidth — simdutf8¶
Baseline section 7 validates twenty-eight megabytes and declines to say how long it took. simdutf8 is std::str::from_utf8 with the loop rewritten for SIMD, in two flavours, and the question is whether the drop-in is really a drop-in:
bytes std::str::from_utf8 simdutf8::basic simdutf8::compat
63 61 66 c3 a9 Ok Ok Ok
63 61 66 e9 20 61 75 Err(3, Some(1)) Err, no position Err(3, Some(1))
e0 b2 Err(0, None) Err Err(0, None)
ed a0 80 Err(0, Some(1)) Err Err(0, Some(1))
f4 90 80 80 Err(0, Some(1)) Err Err(0, Some(1))
input std basic compat basic / std
ascii 14.3 GB/s 17.6 GB/s 17.7 GB/s 1.2×
polish 2-byte 1.35 GB/s 9.6 GB/s 11.1 GB/s 7.1×
cjk 3-byte 1.97 GB/s 11.4 GB/s 11.0 GB/s 5.8×
emoji 4-byte 1.93 GB/s 11.2 GB/s 10.9 GB/s 5.8×
The verdicts agree on every row, including the three Validation is a boundary uses to catch a lax decoder — a surrogate, a truncated sequence and a code point past U+10FFFF — so the crate is not a looser validator, which is the first thing to check of anything faster. compat returns std's own two numbers and can replace it inside a match; basic returns a verdict with no position and is the one to use when the answer is only yes or no.
The numbers say when it matters. On ASCII, std already runs a fast path and the crate gains twenty percent; the moment the text is not ASCII, std drops to under two gigabytes a second and the crate does not — a factor of six or seven, on a laptop, on the text this library is about. That is the answer to does validation cost anything: for a Polish log file, it costs the difference between those two columns, once per read.
8. A pattern — regex¶
There is no regular expression engine in std, and baseline section 8 shows how far a closure over char::is_alphabetic gets: \p{L}+ by hand, and nothing with structure. The regex crate is the engine under ripgrep, it is Unicode by default, and it names its own UTS #18 level on the first screen of its documentation:
pattern string is_match
^\p{L}+$ żółw true
^\p{Cyrillic}+$ Москва true \p{Script=Cyrillic} too
^\w+$ Zażółć_gęślą true
^\d+$ ٣٤ true (?-u)^\d+$ -> false: ASCII is a flag
^[[:alpha:]]+$ żółw false POSIX classes are ASCII even in Unicode mode
(?i)^straße$ STRASSE false simple case folding, one char to one
(?i)^straße$ STRAẞE true
^\p{Emoji}$ U+1FABE true assigned in Unicode 16
^\p{Cn}$ U+1FAE9 false the table knows it is assigned
Regex::new(r"\X") -> error: unrecognized escape sequence
Regex::new(r"(a)\1") -> error: backreferences are not supported
Regex::new(r"(?<=a)b") -> error: look-around, including look-ahead and look-behind, is not supported
Regex::new(r"(?-u)caf\xe9") -> error: pattern can match invalid UTF-8
regex::bytes::Regex::new(r"(?-u)caf\xe9").find(b"caf\xe9 au") -> Some((0, 4))
Level 1 of "Supports Unicode" is a level, not a yes, by its own account, and the rows bear it out: properties and scripts, Unicode \w and \d, simple case folding — and no \X, because graphemes are Level 2 and this engine leaves them to section 4's crate. The two refusals below the table are the design: no backreferences and no look-around, in exchange for a guarantee of linear time on any input, which is the property rg is built on.
The last two lines are the type system reaching into the pattern. A Regex searches a &str, so a pattern that could match a lone e9 is refused at compile time; regex::bytes::Regex searches a &[u8] and takes it. That second engine is how rg searches a file it never decoded, and it is the one to run over bstr's bytes.
9. The locale — icu¶
Baseline section 9: sort() is code point order, and std has nothing to ask about a language. ICU4X — the icu crate — is the Unicode Consortium's own rewrite of ICU in Rust, and it is the one dependency on this page that brings locale data rather than a table:
std sort() Olsztyn Strasse Straße Zawadzki Zebra cmentarz file lza lód resume résumé zabawa zebra Ölmütz ćma Łukasiewicz łza łódź źdźbło żaba file
Collator, locale pl cmentarz ćma file file lód lza łódź Łukasiewicz łza Ölmütz Olsztyn resume résumé Strasse Straße zabawa Zawadzki zebra Zebra źdźbło żaba
Collator, locale en ćma cmentarz file file lód łódź Łukasiewicz lza łza Ölmütz Olsztyn resume résumé Strasse Straße żaba zabawa Zawadzki źdźbło zebra Zebra
Collator, und (root) the same as en
Collator, locale sv ... zebra Zebra Ölmütz ö is a letter after z in Swedish
Collator, locale tr ... Olsztyn Ölmütz ... and a letter after o in Turkish
pl, Strength::Primary lód vs łódź -> Less resume vs résumé -> Equal Strasse vs Straße -> Equal zebra vs Zebra -> Equal
the rest of the same crate:
normalizer ComposingNormalizer::new_nfc().normalize("cafe" + U+0301) == "café" -> true
segmenter graphemes: family 1, 🇵🇱 1, क्षि 1, e + U+0301 1
words, Thai "สวัสดีครับ" (no spaces): 3 pieces Japanese "東京都渋谷区": 東京 · 都 · 渋谷 · 区 a dictionary, not a rule
casemap fold("Straße") -> "strasse" uppercase("istanbul", tr) -> "İSTANBUL" lowercase("İSTANBUL", tr) -> "istanbul"
std: "İSTANBUL".to_lowercase() -> "i" + U+0307 + "stanbul", 9 chars the locale std refuses to take
The Polish row is the one Sorting and collation could only describe: ć after every c word, ł after every l word, ź and ż after z, and it is the same order — word for word — that macOS's pl_PL locale files gave the Python page from a different copy of the same CLDR data. The en row is the untailored algorithm, and it is what pyuca gives in Python; sv and tr are two languages that each put ö somewhere the other does not. Strength::Primary is the knob under ignore case and accents: at that strength résumé equals resume and Straße equals Strasse, but łódź still does not equal lód, because in Polish ł is a primary difference.
The rest of the block is the elephant Case is not a per-character operation pointed at: a normalizer, a segmenter whose word breaker knows where a Thai or Japanese word ends because it carries a dictionary, and a case mapper that takes the language std refuses to — the Turkish İ that to_lowercase turns into two code points comes back as one. The cost is the cost of the data: seventy-odd crates in the build and a binary that carries CLDR. That is the trade the whole page is about, made in full by the one crate that makes it in full.
Four small ones¶
| Crate | Version, released | What it adds | Measured 2026-09-13 |
|---|---|---|---|
unicode-bom ↗ |
2.0.3, 2023-11-13 | a Bom enum from the first bytes, including the marks the Encoding Standard leaves out |
ff fe 00 00 … → Utf32Le, length 4, where encoding_rs::Encoding::for_bom says UTF-16LE and 2; 2b 2f 76 38 → Utf7, where encoding_rs says None |
widestring ↗ |
1.2.1, 2025-10-09 | owned u16 and u32 strings for Windows and other FFI, where std has only the conversion |
U16String::from_str("café 😀").len() → 7 units for 6 chars, as encode_utf16().count(); U16CString::from_str("a\0b") → Err, an interior NUL at 1; a lone surrogate is lossy on both sides |
unicode-script ↗ |
0.5.8, 2025-12-03 | the Script property — the one Confusables and scripts needs and char does not carry |
'а' → Cyrillic, 'a' → Latin, '1' → Common, ZWJ → Inherited |
deunicode ↗ · any_ascii ↗ |
1.6.2, 2025-04-27 · 0.3.3, 2025-06-29 | transliteration, the job the Python page gives three libraries | Zażółć gęślą jaźń → Zazolc gesla jazn from both; 北京 → Bei Jing and BeiJing; 😀 → grinning and :grinning:; Ærøskøbing → AEroskobing and Aeroskobing |
The nine, one row each¶
| # | The job | Crate | Version, released | Licence | What std has instead |
|---|---|---|---|---|---|
| 1 | Decode a legacy encoding | encoding_rs ↗ |
0.8.41, 2026-09-09 | (Apache-2.0 OR MIT) AND BSD-3-Clause | UTF-8, UTF-16, and b as char for Latin-1 |
| 2 | Guess the encoding | chardetng ↗ |
1.0.0, 2026-03-30 | Apache-2.0 OR MIT | from_utf8: yes or no |
| 3 | Normalize | unicode-normalization ↗ |
0.1.25, 2025-10-30 | MIT OR Apache-2.0 | nothing |
| 4 | Count what a person sees | unicode-segmentation ↗ |
1.13.3, 2026-06-01 | MIT OR Apache-2.0 | chars().count(), split_whitespace() |
| 5 | Measure columns | unicode-width ↗ |
0.2.2, 2025-10-06 | MIT OR Apache-2.0 | nothing |
| 6 | Bytes that are mostly text | bstr ↗ |
1.13.1, 2026-08-10 | MIT OR Apache-2.0 | utf8_chunks(), from_utf8_lossy, split, windows |
| 7 | Validate at memory bandwidth | simdutf8 ↗ |
0.1.5, 2024-09-22 | MIT OR Apache-2.0 | from_utf8, at a sixth of the speed off ASCII |
| 8 | A pattern | regex ↗ |
1.13.1, 2026-07-15 | MIT OR Apache-2.0 | char predicates in a closure |
| 9 | The locale | icu ↗ |
2.3.1, 2026-08-20 | Unicode-3.0 | nothing, deliberately |
Left out, and why. The encoding crate was the legacy-encoding answer before encoding_rs and has not published since 2016; simdutf (no 8) wraps the C++ library of the same name and adds transcoding, at the price of a C++ build. And textwrap, unicode-linebreak and unicode-bidi are real text crates that belong to layout and direction rather than to the boundary this page is about.
The same jobs, in Python¶
Where the standard library stops has the table that lines the two languages up, job by job. The short version: Python's standard library covers rows 1 and 3 of the table above, and Rust's covers nothing on Python's list that Python's does not — but every crate here carried a Unicode 17 table on the day the Python interpreter carried 16, and chardetng told two Polish code pages apart where the library requests runs did not.
Try it¶
- Take a file
file --mime-encodingcallsunknown-8bitand run it throughchardetngtwice:guess(None, Allow), thenguess(Some(b"xx"), Allow)with the country code of whoever wrote it. If the two answers differ, the file is short of the letters that separate its candidate tables — which is what a hint is for, and what a confidence number would have told you. - Call
encoding_rs::Encoding::for_labelon everycharset=value in a mailbox or a log of HTTP headers you have, and count theNones. Then look at what the senders of those were actually using. - Print a table with
{:<20}and a Japanese name in it; then pad by20 - s.width()spaces instead. Then colour one cell with an ANSI escape and watchwidth()count it. - Validate the largest text file you own with
std::str::from_utf8and withsimdutf8::basic::from_utf8, timing each; then strip every non-ASCII byte out of a copy and time both again. The ratio is the answer to whether your data would notice.
Practice¶
Six jobs, and which of them std finishes. For each of the six, say yes if std can finish the job alone or no if it needs a crate on this page — and name what std returns.
b"caf\xe9"is Latin-1; decode it.- The same four bytes are Windows-1250; decode them.
- Count what a user sees in the four-person family emoji.
- Is
"café"the same word as"cafe"followed byU+0301? - Find the first byte of a file that is not UTF-8.
- Sort
["łódź", "lód", "zebra"]the way a Polish dictionary does.
Answers
Verified output of where_std_stops_kata_rs.rs — regenerated by tools/run_examples.py, never hand-typed.
1. b"caf\xe9" is Latin-1; decode it
bytes.iter().map(|&b| b as char).collect() -> "café"
YES, std, one line -- because Latin-1 is the first 256 code points.
2. The same four bytes are windows-1250; decode them
the same line gives "café", and it is right only because e9 is é in both tables.
on b"\xbf" it gives "¿"; windows-1250 says ż
NO -- a 128-entry table you do not have. encoding_rs.
3. Count what a user sees in the family emoji
len() 25 chars().count() 7 encode_utf16().count() 11 a user sees 1
NO -- three rulers, and none of them is that one. unicode-segmentation.
4. Is "café" the same word as "cafe" + U+0301?
== -> false
NO -- std has no normalization. unicode-normalization, or icu.
5. Find the first byte of a file that is not UTF-8
from_utf8(b).unwrap_err().valid_up_to() -> 3
YES, std -- and it is the same number simdutf8::compat reports, byte for byte.
6. Sort ["łódź", "lód", "zebra"] the way a Polish dictionary does
sort() -> ["lód", "zebra", "łódź"]
NO -- code point order puts ł after z. icu::collator with locale pl: lód, łódź, zebra.
Score: 2 of 6 std finishes, 4 need a crate -- and both YES rows are the ones
where the answer is arithmetic on bytes rather than a table.
See also¶
- Where the standard library stops — the same page for Python, and the table that lines the two up
- The string crates, and what
stdmakes you do without them ↗ — the sibling library's reading list of fifteen, deliberately unrun - From UTF-8, and lossy — the two numbers
simdutf8::compatandbstrhand back unchanged OsStr,Path, and WTF-8 — the other type for bytes that are mostly text, when the bytes are a filename- Validation is a boundary — the byte strings every validator on this page rejects
- Byte order and the BOM — the marks
decode()reads first, and the one the Standard does not have - Sorting and collation — the page the
plrow answers - Case is not a per-character operation — the locale axis
icu::casemapadds andstdrefuses ripgrep— the Rust grep — theregexcrate'sbytesengine, as a tool- Rust strings in practice — the habits, and the two crates it already names