Where the standard library stops¶
Level: 201 · for anyone who has typed pip install for a text job
One line: The standard library decodes forty-odd encodings, normalizes, folds case and validates — and stops at eight jobs, each with a PyPI library standing where it stopped: guessing an encoding, repairing mojibake, transliterating to ASCII, counting what a person sees, measuring terminal columns, sorting the way a dictionary does, naming a Unicode property in a pattern, and reading a newer Unicode table than the interpreter's. Every one is measured here against the standard library's own attempt, so you can see what you are installing.
Nothing in this page's answer key came from a library. CI installs no package and never will (CONTRIBUTING.md), so the one machine-checked block is the baseline — how far the standard library gets on each job, on the Python 3.14 the runners have — and every library result is in a fence dated and labelled with the versions it ran under, on one machine. The Version, released column of the table at the end was read from PyPI on 2026-09-13; a library that has not published since 2017 is in it, and the date is the point.
The same page exists for Rust — Where std stops — and the two do not line up, which is the most useful thing about reading them together: Python's standard library has the legacy encodings and the normalization that Rust makes you install, and Rust's crates carry a Unicode table a year newer than the one this interpreter ships.
First, the baseline¶
Eight jobs, standard library only. Read this before the install list — half the reason to install something is knowing precisely where the free answer stops, and one of the eight turns out to be nobody's answer rather than a library's.
Verified output of where_the_stdlib_stops_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. DETECTION -- the standard library can VALIDATE, and that is all it can do
file decodes without error under
utf8.txt utf-8, cp1252, iso-8859-2, cp1250, latin-1
latin1.txt cp1252, iso-8859-2, cp1250, latin-1
cp1252.txt cp1252, iso-8859-2, cp1250, latin-1
latin2.txt cp1252, iso-8859-2, cp1250, latin-1
cp1250.txt cp1252, iso-8859-2, cp1250, latin-1
Every 8-bit table accepts every byte it defines, so 'decodes' rules out
UTF-8 and nothing else. Which table is RIGHT is not a question the bytes
can answer; a detector guesses from letter statistics, and the page
measures two of them. For Polish the guess rests on six letters:
letter iso-8859-2 cp1250
ą b1 b9
ś b6 9c
ź bc 9f
Ą a1 a5
Ś a6 8c
Ź ac 8f
6 of the 18 Polish letters differ between the two tables. A text
that happens to use none of them is the same bytes in both, and no
detector can tell the tables apart, because there is nothing to tell.
2. REPAIR -- the recipe is one line, and it has two ways to fail
damaged encode(cp1252).decode(utf-8) encode(latin-1).decode(utf-8)
'café' 'café' 'café'
'Ã\x81lvaro' <UnicodeEncodeError> 'Álvaro'
'café' 'café' <UnicodeEncodeError>
'Za¿ó³æ gêœl¹ jaŸñ' <UnicodeDecodeError> <UnicodeEncodeError>
Row 2: 0x81 is one of cp1252's five unassigned bytes, so the cp1252 recipe
cannot even re-encode it; Latin-1 has all 256 and can.
Row 3: mojibake applied twice needs the recipe twice:
once 'café' twice 'café'
Row 4 is not UTF-8 mojibake at all: it is cp1250 bytes read as cp1252, so
the second half of the recipe names the other table:
encode(cp1252).decode(cp1250) -> 'Zażółć gęślą jaźń'
Knowing WHICH recipe is the whole job. ftfy automates the first kind only.
3. TRANSLITERATION -- NFKD then drop the marks, and the letters it leaves standing
Zażółć gęślą jaźń -> Zazołc gesla jazn still not ASCII: ł
Straße -> Straße still not ASCII: ß
Łódź -> Łodz still not ASCII: Ł
Москва -> Москва still not ASCII: Москва
北京 -> 北京 still not ASCII: 北京
Ærøskøbing -> Ærøskøbing still not ASCII: Æøø
fi -> fi still not ASCII: -
½ -> 1⁄2 still not ASCII: ⁄
Þór -> Þor still not ASCII: Þ
ł has no decomposition (there is no COMBINING STROKE), and neither do ß, æ,
ø, þ, any Cyrillic letter or any Han character. ½ decomposes to 1 FRACTION
SLASH 2, which is not ASCII either. The recipe is a normalization, and
normalization was never about ASCII. A transliterator is a table of
opinions -- Straße -> Strasse, 北京 -> Bei Jing -- and the standard library
ships none.
4. WHAT A PERSON CALLS ONE CHARACTER -- nothing in the standard library counts it
e + acute len = 2 re.findall('.') = 2 a person sees 1
flag PL len = 2 re.findall('.') = 2 a person sees 1
family len = 7 re.findall('.') = 7 a person sees 1
Devanagari kshi len = 4 re.findall('.') = 4 a person sees 1
thumbs up + tone len = 2 re.findall('.') = 2 a person sees 1
re.compile(r'\X') -> raises: the grapheme escape does not exist in re
len() counts code points and so does '.'. UAX #29's boundary rules are in
the third-party regex module, uniseg and grapheme -- measured on the page --
and in nothing that ships with the interpreter.
5. COLUMNS -- f'{s:<8}' pads, and pads by the wrong ruler
|café | len = 4
|日本語 | len = 3
|👨👩👧👦 | len = 7
|🇵🇱 | len = 2
The bars are meant to line up. Python padded each string to 8 CODE POINTS;
a terminal draws 日本語 six columns wide and the family two, and the standard
library has no function that returns either number. unicodedata has the raw
East_Asian_Width property; turning it into columns is a rule set the library
leaves to you, and the property is a table lookup, so its values stay out of
this key.
6. SORTING -- sorted() is code point order, and the C locale has no other
sorted() Zawadzki Zebra cmentarz lód zebra ćma Łukasiewicz łódź żaba
key=str.casefold cmentarz lód Zawadzki zebra Zebra ćma Łukasiewicz łódź żaba
key=locale.strxfrm, C Zawadzki Zebra cmentarz lód zebra ćma Łukasiewicz łódź żaba
Three orders and none of them Polish: ł, ć, ż all land after z. strxfrm is
the standard library's only collation, it says whatever the machine's locale
files say, and under LC_ALL=C -- every container, every CI runner, this
program -- it says byte order. The page shows what a pure-Python UCA and
ICU make of the same words.
7. REGEX -- re knows Unicode, and has no property syntax
re.fullmatch('\\w+' , 'Zażółć_gęślą' ) -> match
re.fullmatch('(?i)straße' , 'STRASSE' ) -> no match
re.fullmatch('(?i)straße' , 'STRAẞE' ) -> match
re.fullmatch('\\d+' , '٣٤' ) -> match
re.fullmatch('\\p{L}+' , 'żółw' ) -> raises -- no such escape
re.fullmatch('\\X' , FAMILY ) -> raises -- no such escape
\w and \d are Unicode-aware; (?i) folds one character to one, so ß never
meets SS; and \p{...} and \X do not exist. That is the whole of the gap the
regex module fills, and the page measures it.
8. THE TABLE -- one Unicode version per interpreter, and nothing to update it
unicodedata.unidata_version has 3 parts and is a property of THIS python3
name('é') -> LATIN SMALL LETTER E WITH ACUTE
name(U+FFFE) -> raises (a noncharacter: never assigned, by policy)
A character assigned AFTER this interpreter's table has no name here and
category Cn, however real it is. How many such characters one version step
adds is on the page, dated -- it is the one number this program must not
print, because next year's interpreter would print a different one.
Three things in that block are worth more than the libraries below. Section 1's six letters are why encoding detection is possible at all for Polish text, and why it is not always possible: the two tables a Polish file is likely to be in agree on twelve of the eighteen letters, so a detector is reading the statistics of ą, ś, ź and their capitals, and a file without them has no evidence in it. Section 2's fourth row is the mojibake no library repairs — 8-bit bytes read under a different 8-bit table — and the standard library does it in one line once you name both tables, which was always the skill and never the code. And section 3 is the recipe every slugifier starts with and the letter it cannot reach: ł has no decomposition, so NFKD and drop the marks leaves one Polish letter standing, and Normalization has the measurement.
1. Guessing the encoding — chardet and charset-normalizer¶
The standard library's whole contribution here is try: b.decode(...), which the baseline shows rules out UTF-8 and nothing else. A detector goes further by guessing: it decodes under every table it knows and scores the result against letter frequencies per language. Two are common. chardet is the old one, a port of Mozilla's detector; charset-normalizer is the one requests switched to in 2021, and so the one most Python programs are already running without knowing it.
The first three files are the ones the base toolbox measured against file and uchardet; the Polish ones are new:
file the bytes are chardet.detect() charset_normalizer.from_bytes().best()
utf8.txt UTF-8, café 1€ utf-8 0.99 utf_8
latin1.txt ISO-8859-1, café Windows-1252 0.02 is cp1006 (Farsi)
cp1252.txt cp1252, Preis… 100€ cp862 0.02 he cp1125 (Russian)
latin2_short ISO-8859-2, 18 chars ISO-8859-2 0.03 pl iso8859_10
latin2_long ISO-8859-2, 96 chars ISO-8859-2 0.11 pl iso8859_10 (Dutch)
cp1250_long cp1250, 96 chars Windows-1250 0.12 pl cp1250 (Dutch)
utf8_pl_long UTF-8, 96 chars utf-8 0.99 pl utf_8 (Dutch)
sjis Shift_JIS CP932 0.25 ja cp932 (Japanese)
utf16_bom UTF-16 with a BOM UTF-16 1.00 utf_16
utf16le_nobom UTF-16LE, no BOM utf-16-le 0.95 utf_16_le
one_byte 63 61 66 e9 Windows-1252 0.02 is utf_16_be
chardet.detect_all(latin2_long) ISO-8859-2 0.11 Windows-1250 0.09 iso8859-16 0.08 MacLatin2 0.07 cp852 0.07
chardet.detect_all(cp1250_long) Windows-1250 0.12 ISO-8859-2 0.10 iso8859-16 0.08 MacLatin2 0.07 cp852 0.07
Read the Polish rows first. chardet tells ISO-8859-2 from Windows-1250 on a 96-character sample, in both directions, and detect_all shows how narrow the margin is — 0.11 against 0.09, which is the six letters from the baseline doing all of the work. charset-normalizer gets the Windows-1250 file and names the ISO-8859-2 one ISO-8859-10, the Nordic table, at 18 characters and at 96; the text it produces under that guess is not Polish. It also calls the Polish pangram Dutch three times, which is a reminder of what these columns are: a language model's opinion, not a measurement.
Then the wrong rows. A 12-byte Windows-1252 file is Hebrew DOS to chardet and, to charset-normalizer, a Ukrainian DOS table it labels Russian; a five-byte Latin-1 file is Farsi. Both libraries answer confidently on input that carries no statistics, and neither says I cannot tell — chardet's confidence of 0.02 is the honest signal, and it is the number nobody reads. The latin1.txt row is the same file uchardet called ISO-8859-2 and file called ISO-8859-1: every detector names a table, and there is nothing in 63 61 66 e9 to name one from.
Two rows are wins over everything in 11_Tools: UTF-16 without a BOM is found by both — the NUL in every second byte is a statistic too — and chardet reports the language beside the encoding. And chardet's detect_all is the API to use: a ranked list with the runner-up visible is what a guess honestly looks like.
Use a detector to narrow a question you then settle by asking, the same rule as file guesses; never to write a file back under the table it guessed.
2. Repairing mojibake — ftfy¶
The baseline has the recipe: mojibake is bytes decoded under the wrong table, so the repair is to re-encode under that table and decode under the right one, and The mojibake round trip is the page about when that can and cannot work. ftfy — fixes text for you — is the recipe with the table-choosing automated, applied as many times as it takes, and explained:
damaged ftfy.fix_text() steps
'café' 'café' encode latin-1 · decode utf-8
'Ã\x81lvaro' 'Álvaro' encode latin-1 · decode utf-8
'café' 'café' encode sloppy-windows-1252 · decode utf-8 · encode latin-1 · decode utf-8
'Zażółć gęślÄ… jaźń' 'Zażółć gęślą jaźń' encode sloppy-windows-1252 · decode utf-8
'5€' '5€' encode sloppy-windows-1252 · decode utf-8
'’' "'" encode sloppy-windows-1252 · decode utf-8 · apply uncurl_quotes
'Za¿ó³æ gêœl¹ jaŸñ' unchanged (none)
'café' unchanged (none)
Four things the rows show. It chooses the table — Latin-1 where that suffices, Windows-1252 where the damage holds a € or a ‚, and never the wrong one on these inputs. sloppy-windows-1252 is a codec ftfy defines: Windows-1252 with its five unassigned bytes filled in from Latin-1, so a repair never dies on the 0x81 that kills the standard-library recipe in the baseline's second row — the five bytes that decide whether a round trip is possible are simply given a value. It repeats: the double-mojibake row is two rounds, found without being told. And it leaves alone what it cannot explain — café and the cp1250-as-cp1252 row come back untouched with an empty step list, because ftfy models one kind of damage, UTF-8 read as an 8-bit table. The baseline's last recipe, encode('cp1252').decode('cp1250'), is the one it does not know.
The ’ row is the one to read before running fix_text over a corpus. It repaired ’ to ’ and then changed ’ to ': uncurl_quotes is on by default, with a handful of other normalizations that are opinions rather than repairs. fix_text(s, uncurl_quotes=False) turns that one off; the ftfy documentation ↗ lists the rest, and fix_and_explain shows which of them fired.
3. Transliteration — anyascii, Unidecode, text-unidecode¶
Baseline section 3 is why this is a library at all: normalization strips marks, and a mark is not what makes ł, ß, ø, Москва or 北京 non-ASCII. A transliterator is a table of opinions about what each character should become, and three libraries ship three tables:
input NFKD, marks dropped anyascii Unidecode text-unidecode
Zażółć gęślą jaźń Zazołc gesla jazn Zazolc gesla jazn Zazolc gesla jazn Zazolc gesla jazn
Straße Straße Strasse Strasse Strasse
Москва Москва Moskva Moskva Moskva
北京 北京 BeiJing 'Bei Jing ' 'Bei Jing '
Ærøskøbing Ærøskøbing Aeroskobing AEroskobing AEroskobing
½ 1⁄2 1/2 ' 1/2' 1/2
Þór Þor Thor Thor Thor
😀 😀 :grinning: '' ''
They agree on the letters and differ at the edges: Unidecode leaves a trailing space after every Han syllable and a leading one before a fraction, writes Æ as AE, and deletes an emoji; anyascii writes the emoji's name in colons. Neither is more correct — transliteration is not reversible, and a romanization is a choice — which is why the decision usually comes down to the column the table does not have: the licence. Unidecode is GPL, which most companies will not link into a product; text-unidecode is the same Perl table republished under the Artistic licence for exactly that reason; anyascii is ISC and the newest. None of it changes what the baseline showed: the standard library gets Zazołc, and the letter it leaves is the one a Polish speaker notices.
4. What a person calls one character — regex, uniseg, grapheme¶
Baseline section 4: len() counts code points, re's . matches one code point, and the standard library has no implementation of UAX #29 ↗, the rules for where one user-perceived character ends. A code point is not a character is the page on the five rulers; this is the three libraries that carry the fourth one.
string len regex \X uniseg grapheme
e + U+0301 2 1 1 1
🇵🇱 two regional indicators 2 1 1 1
family: four people, three ZWJ 7 1 1 1
🧑🧑🧒 family, Emoji 15.1 (2023) 5 1 1 1
👍🏽 thumbs up + skin tone 2 1 1 1
क्षि Devanagari kshi 4 1 1 2
ก + U+0E33 Thai sara am 2 1 1 1
ab CR LF c 5 4 4 4
Three libraries agree on every row but one, and the one is a rule rather than a character. क्षि is two clusters to grapheme because its table is Unicode 13, and the rule that keeps an Indic conjunct together — GB9c — was added in Unicode 15.1, in 2023. Nothing about those four code points changed; the rules did, and a library that has not been released since 2020 does not have them. The table has a version is usually about properties; a grapheme count depends on the rules too.
The choice: regex's \X if you are installing regex anyway (section 7), uniseg for the other UAX #29 boundaries — words and sentences, and line breaks from its cousin UAX #14 — and grapheme only with its date in view.
5. Columns — wcwidth¶
Baseline section 5 is the misaligned table every CLI author has shipped once. The standard library has the raw East_Asian_Width property and nothing that turns it into columns; wcwidth is the C library function of the same name, ported, plus the sequences C never knew about:
string len baseline wcswidth() wcwidth() per code point
café 4 4 4 1 1 1 1
cafe + U+0301 5 4 4 1 1 1 1 0
日本語 3 6 6 2 2 2
family: four people, three ZWJ 7 8 2 2 0 2 0 2 0 2
🇵🇱 2 2 2 2 2
👍🏽 2 4 2 2 2
❤ and ❤ + U+FE0F 1 / 2 1 / 2 1 / 2 1 and 1 0
a TAB b 3 3 -1 1 -1 1
ESC [ 3 1 m r e d ESC [ 0 m 12 12 -1 -1 1 1 1 1 1 1 1 -1 1 1 1
U+200B zero width space 1 0 0 0
The family row is the whole argument for the library over the hand-rolled rule: the rule sums the code points to 8, wcswidth knows a ZWJ sequence draws as one glyph and answers 2, and a skin-tone modifier is absorbed the same way. The heart is the row to remember — ❤ is one column and ❤ followed by VARIATION SELECTOR-16 is two, because the selector asks for emoji presentation, and a table that pads by len() will be off by one on every heart a user pastes.
The two -1 rows are a design decision to know before you call it. wcswidth returns -1 for a string holding any non-printable character — a tab, an escape sequence — rather than a width for the rest; strip the ANSI colouring first, or you will be aligning on a number that is not a width. The Rust crate for the same job made the opposite choice and counts each of them as one column.
6. Sorting the way a dictionary does — pyuca, and PyICU¶
Baseline section 6: three standard-library orders, none of them Polish, and locale.strxfrm is only as good as the machine's locale files, which under the C locale means byte order. Sorting and collation is the page on why the order is a locale property and what that did to a database. pyuca is the Unicode Collation Algorithm ↗ in pure Python, with no locale tailoring; ICU is the full answer.
sorted() Olsztyn Strasse Straße Zawadzki Zebra cmentarz file lza lód resume résumé zabawa zebra Ölmütz ćma Łukasiewicz łza łódź źdźbło żaba file
pyuca, no tailoring ćma cmentarz file file lód łódź Łukasiewicz lza łza Ölmütz Olsztyn resume résumé Strasse Straße żaba zabawa Zawadzki źdźbło zebra Zebra
strxfrm, en_US.UTF-8 ćma cmentarz file file lód łódź Łukasiewicz lza łza Ölmütz Olsztyn resume résumé Strasse Straße żaba zabawa Zawadzki źdźbło zebra Zebra
strxfrm, de_DE.UTF-8 the same as en_US
strxfrm, pl_PL.UTF-8 cmentarz ćma file file lód lza łódź Łukasiewicz łza Ölmütz Olsztyn resume résumé Strasse Straße zabawa Zawadzki zebra Zebra źdźbło żaba
ICU4X, locale pl cmentarz ćma file file lód lza łódź Łukasiewicz łza Ölmütz Olsztyn resume résumé Strasse Straße zabawa Zawadzki zebra Zebra źdźbło żaba
pyuca is exactly the en_US and de_DE order: the untailored algorithm treats ć as a c with a mark and ł as a kind of l, so ćma sorts before cmentarz and the ł words interleave with the l words. That is right for English and German and wrong for Polish, where ć and ł are letters of their own that follow c and l — and the pl_PL row and the ICU row, from two different copies of the locale data, agree on every one of the twenty-one words. The tailoring is the locale, and pyuca has none by design.
pyuca's other column is its date. Its newest table is the DUCET for Unicode 10, from 2017, so every character assigned since sorts by an implicit weight rather than a decided one. It is still the right install for a machine with no locale files, which is every container, because untailored and consistent beats byte order. When the language matters, it is ICU: PyICU on a machine with ICU installed, or the operating system's collation through strxfrm once the locale exists.
pip install pyuca
pip install PyICU # needs ICU and pkg-config on the machine first — on macOS, brew install icu4c pkg-config
7. A property in a pattern — regex¶
Baseline section 7 is the whole gap in three lines: re's \w and \d are Unicode-aware, its (?i) folds one character to one, and \p{...} and \X are escapes that do not exist. "Supports Unicode" is a level, not a yes places re against UTS #18; this is what the third-party regex module — a drop-in with the same API — adds, run side by side:
pattern string re regex
\p{L}+ żółw raises match
\p{Script=Cyrillic}+ Москва raises match
\X family emoji raises one match, the whole thing
[[:alpha:]]+ żółw no match match re reads it as a set of the characters [ : a l p h
(?i)straße STRASSE no match no match simple folding in both
(?if)straße STRASSE no such flag match f is FULLCASE: ß meets SS
(?:café){e<=1} cafe no match match fuzzy: within one edit
\w+ Zażółć_gęślą match match
\d+ ٣٤ match match
\N{LATIN SMALL LETTER L WITH STROKE} ł match match
Properties, scripts, graphemes, POSIX classes, full case folding and fuzzy matching, and nothing you already write changes. The row to notice is the FULLCASE flag: regex matches ß against SS only when asked, because full folding changes a string's length and the default keeps the one-to-one rule re has. It also carries its own Unicode table, and the next section measures how far ahead of the interpreter that table is.
8. A newer table — unicodedata2¶
Baseline section 8: the Unicode version is a property of the interpreter, and a character assigned after that interpreter's table has no name and category Cn however real it is. unicodedata2 is unicodedata rebuilt against the current table and released as a wheel, so a Python that ships Unicode 16 can look up Unicode 17:
unicodedata.unidata_version 16.0.0
unicodedata2.unidata_version 17.0.0
assigned in 17.0, unassigned in 16.0 4,803 code points, in 47 runs
CJK Unified Ideographs Extension J 4,298 U+323B0..U+33479
four new scripts 185 Sidetic 26, Tolong Siki 54, Beria Erfe 50, Tai Yo 55
seven emoji, among the rest U+1F6D8 LANDSLIDE, U+1FA8A TROMBONE, U+1FA8E TREASURE CHEST, U+1FAC8 HAIRY CREATURE, U+1FACD ORCA, U+1FAEA DISTORTED FACE, U+1FAEF FIGHT CLOUD
assigned in both, category changed 1 U+0295 ʕ Ll -> Lo
what the other libraries on this page know, on the same day:
regex 2026.9.10 \p{Assigned} is true for U+088F 17.0
wcwidth 0.8.3 list_versions() -> ('17.0.0',) 17.0
uniseg 0.10.1 unidata_version -> 16.0.0
grapheme 0.6.0 UNICODE_VERSION -> 13.0.0 and it predates the rule that keeps क्षि whole
pyuca 1.2 allkeys-10.0.0.txt 10.0, from 2017
The last block is the point of the section. A Python program with these five libraries installed is running four Unicode tables, five with the interpreter's own, and on this machine they spanned eight years, the interpreter's being neither the newest nor the oldest. unicodedata2 moves one of the five; the honest use of it is the one The table has a version asks for — a lookup whose answer is dated, not a lookup whose answer goes in a test. The one changed category is worth knowing too: ʕ was Ll under every table since it was assigned and is Lo under this one, so a check on its category answers differently under the two.
The eight, one row each¶
| # | The job | Library | Version, released | Licence | What the standard library has instead |
|---|---|---|---|---|---|
| 1 | Guess the encoding | chardet ↗ |
7.6.0, 2026-08-14 | 0BSD | validation: try: b.decode(...) |
| 1 | charset-normalizer ↗ |
3.5.1, 2026-08-15 | MIT | — what requests runs |
|
| 2 | Repair mojibake | ftfy ↗ |
6.3.1, 2024-10-26 | Apache-2.0 | the recipe, once you name the table |
| 3 | Transliterate to ASCII | anyascii ↗ |
0.3.3, 2025-06-29 | ISC | NFKD and drop the marks, which leaves ł |
| 3 | Unidecode ↗ · text-unidecode ↗ |
1.4.0, 2025-04-24 · 1.3, 2019-08-30 | GPL-2.0+ · Artistic | ||
| 4 | Count what a person sees | regex ↗ \X · uniseg ↗ · grapheme ↗ |
2026.9.10 · 0.10.1, 2026-01-09 · 0.6.0, 2020-03-07 | Apache-2.0 and CNRI-Python · MIT · MIT | nothing: len() is code points |
| 5 | Measure columns | wcwidth ↗ |
0.8.3, 2026-08-28 | MIT | the raw east_asian_width property |
| 6 | Sort like a dictionary | pyuca ↗ · PyICU ↗ |
1.2, 2017-09-25 · 2.16.2, 2026-03-20 | MIT · MIT | locale.strxfrm, if the machine has the locale |
| 7 | Name a property in a pattern | regex ↗ |
2026.9.10, 2026-09-09 | Apache-2.0 and CNRI-Python | \w and \d, which are Unicode-aware; no \p{} |
| 8 | Read a newer table | unicodedata2 ↗ |
17.0.1, 2026-02-12 | Apache-2.0 | unicodedata, frozen with the interpreter |
Two that are not in the table, on purpose. python-magic is file(1)'s library behind a Python API and answers file's question, not this page's, and needs libmagic installed beside it. And cchardet is chardet's C-speed cousin, last released in 2020 — its maintained fork is faust-cchardet, and the speed is real, but it is the same guess.
The same jobs, in Rust¶
| The job | Python | Rust | Which standard library has it |
|---|---|---|---|
| Decode a legacy encoding | codecs — standard library |
encoding_rs |
Python. Rust's std decodes UTF-8 and UTF-16 and nothing else |
| Guess the encoding | chardet, charset-normalizer |
chardetng |
neither |
| Repair mojibake | ftfy |
nothing — encoding_rs does the recipe by hand |
neither, and only Python has the library |
| Transliterate | anyascii, Unidecode |
deunicode, any_ascii |
neither |
| Normalize | unicodedata.normalize — standard library |
unicode-normalization |
Python |
| Count what a person sees | regex \X, uniseg |
unicode-segmentation |
neither |
| Measure columns | wcwidth |
unicode-width |
neither, and the two libraries disagree about a control character |
| Sort like a dictionary | pyuca, PyICU |
icu (ICU4X) |
neither — strxfrm is the machine's, not the language's |
| A property in a pattern | regex |
regex |
neither has properties; Python at least ships re |
| A newer table | unicodedata2 |
every crate carries its own | Rust's crates were at 17.0 on the day the interpreter was at 16.0 |
Where std stops is the Rust half, measured the same way; "Handles Unicode" is four questions scores the languages against each other rather than against their libraries.
Try it¶
- Run
chardet.detect_allover a file of yours that once came out as mojibake, and read the second answer as well as the first. If the two confidences are within a few hundredths, the file is short of the letters that would separate its two candidate tables — the baseline's section 1 lists which letters those are for Polish, and the same six lines find them for any pair of tables. - Take a string
ftfyrepaired and callfix_and_explainon it instead. Count the steps that are repairs (an encode and a decode) and the steps that are opinions (anapply), and decide which of the second kind you would have wanted. - Print a two-column table of your own — names on the left, numbers on the right — padded with
f'{s:<20}', and put one emoji and one Japanese name in it. Then pad with20 - wcwidth.wcswidth(s)spaces instead, and compare. - Sort the same list of names three ways —
sorted(),pyuca, andlocale.strxfrmunder a locale you set — and diff the results. Every difference is a decision somebody made about your language, and the third column is the only one that will change when the machine does.
Practice¶
Six jobs, and which of them the standard library finishes. For each of the six, say yes if the standard library can finish the job on its own, no if it needs one of the libraries on this page, or nobody if no program can — and name what the standard library returns.
'café'arrived; get'café'back.- How many characters does a user see in the four-person family emoji?
- Is
'Straße'the same word as'STRASSE', ignoring case? - Which code page is
b'caf\xe9'? - Make
'Łódź'safe for an ASCII URL slug. - Sort
['łódź', 'lód', 'zebra']the way a Polish dictionary does.
Answers
Verified output of where_the_stdlib_stops_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. 'café' arrived; get 'café' back
'café'.encode('cp1252').decode('utf-8') -> 'café'
YES, standard library. Two calls; the skill is knowing to name cp1252.
2. How many characters does a user see in the family emoji?
len(FAMILY) = 7 a user sees 1
NO. Nothing in the standard library implements UAX #29; the regex module's
\X, uniseg or grapheme.
3. Is 'Straße' the same word as 'STRASSE', ignoring case?
lower() == lower(): False
casefold() == casefold(): True
YES, standard library -- but only the second call. lower() is not a fold.
4. Which code page is b'caf\xe9'?
latin-1 -> 'café'
cp1252 -> 'café'
iso-8859-2 -> 'café'
cp1250 -> 'café'
iso-8859-15 -> 'café'
cp1251 -> 'cafй'
koi8-r -> 'cafИ'
mac_roman -> 'cafÈ'
4 different readings from 8 tables, every one of them valid.
NOBODY -- not the standard library, not a detector. One byte carries no
statistics. The answer is whoever wrote the file.
5. Make 'Łódź' safe for an ASCII URL slug
NFKD, drop the marks -> 'Łodz' isascii(): False
NO. ł has no decomposition to strip. anyascii or Unidecode say 'Lodz'.
6. Sort ['łódź', 'lód', 'zebra'] the way a Polish dictionary does
sorted() -> ['lód', 'zebra', 'łódź']
key=locale.strxfrm -> ['lód', 'zebra', 'łódź'] (C locale, as CI runs)
NO. With no locale set, strxfrm is byte order and puts ł after z. pyuca
gets close (lód, łódź, zebra, by treating ł as an l) and ICU with locale
pl gets it right, for the right reason.
Score: 2 of 6 the standard library finishes, 1 nobody can, 3 need a library.
See also¶
- Where
stdstops — the same page for Rust, with the gaps in different places - The five worth installing — the terminal's version of this page, measured against the base toolbox the same way
fileguesses — what a detector is, and the wall every one of them hits- The mojibake round trip — when a repair is possible at all, which
ftfycannot change - Normalization — the recipe transliterators replace, and the letter it leaves
- A code point is not a character — the five rulers, two of which are libraries here
- Sorting and collation — why the order is the locale's, and what that did to a database
- "Supports Unicode" is a level, not a yes —
reandregexplaced against UTS #18 - The table has a version — the rule behind section 8, and behind every dated fence on this page
- Python text in practice — the habits, for the days none of this is needed