The hard strings¶
Level: reference · for anyone who accepts text from outside
One line: A hostile corpus is not a pile of weird text — it is one string per behaviour, there are about twenty behaviours, and this is the list, with a column per check saying which of them reaches each row.
import unicodedata as ud
"caf\u00e9" == "cafe\u0301" # False <- one picture, two values
ud.normalize("NFC", "cafe\u0301") == "caf\u00e9" # True <- a canonical form merges it
"straße".casefold() == "STRASSE".casefold() # True <- this pair needs a fold instead
ud.normalize("NFKC", "10²") # '102' <- and this one loses the exponent
Four lines, four different answers, and nothing in the strings tells you which line you needed. That is the problem this page is a list for.
Why a list, and why a short one¶
The famous hostile corpus is the Big List of Naughty Strings ↗, and it is genuinely useful — five hundred-odd strings, run them through your form, see what breaks. But a pile has two problems. It cannot tell you what it covers, so a green run means "none of these five hundred broke it" and not "the field is sound". And it cannot tell you which strings are the same test twice, so most of a long list is one behaviour wearing different alphabets.
So this list is built the other way round. A string earns a row by breaking one assumption that nothing else on the list breaks — the same rule CAST.md uses for the library's demonstration characters, applied to test data. It comes to nineteen. That is not a claim that nothing else is strange; it is a claim that everything else strange is one of these nineteen in a different script, and if you find one that is not, it has earned a row.
The assumptions, in the order the program below breaks them:
| The assumption | What breaks it |
|---|---|
| Two strings that look the same are the same | composed against decomposed, and three more spellings of the same word |
| Normalizing fixes that | it fixes four of sixteen pairs, and the four it fixes are the four it is for |
| Case-insensitive means one thing | lower(), upper(), casefold() and eq_ignore_ascii_case merge four different sets of pairs |
| Case conversion is a fold | it changes length in both directions, and does not round-trip |
| A trimmed field is trimmed | six invisible characters that no strip() removes — and four more that Python trims and Rust does not |
| Normalization is safe to apply on the way in | NFKC eats an exponent, and NFC rewrites a character used in Japanese names |
| Normalizing the pieces normalizes the whole | it does not; NFC(a) + NFC(b) is not NFC(a + b) |
If it is in a str it can be written out |
a lone surrogate is a legal str and no UTF-8 encoder will take it |
Everything on this page is measured, not asserted — including the counts inside the prose the programs print, which are computed from the same tables they print, so a claim about "four rows" cannot drift away from the four rows above it. The mechanism behind any given row is somebody else's page: Normalization for the four forms and why two spellings exist at all, Confusables and scripts for the two rows no form will ever merge, Case is not a per-character operation for the two case rows, Preparing a string for what a name field should do about all of it.
In Python¶
Verified output of hard_strings_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. FOUR SPELLINGS OF ONE WORD, AND WHY THERE ARE EXACTLY FOUR
------------------------------------------------------------------------
To a reader these are one word, four times. To Python they are four
different values, and nothing about the picture says which you have.
chars bytes code points
A 6 8 0072 00E9 0073 0075 006D 00E9
B 7 9 0072 00E9 0073 0075 006D 0065 0301
C 7 9 0072 0065 0301 0073 0075 006D 00E9
D 8 10 0072 0065 0301 0073 0075 006D 0065 0301
The count is not a curiosity, it is arithmetic. Each accented letter
can be written one way or two, the choices are independent, so a word
with k of them has 2**k spellings -- all of which normalize to one:
distinct values 4
distinct after NFC 1
distinct after NFD 1
the library's Polish pangram, 17 characters, has 8 of them
so it has 2**8 = 256 spellings, every one of them correct
A dictionary keyed on a name therefore has as many slots for that
name as the name has accents, and the user gets a different one
depending on which keyboard, phone or paste they arrived through.
2. ONE PICTURE, MANY VALUES -- AND THE CHECK THAT MERGES EACH PAIR
------------------------------------------------------------------------
16 pairs. Every one of them draws the same, or close enough that
no reader will query it, and every one is two different values. A '='
means that check calls them equal; a '-' means it does not.
left right == NFC NFD NFKC cf cf+K what it is
'caf\xe9' 'cafe\u0301' - = = = - = composed against decomposed
'\xc5' '\u212b' - = = = = = ANGSTROM SIGN against A WITH RING
'\u03a9' '\u2126' - = = = = = OHM SIGN against GREEK CAPITAL OMEGA
'q\u0307\u0323' 'q\u0323\u0307' - = = = - = two marks, written in two orders
'\ufb01le' 'file' - - - = = = the fi ligature
'\xbd' '1\u20442' - - - = - = a vulgar fraction
'10\xb2' '102' - - - = - = a superscript two
'\uff21\uff22' 'AB' - - - = - = fullwidth forms
'\u2168' 'IX' - - - = - = a Roman numeral
'stra\xdfe' 'STRASSE' - - - - = = the German sharp s
'\u03a3' '\u03c2' - - - - = = Greek final sigma
'\u212a' 'k' - - - - = = KELVIN SIGN against the letter k
'I' '\u0131' - - - - - - Turkish dotless i
'a' '\u0430' - - - - - - Cyrillic a
'admin' 'ad\u200bmin' - - - - - - a zero-width space
'admin' 'ad\xadmin' - - - - - - a soft hyphen
merged, of 16 0 4 4 9 6 12
Read it column by column and the checks separate into jobs.
NFC and NFD merge the same four rows and no others, because they are
the CANONICAL forms: they merge spellings of one character and refuse
to merge anything else. Note row 4 -- two combining marks in two
orders -- which is not about composing at all. Canonical ordering
sorts marks by combining class, so both forms fix it and == does not.
NFKC merges five more, and every one of those five is a DIFFERENT
character being flattened onto a letter it merely resembles. That is
why 10**2 becomes 102: the compatibility forms are a lossy fold, and
the loss is the point of them.
The casefold() column is the surprise, and it is why it is worth
printing rather than reasoning about. It merges six, and only three
of those six are about letter case. Full case folding also folds the
ANGSTROM SIGN onto a-ring and the fi ligature onto two letters, so it
is not a pure case operation and never was -- it is the fold that
makes two strings the same KEY, whatever made them differ.
Only the last column -- fold, casefold, fold again -- gets every
family at once, which is what an identifier comparison needs.
And the last four rows are merged by nothing here, which is the
important part of the table. Two of them are two genuinely different
letters that history drew alike; two are a character you cannot see
at all. No normalization form will ever touch either kind, because
neither is a spelling of the other.
3. CASE MAPPING IS NOT A FOLD -- IT CHANGES LENGTH, IN BOTH DIRECTIONS
------------------------------------------------------------------------
char upper() lower() casefold() name
'\xdf' 'SS' '\xdf' 'ss' LATIN SMALL LETTER SHARP S
'\ufb01' 'FI' '\ufb01' 'fi' LATIN SMALL LIGATURE FI
'\u0130' '\u0130' 'i\u0307' 'i\u0307' LATIN CAPITAL LETTER I WITH DOT ABOVE
'\u1e9e' '\u1e9e' '\xdf' 'ss' LATIN CAPITAL LETTER SHARP S
'\u01c8' '\u01c7' '\u01c9' '\u01c9' LATIN CAPITAL LETTER L WITH SMALL LETTER J
Five things in that table are worth saying out loud.
One character can uppercase to two, so upper() is not a per-character
substitution and a fixed-width column can overflow on a name it
accepted yesterday. LATIN CAPITAL LETTER I WITH DOT ABOVE does it in
the other direction: it LOWERCASES to two characters.
Row four is the reason row one is not a typo. A capital sharp s does
exist, and it lowercases to the small one -- but upper() does not
produce it, because the two-letter answer is the one German readers
expect. So the pair is asymmetric on purpose.
Case conversion does not round-trip, and each of the two fails in a
different direction -- which is why one example would have misled:
char upper().lower() lower() agree?
'\xdf' 'ss' '\xdf' False
'\u0130' 'i\u0307' 'i\u0307' True
char lower().upper() upper() agree?
'\xdf' 'SS' 'SS' True
'\u0130' 'I\u0307' '\u0130' False
And lower() is not casefold(). lower() is for display -- it answers
'how is this written in lower case'. casefold() is for comparison and
answers 'what do I hash', which is why it may produce a string nobody
would ever write, and why it is the one to compare with.
The fifth row is the one people do not know is there: a TITLECASE
character, distinct from both its upper and its lower form, so the
mapping is three-way rather than two-way.
What Python cannot show you here is the locale. str.upper() is
locale-independent by design, so 'i'.upper() is 'I' on every machine
in the world. Java, C#, ICU and a database collation are not, and in
a Turkish locale that same call returns a NON-ASCII character. The
page has the measurement; the point for the corpus is that the row
exists and this program is the wrong tool to find it with.
4. THE ONES YOU CANNOT SEE, AND WHICH OF THEM strip() TAKES
------------------------------------------------------------------------
code point cat isspace stripped name
U+0020 Zs True yes SPACE
U+00A0 Zs True yes NO-BREAK SPACE
U+3000 Zs True yes IDEOGRAPHIC SPACE
U+2007 Zs True yes FIGURE SPACE
U+200B Cf False NO ZERO WIDTH SPACE
U+200C Cf False NO ZERO WIDTH NON-JOINER
U+200D Cf False NO ZERO WIDTH JOINER
U+00AD Cf False NO SOFT HYPHEN
U+2060 Cf False NO WORD JOINER
U+FEFF Cf False NO ZERO WIDTH NO-BREAK SPACE
The break is between the two blocks and it is not where the eye puts
it. The first four are spaces: Unicode calls them whitespace, strip()
removes them, and a trimmed field is trimmed. The last six are format
characters -- Cf -- and every one of them is invisible, is not
whitespace to Python, and survives every trim in the standard library.
'admin' 'ad\u200bmin' equal? False
len 5 len 6 and it is not the len that gets read
after strip() 'ad\u200bmin'
after NFKC 'ad\u200bmin'
Two usernames, one picture, one of them still there after every
cleaning step in the standard library. The fix is not a normalization
form; it is a rule that says which categories a name may contain.
And four more, which are here because the ANSWER IS NOT THE SAME IN
EVERY LANGUAGE. These are C0 controls rather than spaces or format
characters, and Python calls them whitespace:
code point cat isspace stripped name
U+001C Cc True yes FILE SEPARATOR (FS)
U+001D Cc True yes GROUP SEPARATOR (GS)
U+001E Cc True yes RECORD SEPARATOR (RS)
U+001F Cc True yes UNIT SEPARATOR (US)
Unicode does not: none of the four has the White_Space property, so
a language that asks Unicode rather than carrying its own table
gives the opposite answer. The Rust section on this page prints its
column for the same fourteen characters, and these are the four rows
where the two blocks disagree -- which matters because U+001E is the
ASCII record separator, reached for precisely BECAUSE it is not
supposed to be text a trim would touch.
5. NORMALIZATION THROWS THINGS AWAY, AND NFC DOES IT TOO
------------------------------------------------------------------------
before NFKC after chars what was lost
'10\xb2' '102' 3 -> 3 an exponent becomes a digit
'\xbd' '1\u20442' 1 -> 3 a fraction becomes three characters
'\u2168' 'IX' 1 -> 2 a numeral becomes two letters
'\u337f' '\u682a\u5f0f\u4f1a\u793e' 1 -> 4 one character becomes four
The champion expansion is an Arabic ligature this library does not
print, because a right-to-left character would reorder the row it
sits in and no page here teaches bidi. Its name and its numbers are
safe to give:
U+FDFA ARABIC LIGATURE SALLALLAHOU ALAYHE WASALLAM
NFKC turns 1 character into 18
NFKC being lossy is well known. This is the one that is not:
'\ufa10' CJK COMPATIBILITY IDEOGRAPH-FA10
NFC -> '\u585a' CJK UNIFIED IDEOGRAPH-585A
NFC -- the safe one, the one everybody recommends storing -- replaces
that character with a different one. It is a CJK compatibility
ideograph, and several of them are in Japanese personal names, which
is why 'just normalize on the way in' is a decision about somebody's
name and not a tidy-up.
And normalizing the pieces is not normalizing the whole:
NFC('e') + NFC('\u0301') 'e\u0301' 2 chars
NFC('e' + '\u0301') '\xe9' 1 chars
equal? False
So a template that normalizes each field and then concatenates has
not produced normalized output, and a stream normalized chunk by
chunk is normalized nowhere except inside the chunks.
6. THE STRING YOUR ENCODER WILL REFUSE
------------------------------------------------------------------------
'\ud800' len 1 Cs a lone surrogate, in a str, legally
.encode() raises UnicodeEncodeError
.encode(errors='surrogatepass') eda080
Python's str is a sequence of code points, and a surrogate is a code
point -- so it goes in, and only the encoder objects. That is the
shape of the whole family: the value is legal in memory, legal in
JSON, and not encodable, so it fails at the boundary rather than at
the assignment. Test the boundary, not the constructor.
7. THE CORPUS, AS ESCAPES YOU CAN PASTE
------------------------------------------------------------------------
Every entry above, written the only way that survives a copy through
an editor, a ticket and a chat window. A bare combining mark or a
zero-width space does not.
nfc_nfd = 'cafe\u0301'
mark_order = 'q\u0323\u0307'
singleton = '\u212b'
ligature = '\ufb01le'
superscript = '10\xb2'
fullwidth = '\uff21\uff22'
sharp_s = 'stra\xdfe'
final_sigma = '\u03c2'
kelvin = '\u212a'
dotless_i = '\u0131'
dotted_I = '\u0130'
confusable = '\u0430'
zero_width = 'ad\u200bmin'
soft_hyphen = 'ad\xadmin'
nbsp = 'a\xa0b'
bom_inside = 'a\ufeffb'
compat_han = '\ufa10'
expanding = '\u337f'
lone_surrogate = '\ud800'
19 strings, one per behaviour. That is the whole list -- not
because nothing else is strange, but because everything else strange
is one of these 19 wearing a different alphabet.
In the terminal¶
The corpus does not arrive in a program. It arrives in a file, and the tools that touch it first have no Unicode tables at all.
Verified output of hard_strings_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
1. FOUR SPELLINGS ARRIVE AS FOUR LINES
------------------------------------------------------------------------
00000000: 72c3 a973 756d c3a9 0a72 c3a9 7375 6d65 r..sum...r..sume
00000010: cc81 0a72 65cc 8173 756d c3a9 0a72 65cc ...re..sum...re.
00000020: 8173 756d 65cc 810a .sume...
bytes per line, newline included 9 10 10 11
Four byte counts for one word. Nothing in that file is malformed
and nothing in it is unusual; it is one name, typed on four
machines.
2. sort -u IS THE DEDUPE EVERY PIPELINE USES
------------------------------------------------------------------------
lines in the file 4
after sort -u 4
grep -c for the whole word 1
grep -c for its first five 2
One of four, then two of four. The prefix search finds the spellings
whose FIRST accent is precomposed and misses the two whose is not --
and the letter it disagrees about is not the letter you searched for.
A report built this way is not wrong in a way anybody will notice.
3. VALID IS NOT THE SAME AS SAME
------------------------------------------------------------------------
iconv -f UTF-8 -t UTF-8, exit 0
Zero: all four lines are well-formed UTF-8, because they are. A
validator answers a question about bytes and this is a question
about meaning, so no amount of validation upstream will help.
4. THE ONE THE EYE CANNOT AUDIT
------------------------------------------------------------------------
00000000: 6164 6d69 6e0a 6164 e280 8b6d 696e 0a admin.ad...min.
grep -c for the exact line 1
after sort -u 2
Two accounts. Printed to a terminal, pasted into a ticket, or read
off a screenshot they are the same five letters -- the three bytes
E2 80 8B draw nothing at all. The only place the difference exists
is the hex dump above.
5. WHAT THE SHELL CAN AND CANNOT DO HERE
------------------------------------------------------------------------
Everything in this script ran under LC_ALL=C, where sort and grep
are pure byte matchers -- which is the right setting for auditing
unknown data, and is what makes the output above the same on every
machine. Turning the locale up does not fix any of it: a UTF-8
locale teaches these tools about character boundaries and collation
order, and none of the five sections above is a boundary question.
So the shell's honest job with this corpus is to SHOW it -- xxd, and
the byte counts -- and to hand the merging decision to a program
that has a Unicode table. Reach for sort -u to find duplicates and
you will get an answer; it just will not be about names.
In Rust¶
Python answers every question this corpus asks, which makes the Python section read like a list of functions. Rust's standard library answers some of them and has no opinion about the rest — so it is the better place to see which half of the problem is a language's job and which half is a policy your program has to choose.
Two of its sections are deliberately the same data as Python's, so the blocks can be read against each other: the sixteen pairs in section 2, and the fourteen invisible characters in section 5. Section 5 is where they disagree, and it is a finding rather than a restatement. Python and Rust give the same answer for ten of the fourteen and the opposite answer for the last four — U+001C–U+001F, the ASCII file, group, record and unit separators. Python's strip() removes them; Rust's trim() keeps them, because char::is_whitespace is Unicode's White_Space property and none of those four has it. So "trim the whitespace" is not one operation across two languages, and the characters it disagrees about are exactly the ones people reach for as delimiters because they are not text. Neither language is wrong and a pipeline that crosses them is.
Verified output of hard_strings_rs.rs — regenerated by tools/run_examples.py, never hand-typed.
1. == IS BYTES, AND THE TYPE SAYS SO
------------------------------------------------------------------------
chars bytes code points
A 6 8 0072 00E9 0073 0075 006D 00E9
B 7 9 0072 00E9 0073 0075 006D 0065 0301
C 7 9 0072 0065 0301 0073 0075 006D 00E9
D 8 10 0072 0065 0301 0073 0075 006D 0065 0301
distinct values 4
str::len() is the BYTE count and has never pretended otherwise,
so the number Python makes you ask for is the one Rust gives you
by default. Neither language merges the four.
2. THE SAME SIXTEEN PAIRS, AND EVERYTHING std CAN DO TO THEM
------------------------------------------------------------------------
left right == lower upper asciieq what it is
"café" "cafe\u{301}" - - - - composed against decomposed
"Å" "Å" - = - - ANGSTROM SIGN against A WITH RING
"Ω" "Ω" - = - - OHM SIGN against GREEK CAPITAL OMEGA
"q\u{307}\u{323}" "q\u{323}\u{307}" - - - - two marks, in two orders
"file" "file" - - = - the fi ligature
"½" "1⁄2" - - - - a vulgar fraction
"10²" "102" - - - - a superscript two
"ＡＢ" "AB" - - - - fullwidth forms
"Ⅸ" "IX" - - - - a Roman numeral
"straße" "STRASSE" - - = - the German sharp s
"Σ" "ς" - - = - Greek final sigma
"K" "k" - = - - KELVIN SIGN against the letter k
"I" "ı" - - = - Turkish dotless i
"a" "а" - - - - Cyrillic a
"admin" "ad\u{200b}min" - - - - a zero-width space
"admin" "ad\u{ad}min" - - - - a soft hyphen
merged, of 16 0 3 4 0
No column reaches more than four; between them they reach 7 of
the sixteen. Nothing std has touches the first row or the fourth:
composing and mark ordering are normalization, std has no
normalize(), and no amount of case conversion is going to help.
The four compatibility rows -- fraction, superscript, fullwidth,
numeral -- are out of reach for the same reason.
The rows it does get are split across the two case directions,
and no single direction gets them all:
"Å" "Å" lower true upper false ANGSTROM SIGN against A WITH RING
"straße" "STRASSE" lower false upper true the German sharp s
"K" "k" lower true upper false KELVIN SIGN against the letter k
"I" "ı" lower false upper true Turkish dotless i
The sharp s merges going UP, because uppercasing it produces
two letters. The KELVIN SIGN merges going DOWN, because its
lowercase is an ordinary k. Fold the other way in either case
and they stay apart. So neither direction of case conversion is
a comparison, and that is what a case FOLD is for: one operation
that catches both, defined so it need not produce a string
anyone would write. Python spells it casefold(). Rust's std has
no spelling for it at all.
And the fourth row above is the reason to want the fold rather
than a direction. Uppercasing merges Turkish dotless i onto
ASCII I -- Unicode's default mapping, and nothing to do with a
Turkish locale -- so a program comparing by to_uppercase() has
just decided that two different letters are one. Python's
casefold() keeps them apart. Comparing by uppercasing does not
merge FEWER pairs than a fold; it merges the WRONG ones.
3. char::to_lowercase RETURNS AN ITERATOR, WHICH IS THE HONEST TYPE
------------------------------------------------------------------------
One character can case-map to several, so the return type cannot
be char. Rust makes you collect it; Python hides the same fact
behind a str that is quietly longer than the one you passed in.
char upper n(up) lower n(lo) code points of lower
'ß' "SS" 2 "ß" 1 00DF
'fi' "FI" 2 "fi" 1 FB01
'İ' "İ" 1 "i\u{307}" 2 0069 0307
'ẞ' "ẞ" 1 "ß" 1 00DF
'Lj' "LJ" 1 "lj" 1 01C9
Both n columns reach 2, in different rows: the sharp s and the
ligature grow going up, and LATIN CAPITAL LETTER I WITH DOT
ABOVE grows going down. A program that wrote c.to_uppercase()
and expected a char would not compile, which is the whole
difference between the two languages here -- Python returns a
str either way and lets you find out at the column width.
4. str::to_lowercase IS NOT A MAP OVER chars
------------------------------------------------------------------------
The Greek final sigma is decided by POSITION, so the str method
has to look at neighbours -- and it does, where the per-char one
cannot:
input "ΣΣ" 03A3 03A3
str::to_lowercase "σς" 03C3 03C2
char by char "σσ" 03C3 03C3
same? false
Two capital sigmas lowercase to a medial sigma and a FINAL one,
because the second ends the word. Doing it a character at a time
gives two medial sigmas and a Greek reader a typo. So even the
case operation std does keep is not a per-character table, and
the convenient-looking .chars().flat_map(...) is the wrong loop.
5. THE SAME FOURTEEN INVISIBLES, AND WHERE THE TWO LANGUAGES PART
------------------------------------------------------------------------
char::is_whitespace() IS Unicode's White_Space property, and
trim() is defined in terms of it. The Python section above asks
str.isspace() and str.strip() over these same fourteen.
code point is_whitespace trimmed name
U+0020 true yes SPACE
U+00A0 true yes NO-BREAK SPACE
U+3000 true yes IDEOGRAPHIC SPACE
U+2007 true yes FIGURE SPACE
U+200B false NO ZERO WIDTH SPACE
U+200C false NO ZERO WIDTH NON-JOINER
U+200D false NO ZERO WIDTH JOINER
U+00AD false NO SOFT HYPHEN
U+2060 false NO WORD JOINER
U+FEFF false NO ZERO WIDTH NO-BREAK SPACE
U+001C false NO FILE SEPARATOR (FS)
U+001D false NO GROUP SEPARATOR (GS)
U+001E false NO RECORD SEPARATOR (RS)
U+001F false NO UNIT SEPARATOR (US)
The first ten rows agree with Python's, value for value. The
last four do not: Python calls all four whitespace and strips
them, and Rust calls none of them whitespace and keeps them.
Neither is wrong, and the reason is worth having. Rust asks
Unicode, which does not give White_Space to any C0 control in
that block; Python carries a table of its own that predates the
property and has to stay compatible with itself. So "trim the
whitespace" is not one operation across two languages, and the
four characters it disagrees about are the ASCII separators --
reached for as delimiters precisely BECAUSE they are not text.
A record separator that survives a Rust trim and vanishes in a
Python one is a framing bug that only shows up at the seam.
6. THE ONE ROW RUST CANNOT EVEN HOLD
------------------------------------------------------------------------
The corpus ends with a lone surrogate, which Python puts in a
str quite happily and refuses only at the encoder. Rust refuses
it at the type:
char::from_u32(0xD7FF) is_some true the code point just below the surrogates
char::from_u32(0xD800) is_some false the first surrogate
char::from_u32(0xDFFF) is_some false the last surrogate
char::from_u32(0xE000) is_some true the code point just above them
char::from_u32(0x10FFFF) is_some true the highest code point there is
char::from_u32(0x110000) is_some false one past the end
str::from_utf8(ED A0 80) Err, valid_up_to 0, error_len Some(1)
A char is a Unicode SCALAR VALUE, which is a code point that is
not a surrogate -- so the literal '\u{D800}' is not a value this
program could contain even if it wanted to; it is a compile
error, which is why the bytes are built by hand above.
That is the sharpest difference on the page. Python's str holds
the whole code point range and fails at the boundary; Rust's
types hold the encodable subset and fail at construction. Both
are defensible, and a corpus has to know which it is testing:
in Python the lone surrogate is a value your code can carry to
the edge of the system, and in Rust it is three bytes that will
never become a String at all.
7. THE CORPUS, AS RUST ESCAPES YOU CAN PASTE
------------------------------------------------------------------------
let nfc_nfd = "cafe\u{301}";
let mark_order = "q\u{323}\u{307}";
let singleton = "\u{212b}";
let ligature = "\u{fb01}le";
let superscript = "10\u{b2}";
let fullwidth = "\u{ff21}\u{ff22}";
let sharp_s = "stra\u{df}e";
let final_sigma = "\u{3c2}";
let kelvin = "\u{212a}";
let dotless_i = "\u{131}";
let dotted_i = "\u{130}";
let confusable = "\u{430}";
let zero_width = "ad\u{200b}min";
let soft_hyphen = "ad\u{ad}min";
let nbsp = "a\u{a0}b";
let bom_inside = "a\u{feff}b";
let compat_han = "\u{fa10}";
let expanding = "\u{337f}";
18 of the 19. The nineteenth, the lone surrogate, has no line here
for the reason section 6 gives -- it cannot be written as a Rust
literal, so a corpus in this language carries it as bytes or not
at all.
8. WHAT IS MISSING IS A LINE, NOT A GAP
------------------------------------------------------------------------
std has no normalize() and no casefold(), and it is worth being
clear that this is a decision rather than an omission.
std does ship Unicode data: the tables behind a char's own
properties and its case mappings, which is why is_whitespace()
and to_lowercase() work on every script -- and those tables
carry a version, the thing this library keeps out of answer
keys. What std stops short of is the algorithms that look past
one char, and the larger data they need: normalization,
segmentation, locales.
So the line is what a char can answer about itself, plus the
one context rule above. Everything past that is a crate:
unicode-normalization for the four forms, caseless or icu for a
fold. The corpus does not get easier in Rust; it gets explicit.
The row that needs a fourth language to show it¶
One behaviour in the list above is stated in prose rather than printed, because neither Python nor Rust can produce it: case mapping that depends on the locale, where a string of pure ASCII uppercases to something that is not ASCII.
Python's str.upper() and Rust's str::to_uppercase are both locale-independent by design, so 'i'.upper() is 'I' on every machine in the world. Java's String.toUpperCase() is not — the no-argument overload uses the JVM's default locale, which is a property of the machine the code happens to run on:
openjdk 25.0.4.1 2026-08-18 (Homebrew), macOS 26 / Darwin 25.6.0, measured 2026-09-08
"title".toUpperCase() TITLE U+0054 U+0049 U+0054 U+004C U+0045
"title".toUpperCase(Locale.ROOT) TITLE U+0054 U+0049 U+0054 U+004C U+0045
"title".toUpperCase(tr) TİTLE U+0054 U+0130 U+0054 U+004C U+0045
"I".toLowerCase(Locale.ROOT) i U+0069
"I".toLowerCase(tr) ı U+0131
equalsIgnoreCase("TITLE") true
toUpperCase(tr).equals("TITLE") false
Five ASCII letters in, and on a Turkish machine one of them comes out as U+0130. The failure this produces is not a mangled name — it is "title".toUpperCase().equals("TITLE") returning false, so an SQL keyword, an HTTP header or a config key stops matching itself, on one developer's laptop, and nothing in the code mentions Turkish. The fix is the argument the first line did not pass: Locale.ROOT when the string is a protocol token, the user's locale only when the string is being shown to them.
The row belongs in the corpus even though this library's own languages cannot reach it, and saying which language produced a measurement is the point of a dated fence.
Using it¶
The escapes in section 7 of the Python output are the deliverable, and the Rust section 7 prints the same list as Rust literals — both written the only way that survives a copy through an editor, a ticket and a chat window, because a bare combining mark or a zero-width space does not. The Rust list is one short: a lone surrogate is not a value a Rust program can hold, which its section 6 measures. Paste them into your test file and drive them through whichever boundary you actually own: the form, the API, the importer, the filename, the search box.
What to look for is not a crash. Every one of these strings is well-formed UTF-8, so a validator passes them all — the shell section measures exactly that. The failures are quieter: a duplicate account, a search that finds nothing, a name that comes back shorter, a column that overflows, two rows in a GROUP BY that a human reads as one.
And when you decide what to do about them, the decision is one line in one place: normalize at the boundary, on the way in, once — not inside the comparison function, where a == b starts depending on who asked. Normalization makes that argument properly; this page's contribution is the evidence that no single call does the whole job.
If you are coming from Python or ABAP¶
Python. unicodedata.normalize for the four forms, str.casefold() for comparison and str.lower() for display, and unicodedata.category(c) to spot the invisibles — the six that survive strip() are all Cf, so a name rule can be written as a category test rather than a blocklist of code points. The identifier fold the standards specify is NFKC_Casefold, and Python has no single call for it; the usual reconstruction is normalize("NFKC", normalize("NFKC", s).casefold()), with the doubled call deliberate. That is an approximation, and the last two rows of the table above are where it comes apart — the real property maps the invisibles to nothing, and no composition of normalize and casefold will do that.
The property is a data file rather than an algorithm, so it can simply be read:
DerivedNormalizationProps.txt, Unicode Public/UCD/latest, fetched 2026-09-08
(an empty third field means: maps to nothing)
00A0 ; NFKC_CF; 0020 # Zs NO-BREAK SPACE
00AD ; NFKC_CF; # Cf SOFT HYPHEN
200B..200F ; NFKC_CF; # Cf [5] ZERO WIDTH SPACE..RIGHT-TO-LEFT MARK
FEFF ; NFKC_CF; # Cf ZERO WIDTH NO-BREAK SPACE
So a conformant identifier fold turns ad<ZWSP>min into admin and Python's two-call version does not, which is the difference between the - in that row and the = a username check needed. It is in a dated fence rather than an answer key because NFKC_Casefold is a derived property with a version, unlike the four normalization forms, which the stability policy ↗ freezes.
ABAP. (Not machine-checked — CI cannot run ABAP.) There is no normalization in the language, so = on two strings is a code-unit comparison with every failure on this page available to it, and the most likely source of a decomposed name is a file from a Mac. TRANSLATE ... TO UPPER CASE is the case operation you have, and its shape is the one this page argues against for comparison: a character-wise mapping, no locale argument, and no fold — so it is the upper column of the Rust matrix rather than the cf one, and that column merged the wrong pairs. Which specific pairs it merges is a question for the system that will run the job, not for a page: test it there, the same way you would verify a code-page number. Practically: do the folding at the interface, in whatever is on the other side of it, and let ABAP receive text that has already been decided about.
Try it¶
- Take the escapes from section 7 and put every one of them through your own login or signup form. Count the accounts you end up with.
- Export a list of names from a system you own and run
python3 -c "import sys,unicodedata as u; [print(l, end='') for l in sys.stdin if not u.is_normalized('NFC', l)]"over it. Anything it prints arrived from somewhere that is not your keyboard. grep -cyour own surname in a file that came from a Mac, then try again with the accent written as a combining mark.- Find the place in your code where two strings from different sources are compared. Decide which of the six columns in the table above that comparison needs, and whether it is doing it.
- Uppercase a config key or an SQL keyword in whatever language you use, with the locale set to
tr-TR. If nothing changes, find out whether that is because the call is locale-independent or because your machine is not Turkish.
Practice¶
Five pairs, and the checks that reach them. None of these five is in the table above, so it cannot be answered by looking one row up. For each pair, write down which of NFC, NFKC, casefold() and NFKC-then-casefold call the two strings equal — all of them, none of them, or some. Then run it.
1. '\u1e9b\u0323' against '\u1e69'
2. '\u0130' against 'i\u0307'
3. '\u01c4' against 'D\u017d'
4. '\xb5' against '\u03bc'
5. '\ufeffdata' against 'data'
Four of the five are merged by something. One is merged by nothing, and it is not the one that looks strangest.
Answers
Verified output of hard_strings_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.
left right NFC NFKC cf NFKC+cf
'\u1e9b\u0323' '\u1e69' - = - =
'\u0130' 'i\u0307' - - = =
'\u01c4' 'D\u017d' - = - =
'\xb5' '\u03bc' - = = =
'\ufeffdata' 'data' - - - -
1. long s with dot above, plus a dot below, against s with both dots
2. capital I with dot above, against i plus a combining dot
3. the DZ-with-caron digraph, against the two letters
4. MICRO SIGN, against GREEK SMALL LETTER MU
5. a BOM in front of an ASCII word
1 is the example UAX #15 uses to show that NFC is not 'the composed
one'. Composing cannot help here -- there is no single character
for a long s with both dots -- so NFC leaves the pair alone and
the COMPATIBILITY form, which is allowed to replace the long s
with an ordinary one, is the first thing that merges them.
2 is merged by casefold() and by nothing before it. The capital I
with dot above has no canonical decomposition, so no amount of
normalizing produces 'i' plus a mark -- but its case mapping
does exactly that, which is a case question wearing a
normalization costume.
3 is a digraph with a compatibility decomposition, so NFKC splits
it into two letters. Note what that costs: the string got
LONGER, and a field with room for one character now holds two.
4 is the trap. casefold() merges MICRO SIGN onto Greek mu, and
there is no case anywhere in that pair -- both are lower case
already. Case folding is not a case operation with a tidy name;
it is the fold that makes two strings one KEY, and it inherits
a handful of compatibility mappings on the way.
5 is merged by nothing, and it is the one that will actually reach
your database. A BOM is a legal character in the middle of a
string, it draws nothing, no normalization form removes it, and
casefold() keeps it too. The check it fails is not a comparison
at all -- it is a rule about which characters a field may hold.
If you got 4 wrong you were reasoning from the NAME of the
function, which is the habit the whole page is arguing against.
See also¶
- Normalization — why
éhas two spellings, what the four forms do, and the stability policy that makes a normalization result safe to write into an answer key - Confusables and scripts — the two rows in the table that no normalization form will ever merge, and the whole-string rule that catches them instead
- Preparing a string — map, normalize, screen, shape-check: the four steps in the order the standards put them
- Case is not a per-character operation — the mechanism behind the two case rows: why a fold can change a string's length, depend on its neighbours, and still not know the language
- PRECIS: stringprep, after stringprep — what the successor standard does with the zero-width-space row, and why its verdict depends on which Unicode the implementation was built against
- Two people, one account — what happens when the corpus reaches a system that hashes before it folds
- The check that ran too early — the same strings as a security property, where the order of canonicalise and check is the whole bug
- A code point is not a character — the other counting question, and the five rulers that disagree about one string
- CAST.md — the demonstration characters this library reuses, and the rule this corpus borrows for choosing rows
- Anki: hexadecimal — the other resource in this chapter, on the other side of the same problem: this page is what you test with, that one is what you review with