Preparing a string¶
Level: 301 · for anyone who stores a name, a login or a domain
One line: Before two strings can be compared they have to be prepared — mapped, normalized, screened and shape-checked — and that step, not the encoding, is the part of the internet that has had to be redesigned twice.
import stringprep, unicodedata
stringprep.in_table_b1('\u200b') # True <- a ZERO WIDTH SPACE: delete it, first of all
stringprep.map_table_b2('ß') # 'ss' <- fold it, and the string gets longer
unicodedata.normalize('NFKC', '℀') # 'a/c' <- and now a path separator is in your name
Everything in chapter 2 so far has been about a character having a number. This page is about the question that comes next and is much harder: are these two strings the same name? Not do these bytes match — chapter 3 settles that — but whether the thing a person typed in Kraków and the thing stored in a database in Ohio are the same identifier, when both people believe they typed café.
Encoding cannot answer it. Two strings can be perfectly valid UTF-8, byte-for-byte different, and identical to every human who looks at them. Somebody has to decide, in writing, which differences count.
Four steps, and the order is normative¶
RFC 3454 ↗ — "Preparation of Internationalized Strings", universally called stringprep — is that decision, written down in December 2002. It defines four steps and insists on their order:
- Map — delete the characters whose presence must not matter, and fold case.
- Normalize — usually NFKC.
- Prohibit — if a forbidden character survives, return an error.
- Check bidi — a string mixing right-to-left and left-to-right has to satisfy a shape rule.
And one constraint that shapes every implementation built on it: a profile returns a string or an error, never both. There is no "prepared with warnings".
RFC 3454 is deliberately incomplete. It defines no policy of its own — it is a framework, and a protocol that wants to use it publishes a profile saying which tables it applies. Nameprep ↗ is the profile for domain names, SASLprep ↗ the one for usernames and passwords. That indirection is the whole design: the same four steps, different answers about what a name may contain.
The tables are already on your machine¶
Python ships RFC 3454's appendices as a stdlib module. Not a reimplementation of the idea — the actual tables, generated from the document, pinned to the Unicode version the RFC pinned. That is what makes this page printable, for the reason The table has a version sets out — with one exception, which the program finds for itself in section 9: the case fold also reads the live table, and until a 2026 fix that leaked into its answers. So the program prints no fold of a character the RFC leaves alone, and every answer below is identical on every machine, before that fix and after it.
Verified output of preparing_a_string_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. THE TABLES OF A 2002 RFC, SHIPPED IN YOUR PYTHON
------------------------------------------------------------------------
import stringprep
membership tests it exposes 17
mapping tables 2 (map_table_b2, map_table_b3)
Every one is an appendix of RFC 3454 turned into a function, and the
module opens by pinning the table it reads them from:
from unicodedata import ucd_3_2_0 as unicodedata
assert unicodedata.unidata_version == '3.2.0'
(A 2026 fix renames the alias unicodedata_320 and adds a second
assert -- on the LIVE table. Section 9 is why it needs one.)
So this is the frozen table from `The table has a version`, with a
protocol built on top of it -- for every table but one: the case
fold reads the live table too. Everything printed below is the same
on every machine, before that fix and after it.
2. FOUR STEPS, AND THE ORDER IS NORMATIVE
------------------------------------------------------------------------
map -> normalize -> prohibit -> check bidi
plain 4 code points 'Cafe' -> 'cafe'
composed 4 code points 'café' -> 'café'
decomposed 5 code points 'café' -> 'café'
case folded 4 code points 'CAFÉ' -> 'café'
Four spellings, one prepared string. That is the entire purpose:
two people who think they typed the same name now have.
Rows 2 and 3 are the same picture on screen, five code points
against six, and the decomposed one is BUILT here rather than
typed, because a decomposed string cannot be typed. NFKC is what
reconciles them; preparation without it would keep them apart.
3. MAPPED TO NOTHING: THE CHARACTERS THAT SIMPLY VANISH
------------------------------------------------------------------------
Table B.1 is deleted before anything else happens, because whether
these are present or absent must not make two names different.
U+00AD SOFT HYPHEN in tables ['b1']
U+200B ZERO WIDTH SPACE in tables ['b1', 'c12']
U+200D ZERO WIDTH JOINER in tables ['b1', 'c22']
U+FEFF ZWNBSP -- the BOM, mid-string in tables ['b1', 'c22']
U+034F COMBINING GRAPHEME JOINER in tables ['b1']
U+FE0F VARIATION SELECTOR-16 in tables ['b1']
U+FEFF is in two of them at once, which is not a contradiction: B.1
deletes it, C.2.2 would reject it, and a profile says which of the
two it applies. Nameprep deletes.
the cast's ZWJ family '👩\u200d👩\u200d👧'
5 code points
after step 1 '👩👩👧'
3 code points
The joiners are gone, so one family became three people. Deleting
an invisible character is the right call for a login name and the
wrong one for text a human will read back. A profile is a choice.
4. FOLDING THAT CHANGES THE LENGTH
------------------------------------------------------------------------
U+00DF ß LATIN SMALL LETTER SHARP S -> 'ss' (2)
U+0130 İ LATIN CAPITAL LETTER I WITH DOT ABOVE -> 'i̇' (2)
U+2126 Ω OHM SIGN -> 'ω' (1)
Sharp s folds to two characters -- so a name gets LONGER during a
step whose job was to make it comparable. A profile must expect
that; RFC 3454 says so in as many words. U+0130 does it with a
combining mark instead.
The capital, U+1E9E, is missing on purpose. It is younger than the
RFC, and what this module folds it to depends on which release of
Python you run -- section 9.
B.2 is upper-to-lower, chosen because Internet protocols had a
tradition of lowercase. The RFC built it by iterating to a fixed
point, which is worth reading as pseudocode because it is four
lines and it invented what Unicode later called NFKC_Casefold:
b = NormalizeWithKC(Fold(a))
c = NormalizeWithKC(Fold(b))
if c is not the same as b, add a mapping for a -> c
Fold once and normalizing can undo it; fold and normalize until
nothing moves, and the table is stable from that point on.
5. THE SAME NAME, PREPARED TWO WAYS, IS TWO DIFFERENT DOMAINS
------------------------------------------------------------------------
'faß.de' .encode('idna') -> b'fass.de'
'STRASSE.de' .encode('idna') -> b'STRASSE.de'
'Bücher.de' .encode('idna') -> b'xn--bcher-kva.de'
'ExAmPlE.com' .encode('idna') -> b'ExAmPlE.com'
Row 1 is the whole argument. Nameprep is a stringprep profile, and
its B.2 folds sharp s to 'ss' -- so the prepared label is pure
ASCII, punycode never runs, and the string a German typed resolves
to a name they did not type. IDNA2008 abolished the mapping step
and keeps the character, giving a DIFFERENT registrable domain from
the same keystrokes. Both are correct implementations of a standard.
Row 4 is the quieter trap. An all-ASCII label skips preparation
entirely (RFC 3490 section 4.1), so the case survives -- while row 3
was folded on its way through. One function, two contracts, chosen
by a character you did not type.
6. PROHIBITION IS A DESIGN TOOL, NOT AN ERROR PATH
------------------------------------------------------------------------
RIGHT-TO-LEFT OVERRIDE 'ad\u202emin' -> error: U+202E prohibited by C.8 changes display / deprecated
a private-use character 'ad\ue000min' -> error: U+E000 prohibited by C.3 private use
a permanent noncharacter 'ad\ufffemin' -> error: U+FFFE prohibited by C.4 noncharacter
a tagging character 'ad\U000e0041min' -> error: U+E0041 prohibited by C.9 tagging
Nine categories of refusal, and not one of them is about a
character being WRONG. They are characters whose presence in an
identifier nobody can verify by looking. U+202E is the one that
reverses the display of everything after it -- the Trojan Source
class of bug, published in 2021 and prohibited by number here in
December 2002.
And two that are NOT refused, which is the more useful half:
NO-BREAK SPACE 'ad\xa0min' -> 'ad min'
ACCOUNT OF '℀' -> 'a/c'
...yet in_table_c12(NO-BREAK SPACE) is True, and C.1.2 is
`prohibit non-ASCII space`. It got through because step 2 had
already turned it into an ordinary space and step 3 never saw it.
That is what `the order is normative` buys, and it is why a
profile may not reorder the steps for convenience.
U+2100 goes the other way: nothing prohibits it, and NFKC expands
it into three characters, one of which is a PATH SEPARATOR.
Preparation does not only filter -- it rewrites, and what it
writes has to be safe in whatever consumes the prepared string.
7. BIDI IS A RULE ABOUT SHAPE, NOT ABOUT RENDERING
------------------------------------------------------------------------
aleph then 1 'ا1' -> error: bidi: an R/AL string must begin and end with R/AL
aleph, 1, beh 'ا1ب' -> 'ا1ب'
aleph then a latin b 'اb' -> error: bidi: an R/AL string may hold no L character
The RFC's own two examples, and they are checkable arithmetic: a
string holding any R/AL character may hold no L character, and must
begin and end with R/AL. No font, no layout engine, no rendering --
just membership in tables D.1 and D.2. Latin digits are in neither,
which is why 'aleph 1 beh' passes and 'aleph 1' does not.
8. THE PIN THAT PROTECTS YOU IS THE PIN THAT LOCKS YOU OUT
------------------------------------------------------------------------
Table A.1 is 'unassigned in Unicode 3.2', and the module computes
it the only way a frozen table can:
def in_table_a1(code):
if unicodedata.category(code) != 'Cn': return False
U+1F600 in_table_a1 -> True GRINNING FACE, assigned 2012
U+1E9E in_table_a1 -> True CAPITAL SHARP S, assigned 2008
U+11DB0 in_table_a1 -> True TOLONG SIKI LETTER I, assigned 2025
U+0378 in_table_a1 -> True genuinely unassigned, still is
Four different truths, one answer -- the identical failure `The
table has a version` found in `Cn`, except that here a standard is
built on top of it. Section 7 had to legislate around it:
a STORED string (a registered name) MUST reject table A.1
a QUERY string (what a user typed) MAY allow it
Read that as the trade it is. Pinning Unicode 3.2 means a domain
can never change meaning because the standard grew -- and it means
an implementation that obeys the pin refuses every character
invented after 2002, permanently, including the one above that
somebody writes their language in.
9. THE ONE STEP THE PIN DID NOT REACH
------------------------------------------------------------------------
U+1E9E LATIN CAPITAL LETTER SHARP S, asked of both tables this
module can reach:
the frozen 2002 table in_table_a1('ẞ') True unassigned
the live table 'ẞ'.lower() 'ß' a capital with a lowercase
Both are right about the table they read: the letter was added in
2008. The case fold has to side with one of them, and B.2 is built
on this function, unchanged in every Python 3:
def map_table_b3(code):
r = b3_exceptions.get(ord(code))
if r is not None: return r
return code.lower()
The dict is frozen into the module when it is generated; lower() is
the live table. Python's frozen table has no case mappings at all,
so the dict has to carry every row where the RFC's answer is not
today's -- and until 2026 it did not. It held only rows the RFC
writes down, the ones where RFC and lower() disagree, so a character
the RFC never mentions fell through to lower(), and U+1E9E folded
to 'ss': a fold for a letter the same module calls unassigned.
RFC 3454's own B.2 is a fixed list of 1,371 rows, and not one of
them is U+1E9E.
CPython fixed it in 2026 (gh-155292). The dict now also holds an
identity row for every character the RFC leaves alone that lower()
would change -- computed against the live table, which is why the
regenerated module asserts that table's version too -- and the same
call returns 'ẞ', the RFC's answer. Which one you get depends on
your interpreter's release, so this program prints neither. Ask
yours:
python3 -c "import stringprep; print(stringprep.map_table_b2('\u1e9e'))"
That is section 8's trap from the other side. A frozen table cannot
tell `not invented yet` from `never will be`; a module that asks a
frozen table one question and a live one the next gives both
answers at once, and nothing it returns says which table it came
from.
One answer above does come from the live table, and is printed on
purpose: 'ẞ'.lower(). U+1E9E has lowercased to U+00DF in every
Unicode that has it, from 5.1 in 2008, and Python 3.0 shipped 5.1,
so no Python 3 says anything else. Nothing in Unicode's stability
policy promises it: ẞ and ß are not a case pair, because ß
uppercases to 'SS'. So it is a bet, and this is it, made out loud.
Section 5 is the one to sit with. faß.de and fass.de are two different registrable domains, and which one you get depends on which decade's standard your library implements — with no error, no warning, and a perfectly successful call either way.
The stdlib codec implements IDNA2003, whose nameprep profile folds sharp s to ss. IDNA2008 abolished the mapping step entirely and keeps the character. Both are correct:
$ python3 -c "print('faß.de'.encode('idna'))"
b'fass.de' # IDNA2003, via nameprep, via stringprep
$ python3 -c "import idna; print(idna.encode('faß.de'))" # idna 3.19 from PyPI
b'xn--fa-hia.de' # IDNA2008: no mapping step, ß survives
$ python3 -c "import idna; print(idna.encode('Bücher.de'))"
idna.core.InvalidCodepoint: Codepoint U+0042 at position 1 of 'Bücher' not allowed
$ python3 -c "import idna; print(idna.encode('Bücher.de', uts46=True))"
b'xn--bcher-kva.de' # UTS-46 puts a mapping step back
Read the third command twice. IDNA2008 has no mapping step at all, so an uppercase B is simply not a legal character in a domain name until UTS-46 ↗ — a Unicode document, not an IETF one — supplies the missing compatibility layer. Between them, one typed string has three defensible answers, and the disagreement is entirely about preparation. Nobody has ever disagreed about how to encode the result.
Rust has to be told¶
Verified output of preparing_a_string_rs.rs — regenerated by tools/run_examples.py, never hand-typed.
1. THE ONE STEP THAT IS JUST A LIST
------------------------------------------------------------------------
RFC 3454 table B.1, transcribed: 27 code points
the widest one U+FEFF
A table small enough to write down is a table that cannot rot.
This one was sealed with the RFC in 2002, so a Rust literal is
an honest way to carry it -- which is NOT true of any of the
other three steps.
2. STRINGS THAT LOOK IDENTICAL AND ARE NOT
------------------------------------------------------------------------
stored as chars bytes == "admin" after B.1
nothing inserted 5 5 true == "admin"
U+200B ZERO WIDTH SPACE 6 8 false == "admin"
U+00AD SOFT HYPHEN 6 7 false == "admin"
U+FEFF the BOM, mid-string 6 8 false == "admin"
Every row renders as `admin` in every font there is. Three of
them are longer, and `==` on &str compares bytes -- so they are
four different names to Rust and one name to the person who
typed them. Deleting B.1 first is what closes that gap, and it
is step one of four for exactly this reason.
3. WHAT RUST'S std WILL AND WILL NOT DO FOR YOU
------------------------------------------------------------------------
the case axis, in full:
'ß'.to_uppercase() "SS" (2 chars)
"CAFÉ".to_lowercase() "café"
eq_ignore_ascii_case false <- and it says its limit in its name
the normalization axis, not at all:
no normalize() in std, so the two spellings of café stay
two different &str values forever
the prohibition axis, partly, and by a different question:
U+E000 char::from_u32 true is_control false is_alphabetic false
U+FFFE char::from_u32 true is_control false is_alphabetic false
U+202E char::from_u32 true is_control false is_alphabetic false
U+0041 char::from_u32 true is_control false is_alphabetic true
Read the third row. U+202E is not a control character to Rust,
is a perfectly good char, and reverses your terminal. std has
no vocabulary for `this character is fine and must not appear
in an identifier` -- that judgement is the RFC's, not the
language's, and no amount of type safety supplies it.
4. THE TABLE IS THE LIBRARY
------------------------------------------------------------------------
Three of the four steps need a table nobody would transcribe:
B.2 is the case-folding database, normalization is the
decomposition database, and C.* plus D.* are properties of
every assigned code point. In Rust each is a crate -- and what
the crate ships is data, not cleverness.
That is why the version question follows preparation around.
A hand-written B.1 is fixed forever; a hand-written B.2 is a
snapshot of somebody's Unicode, and the RFC pinned 3.2 rather
than let it move underneath a domain name.
Is this compiler's table newer than the RFC's 3.2? true
By how much is deliberately not printed -- that would put the
machine that built this page into the answer key.
The split is the same one normalization found, one step further along. Rust's std does the case axis in full — 'ß'.to_uppercase() is "SS", two characters — and the normalization axis not at all. For prohibition it has no vocabulary whatsoever: U+202E is not a control character, is a perfectly valid char, and reverses your terminal. That judgement is a policy question, and policy is what an RFC is for.
Which is the honest summary of the whole subject: the table is the library. Table B.1 is 27 code points and can be a literal in a source file forever. Everything else is a database, and a database has a version.
The pin that protects you is the pin that locks you out¶
stringprep.py opens by pinning its table, with an assertion to prove it (the fix below renames the alias unicodedata_320; this is the spelling in every release before it):
That is the frozen 2002 table, and the reason is sound: a registered domain name must not change meaning because Unicode grew. But look at how the module has to compute Table A.1, "unassigned in Unicode 3.2":
Cn — the same conflation The table has a version found, now load-bearing in a standard. A frozen table cannot distinguish not invented yet from never will be, so 😀, ẞ and U+11DB0 TOLONG SIKI LETTER I are all "unassigned" alongside U+0378, which genuinely is.
RFC 3454 §7 does not solve this. It legislates around it, by splitting strings into two kinds:
| what it is | unassigned code points | |
|---|---|---|
| stored string | a registered name — a domain, a certificate subject | MUST be rejected |
| query | what a user just typed, to match against stored ones | MAY be allowed |
A registry must refuse what it cannot understand, because it is promising to still mean the same thing in twenty years. A lookup may pass it through, because the worst case is failing to match. That is a genuinely good piece of engineering, and it is also an admission: an implementation that obeys the pin refuses every character invented after 2002, permanently. Somebody writes their language in U+11DB0.
The step the pin did not reach¶
The four characters in section 8 get one answer from A.1, and one of them used to get a second answer from the same module. stringprep.in_table_a1('ẞ') is True — unassigned in Unicode 3.2, which is correct, since the capital sharp s was added in 2008. Yet in every CPython release up to 3.14.7, stringprep.map_table_b2('ẞ') returns 'ss': a case fold for a character the module has just called nonexistent. Section 9 of the program prints both tables' halves of that disagreement, and says why it no longer prints the fold itself.
The cause is one function, which the module has carried unchanged through every Python 3:
def map_table_b3(code):
r = b3_exceptions.get(ord(code))
if r is not None: return r
return code.lower()
b3_exceptions is a dict written into the module when it is generated from the RFC; code.lower() is the live table. The frozen table Python ships has no case mappings at all — Case is not a per-character operation counts them, and gets zero — so the fold has to lean on str.lower(), and the dict's job is to hold every row where the RFC's answer is not today's. Until 2026 it held only the rows where the RFC and lower() disagree. A character the RFC never mentions was in neither, fell through to lower(), and got whatever the interpreter's table said. RFC 3454's own table B.2 is a fixed list of 1,371 rows with no row for U+1E9E, so the RFC's answer is no mapping — and a stored string may not contain the character at all, by §7. The 'ss' was CPython's.
How far it reached. Measured on 2026-09-10 by parsing table B.2 out of the RFC's own text and comparing it with map_table_b2 over every code point, on Python 3.14.7 (Unicode 16.0.0): the two disagree on 711 code points. 585 are ẞ's kind, code points A.1 calls unassigned. The other 126 are worse, because the RFC did cover them: letters Unicode 3.2 already had, caseless in 2002, which later gained lowercase partners — Georgian's 38 capitals (Unicode 4.1), the Cyrillic palochka, turned capital F and the reversed Roman hundred (5.0), and 85 Cherokee letters (8.0, 2015). The RFC leaves each of them alone; CPython replaced every one with a character A.1 calls unassigned. And because the stdlib's idna codec runs its nameprep through this module, the domain moved with the fold: on 3.14.7, 'ẞ.de'.encode('idna') is b'ss.de' — a different, ordinary ASCII name, section 5's trap produced by an implementation rather than by a standard — and '\u13a0'.encode('idna') (CHEROKEE LETTER A) is b'xn--kz9a' where the RFC's answer is b'xn--58d'. Cherokee is the case that moved inside Python 3: 3.4 shipped Unicode 6.3, where those letters had no case, so the codec gave the RFC's answer until 3.5.
The fix. CPython tracks this as a security issue, gh-155292 ↗, opened on 2026-08-06; the fix merged on 2026-08-18 and has been backported to the 3.12–3.15 branches. It does not freeze the fold — there is still no frozen case table to freeze it with. It makes the dict hold the whole of the RFC's answer: an identity row for every character the RFC leaves alone that lower() would change, computed against the interpreter's own live table. That is why the regenerated module now opens by asserting two versions, the frozen table it reads and the live one its dict was computed against:
import unicodedata
assert unicodedata.unidata_version == '16.0.0' # the 3.14 branch; each branch asserts its own
from unicodedata import ucd_3_2_0 as unicodedata_320
assert unicodedata_320.unidata_version == '3.2.0'
Imported into the same 3.14.7 interpreter in place of the installed module, the 3.14 branch's regenerated stringprep.py disagrees with the RFC's table B.2 on zero code points, and the two idna calls above return b'xn--kkg.de' and b'xn--58d' — the RFC's answers, the first because a query passes an unassigned code point through untouched. As of 2026-09-10 the only release carrying the fix is 3.15.0rc2; 3.14.7, 3.13.15 and 3.12.14, the latest stable releases, still fold ẞ to ss.
Why the program prints neither answer. Until this page was corrected, section 4 printed U+1E9E ẞ … -> 'ss' (2) as an ordinary row, under a section 1 that said nothing below could vary by machine. Both were true on the machine that recorded them and neither was true of the module: the row was the one fold on the page that came from the live table, and the fix changes it inside a patch release — it is already merged on the 3.14 and 3.13 branches, so their next releases print 'ẞ' on the same machine. So the row is gone, and section 9 prints only the two halves that cannot move: A.1's frozen True, and the live table's 'ẞ'.lower() == 'ß'. That second one is still a lookup, kept deliberately: U+1E9E has lowercased to U+00DF in every Unicode release from 5.1 (2008) to 17.0, and Python 3.0 already shipped 5.1, so no Python 3 answers otherwise. It is not a guarantee — ẞ and ß are not a case pair, because ß uppercases to SS, so Unicode's case-pair stability does not cover them — which makes it a stated bet, of the kind the case page makes out loud.
What it says about the thesis. This is the page's argument in its sharpest form. A frozen table cannot tell not invented yet from never will be; a module that asks a frozen table one question and a live one the next gives both answers at once, and nothing it returns says which table it came from. PRECIS makes the complementary point from the other side: it unpinned the table on purpose, and its section 5 shows a property moving underneath it — U+200B is Zs in the frozen table and Cf in a live one, so the same password rule maps it on one machine and refuses it on another. Here the framework was pinned, and the live table got in anyway, through the one step whose data Python never froze.
Prepare is not encode¶
Preparation and encoding get lumped together as "the Unicode part", and their histories say they are nothing alike. IDNA2003 was three documents; here is what became of them, and of stringprep itself:
| RFC | Status today | |
|---|---|---|
| 3454 ↗ stringprep | the framework | obsolete → 7564, itself obsolete → 8264 ↗ |
| 3491 ↗ Nameprep | the profile for domains | obsolete → 5891 ↗ |
| 3490 ↗ IDNA2003 | the protocol | obsolete → 5890 ↗ / 5891 |
| 4013 ↗ SASLprep | the profile for logins | obsolete → 7613 → 8265 ↗ |
| 3492 ↗ Punycode | the encoding | updated by 5891 — never obsoleted |
(Statuses read from rfc-editor.org on 2026-09-06.)
One row is not like the others. The algorithm that turns code points into xn-- labels was published in March 2003 and has never been replaced — because "write these code points in ASCII" is a question with a right answer. Every layer above it, the layers that decide which string you were entitled to write down, has been replaced twice in the same period. The successor framework is PRECIS ↗, which changed the fundamental approach: stringprep's tables list what is forbidden, PRECIS derives what is allowed from Unicode properties, so it does not have to be reissued every September.
Note the loop that closes there. Stringprep pinned its table to escape the version problem, and inherited a permanent freeze. PRECIS unpinned it to escape the freeze, and took the version problem back.
If you are coming from Python or ABAP¶
Python. stringprep is in the standard library right now, imports without a DeprecationWarning, and implements a document that has been obsolete since 2015 — as does '…'.encode('idna'), which is IDNA2003. Neither is wrong to use if IDNA2003 is what you want; both are wrong to reach for by default, which is exactly what the codec name invites. And until the 2026 fix neither followed it to the letter: the case fold read your interpreter's live table (the step the pin did not reach), so on 3.14.7 and earlier 'ẞ.de'.encode('idna') is b'ss.de', a name no implementation that followed the RFC would produce. If you are validating a domain, use the idna package and decide consciously about uts46=True. If you are comparing usernames, the modern answer is PRECIS (RFC 8265), and the stdlib has nothing for it.
ABAP. (Not machine-checked — CI cannot run ABAP.) There is no stringprep and no IDNA in ABAP, so preparation is whatever your code does before the comparison — which in practice means TRANSLATE … TO UPPER CASE and a hope. Two cautions worth carrying: upper-casing is not case folding (they disagree on ß and on Turkish dotted i, and upper-casing does not compose with normalization the way toCasefold(toNFKC(s)) does), and a CHAR/STRING comparison is a code-unit comparison, so two spellings of the same accented name are simply two different keys. If a system stores names entered by people in more than one place, the preparation rule belongs in one method that every path calls, written down like a profile, rather than inline at each comparison.
Try it¶
python3 -c "import stringprep; print(len([n for n in dir(stringprep) if n.startswith('in_table')]))"— seventeen appendices of a 2002 RFC, in your standard library.- Prepare your own name with the program on this page, then prepare it again with a
U+200Bin the middle. Then decide which of the two your login system would accept. python3 -c "print('faß.de'.encode('idna'))"— and then ask which domain your browser goes to when you type it. They need not be the same.- Take the four steps and reorder them: prohibit before normalize. Find a string that changes answer. (The NO-BREAK SPACE row is one; there are others.)
- Look up whichever profile your stack actually uses — SASLprep for SASL and LDAP,
nameprepfor anything calling.encode('idna')— and check its date against the table above. python3 -c "import stringprep; print(stringprep.map_table_b2('\u1e9e'))"—ssmeans your Python predates the 2026 fix,ẞmeans it has it. Then ask the same interpreter forstringprep.in_table_a1('\u1e9e'), which says the character does not exist either way.
Practice¶
Which of these are the same user? file, file, ADMIN, admin, ａdmin, Ⓐdmin. Run each through NFKC and then casefold(), and write down the groups that collide.
Then the part with no machine answer: for each of a username, a password and a display name, say whether you would apply that folding — and for the one where the answer is "fold to compare, never to store", say what breaks if you get it wrong.
Answers
Verified output of preparing_a_string_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.
input NFKC +casefold code points
file file file FB01 006C 0065
file file file 0066 0069 006C 0065
ADMIN ADMIN admin 0041 0044 004D 0049 004E
admin admin admin 0061 0064 006D 0069 006E
ａdmin admin admin FF41 0064 006D 0069 006E
Ⓐdmin Admin admin 24B6 0064 006D 0069 006E
COLLIDE -> 'file': ['file', 'file']
COLLIDE -> 'admin': ['ADMIN', 'admin', 'ａdmin', 'Ⓐdmin']
Four of the six become 'admin'. That is NFKC doing exactly its job: the
ligature, the full-width letter and the circled letter are COMPATIBILITY
spellings, and the whole point of the K forms is to fold them together.
Whether that is right depends entirely on what the string is for.
* A username -- yes. Two people must not be able to register names
that no human can tell apart, so you fold, and you store the folded
form as the key.
* A password -- no. Folding shrinks the keyspace, and NFKC maps many
characters to fewer, so it hands an attacker collisions for free.
* A display name -- no, and this is the one that gets broken by
accident: normalize in place and you have edited what somebody
wrote. Fold to COMPARE, store what they typed.
And the step order is not decoration. casefold-then-NFKC and
NFKC-then-casefold can differ, which is why the specs pin the order:
'fi' NFKC->casefold 'fi' casefold->NFKC 'fi' same
'İ' NFKC->casefold 'i̇' casefold->NFKC 'i̇' same
See also¶
- The table has a version — the frozen table this page is built on, and why
Cnis a trap - PRECIS: stringprep, after stringprep — what replaced it, with the table unpinned on purpose; its section 5 is this page's section 9 seen from the other side
- Noncharacters and the private use areas — the two classes stringprep prohibits by number in tables C.3 and C.4, explained rather than only listed: both are legal UTF-8 and neither is interchangeable
- Normalization — step 2 on its own, including what NFKC costs
- A code point is not a character — the other question the table alone will not answer
- Validation is a boundary — the same argument for bytes: decide where the check runs
- CAST.md —
ß, the ZWJ family and the BOM all earn their rows here - RFC 3454 ↗ — the document, and it is readable; §7 and §9 are the interesting halves
- RFC 8264 ↗ — PRECIS, what replaced it, and why the approach inverted
- UTS-46 ↗ — the compatibility layer that lets IDNA2008 accept what people actually type
- gh-155292 ↗ — CPython's issue for the case fold that read the live table, and the 2026 fix