Skip to content

Normalization

Level: 201 · for anyone who compares two strings

One line: Unicode encodes é two different ways on purpose, so two strings that print identically can be unequal, and unicodedata.normalize is the step that has to happen before you compare, hash, index or look anything up.

import unicodedata
key = unicodedata.normalize("NFC", s)   # do this once, on the way in

That is the entire practical answer, and if you take nothing else from this page, take the position of that line: on the way in, once, at the boundary — not at the moment you compare. A system that normalizes in the comparison function is a system where a == b depends on which function asked.

Why there are two spellings at all

Unicode had to be adoptable. In 1991 that meant every existing character set had to survive a round trip through it — encode to Unicode, decode back, get your bytes returned unchanged — because otherwise no vendor could switch without risking their data. Latin-1 already had a single byte E9 meaning é, so Unicode needed a single code point meaning é: that is U+00E9, and it exists as a compatibility debt, not as a design.

The design is the other one. é is an e with an acute accent on it, so it is U+0065 LATIN SMALL LETTER E followed by U+0301 COMBINING ACUTE ACCENT — two code points, composable, and the only approach that scales to the accents no legacy charset ever precomposed.

So both exist, both are correct, and neither is going away. Which one arrives at your program is a fact about where the text came from:

Source What you usually get
A keyboard, a web form, a Windows filename NFC — precomposed, U+00E9
A macOS filename read off the disk NFD — decomposed, U+0065 U+0301
A database, a CSV, an API payload whatever the thing that wrote it happened to hold

That third row is the honest one. There is no encoding declaration for normalization form, no BOM equivalent, nothing in the bytes that says which convention was used — so a string arriving from outside your program is in an unknown form, exactly the way a file of unknown encoding is in an unknown encoding. The difference is that a wrong guess about encoding produces mojibake you can see, and a wrong guess about normalization produces a lookup that silently returns nothing.

The four forms

Two independent questions, so four answers.

composed decomposed
canonical — same character, no information lost NFC NFD
compatibility — "close enough", lossy NFKC NFKD

The top row preserves the character's identity and is safe to apply to text you will store — though "canonical" does not mean "reversible", as section 4 below shows. The bottom row is a one-way trip: it folds fi to fi, ½ to 1⁄2, ² to 2. NFKC is right for a search index or a username-uniqueness check, and wrong for anything you will hand back to the person who typed it.

The K is for kompatibility — spelled with a K because C was already taken by composed.

In Python

Verified output of normalization_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. THE LITERAL YOU CANNOT TRUST
------------------------------------------------------------------------
   This page's whole subject is two strings that print the same. So the
   two spellings CANNOT be pasted into the source as literals -- an
   editor, a terminal, a paste buffer or a filesystem may quietly
   normalize one of them on the way in, and then the program proves
   nothing while still looking correct.

   Both of these were typed as 'cafe' plus an accent:
     COMPOSED    = "caf\u00e9"     U+0063 U+0061 U+0066 U+00E9
     DECOMPOSED  = "cafe\u0301"    U+0063 U+0061 U+0066 U+0065 U+0301

   They render as café and café, and: COMPOSED == DECOMPOSED -> False

2. THE FOUR FORMS, ON ONE WORD
------------------------------------------------------------------------
   form   code points  bytes  the code points themselves
   NFC    4            5      U+0063 U+0061 U+0066 U+00E9
   NFD    5            6      U+0063 U+0061 U+0066 U+0065 U+0301
   NFKC   4            5      U+0063 U+0061 U+0066 U+00E9
   NFKD   5            6      U+0063 U+0061 U+0066 U+0065 U+0301

   C is composed, D is decomposed, and the K forms are a second axis
   entirely -- section 6. On a plain accented word the K forms have
   nothing to do, so NFKC == NFC here: True

   All four are idempotent -- running one twice changes nothing:
     True

3. NFC IS NOT 'PUT THE ACCENTS BACK ON'
------------------------------------------------------------------------
   Two characters with no combining mark anywhere in them, which NFC
   nevertheless replaces -- they are SINGLETONS, code points Unicode
   encoded twice by accident of history and now folds together:

   code point  name           NFC gives  which is
   U+212B      ANGSTROM SIGN  U+00C5     LATIN CAPITAL LETTER A WITH RING ABOVE
   U+2126      OHM SIGN       U+03A9     GREEK CAPITAL LETTER OMEGA

   So NFC can change a string that contains no accent at all. If you
   are comparing an angstrom against an angstrom, this is the reason
   one of them lost.

4. AND IT IS NOT A ROUND TRIP
------------------------------------------------------------------------
   NFC(NFD(x)) == x looks like it must hold. It does not, for two
   groups: the singletons above, and the COMPOSITION EXCLUSIONS --
   characters NFD takes apart and NFC is forbidden to reassemble.

   code point  name                    NFD gives      round trips?
   U+0958      DEVANAGARI LETTER QA    U+0915 U+093C  False
   U+0F43      TIBETAN LETTER GHA      U+0F42 U+0FB7  False
   U+00E9      LATIN SMALL LETTER E WI U+0065 U+0301  True

   The last row is a normal character, for contrast. For the first two,
   NFD is a one-way door -- which is why 'normalize on the way in' means
   pick ONE form and keep it, not 'convert freely in both directions'.

5. THE ACCENT THAT COMES OFF, AND THE ONE THAT DOES NOT
------------------------------------------------------------------------
   A Polish word, letter by letter. Not every diacritic is a combining
   mark -- a stroke through a letter is part of the letter's own shape:

   char  code point  name                                 NFD
   ż     U+017C      LATIN SMALL LETTER Z WITH DOT ABOVE  U+007A U+0307
   ó     U+00F3      LATIN SMALL LETTER O WITH ACUTE      U+006F U+0301
   ł     U+0142      LATIN SMALL LETTER L WITH STROKE     U+0142
   w     U+0077      LATIN SMALL LETTER W                 U+0077

   z-with-dot and o-with-acute come apart. l-with-stroke does not, and
   never will: there is no COMBINING STROKE to peel off it.

   Polish has 9 letters with a diacritic. Exactly 1 of them has no
   decomposed form at all: ł. So a rule that says 'strip the combining
   marks to get ASCII' quietly leaves one letter standing.

6. COMPATIBILITY IS A DIFFERENT QUESTION, AND IT IS LOSSY
------------------------------------------------------------------------
   NFKC does everything NFC does, then also replaces characters with
   whatever is 'the same, roughly'. Read the fourth column and decide
   whether roughly is good enough:

   input     NFKC       cp   what was lost
   fi         fi         2    the ligature was a typographic choice
   x²        x2         2    x squared became x 2 -- the meaning, not the look
   ½         1⁄2        3    one character became three, including a slash
   １２３       123        3    fullwidth digits, common in Japanese input
   ℀         a/c        3    one character became a slash -- a path separator
   \u00a0    \u0020     1    a NO-BREAK SPACE became an ordinary space
   ．         .          1    a FULLWIDTH FULL STOP became a plain dot

   The last three are why NFKC belongs on a search index and a username
   uniqueness check, and NOT on text you will store and hand back. Two
   of them turn a harmless character into a path separator or a dot.

7. CASE IS A SEPARATE AXIS, AND IT MOVES FIRST
------------------------------------------------------------------------
   Normalizing does not fold case, and folding case does not normalize.

   char  lower   upper   casefold  code points of casefold
   ß     ß       SS      ss        U+0073 U+0073
   ẞ     ß       ẞ       ss        U+0073 U+0073
   İ     i̇      İ       i̇        U+0069 U+0307
   ς     ς       Σ       σ         U+03C3
   σ     σ       Σ       σ         U+03C3

   Row 1: lower() leaves eszett alone; casefold() gives 'ss'. That is
   the difference, and it is why a caseless comparison uses casefold.
   Rows 4 and 5 are Greek final sigma and ordinary sigma. lower()
   keeps them apart; casefold() makes them one letter, which is what
   a search box wants and what == will not give you.

   Row 3 is the one that decides the ORDER. Casefolding a Turkish
   dotted capital I emits a combining mark that was not there before:

     '\u0130'.casefold()  ->  U+0069 U+0307

   So casefold can CREATE the very thing normalization exists to
   reconcile. Normalize after folding, not before:

     s.casefold() then NFC   -- correct
     NFC then s.casefold()   -- leaves a stray combining mark

   Unicode's own definition of a caseless match puts an NFD on both
   sides of the fold for the same reason. In practice, for a login
   name or a search key:

     key = unicodedata.normalize('NFC', s.casefold())

8. ONE OF THE FEW LOOKUPS YOU MAY WRITE DOWN
------------------------------------------------------------------------
   Chapter 2 said no Unicode table lookup belongs in an answer key,
   because two machines carry two tables. Normalization is an
   exception, as a name is, and it is an exception on purpose.

   Unicode's Normalization Stability Policy, in force since 4.1:
   if a string holds only characters your table already knows, its
   normalized form is the same under every later version, forever.

   Asked of the 2002 table and this one, on characters both know:

   char  frozen 3.2 NFD       live NFD              agree
   é     U+0065 U+0301         U+0065 U+0301         True
   ż     U+007A U+0307         U+007A U+0307         True
   Å     U+0041 U+030A         U+0041 U+030A         True

   And the one place the old table is wrong -- a character it never
   had, so it has no decomposition to apply and returns it untouched:

     U+1B06  BALINESE LETTER AKARA TEDUNG
       frozen 3.2 NFD   U+1B06             (unchanged -- not in that table)
       live NFD         U+1B05 U+1B35

   Read the guarantee's condition again: 'only characters your table
   already knows'. The stability policy is a promise about the past,
   not about characters that had not been invented yet.

Three of those sections are worth pausing on.

Section 3 and 4 together kill the intuition that NFC means "put the accents back on". It also replaces singletons — U+212B ANGSTROM SIGN becomes U+00C5, a character with no combining mark in it either way — and it declines to reassemble the composition exclusions, so NFC(NFD(x)) == x is simply false for a class of characters. Normalization is a projection onto a canonical form, not a reversible transform. Pick one form and keep it.

Section 5 corrects something this page used to claim. Not every diacritic is a combining mark. ż and ó come apart under NFD; ł does not and never will, because there is no COMBINING STROKE to peel off — the stroke is part of the letter's own shape. So the folk technique of "NFD, then throw away the combining marks, and you have ASCII" leaves ł standing, and leaves it standing in a lot of Polish surnames.

Section 7 is the one that decides an ordering. 'İ'.casefold() is two code points — i plus a combining dot — so casefolding can create the very thing normalization exists to reconcile. Fold first, then normalize:

key = unicodedata.normalize("NFC", s.casefold())   # correct order

Unicode's own definition of a canonical caseless match puts an NFD on both sides of the fold, for the same reason one level deeper. It matters for 384 two-character strings this library could find, essentially all of them driven by U+0345 COMBINING GREEK YPOGEGRAMMENI; if your text is not Greek, the one-liner above is the whole of it.

And note what casefold() is not. It is not lower() — 'ß'.lower() is 'ß' and 'ß'.casefold() is 'ss' — and it is not normalization either, since NFKC leaves ß alone. Three separate operations, and only casefold answers "are these the same word ignoring case".

In Rust

Verified output of normalization_rs.rs — regenerated by tools/run_examples.py, never hand-typed.

1. THE TYPE SYSTEM DOES NOT HELP HERE
------------------------------------------------------------------------
   Both of these are a &str, both are valid UTF-8, and both
   print as the same word:

   composed     café    5 bytes  4 chars  U+0063 U+0061 U+0066 U+00E9
   decomposed   café    6 bytes  5 chars  U+0063 U+0061 U+0066 U+0065 U+0301

   composed == decomposed            false

   `String` promises the bytes are valid UTF-8. It promises
   nothing at all about WHICH valid sequence you got, and == is
   a byte comparison, so the answer above is correct and useless.

2. STD DOES THE CASE AXIS, IN FULL
------------------------------------------------------------------------
   char  to_lowercase  to_uppercase  note
   ß     ß             SS            one char in, two out
   ẞ     ß             ẞ             capital sharp S lowercases to the small one
   İ     i̇            İ             and this one GROWS A COMBINING MARK

   The last row is the trap, and it is not Python's:

     '\u{130}'.to_lowercase()  ->  U+0069 U+0307

   Lowercasing produced a combining mark that was not in the
   input, so lowercase-then-compare has the same ordering
   problem in Rust as it has everywhere else.

3. AND STD DOES NOT DO THE NORMALIZATION AXIS AT ALL
------------------------------------------------------------------------
   There is no `str::normalize`. The one caseless comparison in
   std is ASCII-only, and says so in its name:

     "CAFÉ".eq_ignore_ascii_case("café")   false
     "CAFE".eq_ignore_ascii_case("cafe")   true

   That is not a gap somebody forgot. Normalization needs the
   Unicode decomposition tables, and Rust keeps its standard
   library free of data that has a yearly release. The crate is
   `unicode-normalization`, and it is the answer.

4. WHAT THE CRATE IS ACTUALLY CARRYING
------------------------------------------------------------------------
   The algorithm fits on a screen. Here is NFD for five letters:

     café       -> café        U+0063 U+0061 U+0066 U+0065 U+0301
     żółw       -> żółw       U+007A U+0307 U+006F U+0301 U+0142 U+0077
     Ångström   -> Ångström    U+0041 U+030A U+006E U+0067 U+0073 U+0074 U+0072 U+00F6 U+006D

   It is right on the first two words and wrong on the third:
   ö is not in the table, so it comes through composed. That is
   the whole difficulty in one line -- not the loop, the data.

   A real implementation adds three things this has none of:
     - every decomposable code point, not five
     - canonical ordering, so two marks on one letter sort into
       a fixed order and compare equal
     - composition exclusions, for the NFC direction

   Which is why you take the crate. The point of writing the
   loop is to see that the table is the library.

The split is clean and worth remembering: std does case mapping in full Unicode, and normalization not at all. to_uppercase really does turn ß into SS, so the table for case mapping is compiled into the standard library — but there is no str::normalize, and the only caseless comparison in std is eq_ignore_ascii_case, which announces its limit in its own name.

That is a deliberate line rather than an oversight. The decomposition tables are large, they get a release every September, and Rust keeps that kind of data out of std — the same reasoning that keeps char::UNICODE_VERSION a compile-time constant in The table has a version. The crate is unicode-normalization ↗, and section 4's five-entry hand-rolled NFD is there to show what you are actually buying: not the loop, which is trivial, but the table, the canonical ordering of multiple marks, and the exclusion list.

One of the few lookups this library lets you write down

Chapter 2 established that a Unicode table lookup must never become a recorded answer key, because two machines carry two tables — python3 and rustc on the machine these pages were written on are a full Unicode release apart.

Normalization is one of the documented exceptions. The Unicode Normalization Stability Policy ↗, in force since Unicode 4.1:

If a string contains only characters from a given version of Unicode, and it is put into a normalized form in accordance with that version of Unicode, then the results will be identical to the results of putting that string into a normalized form in accordance with any subsequent version of Unicode.

So a decomposition mapping, once assigned, is frozen — which puts normalization in the Guaranteed row of that page's three-row table, next to a character's name, and not in the lookup row next to is_alphabetic. Section 8 of the Python program demonstrates it rather than quoting it: the frozen 2002 table and the live one give byte-identical answers for é, ż and Å.

Read the guarantee's condition, though. It holds for characters your table already knows. Ask the 2002 table to decompose a Balinese letter added in 2006 and it returns the character untouched — not an error, not a hedge, just a string that is not in NFD while claiming to be. Same failure shape as od -a inventing a letter for a byte it cannot name.

The literal you cannot paste

This bit is practical and it bit the writing of this page. Do not put a decomposed string in your source as a literal. Editors normalize on save, terminals normalize on paste, some filesystems normalize on open, and the two spellings look identical at every step — so the test that was supposed to prove composed != decomposed quietly becomes a test that two identical strings are identical, and it passes. Both example programs here write the decomposed form as "cafe\u0301" (Rust: "cafe\u{301}"), which is the only spelling worth trusting in a file that anything might reformat.

If you are coming from Python or ABAP

Python. unicodedata.normalize(form, s) where form is one of the four strings, plus unicodedata.is_normalized(form, s) since 3.8 — which is much faster than normalizing and comparing when you only want to check. str.casefold() is the caseless-comparison operation and str.lower() is not; unicodedata.combining(c) gives a character's canonical combining class, which is 0 for a base letter and non-zero for a mark, and is the honest way to spot combining marks rather than guessing at ranges. If you find yourself needing NFKC_Casefold — the form the identifier rules actually specify — the usual reconstruction is normalize("NFKC", normalize("NFKC", s).casefold()), and the doubled call is not a typo. It is an approximation rather than the property, though: the real NFKC_Casefold maps the default-ignorable characters to nothing, and no composition of these two calls does, so a zero-width space survives it. The hard strings has the UCD lines.

ABAP. (Not machine-checked — CI cannot run ABAP.) There is no ABAP equivalent of unicodedata.normalize in the language itself, which is the first thing to know: a Unicode-enabled ABAP system stores string as UCS-2 code units and takes whatever spelling arrived. So = on two strings is a code-unit comparison with exactly the failure this page describes, and it shows up where text crosses a boundary — an upload from a Mac, an IDoc from a third party, a file dropped on an application server. cl_abap_conv_* transcodes between code pages and does not normalize; cl_abap_char_utilities gives you the control characters, not the tables. Practically: normalize at the interface, in whatever is on the other side of it, before the data reaches ABAP; if you must do it inside, that is an RFC to a service that has the tables, not a REPLACE ALL OCCURRENCES. And treat any code-page number you find in a document as something to verify against the system that will run the job.

Try it

  • Take the two spellings from section 1 and put them in a dict and a set. Count the entries. That is the bug, and it is the shape it takes in real code — not a failed == but a lookup that returns nothing.
  • python3 -c "import unicodedata as u; print(u.is_normalized('NFC', input()))" and paste a filename from a Mac.
  • Run the Rust example's naive_nfd on your own surname. If it comes through unchanged, that is the five-entry table, not a verdict.
  • 'İstanbul'.casefold() — count the code points, then work out what a lowercase-then-compare login check does with it.
  • Find one place in your own code where two strings are compared after arriving from different sources. Decide where the boundary is, and put one normalize there.

Practice

Two strings that draw the same. "café" composed and decomposed: predict len, byte count, equality and hash equality for both.

Then run both through all four normalization forms, plus "file" and "2⁵", and say which two forms are lossy and why that is on purpose. Finish by naming the three places the difference bites — one of them raises nothing at all, and it is the worst.

Answers

Verified output of normalization_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

TWO STRINGS THAT DRAW THE SAME
   NFC  café  len 4  bytes 5  0063 0061 0066 00E9
   NFD  café  len 5  bytes 6  0063 0061 0066 0065 0301
   equal? False     hash equal? False

THE FOUR FORMS
   form   café       file      2⁵     what it does
   NFC    café       file      2⁵     composes, canonical only
   NFD    café      file      2⁵     decomposes, canonical only
   NFKC   café       file     25     composes, and folds compatibility spellings
   NFKD   café      file     25     decomposes, and folds compatibility spellings

   The K forms are LOSSY on purpose: 2⁵ becomes 25 and the ligature
   becomes two letters. That is right for a search index and wrong for
   anything you will display back to the person who typed it.

THE THREE PLACES IT BITES
   1. comparison   NFC == NFD is False
   2. dict/set     {NFC: 1}.get(NFD) -> None   (the key is not there)
   3. length       4 vs 5 code points for one four-letter word
   A dict lookup that misses is the worst of the three, because nothing
   raises -- the key is simply not there, and the two keys print the
   same in the traceback.

WHERE THE TWO SPELLINGS COME FROM
   Not from carelessness. A Mac's filesystem has historically handed
   back decomposed names, most keyboards and most Windows software
   produce composed ones, and both are correct Unicode. So a filename
   that made a round trip through macOS and a filename typed on Windows
   can be the same word and different strings.

THE RULE
   Normalize at the BOUNDARY -- once, on the way in -- and store one
   form. Normalizing to compare is free; normalizing in place is an
   edit to somebody's data, and NFKC in particular cannot be undone.

See also