Skip to content

Normalization is built in

Level: 201 · anyone comparing, storing or indexing text that people typed

One line: café can be spelled with one code point for é or with two, and the two spellings are different strings to == and to a Hash; unicode_normalize — a String method, no gem — makes them one, and refuses a string that is not in a Unicode encoding.

Unicode has a precomposed é, U+00E9, and a plain e followed by U+0301 COMBINING ACUTE ACCENT. Both render the same. Which one a program receives depends on the keyboard, the operating system and every program the text passed through on the way.

Verified output of normalization_is_built_in_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

  composed      U+0063 U+0061 U+0066 U+00E9
  decomposed    U+0063 U+0061 U+0066 U+0065 U+0301

  composed == decomposed                     false
  composed == decomposed.unicode_normalize   true
  decomposed.unicode_normalized?             false

The four forms, on a ligature, a superscript and an accent:
  unicode_normalize(:nfc )  11 code points  U+FB01 U+006C U+0065 U+0020 U+0078 U+00B2 U+0020 U+0063 U+0061 U+0066 U+00E9
  unicode_normalize(:nfd )  12 code points  U+FB01 U+006C U+0065 U+0020 U+0078 U+00B2 U+0020 U+0063 U+0061 U+0066 U+0065 U+0301
  unicode_normalize(:nfkc)  12 code points  U+0066 U+0069 U+006C U+0065 U+0020 U+0078 U+0032 U+0020 U+0063 U+0061 U+0066 U+00E9
  unicode_normalize(:nfkd)  13 code points  U+0066 U+0069 U+006C U+0065 U+0020 U+0078 U+0032 U+0020 U+0063 U+0061 U+0066 U+0065 U+0301

  users[composed]                                      nil
  users.transform_keys(&:unicode_normalize)[composed]  "stored decomposed"
  an ISO-8859-1 string   Encoding::CompatibilityError: Unicode Normalization not appropriate for ISO-8859-1

Reading the output

  • == compares code points and says false. After unicode_normalize — NFC by default — it says true. unicode_normalized? asks the question without building a new string.
  • The four forms. NFC composes and NFD decomposes. The K forms also replace compatibility characters with their plain equivalents: the ligature U+FB01 becomes f i, and the superscript U+00B2 becomes 2. That is lossy — after NFKC, x² and x2 are the same string.
  • A Hash is == with extra steps. The lookup by the composed key misses the decomposed key until the keys are normalized.
  • Only Unicode can be normalized. An ISO-8859-1 string raises Encoding::CompatibilityError instead of being converted behind your back.

What to do

Normalize to NFC once, where text enters — before it is stored, hashed, compared or indexed. Use NFKC for identifiers and search terms, where the ligature and fi ought to match, and never for text you will display back to the person who wrote it.

If you are coming from another language

Python's unicodedata.normalize takes the same four forms by the same names — the Python library's Normalization ↗. Java's is java.text.Normalizer — the Java text library's Normalization and equality ↗. The Encodings library's Preparing a string ↗ puts normalization in order with the other steps that come before a comparison.