Confusables and scripts

Level: 301 · for anyone who accepts a name from a stranger

One line: Two strings can be different code points, different bytes and the same picture — and no normalization form will merge them, because Cyrillic а and Latin a are not two spellings of one letter, they are two letters that history drew the same way.

import unicodedata
'\u00e1' == '\u0430\u0301'                  # False       <- both render 'á'
'a' == 'а'                                  # False       <- U+0061 and U+0430
unicodedata.normalize('NFKC', 'а') == 'a'   # False       <- and no form merges them
unicodedata.name('а').split()[0]            # 'CYRILLIC'  <- the only script test in the stdlib

Preparing a string dealt with the characters you cannot see. This page is about the ones you can see perfectly well, and still read wrong.

The mechanism is not a defect. Latin, Greek and Cyrillic share letter shapes by descent — they are branches of one alphabet — so a table that encodes every writing system at once must contain several characters that a font will draw identically. Unicode did not create the ambiguity; it inherited it, and then had to number both.

What follows from that is the uncomfortable part: equality is doing its job when it says these strings differ, and the reader is doing theirs when they say they don't. No amount of correctness in your string library closes that gap.

Normalization is not the fix

This is the misconception worth killing first, because it is the natural guess and it is wrong in an instructive way.

Verified output of confusables_and_scripts_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. ONE PICTURE, TWO CHARACTERS
------------------------------------------------------------------------
   a   U+0061   LATIN SMALL LETTER A
   а   U+0430   CYRILLIC SMALL LETTER A

   they are equal                False
   their UTF-8 bytes             61  vs  d0 b0
   their lengths in bytes        1  vs  2

   Nothing is broken here. Cyrillic and Latin both took the letter
   from Greek, so the shapes are related by descent -- and a table
   that encodes every writing system has to hold both.

2. NORMALIZATION IS NOT THE FIX -- BUT READ THE EXCEPTION
------------------------------------------------------------------------
   look-alike     name                                           merges under
   U+0430 -> 'a'  CYRILLIC SMALL LETTER A                        nothing
   U+03BF -> 'o'  GREEK SMALL LETTER OMICRON                     nothing
   U+0440 -> 'p'  CYRILLIC SMALL LETTER ER                       nothing
   U+0441 -> 'c'  CYRILLIC SMALL LETTER ES                       nothing
   U+0435 -> 'e'  CYRILLIC SMALL LETTER IE                       nothing
   U+0456 -> 'i'  CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I nothing
   U+217C -> 'l'  SMALL ROMAN NUMERAL FIFTY                      NFKC, NFKD
   U+2215 -> '/'  DIVISION SLASH                                 nothing
   U+2010 -> '-'  HYPHEN                                         nothing

   Eight of the nine survive every normalization form there is, and
   they are RIGHT to. Normalization reconciles two spellings of the
   SAME character; Cyrillic er and Latin p are two different letters
   that a font happens to draw alike, and merging them would corrupt
   every Russian word ever written.

   The ninth is the instructive one. U+217C is a Roman numeral, and
   Unicode itself declared it COMPATIBILITY-equivalent to the letter,
   so NFKC folds it. That is the whole rule: normalization catches a
   look-alike only where the standard already said the two are the
   same character wearing different clothes. Visual similarity is a
   fact about fonts, and no normalization form has ever claimed it.

   The sharpest version of the rule is a pair that LOOKS like a
   normalization problem, because one side really is a combining
   sequence -- Russian marks stress with a real combining acute, so
   this is ordinary dictionary typography, not a contrived string:

      both render as    á   а́

      latin     U+00E1          LATIN SMALL LETTER A WITH ACUTE
      cyrillic  U+0430 U+0301   CYRILLIC SMALL LETTER A + COMBINING ACUTE ACCENT

      form   latin       cyrillic    equal   lengths
      NFC    00e1        0430 0301   False   1 vs 2
      NFD    0061 0301   0430 0301   False   2 vs 2
      NFKC   00e1        0430 0301   False   1 vs 2
      NFKD   0061 0301   0430 0301   False   2 vs 2

   Read the NFD row twice. It gives the two strings the SAME length
   and the SAME combining mark, and they are still unequal -- so the
   difference has been squeezed down to one code point, 0061 against
   0430, which is section 1 again. A reader who reached for
   normalize() because the lengths differed has been walked back to
   the letters, which is where the problem always was.

   And NFC does not close the gap, it WIDENS it: latin composes to a
   single U+00E1, the cyrillic pair stays two. Not a composition
   exclusion -- Unicode never encoded a precomposed cyrillic a with
   acute at all. Nothing in the block carries a plain acute; the
   nearest is U+04F2/U+04F3, which are U with DOUBLE acute. So
   len() reports 1 against 2 forever, and the invariant a beginner
   reaches for -- same picture, same length -- is not available in
   either direction.

3. ONE LETTER, AND IT IS SOMEBODY ELSE'S DOMAIN
------------------------------------------------------------------------
   'apple.com'    U+0061 U+0070 U+0070 ...
                  .encode('idna') -> b'apple.com'
   'аpple.com'    U+0430 U+0070 U+0070 ...
                  .encode('idna') -> b'xn--pple-43d.com'

   The second one is not a spelling of the first; it is a different
   name, and punycode preserves the difference perfectly -- which is
   exactly what an encoding should do. The `xn--` form is why your
   browser shows it: having decided it cannot tell you the names are
   different, it stops showing you the pretty one.

4. THE STANDARD LIBRARY HAS NO script()
------------------------------------------------------------------------
   hasattr(unicodedata, 'script')   False

   Six properties are exposed and script is not among them. Ask the
   ones that are, about the two letters from section 1:

   property             latin a          cyrillic a
   category             Ll               Ll
   bidirectional        L                L
   combining            0                0
   east_asian_width     Na               A
   decomposition        (none)           (none)

   Four of the five cannot tell them apart, and the fifth is a red
   herring worth spending three lines on, because it looks like a
   hit:

      U+0061  east_asian_width Na  LATIN SMALL LETTER A
      U+00E9  east_asian_width A   LATIN SMALL LETTER E WITH ACUTE
      U+017C  east_asian_width N   LATIN SMALL LETTER Z WITH DOT ABOVE
      U+0430  east_asian_width A   CYRILLIC SMALL LETTER A

   It splits LATIN against itself -- three Latin letters, three
   different values -- because it answers `how many columns in a
   CJK terminal`, not `which alphabet`. A property that separates
   two characters is not thereby a script property.

   The only stdlib function that does separate them is name(),
   whose FIRST WORD is a script by convention -- and because a Name
   is a Unicode stability guarantee, that convention is safe to
   build on and safe to record:

      U+0061  LATIN      from LATIN SMALL LETTER A
      U+0430  CYRILLIC   from CYRILLIC SMALL LETTER A
      U+03BF  GREEK      from GREEK SMALL LETTER OMICRON
      U+0030  DIGIT      from DIGIT ZERO
      U+002E  FULL       from FULL STOP

5. A MIXED-SCRIPT DETECTOR, AND WHAT IT CATCHES
------------------------------------------------------------------------
   'apple.com'      ['LATIN']                  single script
   'аpple.com'      ['CYRILLIC', 'LATIN']      MIXED -- refuse
   'paypal.com'     ['LATIN']                  single script
   'рaypal.com'     ['CYRILLIC', 'LATIN']      MIXED -- refuse

   Six lines of stdlib, and it catches the whole classic attack: a
   name that is mostly Latin with one letter borrowed from another
   alphabet cannot help but be mixed-script.

6. AND THE ONE IT WAVES STRAIGHT THROUGH
------------------------------------------------------------------------
   'аррӏе'
   rendered            аррӏе.com
   code points         U+0430 U+0440 U+0440 U+04CF U+0435
   scripts present     ['CYRILLIC']
   mixed?              False
   punycode            b'xn--80ak6aa92e.com'

   Every letter is Cyrillic, so the string is perfectly consistent
   and the detector above is satisfied. This is a WHOLE-SCRIPT
   confusable, and no amount of internal consistency will find it --
   the only thing wrong with the name is that it is not the one the
   reader thinks they are reading.

   That is where the stdlib runs out and UTS #39 begins: a published
   table of confusable sequences, plus restriction levels that ask
   which scripts a USER expects rather than which the string uses.
   And the last of those questions is not answerable inside a
   library at all -- it is a policy a registry or a browser holds.

Section 2 is the one to keep. Eight of the nine look-alikes survive every normalization form there is, and they should — normalization reconciles two spellings of the same character, and Cyrillic er is not a spelling of Latin p. Folding them would corrupt every Russian word ever written.

The ninth is the exception that states the rule. U+217C SMALL ROMAN NUMERAL FIFTY folds to l under NFKC, because Unicode itself had already declared the two compatibility-equivalent. So normalization catches a look-alike exactly when the standard has already said they are the same character in different clothes, and never because they look alike. Visual similarity is a fact about fonts, and no normalization form has ever claimed to know about fonts.

The pair worth carrying away is the one that looks like a normalization problem, because half of it genuinely is one: '\u00e1' and '\u0430\u0301' both render á. The first is a single Latin letter; the second is Cyrillic а followed by a real combining acute — and the two-code-point form is the correct spelling of the two, since Russian marks stress exactly that way. Nothing here is an attack string; it is dictionary typography.

So len() says 1 against 2, and reaching for normalize() is the obvious move. It does not work, and the NFD row is the one to read twice: it gives the two strings the same length and the same combining mark, and they are still unequal — the whole difference has been squeezed down to one code point, 0061 against 0430, which is section 1 again. A reader who reached for normalization because the lengths differed has been walked straight back to the letters, which is where the problem was the entire time. NFC does not close the gap either; it widens it, composing the Latin side to a single U+00E1 while the Cyrillic pair stays two — and not as a composition exclusion, but because Unicode never encoded a precomposed Cyrillic а with acute at all. Nothing in the block carries a plain acute; the nearest characters are U+04F2/U+04F3, which are U with a double acute. The invariant a beginner reaches for — same picture, same length — is unavailable in both directions at once.

That is also why RFC 3454 §9.1 ↗ declines the problem in writing: mapping look-alikes together needs context — which font, which reader — that a protocol does not have. Twenty-four years later that is still the answer.

The standard library has no script()

Section 4 is the practical shock. unicodedata gives you category, bidirectional class, combining class, east-asian width and decomposition, and not the one property this problem is about. Four of those five cannot separate the two letters at all.

The fifth is worth the three lines the program spends on it, because it looks like a hit and is not: east_asian_width says Na for Latin a and A for Cyrillic а, which would be a script test — except that it also says A for Latin é and N for Latin ż. It splits Latin against itself, because it answers how many columns in a CJK terminal. A property that separates two characters is not thereby a script property, and that is a mistake worth only making once.

What is left is unicodedata.name(), whose first word is a script by convention. It feels like a hack and it is a good one: a Name is a Unicode stability guarantee ↗, so unlike almost everything else in this chapter it is safe to build on and safe to write into an answer key — which is why every value on this page could be recorded. See The table has a version for why that distinction decides what a page may print.

Rust knows less, and puts what it knows in the compiler

Verified output of confusables_and_scripts_rs.rs — regenerated by tools/run_examples.py, never hand-typed.

1. TWO chars THAT std CANNOT TELL APART
------------------------------------------------------------------------
                          latin a      cyrillic a
   code point             U+0061       U+0430
   len_utf8()             1            2
   is_alphabetic()        true         true
   is_lowercase()         true         true
   is_ascii()             true         false
   is_alphanumeric()      true         true

   is_ascii() is the only one that splits them, and it is not a
   script test -- it would put 'é' on the Cyrillic side. `char`
   has no script method, no block method, and no name method:
   Rust ships the classification tables and not the names.

2. BLOCKS, TRANSCRIBED -- AND WHY IT IS NOT THE EASY FIX
------------------------------------------------------------------------
   U+0061  a  LATIN (basic, lower)
   U+0041  A  LATIN (basic, upper)
   U+00E9  é  LATIN (1 Supplement + Extended-A/B)
   U+017C  ż  LATIN (1 Supplement + Extended-A/B)
   U+0430  а  CYRILLIC
   U+03BF  ο  GREEK
   U+1D9C  ᶜ  not in the transcribed table
   U+0030  0  not in the transcribed table
   U+002E  .  not in the transcribed table

   Read the LATIN rows in the table above: the alphabet needs
   FIVE ranges and this transcription still misses several, so
   'ᶜ' comes back unknown though it is plainly a Latin letter.
   And the last two rows are in no script at all -- a digit and
   a full stop belong to every alphabet at once, which is why
   any real mixed-script check has to special-case them.

   So a block table is easy to write, easy to verify, and
   wrong in two directions at once: incomplete for one script,
   and silent about the characters that have none.

3. WHERE RUST DOES KNOW -- AND IT IS THE COMPILER
------------------------------------------------------------------------
   let żółw = 4;  compiles       true
   let zolw = 3;  also compiles  true

   rustc accepts non-ASCII identifiers and then does exactly
   what this page argues for: it declines to FOLD them, and
   warns instead. Three lints ship for it --
      uncommon_codepoints
      confusable_idents
      mixed_script_confusables
   -- which is the UTS #39 machinery, in a compiler, aimed at
   the one place where two names that look identical do real
   damage: a diff nobody can read.

   Note where that leaves the two languages. Python NORMALIZES
   identifiers and so cannot warn about them; Rust preserves
   them and so must. Same trade as IDNA2003 against IDNA2008,
   one floor down.

char has no script method, no block method and no name method — Rust ships the classification tables without the names. The program's hand-rolled block table shows why "just check the block" is not the easy fix it looks like: Latin alone needs five ranges in a transcription that still misses ᶜ, and digits and full stops are in no script at all, so every real mixed-script check has to special-case exactly the characters a domain name is full of.

Where Rust does know is rustc itself, and its answer is the interesting one — it accepts non-ASCII identifiers, refuses to fold them, and ships uncommon_codepoints, confusable_idents and mixed_script_confusables to warn instead. Python normalizes identifiers and therefore cannot warn; Rust preserves them and therefore must. That is the same trade as IDNA2003 against IDNA2008 on the previous page, one floor down — and it has its own page waiting in Unicode in identifiers.

The attack the detector does not catch

Six lines of unicodedata will catch the classic homograph: a mostly-Latin name with one letter borrowed from another alphabet cannot help but be mixed-script.

аррӏе.com is not that. Every character is Cyrillic, the string is perfectly self-consistent, and the detector passes it. This is a whole-script confusable, and no test of internal consistency will ever find one — the only thing wrong with the name is that it is not the name the reader thinks they are reading.

That is where a library has to stop and a policy has to start. UTS #39 ↗ publishes the confusables table and defines restriction levels, which work by asking which scripts a user expects rather than which scripts a string contains — a question no function can answer from its argument. In practice the answer is held by registries (which scripts may be mixed in one label) and by browsers (when to show you xn-- instead of the pretty form). Your code's job is usually to preserve the difference faithfully and show it to somebody who can decide.

If you are coming from Python or ABAP

Python. There is no confusables support in the standard library and there will not be; the packages are confusable_homoglyphs and uniseg, or regex (the third-party one) whose \p{Script=Cyrillic} is the script property unicodedata never grew. Before reaching for any of them, be clear which question you are asking, because they are three different jobs: are these two strings confusable (a table lookup), is this one string suspicious (mixed-script or restriction level), and what should I display (a policy). Most bugs here are the third question answered with the first tool.

ABAP. (Not machine-checked — CI cannot run ABAP.) There is no script property and no confusables table, so the practical control is upstream: constrain what may enter the field at all. For master data that should be Latin, an input check that rejects code points outside expected ranges is cruder than UTS #39 and much better than nothing — and note that CP / CA comparisons and TRANSLATE … TO UPPER CASE will not help you here at all, since the strings differ in every byte. If names arrive from an external system, the place to look is the interface's code page: a round trip through a single-byte code page will often destroy the Cyrillic letter rather than pass it through, which is a data-loss bug wearing a security bug's clothes.

Try it

• python3 -c "print('а'.encode())" — paste a Cyrillic а from this page and watch two bytes come back where you expected one.
• Take a domain you trust, replace one letter with its look-alike, and run both through .encode('idna'). Then look at the two xn-- forms and ask which one your eye would have caught.
• Run uni identify on a string you suspect — uni prints the name of every character, which ends the argument in one line.
• Write the six-line detector from section 5 for a field in your own system. Then find the input it waves through, which is the point of section 6.
• Look up whether your registry, your language and your code review each have an answer to this. Most projects have none of the three and have simply not been targeted yet.

Practice

Five checks, and the one that works. Take Cyrillic а (U+0430) and Latin a (U+0061). Predict the result of all five: ==, NFC equality, NFKC equality, casefold() equality, and "same script".

Four of them return the same answer for the same wrong reason. Say what that reason is — then say what check you would actually put in front of a signup form, and why it is a rule about the string, not about the character.

Answers

Verified output of confusables_and_scripts_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

the two letters   'а' and 'a'
code points       U+0430 and U+0061
names             CYRILLIC SMALL LETTER A
                  LATIN SMALL LETTER A
utf-8 bytes       d0b0 and 61

FIVE CHECKS
   a == b           False
   NFC equal        False
   NFKC equal       False
   casefold equal   False
   same script      False

Every one of them says False, and only the last one says it for the
right reason. The first four are asking 'are these two spellings of one
character?' -- and the honest answer is no, so they cannot help. These
are two DIFFERENT characters that history drew the same way, and
normalization exists to merge spellings, not to merge letters.

What does work is a rule about the whole string:
   'paуpal'     scripts ['CYRILLIC', 'LATIN']  MIXED -- refuse it
   'paypal'     scripts ['LATIN']  single script

That is the check: not 'is this character suspicious' but 'does this
identifier mix scripts'. One letter is never wrong on its own -- Cyrillic
a is the right letter in a Cyrillic word.

See also

• Preparing a string — the invisible half, and the RFC that declines this half in writing
• Unicode in identifiers — the same problem where the reader is a code reviewer
• Normalization — what it does reconcile, and why that is a different question
• The table has a version — why a name may be recorded when a property may not
• uni — the character's name — the diagnostic tool for exactly this
• UTS #39, Unicode Security Mechanisms ↗