Unicode in identifiers¶

Level: 301 · deep dive

One line: Python normalizes your variable names — write ﬁle = 2 and you have defined file — while Rust preserves them and warns instead, which is IDNA2003 against IDNA2008 being argued again one floor down, about the text your program is written in.

ns = {}
exec("ﬁle = 2", ns)      # a LATIN SMALL LIGATURE FI, U+FB01
ns["file"]               # 2      <- the name you never typed
"ﬁle" in ns              # False  <- and the one you did is not there

Every other page in this chapter is about text your program handles. This one is about text your program is, and the surprise is that a normalization form runs there too — on your identifiers, before the compiler resolves a single name.

Python normalizes, silently¶

Verified output of unicode_in_identifiers_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. AN IDENTIFIER DOES NOT HAVE TO BE ASCII
------------------------------------------------------------------------
   żółw = 4   ->   4

   Legal since Python 3.0. The rule is Unicode's own, not Python's:
   a name starts with an XID_Start character and continues with
   XID_Continue, which is UAX #31 and is what most languages that
   allow this use. The boolean comes first below because a CJK
   character is two columns wide and would break the alignment:

      isidentifier() True   'żółw'
      isidentifier() True   'café'
      isidentifier() True   '日本語'
      isidentifier() True   '_x'
      isidentifier() True   'ﬁle'
      isidentifier() True   '𝑎'
      isidentifier() False  '2fast'
      isidentifier() False  'a-b'

2. AND PYTHON NORMALIZES IT BEFORE BINDING IT
------------------------------------------------------------------------
   typed '𝑎'        (U+1D44E)
      bound as ['a']
   typed 'ﬁle'      (U+FB01 U+006C U+0065)
      bound as ['file']
   typed 'Ⅰ'        (U+2160)
      bound as ['I']

   Nothing you typed is what the namespace holds. The compiler
   applies NFKC to every identifier, so a mathematical italic letter
   becomes an ordinary one and a ligature becomes two letters --
   which is NFKC -- normalization's aggressive form -- doing exactly
   what it says on the tin, in a place nobody expects to find it.

3. SO TWO SPELLINGS CAN BE ONE VARIABLE
------------------------------------------------------------------------
   exec("ﬁle = ...")  then  ns['file']  ->  'assigned through the ligature'

   exec("file = ...")  then  is 'ﬁle' a key?  False
                             ns['file'] ->  'reassigned through ASCII'

   One variable, two spellings, and the ligature spelling is not even
   a key. A reader who greps the file for `file` finds one of the two
   assignments; a reader who greps for the ligature finds the other.

4. AND THE LOOK-ALIKE IT DOES *NOT* MERGE
------------------------------------------------------------------------
   after binding both spellings of `a`:
      names in the namespace   ['a', 'а']
      how many                 2
      U+0061 LATIN SMALL LETTER A         -> 'latin'
      U+0430 CYRILLIC SMALL LETTER A      -> 'cyrillic'

   NFKC('а') == 'a'   False

   TWO variables, rendered identically, and Python said nothing at
   all. This is the same rule as the previous page: NFKC merges what
   Unicode declared COMPATIBILITY-equivalent -- ligatures, italic
   letters, Roman numerals -- and never merges two letters that are
   merely drawn alike, because they are different letters.

   Put sections 3 and 4 together and the shape is uncomfortable:
   normalization SURPRISES you where the characters are related, and
   ABANDONS you where they are not. The first costs you an afternoon
   of confusion; the second is the one that gets into a code review.

5. WHAT THIS MEANS FOR A DIFF
------------------------------------------------------------------------
   Two assignments that no reviewer can tell apart:

       total = compute()      # ASCII a
       tоtal = compute()      # U+043E CYRILLIC SMALL LETTER O

      'total'    U+0074 U+006F U+0074 U+0061 U+006C
      'tоtal'    U+0074 U+043E U+0074 U+0061 U+006C
      equal?   False

   Python will bind both, run both, and warn about neither. The
   defence is not a language feature here -- it is a linter, a
   pre-commit hook, or a rule that identifiers stay ASCII and the
   other languages live in the strings.

6. THE REPL SESSION THAT LOOKS LIKE A BROKEN INTERPRETER
------------------------------------------------------------------------
   Typed at a prompt, one line after another:

       >>> value = 3
       >>> value = 4          # the `a` here is U+0430, not U+0061
       >>> value
       3

   The second assignment did not overwrite the first, so reading the
   name back gives the value you set TWO lines ago. Nothing is wrong
   with the interpreter and nothing was shadowed: there are two names
   in the namespace and they are the same picture.

      names bound   ['value', 'vаlue']
      as code points
         U+0076 U+0061 U+006C U+0075 U+0065   -> 3
         U+0076 U+0430 U+006C U+0075 U+0065   -> 4

   And the one-line version of the same fact, which is the thing to
   reach for when a name will not resolve and the spelling looks
   right:

       >>> ord('a')
       97
       >>> ord('a')          # pasted from somewhere else
       1072

   97 is U+0061 LATIN SMALL LETTER A.
   1072 is U+0430 CYRILLIC SMALL LETTER A.
   ord() is the whole diagnosis, and it fits on one line.

7. AND THE CHARACTERS THAT CANNOT GET IN AT ALL
------------------------------------------------------------------------
   The letter that slips through is an ORDINARY, ASSIGNED, VISIBLE
   one. The reserved code points are refused at the door:

   code point  what it is                                   isidentifier()  exec
   U+0430      CYRILLIC SMALL LETTER A -- a real letter     True            bound
   U+00E9      LATIN SMALL LETTER E WITH ACUTE              True            bound
   U+E000      a private-use code point                     False           SyntaxError
   U+FFFE      a noncharacter                               False           SyntaxError
   U+0378      unassigned -- may be a letter one day        False           SyntaxError

   Python rejects the last three with the same message -- `invalid
   non-printable character` -- because none of them carries
   XID_Continue, and a code point with no properties cannot be part
   of a name. Which is the reassuring half and also the point: the
   dangerous character in an identifier is never the exotic one. It
   is a real letter from a real alphabet that happens to be drawn
   the same as yours.

PEP 3131 ↗ settled this in 2007: identifiers may be any UAX #31 name, and the compiler applies NFKC to them. So the ligature becomes file, the mathematical italic letter becomes a, and Ⅰ ROMAN NUMERAL ONE becomes the capital letter I. Everything about that is documented, deliberate, and nearly invisible.

Sections 3 and 4 are the pair worth holding together, because the shape is uncomfortable:

Where the characters are related, normalization surprises you — one variable, two spellings, and grep finds a different subset depending on which spelling you search for.
Where they are not related, it abandons you — Cyrillic а and Latin a are two variables, rendered identically, and Python says nothing at all.

That is the same rule as confusables and scripts, met from the other side: NFKC merges what Unicode declared compatibility-equivalent and never merges two letters that are merely drawn alike. The first case costs you an afternoon. The second is the one that gets through a code review.

Sections 6 and 7 are that second case as you actually meet it, which is at a prompt, wondering whether the interpreter is broken. Assign value = 3, assign value = 4 with a Cyrillic а in the middle, ask for value, and get 3 — the value you set two lines ago. Nothing was shadowed and nothing is wrong: there are two names in the namespace and they are the same picture. ord() is the entire diagnosis and it fits on one line — 97 is U+0061, 1072 is U+0430.

Section 7 is the reassuring half, and it sharpens the point rather than softening it. A private-use code point, a noncharacter and a merely unassigned one are all three refused outright, with the same invalid non-printable character — none of them carries XID_Continue, and a code point with no properties cannot be part of a name. So the character that gets into your identifier is never the exotic one. It is a real letter from a real alphabet that happens to be drawn like yours.

Rust preserves, and warns¶

Verified output of unicode_in_identifiers_rs.rs — regenerated by tools/run_examples.py, never hand-typed.

1. RUST TAKES THEM TOO
------------------------------------------------------------------------
   let żółw = 4;   ->   4

   Stable since Rust 1.53, from the same UAX #31 rule Python
   uses. So far the two languages agree completely.

2. AND NORMALIZES THEM -- TO NFC, NOT NFKC
------------------------------------------------------------------------
   bound as   caf\u{e9}       (composed, 4 chars)
   read as    cafe\u{301}     (e + COMBINING ACUTE U+0301, 5 chars)
   value      1

   Two spellings, one binding -- the compiler put the source
   into NFC before it looked at any name. This is the half
   Rust and Python agree on, and it is the uncontroversial
   half: the two spellings really are the same character.

3. BUT THE LIGATURE IS A DIFFERENT NAME
------------------------------------------------------------------------
   let ﬁle  = 1;   (U+FB01 LATIN SMALL LIGATURE FI)   ->  1
   let file = 2;   (f, i as ASCII)                    ->  2
   they are the same binding                          ->  false

   Python binds ONE variable here, because NFKC expands the
   ligature. Rust binds TWO, because NFC does not. Neither is
   wrong; they answer different questions about what a name is.

4. AND SO IS THE CYRILLIC LOOK-ALIKE
------------------------------------------------------------------------
   let а = ...;   (U+0430)   ->  cyrillic
   let a = ...;   (U+0061)   ->  latin
   same binding              ->  false

   Here the two languages AGREE on the outcome -- two
   variables -- and disagree completely about what to do next.

5. THE DIFFERENCE THAT MATTERS IS THE WARNING
------------------------------------------------------------------------
   With the allow() at the top of this file removed, rustc
   emits three lints, all warn-by-default:

      uncommon_codepoints        the ligature is Not_NFKC
      confusable_idents          two names that look alike
      mixed_script_confusables   a script used only for these

   That is UTS #39 machinery, shipped in a compiler, on by
   default. Python has no equivalent and cannot easily have
   one -- having normalized the ligature away, there are no
   longer two names for it to compare.

   So the trade runs: NORMALIZE and the surprise is silent,
   PRESERVE and you owe the user a warning. Rust chose to owe
   the warning. It is the same argument IDNA2008 had with
   IDNA2003 about a domain name, one floor down.

Rust accepts the same UAX #31 identifiers (stable since 1.53) and normalizes to NFC rather than NFKC. Follow what that changes, one row at a time:

	Python (NFKC)	Rust (NFC)
`café` composed vs decomposed	one variable	one binding — they agree
`ﬁle` vs `file`	one variable, silently	two bindings, two warnings
Cyrillic `а` vs Latin `a`	two variables, silently	two bindings, two warnings

The middle row is where the languages genuinely disagree, and neither is wrong — they are answering different questions about what a name is. The bottom row is where they agree on the outcome and disagree completely about what to do next.

Because the three lints are what this page is about, the example file allows them — otherwise every CI run would print warnings forever. Here is what rustc actually says with that line removed:

Measured on one Mac, rustc 1.98.0, 2026-09-06 — not machine-checked: lint wording is not stable enough to record

warning: identifier contains a non normalized (NFKC) character: 'ﬁ'
  = note: this character is included in the Not_NFKC Unicode general security profile
  = note: `#[warn(uncommon_codepoints)]` on by default

warning: found both `ﬁle` and `file` as identifiers, which look alike
  = note: `#[warn(confusable_idents)]` on by default

warning: the usage of Script Group `Cyrillic` in this crate consists solely of mixed script confusables
  = note: the usage includes 'а' (U+0430)
  = note: please recheck to make sure their usages are indeed what you want
  = note: `#[warn(mixed_script_confusables)]` on by default

That is UTS #39 ↗ machinery — the confusables data and the mixed-script restriction from the previous page — shipped inside a compiler and on by default. Note the third one especially: it is not complaining that a Cyrillic letter appears, but that the only use of Cyrillic in the whole crate is a confusable, which is a judgement about the program rather than about a character.

And Python cannot easily have the same thing, which is the real cost of its choice: having normalized the ligature away, there are no longer two names left to compare. You cannot warn about a distinction you have already erased.

The trade, stated once¶

Normalize and the surprise is silent. Preserve and you owe the user a warning. Rust chose to owe the warning; Python chose the quieter source of truth and left the warning to linters. Preparing a string is the same argument at protocol scale — IDNA2003 mapped, IDNA2008 refused to map and pushed the problem to registries — and it is worth noticing that the later design in both cases is the one that stopped folding.

The practical advice is duller than the mechanism, and it is still right for almost every project: keep identifiers ASCII and put the other languages in the strings. Not because non-ASCII names are wrong — a Polish or Japanese team naming domain concepts in their own language is a real gain — but because the moment identifiers are unrestricted, "these two names are the same" becomes a question your reviewers cannot answer by looking, and you need a tool that does. If you do allow them, the tool is rustc's lints, a linter with a confusables rule, or a CI check; the one thing not to rely on is a reader.

If you are coming from Python or ABAP¶

Python. Nothing in the standard library will tell you an identifier was normalized — ast gives you the post-NFKC name, because normalization happens in the tokenizer. To audit a file you have to compare the source text against the parsed names yourself: tokenize, take every NAME token, and flag any where unicodedata.normalize("NFKC", tok) != tok. That is a twenty-line pre-commit hook and it is the only defence the language offers.

ABAP. (Not machine-checked — CI cannot run ABAP.) Identifiers are ASCII and case-insensitive, so this entire class of problem does not exist — which is worth stating plainly rather than treating as a limitation. The nearest equivalent risk lives in data, not code: two customer or material names that render identically and differ in one code point will behave as different keys everywhere, and no CP comparison or TRANSLATE … TO UPPER CASE will bring them together. That is confusables and scripts's problem, and the control is at the interface, not in the ABAP.

Try it¶

Paste ﬁle = 1 into a Python REPL, then type file. Then try to explain it to somebody without using the word "normalization".
python3 -c "import unicodedata as u; print(u.normalize('NFKC', 'ﬁ'))" — the whole mechanism in one call.
Write the twenty-line tokenizer check from the bridge above and run it over a project you did not write.
In Rust, take the example on this page, delete the #![allow(...)] line, and compile it. Three warnings, no configuration.
Ask what your own code review would catch. For most teams the honest answer is "nothing", and the fix is a rule rather than a person.

Practice¶

How many variables is this?

ﬁle = 2      # that is U+FB01, the fi ligature
print(file)  # an ordinary f and i

Predict what the second line does. Then take file, ﬁle, ﬀile, ｆile and 𝐟ile and say which of them bind the same name in Python — and what Rust does with the same five, which is a different answer for a stated reason.

Answers

Verified output of unicode_in_identifiers_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

source text:  \ufb01le = 2        (the name starts with the ligature fi)
then `file` -- typed with an ordinary f and i -- reads back as 2
and the two spellings are different STRINGS: False
One assignment, one variable, two ways of typing its name.

written  NFKC     identifier?  code points
file     file     True         0066 0069 006C 0065
ﬁle      file     True         FB01 006C 0065
ﬀile     ffile    True         FB00 0069 006C 0065
ｆile     file     True         FF46 0069 006C 0065
𝐟ile     file     True         1D41F 0069 006C 0065

   all of ['file', 'ﬁle', 'ｆile', '𝐟ile'] are the identifier 'file'

Python normalizes identifiers to NFKC before binding -- PEP 3131, and it
is not optional. So five different pieces of source text are one name,
and nothing warns you: no error, no lint, no visible difference at the
point of use. A diff shows two lines that look identical.

Rust made the other choice. Identifiers are XID with NFC applied and
mixed-script confusables are a WARNING, so the compiler preserves what
you wrote and tells you when two names could be confused. Neither
choice is wrong; they answer different questions.

It is IDNA2003 against IDNA2008 one floor down -- map-and-fold so that
two spellings cannot both exist, versus preserve-and-refuse so that
nothing is silently changed. Domain names had this argument first, and
had to redesign twice; here it decides what your program is written in.