Skip to content

Comparison has a mode

Level: 201 · for Python programmers

One line: Python has four ways to compare two strings and a name for none of them — == is the ordinal one, and learning that word is what lets you notice that you never chose it.

a = "résumé"       # é written as one code point
b = "résumé"     # é written as e + combining acute accent
print(a, b)                  # résumé résumé   — the same word, twice
a == b                       # False
len(a), len(b)               # (6, 8)

Two strings that a reader would call identical, and == says no. Nothing is broken. == on a str compares code points, left to right, and stops at the first difference — that is the entire rule, it is the same rule on every machine in the world, and Python gives you no way to ask for a different one.

That rule has a name, and Python never says it. Ordinal comparison is the term .NET uses, and .NET uses it constantly, because its string APIs make you pass a StringComparison value — Ordinal, OrdinalIgnoreCase, or a culture-sensitive mode — at the call site. Every Equals, every IndexOf, every StartsWith asks you which question you are asking. Python asks you nothing and answers Ordinal.

Python has the modes; it just spells them as unrelated APIs

The thing worth taking from the .NET framing is not the parameter. It is the observation that "compare these two strings" is at least three different questions, and that a language which does not name them has programmers who do not know they are choosing:

The question .NET spells it Python spells it
Are these the same sequence of code points? StringComparison.Ordinal ==, and every other str operation
The same, ignoring case? StringComparison.OrdinalIgnoreCase a.lower() == b.lower(), or a.casefold() == b.casefold(), or re.IGNORECASE — three answers that disagree
The same, to a reader of some language? StringComparison.CurrentCulture locale.strxfrm, which needs an installed locale and process-global state, or ICU, which is not in the standard library

Row one is free and invisible. Row two is a transformation you apply by hand, in three flavours nothing tells you apart. Row three barely exists. That absence is the honest answer to "is Python good at this?" — it is excellent at the mode you get by default and it hands you the other two as homework.

There is a detail under row three that Python does not have at all, and it is the sharpest one: .NET's culture-sensitive comparison is documented to ignore embedded NUL characters, so two strings differing only by a NUL can compare equal. Section 6 of the run below puts the same pair to Python and measures what happens.

The verified output

Verified output of comparison_has_a_mode_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. == IS ORDINAL -- PYTHON JUST NEVER SAYS THE WORD
   Ordinal means: compare code points, left to right, first difference
   wins. No alphabet, no locale, no case table, no normalization.

     the two strings print as        résumé and résumé
     left == right                   False
     len(left), len(right)           6, 8
     first differing position        1
     the two code points there       U+00E9 and U+0065

     That is the whole rule. Every str comparison in Python obeys it,
     and none of them takes an argument that could change it.

2. FOUR SPELLINGS OF ONE WORD
   To a reader these are the same word, four times:

     A   résumé
     B   résumé
     C   résumé
     D   résumé

   To Python they are four different values:

     A   len 6    8 bytes   0072 00E9 0073 0075 006D 00E9
     B   len 7    9 bytes   0072 00E9 0073 0075 006D 0065 0301
     C   len 7    9 bytes   0072 0065 0301 0073 0075 006D 00E9
     D   len 8   10 bytes   0072 0065 0301 0073 0075 006D 0065 0301

   Every pair, under ==:

              A      B      C      D
     A     True  False  False  False
     B    False   True  False  False
     C    False  False   True  False
     D    False  False  False   True

     4 distinct values, so all 6 pairs are False. One word, four keys:
     len(set(...)) is 4, and a dict built from them has 4 entries.

3. WHERE find() PUTS YOU
   find() is ordinal too, so a needle matches one spelling and not the
   other -- and the offset it returns moves with a letter you did not
   search for.

     needle                           A    B    C    D
     'e'                             -1    5    1    1
     'e' with acute, precomposed      1    1    6   -1
     'e' then U+0301                 -1    5    1    1
     'sum'                            2    2    3    3

     Read the last row first: 'sum' is pure ASCII and appears in all
     four, at index 2 in two of them and index 3 in the other two. The
     accent is nowhere near the needle and it still moved the answer,
     because a code point earlier in the string was written as two.

     Rows one and three are the same four numbers, which is the
     other half of it: in B, C and D the 'e' you find by searching for
     a bare 'e' IS the front of the accented letter. 'e' is not in A at
     all, and in D it is found at an index inside a letter:

     D.find('e')          1
     D[:2] and D[2:]      're' and '\u0301sume\u0301'
     D.replace('e', 'a')  'rásumá'   = 'ra\u0301suma\u0301'
     A.replace('e', 'a')  'résumé'   = 'r\xe9sum\xe9'

     Slicing there leaves a combining accent stranded at the head of
     the tail, and replacing 'e' with 'a' carries both accents onto the
     'a'. In A the same call changes nothing: there is no 'e' in A.

4. THREE CASE-INSENSITIVE ANSWERS, AND NO TWO OF THEM AGREE
   .NET spells this OrdinalIgnoreCase and gives you exactly one.
   Python has three, spelled as unrelated APIs, and picking one is
   picking a mode. WHY lower() and casefold() differ is its own page;
   this table is only about how far apart the three answers land.

     left       right          ==   lower  casefold   re.I   what it is
     'STRASSE'  'straße'    False   False      True  False   German sharp s
     'Σ'        'ς'         False   False      True   True   Greek final sigma
     'K'        'k'         False    True      True   True   KELVIN SIGN U+212A
     'I'        'ı'         False   False     False   True   Turkish dotless i
     'ꭰ'        'Ꭰ'         False    True      True   True   Cherokee A, both cases

     Two rows carry the lesson. re.IGNORECASE will not match STRASSE
     against straße, which casefold() does -- it has no one-to-two
     mappings, so on that row it sides with lower(). And it WILL match
     'I' against Turkish dotless 'ı', which casefold() refuses,
     because those are two different letters -- so on that row it sides
     with neither. Three APIs, three questions, and nothing at the call
     site says which one you asked.

     One more pair, because it shows casefold() is a mapping in its
     own right rather than a thorough lower():

       'ꭰ'.lower()     'ꭰ'   (already lowercase, so nothing to do)
       'ꭰ'.casefold()  'Ꭰ'   (CHEROKEE LETTER A)

     The fold of a Cherokee small letter is a CAPITAL. Whatever
     casefold() is, it is not lower() with more entries.

5. THE LINGUISTIC MODE, AND HOW LITTLE OF IT PYTHON HAS
   .NET's default is culture-sensitive. Python's only stdlib door to
   that is locale.strxfrm, which asks the C library for a sort key.

     locale.strxfrm('résumé')
       -> 'résumé'
       identical to the input?  True

     Under the C locale there are no rules to apply, so the linguistic
     API hands back the ordinal answer -- silently, with no error and no
     flag. That is the state a container, a cron job and a CI runner all
     start in. It is also process-global state, and not thread-safe.

6. THE NUL CHARACTER, WHERE THE MODES DISAGREE MOST
   A NUL inside a string is the case that separates the three modes
   furthest, because a comparison that ignores it will call two
   different strings equal. Here is what Python does with three
   strings that differ by nothing else.

     'admin'       len 5   utf-8 b'admin'       decodes back unchanged: True
     'admin\x00'   len 6   utf-8 b'admin\x00'   decodes back unchanged: True
     'ad\x00min'   len 6   utf-8 b'ad\x00min'   decodes back unchanged: True

     TRIO[0] == TRIO[1]     False
     TRIO[0] == TRIO[2]     False
     len(set(TRIO))         3
     distinct hashes        3
     'admin' in TRIO[2]     False

     Ordinal says three different strings, and that is the right
     answer: NUL is a character like any other, it encodes to one
     ordinary byte, and UTF-8 round-trips it without complaint.

     Now hand the same three to the linguistic API:

     locale.strxfrm on 'admin'       -> 'admin'
     locale.strxfrm on 'admin\x00'   -> ValueError: embedded null character
     locale.strxfrm on 'ad\x00min'   -> ValueError: embedded null character

     Python does not ignore the NUL, and it does not compare it
     either. It refuses the string. That is a third possible behaviour,
     and it is the one worth having: the two strings are not equal, and
     the call that cannot say so fails instead of guessing.

7. SO WHICH MODE FIXES THE FOUR SPELLINGS? NONE OF THEM.
   Count the distinct values the four collapse to under each mode.
   1 would mean all four finally agree.

     == (ordinal)               4
     lower()                    4
     casefold()                 4
     locale.strxfrm, C locale   4
     NFC                        1
     NFD                        1
     NFC then casefold()        1

     Every comparison mode says four. Every normal form says one.
     Comparison mode was never the question here: the four spellings
     are four different strings, and no way of comparing four different
     strings makes them one. Normalization is the fix, and it belongs
     on the way in, once, at the boundary -- not in the comparison.

What the run shows

Four spellings, six comparisons, six Falses. résumé has two accented letters and each can be written two ways, so there are four spellings of one word — lengths 6, 7, 7 and 8, all rendering identically, none equal to any other. Put them in a set and you get four entries; use them as dict keys and you get four rows. This is not a curiosity of the example: composed text and decomposed text meet in ordinary programs, because different sources hand you different spellings and nothing in between them normalizes.

find() moves an offset you did not touch. Section 3 is the part worth remembering. Search all four spellings for 'sum' — pure ASCII, no accent anywhere near it — and the answer is index 2 in two of them and index 3 in the other two, because a letter earlier in the string was written as two code points instead of one. Any code that calls find() and then slices at the result is quietly spelling-dependent. Worse, searching for a bare 'e' succeeds in the decomposed spellings and lands inside a letter: slice there and a combining accent is stranded at the head of the tail, and replace("e", "a") carries both accents onto the a and gives you rásumá.

There are three case-insensitive answers in the standard library, and no two of them agree. Lowercasing is not folding is the page about why lower() and casefold() differ and which of the two you want; this page adds the third and asks a different question — how far apart do the three land. Section 4 puts them side by side, and the two rows that matter both involve re.IGNORECASE. It will not match STRASSE against straße, which casefold() does, because it has no one-to-two mappings — so on that row it sides with lower(). It will match "I" against Turkish dotless "ı", which casefold() refuses, because those are two different letters — so on that row it sides with neither. Three APIs, three questions, and nothing at the call site says which one you asked.

The Cherokee row is the one worth keeping in your head, because it settles what kind of operation folding is: "ꭰ".lower() is "ꭰ" — it is already lowercase, so there is nothing to do — and "ꭰ".casefold() is "Ꭰ", a capital. Whatever casefold() is, it is not lower() with more entries in the table.

The linguistic mode degrades to the ordinal one without telling you. locale.strxfrm is the standard library's only door to culture-sensitive comparison. Under the C locale — which is what a container, a cron job and a CI runner all start in, and what every example in this library runs under — it has no rules to apply and hands back the input unchanged. That is the ordinal answer, returned by the linguistic API, with no error and no flag. Sorting is not comparing is the page about what that costs when you sort.

The NUL, which is the whole point. Three strings — admin, admin with a trailing NUL, and admin with a NUL in the middle — differing by nothing but that one character. UTF-8 encodes it, decodes it back unchanged, and complains about none of it; the sibling library's The NUL byte ↗ is the page on why nothing rejects it. Python's == says three different strings, they hash three different ways, and "admin" in "ad\0min" is False. That is the answer you want, and it is the answer you get for free, because ordinal comparison has no concept of a character that does not count.

The linguistic mode gives a different answer again, and it is the one worth the page. Hand the same three strings to locale.strxfrm and Python does not ignore the NUL and does not compare it either — it raises ValueError: embedded null character. Three platforms, three behaviours: .NET's culture-sensitive comparison is documented to call the strings equal, Python's ordinal comparison calls them different, and Python's linguistic API refuses the question. Refusing is the right third answer. The call cannot represent the distinction, so it fails instead of guessing, which is the opposite of silently equal.

(The .NET half of that sentence is from its documentation, linked in RESOURCES.md — it is not machine-checked here, because CI runs no .NET. Everything attributed to Python above is in the recorded output.)

Normalization is the fix, and comparison mode never was

Section 7 is the payoff, and it is a negative result. Count how many distinct values the four spellings collapse to under each mode: == says four, lower() says four, casefold() says four, locale.strxfrm says four. Every normal form says one.

That is the whole lesson about the four spellings. They are four different strings, and no way of comparing four different strings makes them one — the repair has to happen before the comparison, by rewriting each string into a canonical form. unicodedata.normalize("NFC", s) on the way in, once, at the boundary, is the fix; a comparison helper that normalizes internally is a system where a == b depends on which function asked.

Normalization belongs to the encodings library, which has the written page: Normalization ↗. The stub here holds the URL, and where the boundary between the two libraries finally sits is an open question tracked in the roadmap. Read the written one; this page does not repeat it.

If you are coming from ABAP

(Not machine-checked — CI cannot run ABAP.)

ABAP made the third choice, and it is neither Python's nor .NET's: the mode is baked into which operator you type. = is not CS, and the difference is not what they compare but how. Comparison-mode questions ABAP answers with a token where .NET answers with a parameter and Python does not answer at all — so the ABAP habit to unlearn is not "check the mode", it is "the mode was decided when you picked the operator, so read it again".

Two specifics to verify on your system rather than trust from memory, because both are the kind of thing that has been true for a long time and is easy to misremember. First, the case rule is not uniform across the character operators: the pattern and substring operators (CS, CP, and NS, NP) are documented as not case-sensitive, while the character-set operators (CO, CA, CN, NA) are. Two comparisons one letter apart in the source, two different modes. Second, and with more consequences: a fixed-length c field is blank-padded, so trailing blanks fall out of an = comparison, while string is variable-length and they do not. Python has no equivalent of that at all — "abc" == "abc " is False, always, because there is no padding to ignore and no mode in which ignoring it would be an option.

What transfers: the instinct that a comparison is a choice. What Python enforces that ABAP leaves to habit: exactly one comparison exists, so == in a Python code review is never the line to argue about — the line to argue about is whatever produced the two strings.

If you are coming from Rust

Rust arrives at the same place from the opposite end: == on a str is self.as_bytes() == other.as_bytes(), a byte comparison rather than a code-point one, which is the same ordinal semantics because UTF-8 byte order is code point order. The sharper contrast is one level up — Rust's eq_ignore_ascii_case is deliberately ASCII-only, a language refusing to guess what you meant by "ignoring case" for anything a person typed, where Python's lower() silently applies the whole Unicode table. See Comparing and sorting text ↗ in the Rust library.

Which trade did each language make

.NET's rule — every string API takes an explicit comparison mode — buys one thing and costs another. It buys a code review in which the question is visible: IndexOf(needle) and IndexOf(needle, StringComparison.Ordinal) do not look alike, so a reader can see which question was asked. It costs a parameter on every call site in the language forever, and it means a programmer who has no opinion must still type one, which is how a codebase ends up with the default chosen by whatever the first search result said.

Python made the opposite trade and got the opposite pair. One comparison, no parameter, nothing to get wrong at the call site, and an operator that means the same thing in every line of Python ever written — that is a real and underrated property, and it is why == on a str is never the bug. The cost is that the choice is invisible, so a programmer can work for years without learning that there was one, and then meets it as a bug report from somewhere with accents in the names.

Neither is wrong. .NET decided the risk worth defending against is a wrong mode chosen silently; Python decided it is a wrong mode chosen noisily, by someone who does not know what the modes are. The reason to know the word "ordinal" in Python is that it converts Python's trade into .NET's for free: you cannot pass the mode, but you can see it.

This page was prompted by Microsoft's Best practices for using strings in .NET ↗ and its Character encoding in .NET ↗, both in RESOURCES.md. The topic is theirs; the prose, the program and every number above were written and measured here.

Try it

  1. Take a directory of files whose names came from more than one place — a phone sync, a download, a checkout on someone else's laptop — and list them, then list unicodedata.normalize("NFC", name) beside each. Any name where the two differ is a name your == and your in will disagree with a human about. Filenames are not text is why macOS is the usual source.
  2. Grep your own code for .lower() used as a comparison — if a.lower() == b.lower() and x.lower() in seen. For each hit, decide which of the three case-insensitive answers you actually wanted. If the data is HTTP header names or file extensions, lower() is fine and ASCII is the whole domain; if it is anything a person typed, casefold() is the one that is right.
  3. Find a place where your code calls find or index and then slices at the result. Feed it a string in which an accent is written decomposed and see whether the slice still lands where you meant. Four ways to find it and Slicing is not indexing are the two pages about the halves of that.
  4. Take a text field of yours that reaches a filesystem, a database driver or a subprocess argument — anything with C underneath — and check whether a NUL can get into it. Then decide where the check belongs: == will happily carry the NUL all the way down, because to Python it is content.

Practice

Eight questions, and no two of the three case-insensitive answers agree. Write down what each returns before you run anything. Rows 1–3 are one pair of strings asked three ways; rows 4–5 are a second pair asked two ways.

  1. "straße".lower() == "STRASSE".lower()
  2. "straße".casefold() == "STRASSE".casefold()
  3. re.fullmatch("straße", "STRASSE", re.I) — is it a match?
  4. "ı".casefold() == "I".casefold()
  5. re.fullmatch("ı", "I", re.I) — is it a match?
  6. "ꭰ".casefold().isupper()
  7. "admin" == "admin" + chr(0)
  8. locale.strxfrm("admin" + chr(0))

Then: rows 3 and 5 fall on opposite sides. re.IGNORECASE agrees with lower() on the first pair and with neither on the second. Say what kind of mapping each of the three is doing, and which one you want for a login form.

Answers

Verified output of comparison_has_a_mode_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

PREDICTIONS
     #  expression                                    answer   why
     ------------------------------------------------------------------------------------------------
     1  'straße'.lower() == 'STRASSE'.lower()          False   lower() has no 1-to-2 mapping: 'ß' stays 'ß'
     2  'straße'.casefold() == 'STRASSE'.casefold()     True   casefold() folds 'ß' to 'ss', so both sides become 'strasse'
     3  re.fullmatch('straße', 'STRASSE', re.I)        False   re.IGNORECASE has no 1-to-2 mapping either -- it agrees with lower()
     4  'ı'.casefold() == 'I'.casefold()               False   dotless i is a different letter; folding keeps them apart
     5  re.fullmatch('ı', 'I', re.I)                    True   ...and here re.IGNORECASE disagrees with casefold() instead
     6  'ꭰ'.casefold().isupper()                        True   Cherokee folds UPWARD -- the fold of a small letter is a capital
     7  'admin' == 'admin' + chr(0)                    False   ordinal: a NUL is a character, so the strings differ

8    locale.strxfrm('admin' + chr(0))
     -> ValueError
     Not equal, and not unequal either. The linguistic API refuses a
     string it cannot represent rather than quietly dropping the NUL.

THE FOLLOW-UP: WHY 3 AND 5 GO OPPOSITE WAYS
   Three case-insensitive answers, and no two of them agree on both pairs:

     pair                     lower  casefold   re.I
     straße / STRASSE         False      True  False
     ı / I                    False     False   True

     casefold() implements Unicode case folding, which is defined for
     caseless MATCHING: it expands 'ß' and it deliberately leaves the
     Turkish letters apart. re.IGNORECASE is built from a table of
     single-character equivalences, so it can pair 'ı' with 'I' -- one
     code point for one -- and cannot pair 'ß' with 'ss' at all.
     Neither is a bug. They answer different questions, and the only
     mistake available is not knowing which one you called.

See also