Case mapping is full Unicode, with options instead of a locale¶

Level: 201 · anyone uppercasing, lowercasing or comparing without case

One line: upcase turns ß into SS, downcase(:fold) folds for comparison, and the only way to get Turkish dotless-i rules is to pass :turkic — but casecmp is still ASCII-only, and downcase does not apply the Greek final-sigma rule that Python's lower does.

Unicode case mapping is not a table of one character to one character: a lowercase letter can uppercase to two letters, and a few letters change depending on their neighbours. The Encodings library's Case is not a per-character operation ↗ has the background. Ruby's upcase, downcase, capitalize and swapcase apply Unicode's full mappings, and take options — :ascii, :turkic, :lithuanian, :fold — where other languages consult a locale.

Verified output of case_mapping_is_full_unicode_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

Full mappings: one character can become two
  "stra\u{df}e".upcase                 "STRASSE"
  "stra\u{df}e".upcase.downcase        "strasse"
  "\u{FB00}".upcase                    "FF"
  "\u{130}".downcase.length            2
  "\u{1C6}".capitalize                 "ǅ"
  "\u{3A3}\u{391}\u{3A3}".downcase     "σασ"

Options instead of a locale
  "I".downcase                         "i"
  "I".downcase(:turkic)                "ı"
  "i".upcase(:turkic)                  "İ"
  "\u{C9}COLE".downcase(:ascii)        "École"
  "stra\u{df}e".downcase(:fold)        "strasse"

Comparing without case
  "stra\u{df}e".casecmp?("STRASSE")    true
  "stra\u{df}e".casecmp("STRASSE")     1
  "\u{C9}cole".casecmp?("\u{E9}COLE")  true
  "\u{C9}cole".casecmp("\u{E9}COLE")   -1
  "STRASSE".match?(/stra\u{df}e/i)     true

Reading the output¶

One character can become two. ß uppercases to SS, so the round trip gives strasse. The ligature U+FB00 uppercases to FF, and İ lowercases to i plus U+0307, two code points.
capitalize uses titlecase. ǆ becomes ǅ — one code point that is neither the uppercase Ǆ nor the lowercase ǆ.
No final sigma. Greek writes σ inside a word and ς at the end of one. Ruby's downcase gives σασ; Ruby's case mapping guide ↗ says context-dependent case mapping is not supported.
Options. :turkic maps I to ı and i to İ; :ascii touches only A–Z, so É stays; :fold is Unicode case folding, meant for comparison rather than display.
casecmp? folds; casecmp does not. casecmp? matches ß with SS and É with é. casecmp compares ASCII case only, and orders École before éCOLE with -1. A regex with /i folds like casecmp?.

What to do¶

Compare without case using casecmp?, or downcase(:fold) on both sides — not casecmp. Store downcase(:fold) as the key for a case-insensitive lookup. Pass :turkic when displaying text you know is Turkish or Azerbaijani.

If you are coming from another language¶

Perl gets ß right only when it treats the string as Unicode, and for a string with no character above U+00FF that takes use feature 'unicode_strings' — which use v5.12 and later turn on:

Verified output of case_mapping_is_full_unicode_pl.pl — regenerated by tools/run_examples.py, never hand-typed.

  uc($word), no feature                        STRAßE
  uc("$word \x{263A}"), no feature             STRASSE ☺
  uc($word), unicode_strings                   STRASSE
  fc($word) eq fc('STRASSE'), unicode_strings  true

The second row is the notorious one: adding ☺ to the end of the string changes how the ß in the middle of it is uppercased. The Perl library calls it the Unicode bug ↗, and its fc is how to compare without case ↗ is casecmp? seen from Perl.

Python applies the same full mappings, and the final-sigma rule too:

Verified output of case_mapping_is_full_unicode_py.py — regenerated by tools/run_examples.py, never hand-typed.

  word.upper()                             STRASSE
  word.casefold() == 'STRASSE'.casefold()  True
  sigma.lower()                            σας
  'I'.lower()                              i

Java's toLowerCase() with no argument uses the JVM's default locale — the Java text library's Case is locale-sensitive ↗. The Python library's Lowercasing is not folding ↗ is :fold's counterpart, and the Rust library's Wrong, but not unsafe ↗ includes a sigma bug of its own.