Skip to content

Unicode text

One line: Every regex question about non-ASCII text is one of three questions — is the engine thinking in ASCII, in code units, or in code points? — and one table with ten rows shows which answer each engine gives to each; the Unicode chapter has the depth, and this page is the map.

Level: 301 · your input is not only English

The three questions

  1. ASCII or Unicode? — for \w, \d, \s, \b, (?i) and POSIX classes. Python str, Perl and Rust's regex say Unicode by default; JavaScript, Ruby, Java, Go and PCRE2 (so ABAP) say ASCII, each with its own switch.
  2. Code units or code points? — for . and for what "one character" means. Every engine here is code points except JavaScript without the u flag, which sees UTF-16 and splits an emoji in two.
  3. Code points or grapheme clusters? — for what a person calls one character. Only \X (Perl, PCRE2, Ruby, Java) sees clusters; . never does.

Measured

Verified output of topic_unicode_text_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

construct                    python     perl       node       ruby       java       go         rust       fancy
\w on an accented letter     \u{E9}     \u{E9}     no         no         no         no         \u{E9}     \u{E9}
\d on an Arabic-Indic digit  \u{663}    \u{663}    no         no         no         no         \u{663}    \u{663}
\p{L} on Greek               -          \u{3B1}    no         \u{3B1}    \u{3B1}    \u{3B1}    \u{3B1}    \u{3B1}
\p{L} on Greek, node u       -          \u{3B1}    \u{3B1}    \u{3B1}    \u{3B1}    \u{3B1}    \u{3B1}    \u{3B1}
. on an emoji                \u{1F600}  \u{1F600}  no         \u{1F600}  \u{1F600}  \u{1F600}  \u{1F600}  \u{1F600}
. on an emoji, node u        \u{1F600}  \u{1F600}  \u{1F600}  \u{1F600}  \u{1F600}  \u{1F600}  \u{1F600}  \u{1F600}
. on e + combining accent    no         no         no         no         no         no         no         no
\X on e + combining accent   -          e\u{301}   no         e\u{301}   e\u{301}   -          -          -
(?i) on LONG S vs s          \u{17F}    \u{17F}    -          \u{17F}    no         \u{17F}    \u{17F}    \u{17F}
(?i) on sharp s vs ss        no         \u{DF}     -          \u{DF}     no         no         no         no

Reading the table

  • Rows 1–2, question 1. \w on é and \d on an Arabic-Indic digit: Python, Perl and Rust match; JavaScript, Ruby, Java and Go do not. The switches are re.ASCII / (?a) to turn Unicode off in Python, (?U) to turn it on in Java, (*UCP) in PCRE2, and — for JavaScript — nothing: its \w is ASCII under every flag. What \w and \b match.
  • Rows 3–4, properties. \p{L} is the portable way to say "letter" — except in Python, which refuses it, and in JavaScript, where it needs u. Naming a Unicode property.
  • Rows 5–6, question 2. ^.$ on U+1F600: one character in seven engines; two code units in JavaScript without u, one with.
  • Rows 7–8, question 3. . on e + U+0301 is no for ^.$ everywhere — two code points, and . takes one. \X takes both, in the three engines that have it. Grapheme cluster.
  • Rows 9–10, folding. (?i)^s$ matches U+017F LONG S in six engines and not in Java, whose CASE_INSENSITIVE is ASCII without UNICODE_CASE. (?i)^ss$ matches U+00DF only in Perl and Ruby — the two engines that do full case folding. Case-insensitive matching.

The rules that survive every engine

  1. Say what you mean with a property. \p{L}, \p{Nd}, \p{White_Space} — where the engine has them. Where it does not (Python), the standard library's str.isalpha() / isdecimal() are the honest substitute.
  2. Normalise first, then match. NFC before comparison; casefold() or its equivalent before a case-insensitive comparison that has to be right, and let the regex match the normalised form. A regex engine's (?i) is a simple-folding table and no two are the same.
  3. In JavaScript, always u. It changes ., \p{…}, \u{…}, case folding, and which escapes are errors. Without it the engine is a 1999 engine.
  4. Never truncate with .{n}. It cuts clusters. Use \X{n} or a grapheme iterator.
  5. Bytes are not text. A pattern over a bytes subject (Python), an undecoded string (Perl), or a []byte (Go) is ASCII whatever the flags say, because there are no characters to be Unicode about. Decode first. The Encodings learning library ↗ is where that decoding lives.

Keywords on this page

Shorthand classes · \b · \p{…} · \X · . · POSIX classes · Inline flags · (*UCP).

See also