Skip to content

\w is ASCII, [[:alpha:]] is not, and \b sides with Unicode

Level: 201 · anyone matching words in text that is not all English

One line: On UTF-8 text Ruby's \w, \d and \s match ASCII characters only, while [[:alpha:]], [[:digit:]], [[:space:]] and \p{L} match Unicode — and \b uses the Unicode idea of a word, so /\b\w+\b/ cannot match café at all.

Verified output of w_is_ascii_b_is_not_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

word = "zażółć"
  word.scan(/\w+/)                           ["za"]
  word.scan(/[[:alpha:]]+/)                  ["zażółć"]
  word.scan(/\p{L}+/)                        ["zażółć"]
  word.scan(/(?u)\w+/)                       ["zażółć"]

  "caf\u{e9} au lait".scan(/\b\w+\b/)        ["au", "lait"]
  "caf\u{e9} au lait".scan(/\w+/)            ["caf", "au", "lait"]
  "caf\u{e9}" =~ /caf\b/                     nil

  "\u{661}\u{662}\u{663}" =~ /\d/            nil
  "\u{661}\u{662}\u{663}" =~ /[[:digit:]]/   0
  "a\u{a0}b" =~ /\s/                         nil
  "a\u{a0}b" =~ /[[:space:]]/                1

Reading the output

  • \w+ stops at ż. The POSIX bracket, \p{L} and (?u)\w all take the whole word.
  • \b and \w disagree about é. To \b, é is a word character, so there is no boundary between f and é. To \w, é is not a word character, so \w+ cannot reach past caf. Between them, /\b\w+\b/ finds au and lait and skips café entirely, and /caf\b/ does not match inside café.
  • Arabic-Indic digits are [[:digit:]] but not \d — and Integer() does not read them either, see to_i never fails.
  • A no-break space is [[:space:]] but not \s — and awk-mode split does not split on it, see split has an awk mode.

What to do

In a pattern that may meet text beyond ASCII, write the POSIX brackets or \p{...} — [[:alpha:]], [[:alnum:]], \p{L} — or put (?u) in front. Keep \d where you mean exactly the digits 0 to 9, which is usually what a parser wants.

If you are coming from another language

Perl treats text as Unicode here — given use feature 'unicode_strings', for the reasons in the Perl library's The Unicode bug ↗ — and /a restricts \w, \d and \s to ASCII:

Verified output of w_is_ascii_b_is_not_pl.pl — regenerated by tools/run_examples.py, never hand-typed.

  $word =~ /\w+/g                    zażółć
  $word =~ /\w+/ag                   za
  $cafe =~ /\b\w+\b/g                café au lait
  "\x{661}\x{662}\x{663}" =~ /\d/    match
  "a\x{A0}b" =~ /\s/                 match

Python's re is Unicode by default for a str pattern, and re.ASCII turns that off:

Verified output of w_is_ascii_b_is_not_py.py — regenerated by tools/run_examples.py, never hand-typed.

  re.findall(r'\w+', word)             ['zażółć']
  re.findall(r'\w+', word, re.ASCII)   ['za']
  re.findall(r'\b\w+\b', cafe)         ['café', 'au', 'lait']
  re.search(r'\d', digits)             True
  re.search(r'\s', 'a\xa0b')           True

So Ruby's defaults are the opposite of both, and Ruby is the one of the three where \b and \w disagree. Java makes all four of \w, \d, \s and \b ASCII by default, consistently — the Java text library's \w is ASCII by default ↗. The Encodings library's "Supports Unicode" is a level, not a yes ↗ grades regex engines against Unicode's own checklist.