What \w and \b match once the subject is not ASCII¶
One line: \w is ASCII-only by default in JavaScript, Java, Ruby, Go and PCRE2 — and therefore in ABAP — and Unicode-aware by default in Python str, Perl and Rust's regex; \b is defined in terms of \w, so the same pattern over the same twelve characters finds one word in Ruby, two in Python and three in JavaScript.
Level: 301 · you write validation patterns that other people's names go through
Keywords on this page: \w \d \s · \b · [[:alpha:]] · (?i) · (*UCP) — Topics: Unicode text · Validating input
This is the most expensive Unicode bug there is, because it never raises anything. A pattern written against English test data keeps compiling, keeps matching, and quietly rejects or truncates everyone whose name is not spelled in ASCII.
The subject, and the seven characters¶
Every program on this page asks the same two questions of the same inputs.
The first is a character table. Seven characters, each named by code point rather than printed — under LC_ALL=C a JVM's System.out is US-ASCII, so a page that printed the characters themselves would record a different answer key on macOS than on a GitHub runner. The table asks, per engine, whether \w, \d and \s match:
| Code point | Name | Why it is here |
|---|---|---|
| U+0041 | LATIN CAPITAL LETTER A | the control — every engine says \w |
| U+00E9 | LATIN SMALL LETTER E WITH ACUTE | the ordinary accented letter in a name |
| U+0301 | COMBINING ACUTE ACCENT | a Mark, not a Letter — the engines that say "Unicode" disagree about this one |
| U+0660 | ARABIC-INDIC DIGIT ZERO | a decimal digit that is not 0–9 |
| U+FF11 | FULLWIDTH DIGIT ONE | another one, from a CJK input method |
| U+00A0 | NO-BREAK SPACE | the space that HTML forms paste in |
| U+3000 | IDEOGRAPHIC SPACE | Unicode whitespace, far from ASCII |
The second is a word count. The subject is na + U+00EF LATIN SMALL LETTER I WITH DIAERESIS + ve reader — the word naive spelled with a diaeresis, followed by reader, twelve characters, one of which is not ASCII. The question is how many words \b\w+\b finds in it. There is no ambiguity about the right answer for a human: two.
Python: str is Unicode, bytes is not, and re.ASCII is the switch¶
Verified output of unicode_classes_py.py — regenerated by tools/run_examples.py, never hand-typed.
str, default (Unicode):
\w \d \s
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE yes no no
U+0301 COMBINING ACUTE ACCENT no no no
U+0660 ARABIC-INDIC DIGIT ZERO yes yes no
U+FF11 FULLWIDTH DIGIT ONE yes yes no
U+00A0 NO-BREAK SPACE no no yes
U+3000 IDEOGRAPHIC SPACE no no yes
str, re.ASCII:
\w \d \s
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE no no no
U+0301 COMBINING ACUTE ACCENT no no no
U+0660 ARABIC-INDIC DIGIT ZERO no no no
U+FF11 FULLWIDTH DIGIT ONE no no no
U+00A0 NO-BREAK SPACE no no no
U+3000 IDEOGRAPHIC SPACE no no no
bytes have no Unicode mode at all:
re.fullmatch(rb'\w', b'\xc3') -> no (one byte of the two that encode U+00E9)
re.compile(rb'\w', re.UNICODE) -> ValueError
\b\w+\b over 'na' U+00EF 've reader' -- how many words?
str, default -> 2 na\u00EFve reader
str, re.ASCII -> 3 na ve reader
bytes -> 3 na ve reader
\d and the number that follows it (subject is U+0660 U+0661 U+0662):
re.fullmatch(r'\d+') -> yes
re.fullmatch(r'(?a)\d+') -> no
int() -> 12
float() -> 12.0
json.loads() -> JSONDecodeError
.encode('ascii') -> UnicodeEncodeError
Python is the engine that gets this most nearly right by default, and it is still worth reading carefully.
\w on a str matches U+00E9 and both non-ASCII digits, \d matches U+0660 and U+FF11, \s matches both exotic spaces. re.ASCII (or the inline (?a)) turns all of that off in one move, which is the right thing to reach for when the pattern is meant to validate an identifier rather than a name.
\w does not match U+0301 COMBINING ACUTE ACCENT. Python's \w is "what str.isalnum() says, plus underscore", and a combining mark is not alphanumeric. Perl, Rust and PCRE2 under (*UCP) all say yes to the same character, because their \w is the Unicode Word property, which includes Marks. So "Unicode-aware \w" is not one definition either — caf + e + U+0301 and caf + U+00E9 look identical on screen and are two different answers in Python.
The bytes rows are the important half. A bytes pattern has no Unicode mode at all — re.compile(rb"\w", re.UNICODE) is a ValueError, which is Python being honest: a byte is not a character. So the same source file, the same pattern text, and the only difference is whether the subject was decoded, finds two words or three.
Perl: decoded text or bytes, and the feature that settles it¶
Verified output of unicode_classes_pl.pl — regenerated by tools/run_examples.py, never hand-typed.
decoded text, default (unicode_strings, from use v5.36):
\w \d \s
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE yes no no
U+0301 COMBINING ACUTE ACCENT yes no no
U+0660 ARABIC-INDIC DIGIT ZERO yes yes no
U+FF11 FULLWIDTH DIGIT ONE yes yes no
U+00A0 NO-BREAK SPACE no no yes
U+3000 IDEOGRAPHIC SPACE no no yes
the /a flag restricts the same pattern to ASCII:
U+0041 LATIN CAPITAL LETTER A \w yes \d no
U+00E9 LATIN SMALL LETTER E WITH ACUTE \w no \d no
U+0301 COMBINING ACUTE ACCENT \w no \d no
U+0660 ARABIC-INDIC DIGIT ZERO \w no \d no
U+FF11 FULLWIDTH DIGIT ONE \w no \d no
U+00A0 NO-BREAK SPACE \w no \d no
U+3000 IDEOGRAPHIC SPACE \w no \d no
characters up to U+00FF without the unicode_strings feature (pre-5.12):
U+0041 LATIN CAPITAL LETTER A \w yes \s no
U+00E9 LATIN SMALL LETTER E WITH ACUTE \w no \s no
U+00A0 NO-BREAK SPACE \w no \s no
the same twelve characters of text, decoded and not decoded:
decoded text length 12 \b\w+\b -> 2 na\u00EFve reader
raw UTF-8 length 13 \b\w+\b -> 3 na\xC3 ve reader
a number that \d accepts and arithmetic does not (U+0660 U+0661 U+0662):
=~ /^\d+$/ -> yes
=~ /^\d+$/a -> no
0 + $n -> 0
sprintf %d -> 0
Perl is Unicode-aware by default in the sense that matters, and /a restricts a pattern to ASCII — the same idea as Python's re.ASCII, spelled as a flag.
The two sections below the tables are the ones to read twice.
characters up to U+00FF without the unicode_strings feature is the historical Perl Unicode bug. For code points 128–255 the answer used to depend on how Perl happened to be storing the string internally, which is not something a pattern should ever be able to see. use v5.36 (anything from use v5.12) turns on the unicode_strings feature and settles it; turning the feature back off shows the old answer — U+00E9 stops being a word character. A module that says use strict; use warnings; and nothing else is still living in the old world.
the same twelve characters of text, decoded and not decoded is the trap this page is named for. Read the file, decode it, and the text is twelve characters and \b\w+\b finds two words. Read the same file and forget to decode it, and the text is thirteen characters, \b\w+\b finds three, and the first of them is na\xC3 — because byte 0xC3 on its own is U+00C3 LATIN CAPITAL LETTER A WITH TILDE, which is a word character. Nothing fails. The pattern matched; it matched the wrong thing.
JavaScript: the u flag does not do what its name suggests¶
Verified output of unicode_classes_js.js — regenerated by tools/run_examples.py, never hand-typed.
no flags:
\w \d \s
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE no no no
U+0301 COMBINING ACUTE ACCENT no no no
U+0660 ARABIC-INDIC DIGIT ZERO no no no
U+FF11 FULLWIDTH DIGIT ONE no no no
U+00A0 NO-BREAK SPACE no no yes
U+3000 IDEOGRAPHIC SPACE no no yes
the u flag -- identical, because u is about code points, not classes:
\w \d \s
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE no no no
U+0301 COMBINING ACUTE ACCENT no no no
U+0660 ARABIC-INDIC DIGIT ZERO no no no
U+FF11 FULLWIDTH DIGIT ONE no no no
U+00A0 NO-BREAK SPACE no no yes
U+3000 IDEOGRAPHIC SPACE no no yes
what you have to write instead (u flag required):
U+0041 LATIN CAPITAL LETTER A \p{L} yes \p{Nd} no \p{White_Space} no
U+00E9 LATIN SMALL LETTER E WITH ACUTE \p{L} yes \p{Nd} no \p{White_Space} no
U+0301 COMBINING ACUTE ACCENT \p{L} no \p{Nd} no \p{White_Space} no
U+0660 ARABIC-INDIC DIGIT ZERO \p{L} no \p{Nd} yes \p{White_Space} no
U+FF11 FULLWIDTH DIGIT ONE \p{L} no \p{Nd} yes \p{White_Space} no
U+00A0 NO-BREAK SPACE \p{L} no \p{Nd} no \p{White_Space} yes
U+3000 IDEOGRAPHIC SPACE \p{L} no \p{Nd} no \p{White_Space} yes
\b\w+\b over 'na' U+00EF 've reader' -- how many words?
\b\w+\b -> 3 na ve reader
\b\w+\b with u -> 3 na ve reader
[\p{L}\p{N}_]+ -> 2 na\u00EFve reader
\b[\p{L}\p{N}_]+\b -> 2 na\u00EFve reader
\d and the number that follows it (subject is U+0660 U+0661 U+0662):
/^\d+$/ -> no
/^\d+$/u -> no
/^\p{Nd}+$/u -> yes
Number() -> NaN
parseInt() -> NaN
\w is [A-Za-z0-9_] in JavaScript. With the u flag it is still [A-Za-z0-9_]. There is no switch, and the two tables above are printed side by side precisely because they are identical — the u flag is about how the pattern is parsed (code points rather than UTF-16 code units, \p{...} becoming available), not about what the shorthand classes mean.
\d is [0-9], always. That makes JavaScript the safest of these engines for validating a number and the most annoying for matching a name.
And then \s is Unicode-aware, with no flag at all: U+00A0 and U+3000 both match. So within one engine, \s follows Unicode and \w and \d do not. If you have an internal model of "this engine is Unicode-aware" or "this engine is ASCII", JavaScript does not fit it.
The workaround is the last two rows of the word count: [\p{L}\p{N}_] with the u flag, which is ES2018 and therefore safe in any runtime you are likely to meet. Wrapping it in \b still finds two words here — JavaScript's \b is ASCII, but every boundary this subject needs happens to fall next to an ASCII letter or a space. Put a non-ASCII letter at the start or end of a word and that stops being true, which is the argument for (^|[^\p{L}\p{N}_]) over \b when the class is Unicode and the boundary is not.
Java: ASCII until you say (?U)¶
Verified output of unicode_classes_java.java — regenerated by tools/run_examples.py, never hand-typed.
default -- \w, \d and \s are ASCII:
\w \d \s
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE no no no
U+0301 COMBINING ACUTE ACCENT no no no
U+0660 ARABIC-INDIC DIGIT ZERO no no no
U+FF11 FULLWIDTH DIGIT ONE no no no
U+00A0 NO-BREAK SPACE no no no
U+3000 IDEOGRAPHIC SPACE no no no
(?U), which is Pattern.UNICODE_CHARACTER_CLASS:
(?U)\w (?U)\d (?U)\s
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE yes no no
U+0301 COMBINING ACUTE ACCENT yes no no
U+0660 ARABIC-INDIC DIGIT ZERO yes yes no
U+FF11 FULLWIDTH DIGIT ONE yes yes no
U+00A0 NO-BREAK SPACE no no yes
U+3000 IDEOGRAPHIC SPACE no no yes
the \p{...} properties, which never needed (?U):
\p{L} \p{Nd} \p{IsWhite_Space}
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE yes no no
U+0301 COMBINING ACUTE ACCENT no no no
U+0660 ARABIC-INDIC DIGIT ZERO no yes no
U+FF11 FULLWIDTH DIGIT ONE no yes no
U+00A0 NO-BREAK SPACE no no yes
U+3000 IDEOGRAPHIC SPACE no no yes
\b\w+\b over 'na' U+00EF 've reader' -- how many words?
\b\w+\b -> 3 na ve reader
(?U)\b\w+\b -> 2 na\u00EFve reader
\b[\p{L}\p{N}_]+\b -> 2 na\u00EFve reader
\d and the number that follows it (subject is U+0660 U+0661 U+0662):
\d+ -> no
(?U)\d+ -> yes
\p{Nd}+ -> yes
Integer.parseInt() -> 12
Double.parseDouble() -> throws NumberFormatException
java.util.regex is ASCII by default and has one switch that moves everything at once: Pattern.UNICODE_CHARACTER_CLASS, spelled inline as (?U). Turn it on and \w, \d, \s and \b all become Unicode-aware together, and the word count goes from three to two. Turn it off and you have [a-zA-Z_0-9].
The \p{...} table is the alternative: \p{L} and \p{Nd} are Unicode whatever the flags say, so a pattern that spells its classes out does not depend on whoever compiled it remembering the flag.
Then the last two lines, which are the sharpest thing on this page:
The same string of three ARABIC-INDIC DIGITs. Integer.parseInt reads it as 12, because Character.digit is Unicode-aware. Double.parseDouble refuses it. Same JDK, same string, two answers — so "did it pass the regex" tells you nothing about whether the next line of code will work.
Ruby: \w and \b do not agree with each other¶
Verified output of unicode_classes_rb.rb — regenerated by tools/run_examples.py, never hand-typed.
default -- \w, \d and \s are ASCII:
\w \d \s
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE no no no
U+0301 COMBINING ACUTE ACCENT no no no
U+0660 ARABIC-INDIC DIGIT ZERO no no no
U+FF11 FULLWIDTH DIGIT ONE no no no
U+00A0 NO-BREAK SPACE no no no
U+3000 IDEOGRAPHIC SPACE no no no
the POSIX bracket classes are not:
[[:word:]] [[:digit:]] [[:space:]]
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE yes no no
U+0301 COMBINING ACUTE ACCENT yes no no
U+0660 ARABIC-INDIC DIGIT ZERO yes yes no
U+FF11 FULLWIDTH DIGIT ONE yes yes no
U+00A0 NO-BREAK SPACE no no yes
U+3000 IDEOGRAPHIC SPACE no no yes
and (?u) switches the shorthands over:
(?u)\w (?u)\d (?u)\s
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE yes no no
U+0301 COMBINING ACUTE ACCENT yes no no
U+0660 ARABIC-INDIC DIGIT ZERO yes yes no
U+FF11 FULLWIDTH DIGIT ONE yes yes no
U+00A0 NO-BREAK SPACE no no yes
U+3000 IDEOGRAPHIC SPACE no no yes
\b\w+\b over 'na' U+00EF 've reader' -- how many words?
\b\w+\b -> 1 reader
\w+ (no \b) -> 3 na ve reader
(?u)\b\w+\b -> 2 na\u00EFve reader
\b[[:word:]]+\b -> 2 na\u00EFve reader
is U+00EF a word character to \b, when \w says it is not?
'na' U+00EF 've' =~ /na\b/ -> no
'na' U+00EF 've' =~ /\bve/ -> no
\d and the number that follows it (subject is U+0660 U+0661 U+0662):
/\A\d+\z/ -> no
/\A[[:digit:]]+\z/ -> yes
/\A(?u)\d+\z/ -> yes
String#to_i -> 0
Integer() -> raises ArgumentError
Ruby's \w, \d and \s are ASCII-only, like Java's. [[:word:]], [[:digit:]] and [[:space:]] — the POSIX bracket classes, which read like the older and narrower spelling — are Unicode-aware. (?u) switches the shorthands over.
Now read the word count:
Adding \b to a pattern removed two of its three matches. That is not a typo. Ruby's \b is Unicode-aware even though Ruby's \w is not: U+00EF counts as a word character when deciding where a boundary is, and does not count as one when \w is asked to match it. So \b\w+\b demands a boundary that \w can never reach, and the whole first word disappears. The two lines below the count prove the mechanism directly — neither na\b nor \bve matches.
Of everything measured on this page this is the one most likely to be mistaken for a bug in your own code, because the pattern looks more careful than the one that works.
Go: ASCII, with no switch and no apology¶
Verified output of unicode_classes_go.go — regenerated by tools/run_examples.py, never hand-typed.
the shorthands, and there is no flag that changes them:
\w \d \s
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE no no no
U+0301 COMBINING ACUTE ACCENT no no no
U+0660 ARABIC-INDIC DIGIT ZERO no no no
U+FF11 FULLWIDTH DIGIT ONE no no no
U+00A0 NO-BREAK SPACE no no no
U+3000 IDEOGRAPHIC SPACE no no no
the Unicode categories, which is all RE2 offers:
\pL \p{Nd} \p{Zs}
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE yes no no
U+0301 COMBINING ACUTE ACCENT no no no
U+0660 ARABIC-INDIC DIGIT ZERO no yes no
U+FF11 FULLWIDTH DIGIT ONE no yes no
U+00A0 NO-BREAK SPACE no no yes
U+3000 IDEOGRAPHIC SPACE no no yes
RE2 has no binary Unicode properties -- ERR means "failed to compile":
\p{White_Space} ERR
\p{Alphabetic} ERR
\p{Word} ERR
\b\w+\b over 'na' U+00EF 've reader' -- how many words?
\b\w+\b -> 3 na ve reader
\b[\pL\pN_]+\b -> 2 na\u00EFve reader
\d and the number that follows it (subject is U+0660 U+0661 U+0662):
\d+ -> no
\p{Nd}+ -> yes
strconv.Atoi -> error
RE2 gives \w, \d, \s and \b the ASCII definitions and offers no flag to change them. What it offers instead is the Unicode categories: \pL, \p{Nd}, \p{Zs}. That is a deliberate, consistent position, and \b[\pL\pN_]+\b gets the right answer.
The middle block is the limit worth knowing before you need it: RE2 has the Unicode categories and scripts, and not the binary properties. \p{White_Space}, \p{Alphabetic} and \p{Word} all fail to compile. "Any Unicode whitespace" in Go means \p{Zs} plus the ASCII controls plus U+0085 and U+2028 and U+2029, written out. There is no shorthand for it.
Rust's regex: Unicode by default, (?-u) to leave¶
Verified output of unicode_classes_rs_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
regex crate, default -- Unicode:
\w \d \s
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE yes no no
U+0301 COMBINING ACUTE ACCENT yes no no
U+0660 ARABIC-INDIC DIGIT ZERO yes yes no
U+FF11 FULLWIDTH DIGIT ONE yes yes no
U+00A0 NO-BREAK SPACE no no yes
U+3000 IDEOGRAPHIC SPACE no no yes
the same pattern inside (?-u:...):
(?-u:\w) (?-u:\d) (?-u:\s)
U+0041 LATIN CAPITAL LETTER A yes no no
U+00E9 LATIN SMALL LETTER E WITH ACUTE no no no
U+0301 COMBINING ACUTE ACCENT no no no
U+0660 ARABIC-INDIC DIGIT ZERO no no no
U+FF11 FULLWIDTH DIGIT ONE no no no
U+00A0 NO-BREAK SPACE no no no
U+3000 IDEOGRAPHIC SPACE no no no
\b\w+\b over 'na' U+00EF 've reader' -- how many words?
\b\w+\b -> 2 na\u00EFve reader
(?-u:\b\w+\b) -> 3 na ve reader
\b\w+\b -> 2 na\u00EFve reader (fancy-regex)
\d and the number that follows it (subject is U+0660 U+0661 U+0662):
\d+ -> yes
(?-u:\d+) -> no
parse::<u32>() -> invalid digit found in string
what the crate refuses, and why it is not arbitrary:
(?-u:\w) compiles
(?-u:\W) refused -- it could match a byte that is not UTF-8
(?-u:.) refused -- it could match a byte that is not UTF-8
The regex crate is the other end of the scale: \w, \d, \s and \b are Unicode by default, \w is the full Word property (so U+0301 matches, as in Perl), and (?-u:...) restricts a span of the pattern to ASCII. fancy-regex is built on the same character classes and gives the same answer.
Two details the run turns up:
\d+ matches the ARABIC-INDIC digits and "...".parse::<u32>() then answers invalid digit found in string. Rust's integer parsing is ASCII-only. This is the cleanest statement of the whole problem: the regex said it was a number and the number parser said it was not.
(?-u:\w) compiles, (?-u:\W) and (?-u:.) do not. The refusal is not arbitrary — a Regex matches &str, which is guaranteed UTF-8, and a non-Unicode \W or . could match a single byte in the middle of a multi-byte character and hand you back something that is not valid UTF-8. The crate would rather not compile the pattern.
PCRE2, and therefore ABAP: (*UCP)¶
PCRE2's \w is ASCII unless the PCRE2_UCP option is set, and the portable way to set it from inside a pattern is (*UCP) at the very start. Since ABAP's PCRE addition (7.55 and later) is the PCRE2 library in the kernel, this is ABAP's story too.
Real run — PCRE2 10.48 (pcre2grep), macOS 26.6.2 x86-64, 2026-09-22. w.txt holds na + U+00EF + ve as UTF-8; d.txt holds U+0660 U+0661 U+0662.
$ printf 'na\xc3\xafve\n' > w.txt
$ printf '\xd9\xa0\xd9\xa1\xd9\xa2\n' > d.txt
$ pcre2grep -c '^\w+$' w.txt # no UTF mode at all
0
$ pcre2grep --utf --no-ucp -c '^\w+$' w.txt # UTF on, UCP off: the library's default
0
$ pcre2grep --utf --no-ucp -c '(*UCP)^\w+$' w.txt # the switch, written in the pattern
1
$ pcre2grep --utf -c '^\w+$' w.txt # pcre2grep 10.48 turns UCP on for you
1
$ pcre2grep --utf --no-ucp -c '^\d+$' d.txt
0
$ pcre2grep --utf --no-ucp -c '(*UCP)^\d+$' d.txt
1
$ pcre2grep --utf --posix-digit -c '^\d+$' d.txt # UCP on, but \d kept to [0-9]
0
Three things in that transcript are worth separating.
UTF mode and UCP are different options. Line 2 has the subject correctly decoded as UTF-8 and still says no: knowing that those two bytes are one character does not make that character a word character. That is the library default, and it is what a pattern inherits unless something sets PCRE2_UCP.
pcre2grep 10.48 sets UCP for you when you pass --utf. Line 4 is the tool being helpful, not the library changing its mind — which is why --no-ucp exists, and why a result from a command line is not evidence about what an embedded PCRE2 will do. When testing a pattern destined for ABAP, put (*UCP) in the pattern rather than relying on a flag the host may not pass.
--posix-digit keeps \d at [0-9] with UCP otherwise on, which is exactly the combination most validation patterns want: Unicode letters in the name field, ASCII digits in the amount field. Inside a pattern there is no such switch, so spell the digits out as [0-9] and leave (*UCP) to the letters.
The same measurements, run over the na U+00EF ve reader subject: \b\w+\b finds 3 words without UCP and 2 with (*UCP). \s does not match U+00A0 without UCP and does with. \w matches U+0301 COMBINING ACUTE ACCENT under UCP, like Perl and Rust and unlike Python.
z_unicode_classes.prog.abap in full — pasted here by tools/run_examples.py from the file CI runs.
REPORT z_unicode_classes.
" ABAP's PCRE addition (7.55 and later) is the PCRE2 library, and PCRE2's \w is
" ASCII-only until the UCP option is switched on. (*UCP) at the very start of a
" pattern is the portable way to ask for it, and it is the same switch this
" lesson measures with pcre2grep.
"
" Nothing in this repository can run ABAP, so this file is checked by abaplint
" and printed as source only -- there is no recorded output beside it.
"
" The source has to stay 7-bit ASCII (abaplint enforces that), so the one
" non-ASCII character in the subject is built from its code point rather than
" typed: on a Unicode ABAP system a character field holds UTF-16 code units, so
" casting a two-byte hex field to a character field yields U+00EF directly.
DATA hex TYPE x LENGTH 2 VALUE '00EF'. " U+00EF LATIN SMALL LETTER I WITH DIAERESIS
FIELD-SYMBOLS <char> TYPE c.
ASSIGN hex TO <char> CASTING TYPE c.
" "naive reader", with U+00EF in the middle of the first word.
DATA(text) = |na{ <char> }ve reader|.
" \w without UCP is [A-Za-z0-9_], so \b\w+\b finds three "words" here --
" na, ve and reader -- and the first name in your customer table is silently
" cut in half.
FIND ALL OCCURRENCES OF PCRE `\b\w+\b`
IN text RESULTS DATA(ascii_words).
" With UCP, \w is the Unicode word property, so the same pattern finds two.
FIND ALL OCCURRENCES OF PCRE `(*UCP)\b\w+\b`
IN text RESULTS DATA(unicode_words).
DATA(ascii_count) = lines( ascii_words ).
DATA(unicode_count) = lines( unicode_words ).
WRITE / |words found without UCP: { ascii_count }|.
WRITE / |words found with (*UCP): { unicode_count }|.
" \d is the same question with sharper consequences: (*UCP) applies to \d as
" well, so (*UCP)\d+ also accepts U+0660..U+0669 ARABIC-INDIC DIGIT ZERO..NINE
" -- measured here with pcre2grep, not on an ABAP system. A pattern that wants
" Unicode letters and ASCII digits therefore has to spell the digits out as
" [0-9]; if the pattern is a validation, that is the safe default.
The ABAP file above is source only
Nothing in this repository can run ABAP, so there is no recorded output beside it — only what abaplint will accept. The counts it would print are the PCRE2 counts measured above: 3 without (*UCP), 2 with. If you run it on a real system, record the transcript with the release, the way the ABAP learning library ↗ does.
\d is the same question with a bill attached¶
\w getting a name wrong is a bad afternoon. \d getting a number wrong reaches the database. Here is what every engine above answered for the string U+0660 U+0661 U+0662 — three ARABIC-INDIC DIGITs that a human reads as 012 — and what the host language's own number parser then did with it:
| Engine | \d+ matches? |
the language's parser |
|---|---|---|
Python str |
yes | int() → 12, float() → 12.0, json.loads() → JSONDecodeError |
Python bytes, or (?a) |
no | — |
| Perl, decoded | yes | 0 + $n → 0 |
Perl, /a |
no | — |
JavaScript, with or without u |
no | Number() → NaN |
| Java, default | no | — |
Java, (?U) |
yes | Integer.parseInt() → 12, Double.parseDouble() → throws |
| Ruby, default | no | — |
Ruby, [[:digit:]] or (?u) |
yes | to_i → 0, Integer() → raises |
Go regexp |
no | — |
Go, \p{Nd} |
yes | strconv.Atoi → error |
Rust regex |
yes | parse::<u32>() → invalid digit found in string |
Rust, (?-u:\d) |
no | — |
PCRE2 / ABAP with (*UCP) |
yes | not measured — nothing here can run ABAP |
Read the yes rows: in Perl the value silently becomes 0; in Ruby, Go and Rust the parse fails after validation passed; in Java one parser succeeds and another throws. Python is the only row where the regex and the primary parser genuinely agree — and even there the value cannot be serialised as JSON or encoded as ASCII.
The rule that falls out of the table: if a \d is going to become a number, keep it ASCII. Write [0-9], not \d, and the question never arises in any engine. Save the Unicode-aware classes for text that stays text.
The measured summary¶
| Engine (version) | \w by default |
how to switch | \d matches U+0660? |
\b\w+\b finds |
|---|---|---|---|---|
Python 3 re, str |
Unicode | re.ASCII / (?a) → ASCII |
yes | 2 |
Python 3 re, bytes |
ASCII | no switch exists | n/a — no such character in a byte string | 3 |
| Perl 5.36+, decoded | Unicode | /a → ASCII |
yes | 2 |
| Perl 5.36+, undecoded bytes | each byte as Latin-1 | decode the string | n/a — no such character in a byte string | 3 |
| JavaScript (ES2018+) | ASCII | none — [\p{L}\p{N}_] with u |
no | 3 |
| Java 25 | ASCII | (?U) → Unicode |
no ((?U): yes) |
3 ((?U): 2) |
| Ruby 3.2+ | ASCII | (?u) or [[:word:]] |
no ((?u): yes) |
1 |
Go 1.25 regexp |
ASCII | none — \pL, \p{Nd} |
no | 3 |
Rust regex 1.13.1 |
Unicode | (?-u:…) → ASCII |
yes | 2 |
PCRE2 10.48 / ABAP PCRE |
ASCII | (*UCP) → Unicode |
no ((*UCP): yes) |
3 ((*UCP): 2) |
Every cell was produced by running the pattern; none of it is transcribed from documentation. The Ruby 1 is the row that surprised this page's author, and it survived three re-runs.
For what compiles where rather than what matches what, see who supports what, which is built the same way.
What to do about it¶
- Decide which one you mean, then say it. "A word character" is not a specification.
[A-Za-z0-9_]and[\p{L}\p{N}_]are, and both are shorter to reason about than remembering ten defaults. - Never let
\dreach a number parser.[0-9]costs three more characters and removes an entire class of production incident. - Decode at the edge. Every bytes-versus-text answer on this page is the same bug in a different language: the subject arrived as bytes and was matched as if it were text. Decode when you read, encode when you write, and match in the middle.
- Test with a subject that is not ASCII. One accented letter and one Arabic-Indic digit in a fixture catches every difference on this page. All of them are silent without it.
- In ABAP, put
(*UCP)in the pattern, not in a flag. It travels with the pattern into whatever calls it, and it is the only form a code reviewer can see.
See also¶
- Naming a Unicode property:
\p{...}— the workaround this page reaches for in six languages, done properly: what\p{L},\p{Nd}and\p{Alphabetic}mean and which engines accept which spelling. - Who supports what — the same comparison for constructs rather than character classes, and the engine families that predict these answers.
- Forward references — the other page in this library about one pattern meaning different things in two engines.
- Resources — where each engine documents its own character classes, which is the only place these defaults are written down.
- Encodings learning library ↗ — what a "character" actually is, which is the question underneath this whole page: code point, code unit, grapheme, and why U+00E9 and
e+ U+0301 are not the same string. - Python learning library ↗ —
strversusbytesin the rest of Python's text handling, not just inre. - Java text learning library ↗ —
UNICODE_CHARACTER_CLASSalongside Java's other Unicode defaults. - Ruby text learning library ↗ — Ruby's per-string encodings, which is the layer under
[[:word:]]. - Perl learning library ↗ —
use utf8,binmode, and where a Perl one-liner decides a string is text. - Rust learning library ↗ — why
&stris guaranteed UTF-8, which is why(?-u:.)will not compile.