Skip to content

The Unicode bug

Level: 201 · your script has no use v5.12 or later — or you write one-liners

One line: Without the unicode_strings feature, é =~ /\w/ and uc 'é' give different answers for the same string depending on whether perl stores it one byte per character or as UTF-8; use v5.12 or later — or -E in a one-liner — fixes that, and in exchange \w, \d, \s and /i mean all of Unicode, including an Arabic-Indic digit that + 0 reads as 0, until /a takes them back.

perlunicode names it itself: "The Unicode Bug". Perl stores a string one of two ways — one byte per character when every character is below U+0100, or UTF-8 — and normally the difference is invisible: eq says the two are the same string, and they are. But before 5.12, perl used the storage to choose the rules: the one-byte form got ASCII rules, the UTF-8 form got Unicode rules. That is still what happens in any code that has not asked otherwise.

Measured

unicode_bug_pl.pl builds é both ways — utf8::downgrade and utf8::upgrade change only the storage — then asks the same questions with the feature off and on:

Verified output of unicode_bug_pl.pl — regenerated by tools/run_examples.py, never hand-typed.

--- without the unicode_strings feature ---
  the same string?          yes
  /\w/ on the one-byte form  no
  /\w/ on the UTF-8 form     yes
  uc of the one-byte form    é
  uc of the UTF-8 form       É

--- with use v5.36, which turns it on ---
  /\w/ on the one-byte form  yes
  /\w/ on the UTF-8 form     yes
  uc of the one-byte form    É
  uc of the UTF-8 form       É

--- so \w, \d, \s and /i mean Unicode, and /a takes it back ---
  'Kraków' =~ /^\w+$/        yes   with /a: no
  '٣' =~ /\d/                yes   with /a: no
  '٣' + 0                    0    warned: yes
  NBSP =~ /\s/               yes   with /a: no
  KELVIN SIGN =~ /k/i        yes   with /aa: no

--- a one-liner has no use v5.36: -e has the bug, -E does not ---
  perl -e 'print "\x{E9}" =~ /\w/ ? "yes\n" : "no\n"'   no
  perl -E 'print "\x{E9}" =~ /\w/ ? "yes\n" : "no\n"'   yes

Where each form comes from

A string is in the one-byte form when nothing has upgraded it: bytes read with no layer, chr(233), "\x{E9}". It is in the UTF-8 form after a decode, as a use utf8 literal with a non-ASCII character in it, or after being joined with any character above U+00FF. So without the feature, the same $name =~ /^\w+$/ accepts José from one source and rejects it from another, and no printout of $name shows why.

The fix, and what it turns on

use v5.12 or any later version turns on unicode_strings for the rest of the file, and use v5.36 does it along with everything else this library uses. The last section of the output is the catch: a one-liner has no use line, so perl -e still has the bug. perl -E enables the current version's features, and gets the answer right.

With the feature on, the classes mean Unicode — and that has consequences of its own:

  • \w matches ó, so 'Kraków' =~ /^\w+$/ is true. Right for names.
  • \d matches ٣, ARABIC-INDIC DIGIT THREE — and '٣' + 0 is 0, with a warning. A "digits only" check written with \d lets through a string perl cannot turn into the number it looks like.
  • \s matches NO-BREAK SPACE, which a web form or a PDF copy puts into data.
  • /i lets k match KELVIN SIGN, U+212A, because it folds to k.

/a restricts \d, \s and \w to ASCII; /aa also stops /i from matching an ASCII letter against a non-ASCII one.

What to do instead

  • Start every file with use v5.36 (at the least use v5.12), and write one-liners with -E when they match text.
  • Validate formats — IDs, numbers, protocol fields — with /a or explicit [0-9]; see Numbers from text.
  • Match words people wrote with the default Unicode classes, on text you decoded at the edge.

If you are coming from another language

See also