Skip to content

Unicode text

One line: Perl does everything Unicode asks — but only for strings it has been told are text: a literal is bytes until use utf8, input is bytes until a layer or decode, output depends on the string until a layer is on the handle — and every one of those defaults is silent until the text stops being ASCII.

A Perl string is a sequence of numbers. It does not record whether those numbers are bytes read from a file or characters decoded from them — only how perl happens to be storing them. So the program has to know, and the only way it knows is by where it decoded. Decode too late and length counts bytes; decode twice and ó becomes ó; decode never and everything looks right until the first accented name.

The lessons follow a string through a program: where it comes from (the source file, the input), where it goes (the output), and what can be asked of it once it is text.

Lesson Level What it settles
use utf8 is about the source file 101 Why a literal "zażółć" is ten bytes, what lc and reverse do to them, and why printing them raw hides the bug
"Wide character in print" 101 Why the same é is written as one byte or two, and why the warning is the good case
Decode at the edges 201 The layer, decode, FB_CROAK and :utf8 on one bad byte — five different results — and what decoding twice does
length counts code points 201 Bytes, code points and graphemes; the two spellings of café; and what substr and reverse cut
fc is how to compare without case 201 lc against fc on Straße and ΣΑΣ, letters that change length, title case, and the language rules Perl does not have

Where this connects