Skip to content

use utf8 is about the source file

Level: 101 · your script has a letter outside ASCII in a string

One line: Without use utf8, a literal "zażółć" in a UTF-8 source file is ten bytes — length says 10, lc breaks it, reverse scrambles it — and printing it raw looks perfect, which is why the bug survives until someone adds an output layer and sees Kraków.

use utf8 does one thing: it tells perl that the program's source code is UTF-8, so that string literals decode into characters. It does nothing to input, output or filehandles — that is the next lesson and the one after. And it is lexical, which is how this program can hold both versions of the same literal:

use_utf8_pl.pl in full — pasted here by tools/run_examples.py from the file CI runs.

# A string literal is bytes unless `use utf8` says the source file is UTF-8.
# This file IS saved as UTF-8, and the same letters below make two different
# strings depending on whether the pragma is in scope. `use utf8` is lexical,
# so both halves fit in one file.
use v5.36;

my $bytes       = do { no utf8;  "zażółć" };
my $chars       = do { use utf8; "zażółć" };
my $upper_bytes = do { no utf8;  "ZAŻÓŁĆ" };

sub bytes_of ($s)       { join ' ', map { sprintf '%02x', ord } split //, $s }
sub code_points_of ($s) { join ' ', map { sprintf 'U+%04X', ord } split //, $s }

say "--- one literal, two strings ---";
say "no utf8:   length ", length($bytes), "  ", bytes_of($bytes);
say "use utf8:  length ", length($chars), "   ", code_points_of($chars);
say "";

say "--- string functions work on what they were given ---";
say "uc  no utf8:   ", bytes_of(uc $bytes), "   only z and a changed";
say "uc  use utf8:  ", code_points_of(uc $chars);
say "lc  no utf8:   ", bytes_of(lc $upper_bytes), "   c5 c3 c4 lower-cased as Latin-1 letters";
say "rev no utf8:   ", bytes_of(scalar reverse $bytes), "   the bytes of each letter reversed too";
say "rev use utf8:  ", code_points_of(scalar reverse $chars);
say "";

say "--- and printing them ---";
my $city_bytes = do { no utf8;  "Kraków" };
my $city_chars = do { use utf8; "Kraków" };
print "no layer, no utf8:        $city_bytes   (bytes in, the same bytes out)\n";
binmode STDOUT, ':encoding(UTF-8)';
print "UTF-8 layer, use utf8:    $city_chars\n";
print "UTF-8 layer, no utf8:     $city_bytes   (each byte encoded a second time)\n";

Measured

Verified output of use_utf8_pl.pl — regenerated by tools/run_examples.py, never hand-typed.

--- one literal, two strings ---
no utf8:   length 10  7a 61 c5 bc c3 b3 c5 82 c4 87
use utf8:  length 6   U+007A U+0061 U+017C U+00F3 U+0142 U+0107

--- string functions work on what they were given ---
uc  no utf8:   5a 41 c5 bc c3 b3 c5 82 c4 87   only z and a changed
uc  use utf8:  U+005A U+0041 U+017B U+00D3 U+0141 U+0106
lc  no utf8:   7a 61 e5 bb e3 93 e5 81 e4 86   c5 c3 c4 lower-cased as Latin-1 letters
rev no utf8:   87 c4 82 c5 b3 c3 bc c5 61 7a   the bytes of each letter reversed too
rev use utf8:  U+0107 U+0142 U+00F3 U+017C U+0061 U+007A

--- and printing them ---
no layer, no utf8:        Kraków   (bytes in, the same bytes out)
UTF-8 layer, use utf8:    Kraków
UTF-8 layer, no utf8:     Kraków   (each byte encoded a second time)

What happened

  • Ten against six. Without the pragma the literal is the file's own bytes: ż is c5 bc, ó is c3 b3, and each byte is one element of the string. With it, each letter is one code point.
  • uc looked harmless. Only z and a changed. The other eight bytes, read as Latin-1 characters, are Å, ¼, Ã, ³ and friends — letters that are already upper case, or symbols that have none — so uc left them alone.
  • lc was not harmless. The upper-case word's lead bytes c5, c3 and c4 are Å, Ã and Ä as Latin-1 letters, and lc lower-cased them to e5, e3 and e4. The result is no longer UTF-8 for anything.
  • reverse reversed bytes, so each two-byte letter came out with its bytes swapped.
  • Printing is where it hides. With no layer on STDOUT, the ten bytes go out as the same ten bytes, and a UTF-8 terminal shows Kraków — correct, by accident. Add binmode STDOUT, ':encoding(UTF-8)' and each byte is encoded as a character in its own right: c3 becomes Ã, b3 becomes ³. The encodings library explains that pattern in Mojibake ↗.

What to do instead

At the top of every program that has text in it:

use v5.36;
use utf8;                              # this source file is UTF-8
use open qw(:std :encoding(UTF-8));    # so are STDIN, STDOUT, STDERR, and every open()

In a one-liner, -Mutf8 is the pragma and -CSDA does the job of the third line: UTF-8 on the standard handles, on files opened inside the program, and on @ARGV.

If you are coming from another language

  • Python. Python 3 reads source as UTF-8 unless the file says otherwise, so a literal is always text — what use utf8 switches on. See String literals ↗.
  • Rust. Source must be UTF-8, and a &str literal is guaranteed valid UTF-8 by the compiler. See Six kinds of string ↗.
  • C. A C literal is never decoded, so strlen always gives the without-use utf8 answer. See A char is a byte, not a character ↗.

See also