Skip to content

"Wide character in print"

Level: 101 · you have seen the warning — or worse, garbage with no warning

One line: A filehandle with no encoding layer writes a string in perl's internal form, so café goes out as the single byte e9 — not UTF-8, and no warning — unless the same string contains any character above U+00FF, in which case the whole string goes out as UTF-8 with a "Wide character" warning; one layer on the handle makes both right.

Measured

Every command's output is piped through hex, a byte dump written in perl, so the bytes are the evidence and the terminal's rendering is not:

Verified output of wide_character_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ perl -e 'print "caf\x{E9}\n"' | hex
63 61 66 e9 0a

$ perl -e 'print "caf\x{E9} \x{2192} ok\n"' 2>warnings.txt | hex; cat warnings.txt
63 61 66 c3 a9 20 e2 86 92 20 6f 6b 0a
Wide character in print at -e line 1.

$ perl -e 'print "caf\x{E9}\n", "caf\x{E9} \x{2192}\n"' 2>/dev/null | hex
63 61 66 e9 0a 63 61 66 c3 a9 20 e2 86 92 0a

$ perl -CS -e 'print "caf\x{E9} \x{2192} ok\n"' 2>warnings.txt | hex; cat warnings.txt
63 61 66 c3 a9 20 e2 86 92 20 6f 6b 0a

$ perl -e 'binmode STDOUT, ":encoding(UTF-8)"; print "caf\x{E9}\n"' | hex
63 61 66 c3 a9 0a

$ perl -e 'use open qw(:std :encoding(UTF-8)); print "caf\x{E9} \x{2192} ok\n"' 2>warnings.txt | hex; cat warnings.txt
63 61 66 c3 a9 20 e2 86 92 20 6f 6b 0a

Reading the bytes

  1. caf\x{E9} came out as 63 61 66 e9. Perl stores a string whose characters all fit below U+0100 at one byte per character, and with no layer it writes that storage. e9 alone is not valid UTF-8, so a UTF-8 terminal or editor shows a replacement mark — and there was no warning, because perl had no trouble writing it.
  2. Add → (U+2192) and it cannot fit in a byte, so perl writes the string's other internal form, UTF-8: é becomes c3 a9, the arrow e2 86 92, and perl warns Wide character in print.
  3. Both in one print: e9 in the first string, c3 a9 in the second. The same character, as two different byte sequences, in one line of output. A program that prints data this way writes files that are neither Latin-1 nor UTF-8, and which one a line is depends on what else was on it.
  4. -CS, binmode STDOUT, ":encoding(UTF-8)" and use open qw(:std :encoding(UTF-8)) each put an encoding layer on the handle. Every string is encoded as UTF-8 on the way out, é is always c3 a9, and the warnings file stays empty.

The warning is the good case. It appears only when perl had to guess — and its guess, UTF-8, is usually right. The silent e9 is the one that corrupts a file.

What to do instead

In a program, use open qw(:std :encoding(UTF-8)); at the top, and a layer on every handle you open yourself:

open my $out, '>:encoding(UTF-8)', $path or die "$path: $!";

In a one-liner, -CS, or -CSDA to decode input and @ARGV as well. And do not silence the warning: it is telling you that a handle has no layer, which means some other line through the same handle is already wrong.

If you are coming from another language

See also