Skip to content

Decode at the edges

Level: 201 · you read files, sockets or command output that should be UTF-8

One line: Bytes become text in exactly one place — an :encoding(UTF-8) layer or Encode::decode — and that place also decides what an invalid byte becomes: the layer warns and turns it into the four characters \xE9, decode substitutes U+FFFD without a word, FB_CROAK throws (and empties its argument unless you add LEAVE_SRC), and :utf8 does not check at all.

Measured

decode_at_the_edges_pl.pl writes a two-line file — Kraków in UTF-8, and café from a Latin-1 file, whose é is the single byte e9 — and reads it back every way perl offers. Warnings are printed where they happen:

Verified output of decode_at_the_edges_pl.pl — regenerated by tools/run_examples.py, never hand-typed.

--- 1. no layer: bytes ---
  length 7: 4b 72 61 6b c3 b3 77
  length 4: 63 61 66 e9
--- 2. the :encoding(UTF-8) layer: characters, and a warning ---
    warning: UTF-8 "\xE9" does not map to Unicode
  length 6: Kraków
  length 7: caf\xE9
--- 3. decode(), no check: U+FFFD, and no warning ---
  U+004B U+0072 U+0061 U+006B U+00F3 U+0077
  U+0063 U+0061 U+0066 U+FFFD
--- 4. decode() with FB_CROAK: an exception to catch ---
  decoded: Kraków
  died:    UTF-8 "\xE9" does not map to Unicode
  without LEAVE_SRC, the variable is consumed: decoded 'Kraków', $input is now ''
  and a constant cannot be: Modification of a read-only value attempted
--- 5. the :utf8 layer does not check at all ---
    warning: utf8 "\xE9" does not map to Unicode
  line 2 is valid perl text: no
--- 6. twice is wrong in both directions ---
  decoded once:  Kraków   U+004B U+0072 U+0061 U+006B U+00F3 U+0077
  decoded twice: Krak�w   U+004B U+0072 U+0061 U+006B U+FFFD U+0077
  encoded twice: 4b 72 61 6b c3 83 c2 b3 77   read back: Kraków
  decode() on text above U+00FF: died: Wide character

One bad byte, five results

  1. No layer. Both lines are bytes: Kraków is 7, café is 4. Nothing is wrong yet, and nothing is text yet.
  2. <:encoding(UTF-8). Kraków is 6 characters. The bad byte became the four characters \, x, E, 9 — so café is now caf\xE9, 7 characters of text that will be written back out as exactly that. The warning came before line 1 was printed, because the layer decodes a buffer at a time, not a line.
  3. decode('UTF-8', $bytes). The bad byte became U+FFFD, the replacement character, and nothing was printed at all. The data is changed and the log is clean.
  4. decode('UTF-8', $bytes, FB_CROAK | LEAVE_SRC). The good line decodes; the bad one throws UTF-8 "\xE9" does not map to Unicode, which eval catches. FB_CROAK has a trap of its own: a decode given any check flag removes the input it consumed from the variable, so without LEAVE_SRC $input is empty afterwards — and given a constant, it dies with Modification of a read-only value attempted.
  5. <:utf8. A warning, and a string that utf8::valid says is not valid perl text. :utf8 marks the bytes as UTF-8 without checking that they are, and the functions that read the string afterwards trust that mark.

Twice is wrong in both directions

  • Decoding text again. ó is U+00F3, which fits in a byte, so decode takes it as the byte f3 — not UTF-8 on its own — and replaces it with U+FFFD. The damage is silent and cannot be undone.
  • Encoding bytes again. c3 b3 encoded a second time is c3 83 c2 b3, which reads back once as Kraków. That one is repairable: the encodings library's The mojibake round trip ↗ shows how.
  • Decoding a string with a character above U+00FF dies with Wide character — perl's way of saying this was already text.

What to do instead

Decode once, where the bytes arrive, and choose the failure you want:

use v5.36;
use utf8;
use open qw(:std :encoding(UTF-8));
use Encode qw(decode FB_CROAK LEAVE_SRC);

# Bytes you do not trust: read them raw, and fail loudly on the first bad one.
open my $in, '<:raw', $path or die "$path: $!";
my $bytes = do { local $/; <$in> };
my $text  = eval { decode('UTF-8', $bytes, FB_CROAK | LEAVE_SRC) }
    // die "$path is not UTF-8: $@";

For input you do trust, <:encoding(UTF-8) is fine — but treat its warning as an error, because the data behind it has already changed. Never use :utf8 on input.

If you are coming from another language

See also