Mixing encodings raises — unless it is all ASCII¶

Level: 101 · anyone who has met Encoding::CompatibilityError

One line: Joining a UTF-8 string and an ISO-8859-1 string raises once both hold a non-ASCII byte, works while either side is pure ASCII, and comparing them never raises at all — == returns false and a Hash lookup quietly misses.

A language with one internal encoding converts everything into it and never has to choose. Ruby keeps each string in the encoding it came in, so a + b has to pick a label for the result. The rule: when both encodings are ASCII-compatible and one side holds only ASCII, the result takes the other side's encoding. Otherwise Ruby raises rather than guess.

Verified output of mixing_encodings_raises_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

utf8   holds 5 bytes, UTF-8
latin1 holds 4 bytes, ISO-8859-1, the same four characters
plain  is ASCII only: ascii_only? true

Joining:
  utf8 + latin1          Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ISO-8859-1
  "#{utf8} #{latin1}"    Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ISO-8859-1
  plain + latin1         "menu: caf\xE9", ISO-8859-1
  plain + utf8           "menu: café", UTF-8

Asking first:
  Encoding.compatible?(utf8, latin1)   nil
  Encoding.compatible?(plain, latin1)  #<Encoding:ISO-8859-1>

Comparing, which never raises:
  utf8 == latin1                         false
  utf8 == latin1.encode("UTF-8")         true
  "abc" == "abc".encode("ISO-8859-1")    true
  { utf8 => 3 }[latin1]                  nil

Reading the output¶

The error is raised at the join, not where the ISO-8859-1 string was made — often a long way from the actual mistake. Interpolation is a join and raises the same way.
ASCII hides it. plain + latin1 succeeds, and the result is ISO-8859-1 (inspect shows é as \xE9 because the string is not in the output's encoding). So a UTF-8 template joined to names from an ISO-8859-1 file passes every test whose names are ASCII, and raises on the first customer called José.
Encoding.compatible? asks first. It returns nil, or the encoding the joined string would have.
Comparison never raises. utf8 == latin1 is false: the same characters, different bytes. Two ASCII-only strings in ASCII-compatible encodings compare equal. A Hash compares keys the same way, so { utf8 => 3 }[latin1] is nil.

What to do¶

Pick one encoding for the inside of the program — UTF-8 — and convert at the edge, once, with encode: the output shows utf8 == latin1.encode("UTF-8") is true. Reading files with an explicit encoding is chapter 02.

If you are coming from another language¶

Perl has no error to raise. A byte string joined to a character string has each of its bytes read as a Latin-1 character, so the result is valid text that says the wrong thing:

Verified output of mixing_encodings_raises_pl.pl — regenerated by tools/run_examples.py, never hand-typed.

  "$bytes $chars"     cafÃ© café
  length             10
  $bytes eq $chars   false
  $prices{$bytes}    undef

That is the cafÃ© from the previous lesson's mojibake row, produced without a word, and a hash lookup that misses just as Ruby's does. The Perl library's "Wide character in print" ↗ is the same mix-up on the way out.

Python makes the two kinds different types, so the join fails before any wrong text exists:

Verified output of mixing_encodings_raises_py.py — regenerated by tools/run_examples.py, never hand-typed.

  data + text                   TypeError: can't concat str to bytes
  data == text                  False
  data.decode('utf-8') == text  True

Java and Rust strings hold one encoding inside, so two strings are never incompatible; the equivalent mistake happens earlier, when bytes are decoded with the wrong charset — see the Java text library's The default charset ↗. The Encodings library's Mojibake ↗ shows what each wrong guess looks like.