A string is bytes plus a label

Level: 101 · anyone who has called force_encoding to make an error go away

One line: A Ruby string is a sequence of bytes and the name of the encoding to read them with; force_encoding changes the name and never the bytes, and encode changes the bytes and keeps the text.

Every String answers encoding, and every method that talks about characters — length, [], reverse, upcase, a regex match — reads the bytes through that encoding. So the same five bytes can be four characters, five characters, or three, depending only on the label.

Verified output of bytes_plus_a_label_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

"café" is labelled UTF-8

force_encoding: same bytes, new label
  UTF-8 (as written)           63 61 66 C3 A9          length 4, valid true
  force_encoding(ISO-8859-1)   63 61 66 C3 A9          length 5, valid true
  force_encoding(BINARY)       63 61 66 C3 A9          length 5, valid true
  force_encoding(UTF-16LE)     63 61 66 C3 A9          length 3, valid false

encode: same text, new bytes
  encode(ISO-8859-1)           63 61 66 E9             length 4, valid true
  encode(UTF-16LE)             63 00 61 00 66 00 E9 00 length 4, valid true

relabel as ISO-8859-1, then encode to UTF-8:  café
word.b is force_encoding(BINARY) on a copy:   ASCII-8BIT
which inspect names:                          #<Encoding:BINARY (ASCII-8BIT)>
and word itself was never touched:            UTF-8

Reading the output

force_encoding moves no bytes. All four rows of the first group hold 63 61 66 C3 A9. Under UTF-8, C3 A9 is one character, é, so the length is 4. Under ISO-8859-1 every byte is a character, so the same bytes are five: c a f à ©. That row is also valid true — every byte means something in ISO-8859-1, so no check will ever flag the mistake. Under UTF-16LE the five bytes make two code units and a stray byte, and valid_encoding? finally says no.

encode moves bytes and keeps the text. encode("ISO-8859-1") writes é as the single byte E9; encode("UTF-16LE") writes every character as two bytes. Both still have length 4.

Mojibake is the two operations in the wrong order. Relabel UTF-8 bytes as ISO-8859-1, convert that "text" to UTF-8, and café becomes café — each step correct for the label it was handed.

BINARY has two names. String#b is force_encoding(Encoding::BINARY) on a copy. Encoding#to_s still prints the old name, ASCII-8BIT; inspect shows both.

force_encoding changes the string it is called on, which is why the example calls dup first. On a frozen string it raises — and string literals are on their way to being frozen, see String literals are chilled, not frozen.

Which one to call

• Bytes arrived without a trustworthy label — a socket read, File.binread, a body whose header said charset=iso-8859-1: force_encoding to the encoding you know they are in, then check valid_encoding?.
• You need different bytes — a legacy system wants ISO-8859-1, an API wants UTF-16: encode.
• An exception went away when you added force_encoding: the bytes are still what they were. Read Invalid bytes get in, and fail later before shipping it.

If you are coming from another language

Perl keeps one bit per string instead of a name: the string holds bytes, or it holds characters. decode and encode are the ways across.

Verified output of bytes_plus_a_label_pl.pl — regenerated by tools/run_examples.py, never hand-typed.

  bytes, as read             63 61 66 C3 A9 length 5, utf8 flag 0
  decode('UTF-8', $bytes)    63 61 66 E9    length 4, utf8 flag 1
  encode('ISO-8859-1', ...)  63 61 66 E9    length 4, utf8 flag 0

The first row is Ruby's BINARY in all but name. The second has no encoding at all — its "bytes" column is really code points, E9 for é — which makes it closer to Python's str than to any Ruby string. The third is bytes again, in ISO-8859-1. The Perl library's Decode at the edges ↗ is the discipline that follows from keeping only a flag.

Python splits the two cases into two types, so there is nothing to relabel: bytes.decode does what force_encoding does plus a validity check, and str.encode does what encode does. See the Python library's Encode and decode ↗.

Java and Rust store one encoding inside every string — UTF-16 and UTF-8 — and convert at the boundary, so the label question comes up once, when bytes become a string: the Java text library's Encodings chapter ↗ and the Rust library's Strings chapter ↗.

The Encodings library's Encode and decode are verbs ↗ and Mojibake ↗ show the same two operations with no programming language in the way.