Invalid bytes get in, and fail later¶

Level: 201 · anyone reading text they did not write

One line: Ruby lets a byte that is invalid in the string's encoding into the string without a word; length, include? and sub then work around it while split, =~, upcase and encode raise — so the error surfaces wherever the first character-level method happens to run, unless you check or scrub where the bytes came in.

Nothing checks bytes when a string is made. File.read, a socket, force_encoding, even a literal like "caf\xE9" in a UTF-8 source file will hand over a string whose valid_encoding? is false. What happens next depends on which method touches it first.

Verified output of invalid_bytes_fail_late_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

"price: 12\xFF EUR"  encoding UTF-8, valid_encoding? false

  line.length                                  14
  line.bytesize                                14
  line.include?("EUR")                         true
  line.start_with?("price")                    true
  line.sub("EUR", "USD")                       "price: 12\xFF USD"
  line.split(" ")                              ArgumentError: invalid byte sequence in UTF-8
  line.split(/ /)                              ArgumentError: invalid byte sequence in UTF-8
  line =~ /EUR/                                ArgumentError: invalid byte sequence in UTF-8
  line.upcase                                  ArgumentError: input string invalid
  line.unicode_normalize                       ArgumentError: invalid byte sequence in UTF-8
  line.encode("UTF-16LE")                      Encoding::InvalidByteSequenceError: "\xFF" on UTF-8

Repairs:
  line.scrub                                   "price: 12� EUR"
  line.scrub("?")                              "price: 12? EUR"
  line.scrub { |b| "<#{b.unpack1("H*")}>" }    "price: 12<ff> EUR"
  line.encode("UTF-8", invalid: :replace)      "price: 12� EUR"
  line.encode("UTF-16LE", invalid: :replace)   "price: 12� EUR"

A literal can hold one too: "caf\xE9" is UTF-8, valid_encoding? false

Reading the output¶

Byte-level and substring methods carry on. length counts the bad byte as one character; include?, start_with? and sub with a string argument compare bytes and do not care.
Anything that has to classify characters raises ArgumentError. A regex must decode, and so must split — including split(" "), which looks like a substring but is the whitespace-classifying awk mode. upcase raises too, with a different message: input string invalid.
encode raises its own class, Encoding::InvalidByteSequenceError, and names the byte.
The repairs return a new, valid string. scrub replaces each invalid sequence with U+FFFD, with a string you give it, or with whatever its block returns for the bytes. encode with invalid: :replace does the same while converting — including, as the output shows, "converting" UTF-8 to UTF-8.

What to do¶

Decide the policy where the bytes come in, once: reject (raise unless s.valid_encoding?) or repair (s = s.scrub). Everything past that line can then assume valid text, and no ArgumentError turns up three calls deep inside a regex.

If you are coming from another language¶

Most languages validate at the moment bytes become a string: Python's bytes.decode raises there unless told errors="replace", and Rust's String::from_utf8 returns an error. The Encodings library's Validation is a boundary ↗ runs the same bad bytes through each language it covers and checks the result.

Perl decides what an invalid byte becomes at the one place bytes become text — an :encoding layer or Encode::decode — see the Perl library's Decode at the edges ↗.

Java chooses a policy per API rather than per program — see the Java text library's Malformed input has two policies ↗. Ruby's version of that choice is the one between the method that raises and the method that does not, made by whichever method runs first.