Skip to content

String.new is binary; "" is not

Level: 201 · anyone building a string out of bytes

One line: "", +"" and String.new("") are UTF-8, but String.new with no argument is BINARY, and so is what read(n), File.binread, pack("C") and unpack1("m") return; a BINARY buffer that already holds a byte above 127 then refuses UTF-8 text.

Most strings get their encoding from where they came from: a literal from its source file, a line of a file from the default external encoding. A handful of constructors and readers produce bytes instead, and label them BINARY — which is the right answer for bytes, and a surprise when you meant text.

Verified output of string_new_is_binary_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

Empty strings:
  ""                                 UTF-8
  +""                                UTF-8
  String.new                         ASCII-8BIT
  String.new("")                     UTF-8
  String.new(capacity: 4096)         ASCII-8BIT
  String.new(encoding: "UTF-8")      UTF-8

Strings that come back from byte-level methods:
  [233].pack("C")                    ASCII-8BIT
  [233].pack("U")                    UTF-8
  "aGk=".unpack1("m")                ASCII-8BIT
  File.binread(path)                 ASCII-8BIT
  File.open(path) { |f| f.read }     UTF-8
  File.open(path) { |f| f.read(3) }  ASCII-8BIT

Appending UTF-8 text to a buffer:
  String.new << "café"               UTF-8
  String.new << "\xFF" << "café"     Encoding::CompatibilityError: incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8
  +"".append_as_bytes(0xFF, "café")  UTF-8, bytes ff636166c3a9, valid_encoding? false

Reading the output

  • The constructor with no argument makes a byte buffer. String.new and String.new(capacity:) are BINARY. Give it a string or encoding: and it takes that.
  • Reading with a length reads bytes. f.read with no argument returns text in the default external encoding — UTF-8 here, and where that comes from is its own lesson. f.read(3) returns BINARY, because three bytes need not end on a character boundary. binread is BINARY by name.
  • pack labels by letter. C produces a byte, so BINARY; U produces a code point, so UTF-8. More in pack speaks Perl.
  • An empty BINARY buffer adopts UTF-8. String.new << "café" works, because the buffer held only ASCII (nothing) and the compatibility rule lets the result become UTF-8. Once a single high byte is inside, the next UTF-8 append raises.
  • append_as_bytes, added in Ruby 3.4, appends bytes and integers whatever their encodings and leaves the receiver's label alone — so the result above is UTF-8 and invalid, which is correct for a buffer you will check once it is complete.

What to do

  • Building text: start from +"" or String.new(encoding: "UTF-8").
  • Building bytes — a protocol frame, a binary file: String.new and append_as_bytes, and one force_encoding at the end if the finished thing is text.
  • Reading text in chunks: join the chunks first, then force_encoding and valid_encoding? on the whole — never on a chunk, which may end in the middle of a character.

If you are coming from another language

Python and Rust give the two kinds of buffer different types — bytearray and io.StringIO, Vec<u8> and String — so this mix-up fails the first time the code runs, or does not compile. (Not machine-checked here.) The C library's Text and binary are both bytes ↗ is the same distinction with no type to help at all.