What File.read returns depends on the locale

Level: 201 · anyone whose script works in a terminal and fails under cron, in a container or in CI

One line: File.read labels what it reads with Encoding.default_external, which Ruby takes from the locale — so under LC_ALL=C a UTF-8 file arrives labelled US-ASCII, invalid, and the first regex raises; -E UTF-8 or an encoding: argument fixes reading files, but not ENV and not ruby -e.

A terminal on a Mac or a desktop Linux usually runs with a UTF-8 locale, and there Ruby's default is UTF-8. Cron jobs, containers whose image sets no LANG, and minimal servers often run with the C locale instead, and there the default is US-ASCII. The program below writes one small file and reads it in child processes whose environment it spells out.

Verified output of default_external_follows_the_locale_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

$ LC_ALL=C ruby child.rb
  Encoding.default_external   US-ASCII
  File.read(path).encoding    US-ASCII, valid_encoding? false, length 6
  p the line                  "caf\xC3\xA9"
  line =~ /é/                 ArgumentError: invalid byte sequence in US-ASCII
  line.upcase                 "CAF\xC3\xA9"
  File.read(path, encoding:)  UTF-8, length 5
  ENV["DISH"].encoding        ASCII-8BIT, valid_encoding? true

$ LC_ALL=C ruby -E UTF-8 child.rb
  Encoding.default_external   UTF-8
  File.read(path).encoding    UTF-8, valid_encoding? true, length 5
  p the line                  "café"
  line =~ /é/                 3
  line.upcase                 "CAFÉ"
  File.read(path, encoding:)  UTF-8, length 5
  ENV["DISH"].encoding        ASCII-8BIT, valid_encoding? true

$ LC_ALL=C ruby -e 'puts "café".length'
  stderr: -e: -e:1: syntax error found (SyntaxError)
  [exit status 1]

$ LC_ALL=C ruby literal.rb    # the same line, saved in a file
4

Reading the output

• Under LC_ALL=C, nothing fails at the read. The string is labelled US-ASCII, is invalid — US-ASCII has no bytes above 127 — and has length 6.
• The first regex raises, far from the read that caused it. upcase is worse: it raises nothing, uppercases caf, and leaves the two bytes it cannot read untouched.
• An explicit encoding: always wins. The same process, the same locale, length 5.
• -E UTF-8 changes the default for the whole process, and every read without its own encoding: follows it.
• ENV is not a file. ENV["DISH"] came back labelled ASCII-8BIT in both runs; -E did not change it.
• ruby -e reads its program in the locale's encoding, so a one-liner containing é is a SyntaxError under LC_ALL=C. The same line saved in a file runs: a source file is UTF-8 unless a magic comment says otherwise.

What to do

• Pass encoding: on every read of text you did not create: File.read(path, encoding: "UTF-8"), File.foreach(path, encoding: "BOM|UTF-8"), File.open(path, "r:UTF-8").
• Set a UTF-8 locale — LANG=C.UTF-8 is the usual one — in every container image and service definition that runs Ruby, or RUBYOPT=-EUTF-8 where the locale cannot be changed.
• Treat a value from ENV as bytes of unknown encoding: ENV["X"].dup.force_encoding("UTF-8"), then valid_encoding?.

If you are coming from another language

Perl does not decode a file at all unless the handle has an :encoding layer, so the locale never enters into it — every read below is bytes until the layer is added. The runner runs it under LC_ALL=C:

Verified output of default_external_follows_the_locale_pl.pl — regenerated by tools/run_examples.py, never hand-typed.

  open with '<'                    length 5
  open with '<:encoding(UTF-8)'    length 4

perl -C and the open pragma are how a Perl program asks for layers by default; see perlrun ↗. The Perl library's Decode at the edges ↗ is the pattern to follow instead.

Java removed this dependency in Java 18: the default charset is UTF-8 whatever the locale — the Java text library's The default charset ↗. Python has overridden the C locale for text I/O since 3.7, and the Python library's Opening a file ↗ takes apart the decisions open() makes. (Not machine-checked here.)