Ruby text — a learning library

Every Ruby string carries its own encoding. Not "Ruby strings are UTF-8" and not "Ruby strings are UTF-16": a Ruby string is a run of bytes plus the name of the encoding to read them with, and two strings in one program can carry different names. One idea per page. Every claim on every page is backed by a program that runs, and whose output is checked against a recorded answer key in CI on Ubuntu and macOS.

That design arrived in Ruby 1.9, in December 2007, so that text could stay in the encoding it arrived in — Shift_JIS, EUC-JP, ISO-8859-1 — instead of being converted to Unicode on the way in. Most of what surprises people about Ruby text follows from it: what force_encoding does and does not do, why joining two strings can raise, why a bad byte gets in and complains later. Most of the rest came from Perl — the one-liner switches, split, pack, the regex syntax — and still behaves like Perl, including in several of the places where Perl surprises people.

Start here

00 — Start here — who this is for, the five pages to read first, and Ruby's text scorecard.

The chapters

	Chapter	What it covers
01	Strings carry an encoding	force_encoding and encode, CompatibilityError, invalid bytes, BINARY
02	Reading and writing	the locale's default encoding, the BOM, chomp and CRLF
03	Unicode	code points and graphemes, normalization, full case mapping
04	Regex	line anchors, ASCII \w, \h, named-capture variables, the backtracking cache
05	Literals	chilled string literals, <<~ heredocs
06	Perl heritage	-n -p -a -F -i, split's awk mode, succ, pack
07	Parsing and formatting	to_i and Integer(), padding by code points
08	Resources	the docs, the NEWS entries, the books, and every lesson's counterparts in the sibling libraries

Running the examples

You need Ruby 4.0 or later. macOS still ships Ruby 2.6 as /usr/bin/ruby; brew install ruby puts a current one in Homebrew's prefix. Every example is a single file that needs nothing beyond what Ruby ships:

ruby 01_Strings_Carry_an_Encoding/bytes_plus_a_label/examples/bytes_plus_a_label_rb.rb

Beside many of the Ruby programs sits a Perl one and a Python one making the same point, so that a comparison on a page is a measurement rather than a memory. They need Perl 5.26 or later and Python 3; macOS and Ubuntu both include them.

To run all of them and check every recorded output:

python3 tools/run_examples.py --check

The tool finds a Ruby 4.0 by itself — $RUBY, then ruby on PATH, then Homebrew's — and refuses anything older.

The one rule

No page hand-types what a program prints. A lesson marks the spot and the runner fills it from a real run, so an example that changes behaviour in a new Ruby breaks the build instead of quietly making a page wrong. See CONTRIBUTING.md.

Sibling libraries

The same house rule, and many of the same questions asked of another language. Resources maps each lesson here to its counterparts there.

• Encodings ↗ — bits, bytes, code points and UTF-8 from the bottom up. Read it first if "code point" is a new word.
• Python ↗ — str and bytes: two types where Ruby has one type and a label.
• Java text ↗ — UTF-16 underneath, locale-sensitive defaults on top.
• Rust ↗ — one encoding, UTF-8, enforced by the type system.
• C ↗ — bytes up to a NUL, and ICU for everything else.
• Perl ↗ — where Ruby's one-liner switches, split, chomp and pack came from, and a different answer to text: one flag per string, and an :encoding layer at the edge.

The Perl comparisons on these pages are runnable .pl programs beside the Ruby ones, and each links the Perl library's page on the same idea where there is one; chapter 06 is where Ruby's debt to Perl is most direct.