00 — Start here

Level: 101 · read this first

This library is for someone who writes Ruby and has been surprised by text: an Encoding::CompatibilityError from a line that worked yesterday, a script that is fine in a terminal and broken under cron, a CSV whose row["id"] is nil, a validation regex that let a newline through.

What it assumes

Ruby syntax, and the words byte, code point and UTF-8. The Encodings library ↗ builds those up from nothing if they are new. Everything else a page needs is on that page.

The five to read first

1. A string is bytes plus a label — the model everything else follows from.
2. What File.read returns depends on the locale — why it works on your machine and not in the container.
3. ^ and $ match at every line — the one with a security bug in it.
4. The BOM is kept unless you ask — the one you will hit this month.
5. String literals are chilled, not frozen — what Ruby 3.4 and 4.0 changed about literals, and what they did not.

How to read a page

Each has the same shape: a one-line claim, the idea in prose, an output block from a program in that page's examples/ folder, a reading of that output, and what to do about it. Most end with the same question put to Perl and Python, with their output too, and links to the sibling libraries. The output blocks are generated, never typed — if a page shows a value, Ruby 4.0 printed it.

Ruby's text scorecard

A summary of the whole library, so you know what you are walking into.

Genuinely good

• Normalization, grapheme clusters, full case mapping and case folding are String methods — unicode_normalize, grapheme_clusters, upcase, downcase(:fold) — with no gem. See chapter 03.
• Case mapping has no locale to go wrong: Turkish rules are an option you pass, :turkic. See Case mapping is full Unicode.
• Joining strings in incompatible encodings raises instead of producing mojibake — Perl prints café in the same situation. See Mixing encodings raises.
• unicode_normalize refuses a string that is not in a Unicode encoding rather than guessing. See Normalization is built in.
• chomp removes a Windows \r\n; Perl's does not. See chomp knows about CRLF.
• Since Ruby 3.2 the regex engine caches its backtracking, so the classic ReDoS pattern runs in linear time, and there is a timeout for the patterns the cache cannot help. See Backtracking has a cache.
• pack's letters are Perl's, so a Perl record layout ports letter for letter. See pack speaks Perl.

Genuinely surprising, and sometimes bad

• ^ and $ are line anchors, always. See ^ and $ match at every line.
• \w, \d and \s are ASCII-only on UTF-8 text, while \b is not. See \w is ASCII.
• A byte that is invalid in its string's encoding gets in silently, and only some methods notice. See Invalid bytes get in, and fail later.
• File.read labels text with an encoding taken from the locale — US-ASCII under LC_ALL=C. See the locale lesson.
• A UTF-8 byte order mark is kept as a character. See The BOM is kept.
• String.new and read(n) return BINARY. See String.new is binary.
• "42abc".to_i is 42 and "abc".to_i is 0, with no error. See to_i never fails.
• split(" ") is not a split on a space. See split has an awk mode.
• A string literal is still mutable in Ruby 4.0, with a warning only under -W:deprecated. See String literals are chilled.
• ljust and %-8s pad by code points, not by terminal columns. See Padding counts code points.
• downcase does not apply the Greek final-sigma rule; Python's lower does. See Case mapping is full Unicode.
• \h is a hex digit in Ruby and horizontal whitespace in Perl. See \h is a hex digit.

The pattern: Ruby is right about Unicode content — what a character's case is, how it composes, where a grapheme ends — and Perl-shaped about syntax: anchors, split, \h, to_i. Most of the bugs live in the Perl-shaped half, and in encoding labels picked up from the environment instead of stated in the code.