Ruby Text — Learning Library

03 — Unicode

Ruby Text — Learning Library

Ruby text — a learning library
Start Here
Start Here
Strings Carry an Encoding
Strings Carry an Encoding
- A string is bytes plus a label
  A string is bytes plus a label
- Mixing encodings raises — unless it is all ASCII
  Mixing encodings raises — unless it is all ASCII
- Invalid bytes get in, and fail later
  Invalid bytes get in, and fail later
- String.new is binary; "" is not
  String.new is binary; "" is not
Reading and Writing
Reading and Writing
- What File.read returns depends on the locale
  What File.read returns depends on the locale
- The BOM is kept unless you ask
  The BOM is kept unless you ask
- chomp knows about CRLF
  chomp knows about CRLF
Unicode
Unicode
- length counts code points
  length counts code points
- Normalization is built in
  Normalization is built in
- Case mapping is full Unicode, with options instead of a locale
  Case mapping is full Unicode, with options instead of a locale
Regex
Regex
- ^ and $ match at every line
  ^ and $ match at every line
- \w is ASCII, [[:alpha:]] is not, and \b sides with Unicode
  \w is ASCII, [[:alpha:]] is not, and \b sides with Unicode
- \h is a hex digit in Ruby and a blank in Perl
  \h is a hex digit in Ruby and a blank in Perl
- Named captures become local variables — sometimes
  Named captures become local variables — sometimes
- Backtracking has a cache, and a timeout
  Backtracking has a cache, and a timeout
Literals
Literals
- String literals are chilled, not frozen
  String literals are chilled, not frozen
- <<~ strips indentation — by a different rule than Perl's
  <<~ strips indentation — by a different rule than Perl's
Perl Heritage
Perl Heritage
- ruby -n -p -a -F -l -i came from Perl
  ruby -n -p -a -F -l -i came from Perl
- split has an awk mode
  split has an awk mode
- succ counts in letters
  succ counts in letters
- pack and unpack speak Perl's template language
  pack and unpack speak Perl's template language
Parsing and Formatting
Parsing and Formatting
- to_i never fails; Integer() does
  to_i never fails; Integer() does
- ljust and %-8s pad by code points, not columns
  ljust and %-8s pad by code points, not columns
Resources
Resources

03 — Unicode¶

Ruby's String has Unicode's hard parts built in, as methods: segmentation into grapheme clusters, the four normalization forms, full case mapping and case folding. Ruby 4.0 carries the Unicode 17.0 tables. The traps are in the defaults beside them: length and reverse work in code points, == compares code points without normalizing, and casecmp is ASCII-only.

Lesson	The one thing
`length` counts code points	and what `reverse` and `[]` do to a flag or an accent
Normalization is built in	two spellings of `café`, and `unicode_normalize`
Case mapping is full Unicode, with options instead of a locale	`ß`, `:turkic`, `:fold`, and the ASCII-only `casecmp`