Text lives in the Ruby text library¶
Level: 101 · a map for anyone who came to this library looking for strings, encodings, regex or literals
One line: a Ruby String is bytes plus an encoding label — one type, where Python has str and bytes — and everything that follows from that is measured in the Ruby text library ↗, lesson by lesson, so this library links there instead of repeating it; the table below puts each of its twenty-three lessons beside the Python library page on the same idea.
The one measured thing on this page is the label. "café" under -E UTF-8 is a String whose encoding is UTF-8, whose length is 4 characters and whose bytesize is 5 bytes. force_encoding changes the label and leaves the bytes; encode changes the bytes to say the same text under another label; .b is a copy labelled BINARY (ASCII-8BIT); an invalid byte is allowed in and only fails when something needs to read it as characters; two Strings with different non-ASCII encodings cannot be concatenated; and a character that US-ASCII cannot write is an Encoding::UndefinedConversionError unless you say what to replace it with. Characters, bytes and code points are three different counts, \w matches ASCII only while \p{L} matches letters, padding counts characters, and slicing a String by index gives characters while byteslice gives bytes. Every one of those rows is a page in the text library, where it is measured properly.
Verified output of text_lives_in_the_text_library_rb.rb — regenerated by tools/run_examples.py, never hand-typed.
1. a String: inspect, class, encoding "café"; String; UTF-8
2. length counts characters; bytesize counts bytes 4; 5
3. bytes; codepoints [99, 97, 102, 195, 169]; [99, 97, 102, 233]
4. valid_encoding?; ascii_only? true; false
5. s.b is the same bytes under a BINARY label: encoding, length, == s ASCII-8BIT; 5; false
6. force_encoding relabels (length); encode converts (bytes) 5; [99, 97, 102, 233]
7. a lone "\xE9": valid_encoding?; relabelled ISO-8859-1, then encoded false; "é"
8. encode("US-ASCII") raises; undef: :replace; fallback: Encoding::UndefinedConversionError: U+00E9 from UTF-8 to US-ASCII; caf?; cafe
9. UTF-8 + ISO-8859-1, non-ASCII on both sides, raises Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ISO-8859-1
10. default_external (-E UTF-8 here); __ENCODING__; String.new; "" UTF-8; UTF-8; ASCII-8BIT; UTF-8
11. e + U+0301: length; grapheme clusters; NFC == "\u{e9}"; upcase 2; 1; true; CAFÉ; STRASSE
12. \w is ASCII: s[/\w+/]; \p{L} is Unicode: match?(/\A\p{L}+\z/) "caf"; true
13. padding counts characters: "%-6s|" % s café |
14. s[3], s[-1] are characters; byteslice(3, 2) bytes; byteslice(3, 1) valid? "é"; "é"; "é"; false
Reading the output¶
Rows 1–4 are the four numbers a String carries: one label (UTF-8), 4 characters, 5 bytes (é is 0xC3 0xA9), 4 code points, valid, and not ASCII-only. Row 5: .b is the same five bytes under the BINARY label, so its length is 5 and it is not == to the original — equality compares bytes and compatible labels. Row 6 is the difference between relabelling and converting: force_encoding("ISO-8859-1") makes the same bytes read as five characters, encode("ISO-8859-1") makes four bytes with é as 233. Row 7: a lone "\xE9" is a String that is not valid UTF-8 — it exists, and valid_encoding? says so — and once relabelled as Latin-1 it converts to a proper é. Row 8: US-ASCII cannot hold é, so encode raises Encoding::UndefinedConversionError with the code point in its message, undef: :replace writes ?, and fallback: lets you choose e. Row 9: Encoding::CompatibilityError when both sides have non-ASCII bytes under different labels. Row 10: the three defaults — external (set to UTF-8 by this library's runner), the source file's own, and the empty String's — and String.new, which is BINARY. Row 11: e plus a combining acute is 2 code points but 1 grapheme cluster, NFC normalization folds it into the precomposed é, and upcase is full Unicode — ß becomes SS. Row 12: \w stops before é, \p{L} matches the whole word. Row 13: %-6s pads to six characters. Row 14: s[3] and s[-1] are the character é, byteslice(3, 2) is its two bytes, and byteslice(3, 1) is half a character, which is not valid.
The map¶
Every lesson of the Ruby text library, with the Python library page (or Regex library page) that measures the same idea from its side. A dash means there is no such page yet.
Two more shelves: the Regex library ↗ asks one regex question per page of several engines at once, Ruby among them, and the Encodings library ↗ is where UTF-8 itself, byte order marks, mojibake and the rest are taken apart.
Compared with Python¶
Python has two types where Ruby has one type and a label. A str is text and carries no encoding at all (row 1 prints n/a where Ruby printed UTF-8); bytes is the other type, and s.encode() crosses from one to the other (row 5: a bytes of length 5, never == to the str). There is therefore nothing to relabel (row 6), no valid_encoding? because a str cannot be invalid (row 4), and the failures move to the boundary: b"\xe9".decode() raises UnicodeDecodeError at once (row 7) where Ruby let the byte in and reported it later, encode("ascii") raises UnicodeEncodeError with errors="replace" and "xmlcharrefreplace" as the escape hatches (row 8), and str + bytes is a TypeError whatever the bytes hold (row 9), where Ruby raised only because both sides had non-ASCII characters. Rows 11–14 agree on the counts and disagree on two defaults: the standard library has no grapheme clusters, and \w matches é because Python's regex is Unicode by default — the opposite of Ruby's ASCII \w. The Python library's str is not bytes ↗ and Encode and decode ↗ are the two pages behind this section, and the table above pairs the rest.
Verified output of text_lives_in_the_text_library_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. a str: repr, type; a str carries no encoding label 'café'; str; n/a
2. len counts characters; len(s.encode()) counts bytes 4; 5
3. list(s.encode()); [ord(c) for c in s] [99, 97, 102, 195, 169]; [99, 97, 102, 233]
4. a str is always valid text (no valid_encoding); isascii n/a; False
5. s.encode() is a second type: type, len, == s bytes; 5; False
6. no relabel (bytes carry no label); encode('latin-1') converts n/a; [99, 97, 102, 233]
7. b'\xe9'.decode() fails at the boundary; .decode('latin-1') UnicodeDecodeError: unexpected end of data; 'é'
8. encode('ascii') raises; errors='replace'; 'xmlcharrefreplace' UnicodeEncodeError: ordinal not in range(128); b'caf?'; b'café'
9. str + bytes raises, whatever the bytes hold TypeError
10. getdefaultencoding; utf8_mode; type(b''); type('') utf-8; 1; bytes; str
11. e + U+0301: len; no grapheme clusters; NFC ==; upper 2; n/a; True; CAFÉ; STRASSE
12. \w is Unicode here: re.match(r'\w+', s); [^\W\d_]+ for letters 'café'; True
13. padding counts characters: '%-6s|' % s café |
14. s[3], s[-1] are characters; s.encode()[3:5] bytes; [3:4] decoded 'é'; 'é'; b'\xc3\xa9'; '�'
Try it¶
- Write
describe(name, str)that prints a String's encoding label, character count, byte count,valid_encoding?andascii_only?, and call it on: a UTF-8 literal, its.bcopy, itsencode("ISO-8859-1"), the same bytes relabelled withforce_encoding("ISO-8859-1"), a String holding an invalid byte, and a plain ASCII String. - Take the invalid String from exercise 1 and make it valid three ways —
scrub,encode("UTF-8", invalid: :replace), andforce_encoding("ISO-8859-1").encode("UTF-8")— then print what each produced and explain which one guessed and which one knew.
Solution to 1
text_lives_in_the_text_library_kata_rb.rb in full — pasted here by tools/run_examples.py from the file CI runs.
# Kata: describe(name, str) -- encoding label, characters, bytes, validity --
# for strings that share bytes or characters but not labels.
def describe(name, str)
puts format("%-11s %-10s chars=%d bytes=%d valid=%-5s ascii_only=%s", name, str.encoding, str.length, str.bytesize, str.valid_encoding?, str.ascii_only?)
end
utf8 = "caf\u{e9}"
describe "utf8", utf8
describe "utf8.b", utf8.b
describe "latin1", utf8.encode("ISO-8859-1")
describe "relabelled", utf8.dup.force_encoding("ISO-8859-1")
describe "invalid", "caf\xE9"
describe "ascii", "cafe"
Verified output of text_lives_in_the_text_library_kata_rb.rb — regenerated by tools/run_examples.py, never hand-typed.
utf8 UTF-8 chars=4 bytes=5 valid=true ascii_only=false
utf8.b ASCII-8BIT chars=5 bytes=5 valid=true ascii_only=false
latin1 ISO-8859-1 chars=4 bytes=4 valid=true ascii_only=false
relabelled ISO-8859-1 chars=5 bytes=5 valid=true ascii_only=false
invalid UTF-8 chars=4 bytes=4 valid=false ascii_only=false
ascii UTF-8 chars=4 bytes=4 valid=true ascii_only=true
See also¶
#{}callsto_s;formatis printf — the formatting this library does keep, with the padding row that the text library measures in depthFile.openwith a block closes the file for you — the encoding argument onFile.openandFile.readgetsreturns nil at EOF, and readsARGV's files first — the input side, where the text library's one-liner flags apply- Strings are mutable — the other thing a Ruby String is that a Python
stris not - Ruby text library ↗ — the whole shelf this page maps
- Regex library ↗ — one regex question per page, Ruby among the engines
- Encodings library ↗ — UTF-8 itself, byte by byte
stris notbytes↗ — Python's two types, measured- Encode and decode ↗ — the boundary where Python's failures happen
- Twenty-three Ruby traps for a Python programmer — The twenty-three places where a Python reflex gives the wrong answer in Ruby