Skip to content

length counts code points

Level: 101 · anyone who has truncated a name or reversed a string

One line: length counts code points — 5 for a family emoji a reader sees as one character — and reverse and [] cut in code points too, so they separate an accent from its letter and turn a flag into two letters; grapheme_clusters and /\X/ work in the characters a reader sees.

Verified output of length_is_code_points_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

                  bytesize length UTF-16 units graphemes
  caf + U+00E9           5      4            4         4
  cafe + U+0301          6      5            5         4
  flag of Poland         8      2            4         1
  family emoji          18      5            8         1

  flag                            U+1F1F5 U+1F1F1
  flag.reverse                    U+1F1F1 U+1F1F5
  cafe                            U+0063 U+0061 U+0066 U+0065 U+0301
  cafe.reverse                    U+0301 U+0065 U+0066 U+0061 U+0063
  cafe.grapheme_clusters.reverse  U+0065 U+0301 U+0066 U+0061 U+0063
  cafe[0, 4]                      U+0063 U+0061 U+0066 U+0065
  cafe.scan(/\X/).first(4).join   U+0063 U+0061 U+0066 U+0065 U+0301

  Unicode version of this Ruby    17.0.0

Reading the output

The table is four rulers laid along the same strings:

  • bytesize is storage in the string's encoding, UTF-8 here.
  • length is code points.
  • UTF-16 units is what a language that stores UTF-16 reports as its length — see the Java text library's Length is three different numbers ↗.
  • grapheme_clusters.size is user-perceived characters, by Unicode's segmentation rules in UAX #29 ↗.

Below the table:

  • A flag is two code points, regional indicators for P and L. reverse swaps them into L then P, which is not the flag of anywhere.
  • A combining accent follows its letter. reverse moves U+0301 to the front of the string, where it has no letter to sit on; reversing the grapheme clusters keeps e and its accent together.
  • cafe[0, 4] drops the accent; scan(/\X/).first(4) keeps it.
  • The grapheme rules change with Unicode versions — emoji sequences especially — so the Ruby matters: this one carries Unicode 17.0.0.

What to do

Truncate, reverse and count for display in grapheme clusters: s.grapheme_clusters.first(n).join, s.grapheme_clusters.reverse.join, s.grapheme_clusters.size. Use bytesize for a limit measured in bytes, such as a protocol field. length is right when the other side counts code points.

If you are coming from another language

Python's len counts code points as Ruby's length does, and its standard library has no grapheme segmentation. (Not machine-checked here.) The Perl library's length counts code points ↗, the Python library's Counting characters ↗, the Rust library's Four lengths ↗ and the Encodings library's A code point is not a character ↗ lay the same rulers along strings in their own languages.