Skip to content

length counts code points

Level: 201 · you count, cut, pad or reverse text a person will read

One line: length counts code points, which is neither what a reader counts nor what a file stores: café is 4 or 5 depending on how it was spelled, a flag is 2 and a family emoji 5 — /\X/g counts what a reader sees, Encode::encode counts bytes, and Unicode::Normalize makes the two spellings of café one string.

Measured

Verified output of length_pl.pl — regenerated by tools/run_examples.py, never hand-typed.

--- three rulers ---
                         bytes length graphemes
café, composed (NFC)         5      4         4
café, decomposed (NFD)       6      5         4
zażółć                      10      6         6
flag of Poland               8      2         1
family emoji                18      5         1

--- two spellings of café ---
nfc eq nfd:              no
NFC(nfd) eq nfc:         yes
nfd =~ /^cafe/:          yes   the e is there, with its accent after it
index(nfd, 'cafe'):      0
length NFD('zażółć'):    9   ł has no decomposition

--- cutting by code point cuts letters ---
substr(nfd, 0, 4):       [cafe]
reverse nfd:             [́efac]
reverse by \X:           [éfac]
reverse flag:            [🇱🇵]   PL reversed is LP
sprintf '%-6s|':         [café  |] [café |]

Three rulers

  • Bytes — length encode('UTF-8', $s) — what the string costs in a file or a database column.
  • Code points — length $s — what perl stores.
  • Graphemes — scalar(() = $s =~ /\X/g) — what a reader would count. \X matches one extended grapheme cluster: a base character with the marks that attach to it, a pair of regional indicators that make one flag, or a run of emoji joined by U+200D ZERO WIDTH JOINER.

The three agree on zażółć, because every Polish letter has a single precomposed code point. They stop agreeing as soon as a combining mark, a flag or an emoji arrives — and all three arrive in names, messages and form fields.

Two spellings of café

é can be one code point, U+00E9, or two: e followed by U+0301 COMBINING ACUTE ACCENT. They look identical and compare unequal. NFC composes where it can, and NFC($nfd) eq $nfc is true. The decomposed spelling also contains a plain e, so a search for cafe finds café — and substr can split the e from its accent.

Normalization does not decompose everything: ł is its own letter with no decomposition, so NFD makes zażółć 9 code points, not 12.

Cutting by code point cuts letters

  • substr($nfd, 0, 4) is cafe: the accent was the fifth code point.
  • reverse reverses code points, so the accent comes first, attached to nothing — or to whatever the terminal prints before it, here the bracket.
  • Reversing by \X keeps each letter whole.
  • A flag is two regional-indicator letters, P and L. Reversed they spell LP, which is not a flag.
  • sprintf '%-6s' pads to six code points, so the decomposed café is one column short.

What to do instead

  • Normalize at the edge, once: my $text = NFC(decode('UTF-8', $bytes, FB_CROAK)).
  • Count, cut and reverse by grapheme: my @letters = $s =~ /\X/g;.
  • Limit a database column by bytes, not by length.
  • Align in a terminal by display width, which is a fourth ruler again — Python's Padding is not alignment ↗ measures it.

If you are coming from another language

See also