Skip to content

Counting characters

Level: 201 · for Python programmers

One line: "How long is this string?" has at least four correct answers, len() only ever gives one of them, and wc gives three — which is why wc -c and wc -m are separate flags.

Write a word-count program and you will discover this within an hour. Counting bytes is easy and counting words looks easy, but the moment the input leaves ASCII the two numbers come apart, and you have to decide which one the user asked for. The Unix answer is to refuse to choose: wc -c counts bytes, wc -m counts characters, and they are different flags because they are different questions.

Python has the same split, just spelled differently — len(raw) against len(text) — and it has a third answer that neither wc nor len() can give you.

Verified output of counting_characters_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. FOUR ANSWERS TO 'HOW LONG IS THIS?'
     sample              len()  utf-8  utf-16  graphemes
                        points  bytes   units    (human)
     ----------------------------------------------------
     plain ASCII             4      4       4          4
     NFC - one char          4      5       4          4
     NFD - two chars         5      6       5          4
     Polish                  6     10       6          6
     emoji                   1      4       2          1
     flag                    2      8       4          2
     family                  5     18       8          1

     Read the 'family' row: one thing on your screen, five code points,
     eighteen UTF-8 bytes. Every column is a legitimate answer to a
     different question, and len() only ever answers one of them.

2. THE TRAP THAT LOOKS LIKE A BUG
     nfc = 'café'   len = 4
     nfd = 'café'   len = 5
     they print identically:      café == café
     nfc == nfd                   False
     after normalize('NFC', ...)  True

     Two strings that look the same, print the same, and compare False.
     Normalize before comparing text that came from somewhere else.

3. WHAT wc COUNTS, AND WHICH len() MATCHES IT
     text  = 'Zażółć gęślą jaźń\n'
     bytes = 27   <- wc -c   len(text.encode('utf-8'))
     chars = 18   <- wc -m   len(text)
     words = 3    <- wc -w   len(text.split())
     lines = 1    <- wc -l   counts NEWLINES, not lines

     wc -c and wc -m differ by 9 here, and a wc that only reads bytes
     cannot tell you the second number at all.

4. THE LINE COUNT IS A NEWLINE COUNT
     with trailing \n   'a\nb\n'   count('\n') = 2   but there are 2 lines
     without            'a\nb'     count('\n') = 1   but there are 2 lines

     Without a trailing newline both wc -l and count(chr(10)) say 1 where a
     person says 2 -- they agree because they are the same rule. splitlines()
     is the one that answers the question a person actually asked.

5. str.split() IS NOT A WORD DEFINITION
     'one  two'                 ->  2 ['one', 'two']
     'hyphen-ated'              ->  1 ['hyphen-ated']
     "don't"                    ->  1 ["don't"]
     'łódź\xa0łódź'             ->  2 ['łódź', 'łódź']

     The last one holds a NO-BREAK SPACE (U+00A0). str.split() treats it
     as whitespace, so Python says two words; a tool splitting on ASCII
     space alone would say one. 'Word' is a policy, not a fact.

Read the family row first. One thing on the screen, 5 code points, 18 UTF-8 bytes, and len() says 5. Nothing in the standard library will tell you it is one character, because segmenting text into what a reader would call characters is UAX #29 ↗ and needs a table Python does not ship. If your program slices a string to fit a column, this is where it cuts a family in half.

And the graphemes column above is a deliberately crude rule, which the program says in a comment. It counts a new cluster at anything that is not a combining mark, not a ZWJ and not preceded by one — enough to show that the gap exists, and wrong in both directions the moment you leave these seven samples. It reads the flag row as 2 rather than 1, and a skin-toned emoji as 2, because it knows nothing about regional indicators or emoji modifiers. The proper treatment — five rulers rather than four, the UAX #29 boundary rules named one by one, and four real segmenters measured against each other — is A code point is not a character ↗ in the encodings library.

The NFC/NFD pair is the trap that looks like a bug. Two strings that render identically, print identically, and compare False. It reaches you through real doors: HFS+ handed back NFD, most Linux tools and most databases hand back NFC, so a filename compared against a database row can miss for a reason invisible on screen. (APFS does not normalize — it preserves what you wrote and matches both spellings, which is a different bug with the same cure; measured in Filenames are not text.) Normalize before comparing anything that came from outside your process.

wc -l counts newlines, not lines. So does the obvious Python. They agree with each other and disagree with a person reading the file, whenever the last line has no trailing newline. That is not a bug in either — it is the definition — but splitlines() answers the question that was actually asked.

str.split() is a policy, not a fact. It splits on Unicode whitespace, which includes the no-break space U+00A0 that a word processor puts between a number and its unit. A tool splitting on ASCII space alone counts that as one word. Neither is wrong; they are different definitions of "word", and a word-count program has to pick one and say so.

The Rust view

Rust asks the same question and gets the same numbers, with one difference worth knowing: len() means something else there. Python's counts code points; Rust's counts bytes, because it is O(1) and Rust would rather be honest about the cost than fast and vague about the meaning. Zażółć is 6 in Python and 10 in Rust, from a method with the same name — so a loop translated between them keeps working on ASCII and changes meaning at the first Polish letter.

The sibling library has two pages on this. Meet the char ↗ is the same three counts, and the same e + combining-accent pair that renders as one character and compares unequal. Four lengths ↗ adds the column this page does not have — UTF-16 code units, which is what JavaScript, Java, C# and SQL Server mean by "length" — then works through which system means which count, and the byte limit that panics mid-letter. Neither is repeated here: Rust's std ships UTF-8 correctness but not the Unicode character database, so normalization and grapheme segmentation are crates there rather than methods.

If you are coming from ABAP

strlen( ) counts characters, and xstrlen( ) counts bytes on an xstring — the same split, with the type carrying it rather than a flag. The genuine difference is that ABAP's internal representation is UTF-16, so strlen( ) returns code units, and a character above U+FFFF counts as 2 where Python's len() says 1. An emoji in a CHAR field is where this shows up. (Not machine-checked — CI cannot run ABAP.)

Try it

  1. Write the smallest wc you can: read stdin as bytes, decode once, and print lines, words, chars, bytes. Compare it against the real wc on a file with Polish text. Which column disagrees, and why?
  2. Now feed your program a file that is not valid UTF-8. The real wc still prints numbers. What does yours do — and is raising actually the wrong answer here?
  3. len("👨‍👩‍👧") says 5. Find the two U+200D characters in it with [hex(ord(c)) for c in s]. Then remove them and see what renders.

See also