A token is not a character¶

Level: 201 · for anyone who has watched a model miscount letters

One line: A language model does not read characters — it reads tokens, learned by counting byte pairs in a training corpus — so the letters are sealed inside the tokens, and the same sentence costs several times more in Polish than in English because of whose text the counting was done on.

This library has been about rulers. A code point is not a character lays five of them along one string — bytes, code units, code points, grapheme clusters, terminal columns — and gets five different answers to how long is this string, every one of them correct. CAST.md puts four of them in one table for exactly that reason.

A language model brings a sixth, and it is different in kind from the other five. Every ruler so far was fixed by a standard — you can look up how many bytes é takes and the answer is the same everywhere. A tokenizer's answers come from a training corpus, so two correct implementations disagree, and the disagreement is not a bug anyone can fix. This is the library's thesis — it depends who is counting — arriving somewhere it has consequences for how much your text costs.

Byte pair encoding, in twenty lines¶

The algorithm is embarrassingly simple, and knowing it is most of what you need:

Start with every byte as its own token. There are 256, so nothing is ever unrepresentable.
Find the commonest adjacent pair of tokens in the corpus. Glue it into one new token.
Repeat, until you have spent your vocabulary budget.

That is it. There is no linguistics in there, no dictionary, no notion of a word. the becomes one token because those four bytes keep turning up together, and so does berry if the corpus talks about fruit. GPT-2 ran exactly this to a budget of 50,257 tokens; the program below runs it to a budget of 48, on a paragraph you can read.

Why the letters are unreachable¶

Run strawberry through a tokenizer trained on English and it comes out as a handful of pieces — and what the model actually receives is not the pieces, it is their ids: a list of integers, each one a position in a lookup table.

Ask how many r's are in strawberry and the honest answer is that the question is not answerable from that input. Two of the r's are sealed inside one token, one inside another, and nothing in a token id says how the token is spelled. This is not a limitation of any particular model — it is a property of what a model is handed, and it is visible in twenty lines of Python.

That has a corollary worth stating plainly: a model that answers such a question correctly is not reading the spelling. It has learned the spelling as a fact about the token, the way you know how to spell a word you have never seen written down letter by letter. Which is why this class of question is unreliable in a way that has nothing to do with how capable the model is.

In Python¶

Verified output of a_token_is_not_a_character_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. WHAT THE TRAINING LEARNED, IN ORDER
------------------------------------------------------------------------
   corpus: 295 characters of English, 295 bytes
   budget: 48 merges      learned: 48

   The first twelve pairs it glued together:

      256  't'        + 'h'        -> 'th'
      257  'e'        + ' '        -> 'e '
      258  ' '        + 'th'       -> ' th'
      259  't'        + ' '        -> 't '
      260  'a'        + 't '       -> 'at '
      261  ' th'      + 'e '       -> ' the '
      262  ' '        + 'a'        -> ' a'
      263  'm'        + 'a'        -> 'ma'
      264  'ma'       + 'l'        -> 'mal'
      265  's'        + 'mal'      -> 'smal'
      266  'smal'     + 'l'        -> 'small'
      267  'small'    + ' '        -> 'small '

   Nobody chose those. `small` is one token because that paragraph
   keeps saying it, and `berry` becomes one for the same reason. A
   different paragraph makes a different tokenizer, and there is no
   standard anywhere in this section -- only counting.

2. WHY THE MODEL CANNOT COUNT THE LETTERS
------------------------------------------------------------------------
   'strawberry'   10 characters, 10 bytes, 5 tokens

   token         id   what the model receives
   's'          115   an opaque integer
   't'          116   an opaque integer
   'ra'         297   an opaque integer
   'w'          119   an opaque integer
   'berry'      283   an opaque integer

   the model sees:  [115, 116, 297, 119, 283]

   Count the letter `r` in that list. You cannot, and neither can the
   model, because the r's are INSIDE the tokens. Two of them live in
   token 283 and one in another, and a token id is a position in a
   lookup table, not a string. Nothing in the model's input says how
   any token is spelled.

   What is demonstrated here is the input, not the model: nothing
   in that list of five integers carries a spelling. It follows that
   a model answering `how many r's` correctly is drawing on something
   other than what it was handed -- the spelling has to have been
   learned as a fact ABOUT the token rather than read OFF it.

3. THE SAME MEANING COSTS DIFFERENT AMOUNTS
------------------------------------------------------------------------
   Tokenized by the ENGLISH-trained merges above.

                                    chars  bytes  tokens  bytes/token
   'the cat sat in the warm house'     29     29       8   3.62   English
   'żółw siedzi w ciepłym domu'        26     30      28   1.07   Polish

   Two sentences of about the same length. One costs 8 tokens and the
   other 28 -- three and a half times as much to say a comparable
   thing. The Polish sentence is barely compressed at all: almost
   every byte is still its own token, because none of its byte pairs
   was common enough in an English paragraph to earn a merge.

   That is the whole mechanism behind `the same text costs more in
   Polish`. It is not about the language being harder, and only
   partly about UTF-8 being wider. It is about whose text the merges
   were learned from.

4. AND THE BUDGET IS ZERO-SUM
------------------------------------------------------------------------
   The same 48 merges, learned from English + Polish instead:

                                     English corpus   mixed corpus
   'the cat sat in the warm house'                8             11
   'żółw siedzi w ciepłym domu'                  28             15

   The Polish sentence got much cheaper. The ENGLISH one got dearer.

   That is not a flaw in the experiment, it is the point. A vocabulary
   is a fixed budget, and a merge spent on a Polish byte pair is a
   merge not spent on an English one. Every language in the corpus is
   competing for the same slots, and the majority language wins them.

   Which is why a real tokenizer, trained on text that is mostly
   English, is cheap for English -- not by design, and not by
   anybody's decision, but as arithmetic.

5. A TOKEN IS BYTES, SO IT NEED NOT BE A CHARACTER
------------------------------------------------------------------------
   'żółw' through the English merges -> 7 tokens
     c5       c5
     bc       bc
     c3       c3
     b3       b3
     c5       c5
     82       82
     77       'w'

   'berry' through the English merges -> 1 token
     6265727279 'berry'

   Every token of the Polish word is a lone byte, and six of the seven
   are HALF A CHARACTER -- the first or second byte of a two-byte
   UTF-8 sequence, which no decoder will accept on its own. That is
   why `show` above has to fall back to hex.

   This is the same boundary problem as a chunked read cutting a
   character in two, arriving in a new place. A byte-level tokenizer
   can never fail on unfamiliar text -- there is always a token for
   every byte -- and the price of never failing is that the pieces
   it hands the model are not characters.

6. WHAT THIS PROGRAM IS NOT
------------------------------------------------------------------------
   Everything above is real BPE and none of it is a real tokenizer.
   The differences that matter:

     scale       48 merges over one paragraph; a production vocabulary
                 is 50,000 to 200,000 tokens over a corpus measured in
                 terabytes. Every number here is smaller than a real
                 one; the RATIOS are the transferable part.
     pre-tokens  real implementations split on a regex FIRST, so that
                 a merge can never cross a word boundary. This one has
                 no such rule, which is why `the ` includes its space
                 and why a merge could in principle span two words.
     the mapping GPT-2 remaps the 256 bytes into printable code points
                 before merging, so that a vocabulary file is text.
                 That is cosmetics over the same algorithm.

   And one thing this page deliberately does NOT contain: a token
   count from a real model. There is no tokenizer library here, no
   network, and no vocabulary file -- so a number attributed to a
   named model would be a number nothing in this repository can
   check, which is the one kind of claim this library does not make.
   The mechanism is the claim, and the mechanism is on this page.

The cost asymmetry, and where it actually comes from¶

Section 3 measures it: two sentences of about the same length, tokenized by the same English-trained merges, cost 8 tokens and 28. Section 4 is the part that explains it, and it is the half usually left out.

Re-train the same 48-merge budget on a corpus that is half Polish, and:

the Polish sentence drops from 28 tokens to 15;
the English sentence rises from 8 to 11.

A vocabulary is a fixed budget, and it is zero-sum across languages. A merge spent on a Polish byte pair is a merge not spent on an English one. So a tokenizer trained on text that is mostly English is cheap for English — not by anyone's decision, and not because English is simpler, but as arithmetic about which byte pairs were commonest.

Three things follow, and they are the practical content of this page:

A cost per token is a cost per token, not a cost per word. If you are billed or rate-limited by token, the same document in Polish, Turkish, Hindi or Japanese costs more than its English translation, and the multiplier is a property of the tokenizer rather than of your text.
A context window is smaller in some languages than others. The same fixed window holds noticeably less Polish than English.
UTF-8's width is only part of it. żółw is 4 characters and 7 bytes, so some of the cost is the encoding — but the Polish sentence measured here costs 1.07 bytes per token against English's 3.62, and that ratio is about the merges, not the encoding. A language written entirely in ASCII would still be expensive if the corpus had not seen it.

The token boundary falls inside a character¶

Section 5 is the one that ties this page back to the rest of the library. żółw through English-trained merges is seven tokens — and six of them are half a character, the lead or trailing byte of a two-byte UTF-8 sequence, which no decoder will accept on its own.

That is the same failure this library has met twice already: a boundary drawn at a place the encoding does not permit. Validation is a boundary is the general form, and a chunked read cutting a sequence in half is the classic instance.

The reason a byte-level tokenizer does it deliberately is a good one: starting from all 256 bytes means it can never fail. There is no text it cannot represent, no unknown-token escape hatch, no crash on an unfamiliar script. The price of never failing is that the pieces it hands the model are not characters, and sometimes not even whole ones.

What is on this page and what is not¶

Everything above is real BPE, run here, and none of it is a real tokenizer. The scale is a paragraph against terabytes; a production implementation splits on a regex first so a merge cannot cross a word boundary; and GPT-2 remaps the 256 bytes into printable code points so the vocabulary file is text.

And one thing is deliberately absent: a token count attributed to a named model. There is no tokenizer library in this repository, no network, and no vocabulary file, so such a number would be one nothing here can check — and this library's rule is that a number on a page has been run. The mechanism is the claim, and the mechanism is measured. If you want the numbers for a specific model, run its own tokenizer against your own text; the ratio you find will have the shape section 4 explains.

If you are coming from Python or ABAP¶

Python. len(text) counts code points and is not a token count, and there is no way to estimate one without the specific model's vocabulary — the rule of thumb that "a token is about four characters" is an English rule of thumb, and section 3 is what it looks like when you apply it to Polish. If you are budgeting for an API, tokenize a real sample of your own text with the model's own tokenizer; if you are chunking documents to fit a window, chunk on token counts rather than characters, and expect a chunk size tuned on English to overflow on anything else.

ABAP. (Not machine-checked — CI cannot run ABAP.) The relevant instinct transfers directly, and it is one this library has already argued in a different register: STRLEN counts characters, XSTRLEN counts bytes, and neither is what a model charges you for. If a system sends text to a model — a long text field, an incident description, a material description translated into fourteen languages — the length check that keeps it under a limit belongs on the token count and cannot be computed in ABAP. Send a sample through the tokenizer once, measure the ratio per language, and treat that as a configuration value rather than a constant, because a model change moves it. Note also that the SAP text you are most likely to send is precisely the expensive kind: short fields in many languages, where the non-English ones carry the diacritics and the tokenizer has seen the least of them.

Try it¶

Take a paragraph of your own writing and its translation into another language, and send both through the same model's tokenizer. Write down the ratio. That number is your multiplier for everything you do with that model.
Change the corpus in the program on this page — paste in a paragraph of your own — and watch which merges it learns. The first twelve tell you what your text is made of.
Raise BUDGET from 48 and find the point at which the Polish sentence stops shrinking. That is your corpus running out of repeated Polish byte pairs.
Ask a model you use how many of some letter are in a longish word, then ask it to spell the word out one letter per line first, and compare. The second framing puts the letters into its input.
If you chunk documents for retrieval, check whether your chunk size is in characters or tokens, and what it does to your least-English document.