03_Encodings — how a number becomes bytes¶
Level: 101 → 201 · the chapter this library is built around
A code point is a number, and a file is bytes. An encoding is the rule for writing one as the other, there are several rules, and — the fact the last two lessons exist for — the file does not record which rule was used. This chapter is UTF-8 done by hand, then who checks that a file really follows it, then the sequences it may never contain, then the other rules you will meet, then what it looks like when a file is read under the wrong one — then the channels that will not carry bytes at all, where the text is encoded a second time to get through: once character by character, and once for the whole stream — then the four-layer model that names everything the chapter has been walking through, including the two layers UTF-16 refers to at once — and last, the encoding that answered the same seven-bit problem a third way and had to be deleted from the web for it.
| # | Lesson | The question it answers | Status |
|---|---|---|---|
| 1 | UTF-8 by hand | How does U+00E9 become C3 A9, with a pencil? |
written, 2026-09-07 |
| 2 | Validation is a boundary | Who checks that a run of bytes really is UTF-8, and what is left of the check afterwards? | written, 2026-09-05 |
| 3 | Overlong sequences | Why is C1 BD not just another way of writing }, and what else does that rule forbid? |
written, 2026-09-05 |
| 4 | UTF-16 and surrogates | Why do Windows, Java, JavaScript and SAP say an emoji is two characters long? | written |
| 5 | Byte order and the BOM | Which byte of a two-byte number comes first, and what are EF BB BF doing at the top of my CSV? |
written, 2026-09-05 |
| 6 | Encode and decode are verbs | What exactly are the two operations, and where in a program do they belong? | written, 2026-09-05 |
| 7 | Mojibake | Why é, and how do I name the culprit from the garbage alone? |
written, 2026-09-05 |
| 8 | Escaping into ASCII | Why is one emoji \ud83d\ude00 in JSON, %F0%9F%98%80 in a URL, and neither in an email header? |
written, 2026-09-06 |
| 9 | Binary to text | How does a PNG get through a channel that carries only ASCII, and why is base64 of a word not one answer? |
written, 2026-09-06 |
| 10 | The alphabet is not the encoding | A menu offers four kinds of Base32 and prints four different strings — are those four encodings, or one? | written, 2026-09-07 |
| 11 | An encoding is four layers | UTF-16 and UTF-16LE are not a name and its default — what are they? |
written, 2026-09-08 |
| 12 | UTF-7, and the seven-bit transport | Why does an encoding exist that writes Unicode in ASCII, and why did everyone delete it? | written, 2026-09-08 |
The through-line¶
Every encoding bug is one of two verbs with the wrong table. Once UTF-8 is something you can do by hand, mojibake stops being mysterious: it is bytes that were encoded under one table and decoded under another, and you can read which two from the shape of the damage.