What to write next — the prioritized backlog¶

Level: reference

One line: Fifty-one pages in the order they are worth writing, ranked against the four checkpoints this library was built around — because the topic map in TOPICS.md lists over 1,500 terms and a list that long is a way of not choosing.

The finding that decides the order¶

00_Start_Here names four checkpoints — the four things Adam said on day one he could not yet do. On 2026-09-06, counting the pages on disk:

Checkpoint	The page that settles it	Status
`0x41` ↔ 65 ↔ `0b01000001` by hand	chapter 1, three lessons	written
Code point vs UTF-8 bytes	UTF-8 by hand	written, 2026-09-07
Python `str` vs `bytes`	Encode, decode and errors	written, 2026-09-07
Rust `String` vs `&str` vs `char`	`char` is four bytes	written, 2026-09-07

All four checkpoint pages are now written — the last two, UTF-8 by hand and char is four bytes, on 2026-09-07, a day that started with three of the four still outlined. The promise the library was built around is kept; what is left below is the work that grew around it (ROADMAP.md explains each detour as it was taken).

So the ranking is: finish the promise, then the goal that has no chapter yet (real SAP data), then the remaining stubs, then the pages the topic map exposed. 25 of the 51 rows name a page that already exists, and two of those are still stubs — rows 4 and 22, with their questions written down and their URLs minted; graduating one of those beats minting a new page almost every time. (Re-counted 2026-09-10, counting a link in a row's title cell only, which is the definition a script can repeat: rows naming an existing page and of those, still stubs. The previous wording named rows 19, 20 and 21 as stubs and all three had graduated with the terminal chapter — which is the argument for naming the members rather than carrying a count.)

Every hook below is a claim to verify, not a fact this library has checked. That is the whole difference between this page and every other page here (CONTRIBUTING.md). Expect roughly one in three to come out differently once a program is pointed at it — which is the reason to point a program at it.

Tier 1 — the checkpoints (1–7)¶

Nothing else in this library is worth more than these seven. One is a stub — row 4, A str vs bytes — and its questions are already written on the page.

#	Page	Lands in	The hook to verify
1	~~UTF-8 by hand~~	03	Written 2026-09-07. All four languages, and Table 3-7 walked from both ends rather than quoted — which is where the row's hook came out sharper than it was written. 1,112,064 scalar values encoded by hand, and separately 1,112,064 byte sequences enumerated from the nine rows' byte columns: the two counts meeting is what makes the table a bijection, and `0x110000 − 2048` and the nine byte products are the same number arrived at two incompatible ways. The timing was wrong: about three seconds in Python, not one, precisely because the encoder is hand-written rather than called (0.09 s in Rust, 0.05 s in C, and 0.28 s for the same Python loop using `.encode()`)
2	~~Encode, decode and errors~~	04	Written 2026-09-07. Eight handlers, not six, and the blast radius came out sharper than the row predicted: `replace` is not visible data loss either, because a real `U+FFFD` in the input produces the same string
3	~~`char` is four bytes~~	05	Written 2026-09-07. The checkpoint page, and it came out with two findings the row did not predict: the surrogate hole that makes `char` strict is the same hole that makes `Option<char>` free — 4 bytes, against 8 for `Option<u32>` — and `char as u8` compiles, truncates, and is correct for `é`, so the case a European developer tests first is the one that hides it
4	A `str` vs `bytes`	04	The page checkpoint 3 rests on. `len()` answers two different questions and neither is "characters"
5	~~UTF-16 and surrogates~~	03	Written 2026-09-07. `U+1F600` → `D83D DE00` by hand, and the sting landed as written: `json.dumps` emits that surrogate pair into a format that is UTF-8 by RFC 8259 ↗, and `json.loads('"\ud800"')` succeeds
6	~~A code point is not a character~~	02	Written 2026-09-07. The five lengths landed as described; the row's proposed method did not. `unicodedata.east_asian_width` would have put a table lookup in the answer key, so the fifth length is computed from a width table in the source and the interpreter's reading goes in a dated fence — the four rulers that are arithmetic are the four that CI checks
7	~~`from_utf8` and lossy~~	05	Written 2026-09-07. `valid_up_to()` is `UnicodeDecodeError.start`, on every input tried, and `error_len()` is `end - start` wherever Rust reports one. They part in exactly one place, and it is the useful one: Rust's `None` says incomplete, a verdict Python's exception has no field for

Tier 2 — real data, the goal with the emptiest chapter (8–14)¶

Goal three was real SAP data, and 07_Real_Data is now finished: every row in this tier has landed, rows 8 to 14, between 2026-09-07 and 2026-09-08. Every one of these is a bug Adam will actually meet.

#	Page	Lands in	The hook to verify
8	~~Windows-1252 vs Latin-1~~	07	Written 2026-09-07. The replacement claim came out stronger than the row: cp1252 keeps zero of the disputed 32 mappings, so it is not a superset by any reading. The glossary was already correct ("reassigned"). New: `é` mojibake is identical under both tables, so only `€` names which one was used
9	~~The mojibake round trip~~	07	Written 2026-09-07. The five holes are UTF-8 continuation bytes, so what cp1252 cannot repair is `Á Í Ï Ð Ý` and `Ł` — ordinary European capitals. And the guard has a measured false-positive rate: 10.4% at two characters, 1.7% at three, so it is safe on a sentence and risky on a short field
10	~~SAP code pages~~	07	Written 2026-09-07. Three numbering catalogues, and Python's `cp` follows IBM/Microsoft — `cp1100` is a `LookupError`, so SAP's numbers are not typeable at all. Only 1 of the 10* euro twins is in the stdlib, and the page says so; EBCDIC also moves the newline to `0x25`, so `wc -l` reports 0 lines
11	~~Fixed-width byte fields~~	07	Written 2026-09-07. The hook held and the number underneath it is the lesson: 9 of the 26 possible widths split this one value and 17 do not, so whether `encode()[:n]` is a bug is a property of the data, not the code . And a sibling landed 2026-09-08. Packing a record takes the same byte budget into a binary record — where the layout lives in four languages, byte order, alignment padding — and answers it the other way: check the encoded length and refuse before packing, because a binary field has no column to truncate into. The two pages each say which half they own
12	~~CRLF vs LF~~	07	Written 2026-09-07. Python and shell; `splitlines()` splitting on eleven sequences where `split('\n')` sees one, the byte that cannot appear in its own output, and git reporting a tree clean when every line on disk differs from the blob
13	~~Sorting and collation~~	07	Written 2026-09-07. Three orders was an undercount — one list of nine words gets five, and the two libcs agree on every letter, so the split is in the punctuation: `strcoll("a-b","ab")` changed sign across glibc 2.28, and PostgreSQL could not warn about it until 13, two years later. Extended 2026-09-08 with the model and the second failure: UTS #10's three levels, and the finding that a collation which is correct need not be repeatable — it makes ties deliberately, and a stable sort's promise about a tie is a restatement of the input
14	~~Interfaces and storage~~	10	Written 2026-09-07. The closing rule, and the fourth clause is the one that matters. The map came out with three groups rather than two: four channels carry the declaration inside the data, JSON needs no charset parameter at all because RFC 8259 §11 ↗ records that adding one has no effect on a compliant recipient, and CSV, fixed-width and a URL query have nowhere to put it and nothing fixing it

Tier 3 — the remaining stubs (15–22)¶

Cheap: the questions are already written, and each is one program away.

#	Page	Lands in	The hook to verify
15	~~Slicing by byte~~	05	Written 2026-09-07. The panic is caught with `catch_unwind` and `is_char_boundary` checked against the outcome at every index — it predicts all of them. The message itself is in a dated fence naming four builds rather than in a key, per CONTRIBUTING.md
16	~~A `str` in memory~~	04	Written 2026-09-07 — A `str` in memory. The hook held up (a 1,000-character ASCII string goes from 1041 bytes to 4060 when one `😀` joins it) but no such number is in the key: the program prints strides, ratios and thresholds only. Two findings came out of writing it — `sys.audit()` fills a `str`'s UTF-8 cache on CPython 3.14 and not on 3.11–3.13, a version split that first read as a platform one; and the ASCII/Latin-1 boundary at `U+0080` is a header step, not a stride step, which is why the buffer table has four rows and three widths
17	~~Opening a file~~	04	Written 2026-09-07 — Opening a file. The trap was real and is disclosed on the page: the harness runs Python in UTF-8 Mode — not through the `PYTHONUTF8=1` it sets, which `-I` discards, but through `LC_ALL=C`, which PEP 540 turns into UTF-8 Mode by itself — so every default is read from a child interpreter instead. `io.text_encoding()` and PEP 597's `EncodingWarning` both earned a section. Two values were kept out of the key for being facts about the runner — the C locale's name (`US-ASCII` / `ANSI_X3.4-1968`) and, with UTF-8 Mode off, `sys.getfilesystemencoding()`, `utf-8` on macOS against `ascii` on glibc, the Python-side face of `find`'s APFS finding
18	~~`bytes`, hex and int~~	04	Written 2026-09-07 — Bytes, hex and int. The hook was out of date and the page says so: `int.from_bytes` required `byteorder` from Python 3.2 until 3.11 gave it a default of `'big'`, so the argument that prevented the bug now has to be written down out of discipline instead. Chapter 1's hex lessons are linked rather than rebuilt; the PNG header is built in the program (length and CRC computed) so its `0D 0A` pair can be read as what it is — a file format defending itself against line-ending translation
19	~~`iconv`~~	06	Written 2026-09-07. The third split held and got sharper — it is not a BSD/GNU split at all: both iconvs accept `f4 90 80 80`, `f5 90 80 80` and a five-byte sequence, agreeing with each other and disagreeing with Python and Rust. Two more came out of the build: `//IGNORE` exits 1 on both platforms after doing exactly what it was asked, and the `-c` split is narrower than recorded — macOS only loses the tail when it is a single byte
20	~~`printf` writes bytes~~	06	Written 2026-09-07. Finding 13 reproduced exactly, and a new version split beside it: `$( )` drops a NUL byte on every bash, and only 4.4 and later say so — the silent build is the one on the Mac
21	~~`file` guesses~~	06	Written 2026-09-07. Five answer shapes, not four — `unknown-8bit` is the fifth — and the finding was elsewhere: `file` reads 64 KiB, so 65534 bytes of padding before an `é` reports `utf-8`, 65535 reports `iso-8859-1` because the window ends mid-character, and 65536 reports `us-ascii`
22	Logical and visual order	02	A terminal screenshot is not evidence about a string. UAX #9 ↗, embedding levels, and the nine controls written only as escapes

Tier 4 — the pages the topic map exposed (23–51)¶

New pages. Ordered by how often the problem shows up in Adam's three goals, not by how interesting the idea is.

#	Page	Lands in	The hook to verify
23	~~Bytes that are not text — `surrogateescape`~~	04	Written 2026-09-07 — Bytes that are not text. It took the three things this row still listed as owed (`os.fsdecode`, `sys.getfilesystemencoding`, the Windows half via `surrogatepass` and PEP 529 ↗) and left the `x-user-defined` comparison where it already is, on "Handles Unicode" is four questions
24	Best-fit mapping — the quote that appeared from nowhere	12	Windows converts `U+2018` to `'` when encoding to a code page that lacks it. A filter that checked the original string has been bypassed by a conversion. Currently described-not-demonstrated on canonicalize then check
25	The space that is not a space	07	NBSP survives copy-paste from Excel and Word, is not stripped by every language's `strip()`, and makes a numeric field fail to parse. The most common invisible bug in business data
26	The Norway problem — when a type is guessed	07	YAML 1.1 reads `no` as `false` and `12:30` as 750 in base 60. A country code becomes a boolean. Cross-links the voting library ↗, which machine-checks for exactly this
27	Excel and the CSV	07	Three separate facts: Excel needs a UTF-8 BOM to read UTF-8, uses the locale's list separator (semicolon across Europe), and executes a leading `=` — CSV injection
28	Git and your filenames	07	`core.quotepath` (the octal escapes in `git status`), `core.precomposeunicode` (macOS hands git a decomposed name), and `working-tree-encoding` for UTF-16 files
29	The filename inside the zip	07	CP437 unless the EFS bit is set. The garbled-Japanese-zip classic, and a flag most tools never write
30	~~Case is not a per-character operation~~	02	Written 2026-09-08, on the transformation half the row had been narrowed to. The matching half — `re.IGNORECASE` puts `i`, `I`, `İ` and `ı` in one class where `casefold()` makes three — is "Supports Unicode" is a level, not a yes, and the page links it. All three axes measured, plus one the matcher cannot see: `re.fullmatch('ß', 'SS', re.IGNORECASE)` is `False` both ways, because a case-insensitive regex compares one character with one character. The Rust half found the sharpest form of the claim: `"ΟΔΟΣ".to_lowercase()` and `"ΟΔΟΣ".chars().flat_map(char::to_lowercase).collect::<String>()` return different strings — U+03C2 against U+03C3 — because a `char` cannot see its neighbours, and std documents exactly that. Turkish is the refusal the row predicted: `LC_ALL=tr_TR.UTF-8` with the locale actively set changes nothing, and the stdlib's locale-sensitive text operations are about sorting
31	~~Where a line may break~~	02	Written 2026-09-08. Four UAX #14 ↗ classes and three rules hand-rolled, which is enough for Japanese and provably not enough for Thai — the split the row predicted, with the reason: Japanese is a rule over adjacent characters, Thai is a dictionary. `textwrap` orphans the ideographic full stop onto its own line, and on the Thai string is correct at width 4 and wrong at 3, 5 and 6
32	What a charset label means	03	The WHATWG Encoding Standard: `iso-8859-1` must be decoded as windows-1252, label matching is case- and whitespace-insensitive, and unknown labels map to the replacement encoding on purpose. The document the web actually obeys
33	How many `U+FFFD`?	12	Two decoders, one invalid input, different numbers of replacement characters (the maximal-subpart rule). A parser differential hiding in the error path. The census half was measured on 2026-09-10 for `rune` is an `int32`: eleven decoders, five answers — the maximal-subpart majority, Go per byte, Go's `ToValidUTF8` per run, Java folding an encoded surrogate into one, Perl folding anything with UTF-8's shape. The attack half is still this page's
34	ICU — the library underneath	03	Collation, break iteration, normalization, transliteration, formatting. Most languages' text handling is ICU wearing a hat; Rust's std deliberately is not, which is why the crates exist. The claim is now asserted but not demonstrated at the foot of "Handles Unicode" is four questions, which is exactly the gap this page closes
35	~~PRECIS: stringprep, after stringprep~~	02	Written 2026-09-08, built on Preparing a string as the row asked. Both profiles implemented from RFC 8265 ↗. The page's own finding: `U+200B` is `Zs` in the frozen 2002 table Python ships and `Cf` in a modern one — so PRECIS's space-mapping rule maps it under one Unicode and disallows it under another, which is the freeze problem returning by the front door
36	Passwords are not text	12	bcrypt truncates at 72 bytes, so a passphrase in a non-Latin script carries less than it looks; and normalizing a password is a decision with no free option — normalize and two spellings both log in, don't and one silently never works
37	Text you cannot see	12	The `U+E0000` tag block: invisible to a human reviewer, fully present to a parser or a model. Trojan Source's 2026 descendant, and the same house rule applies — escapes only, never a raw control
38	The filename that reads backwards	12	An override in a filename displays `...exe` as `...pdf`. Decades old, still shipping. The bidi lesson with a consequence attached
39	A log viewer is an interpreter	12	Untrusted text printed to a terminal can move the cursor, rewrite earlier lines, set the title, and on some terminals request a reply that is then typed as input. Stripping ANSI with the usual regex misses OSC
40	~~A pipe is not a terminal~~	06	Written 2026-09-07, and the hook was wrong — the useful kind. Measured with a `pty` and a pipe running the same child on Python 3.11–3.14: `isatty()` moves `line_buffering` and nothing else; encoding and error handler are identical on both roads, and the `UnicodeEncodeError` fires on both or neither. The locale decides the encoding; the tty decides only whether your output survives the process. Handed over on 2026-09-10 by the Python library's stdin, stdout and pipes ↗ stub, which now keeps only the call (`BrokenPipeError`, `sys.stdout.buffer`): its encoding questions are answered on this page, except one that no page here answers yet — `PYTHONIOENCODING` against `sys.stdout.reconfigure(encoding=...)`, and which to reach for
41	A character split across two reads	03	A chunked read can cut a UTF-8 sequence in half, so `chunk.decode()` per chunk is wrong. Incremental decoders exist for this, in every language. The Python-module half is now written next door — The codecs registry ↗ has the split-sequence bug, the `errors='replace'` version that corrupts instead of raising, and `final=True`; this page is the language-agnostic half and should not repeat them
42	The filename in the HTTP header	08	Headers are ISO-8859-1 by RFC 7230 ↗ — but 7230 is obsoleted by RFC 9110 ↗, which calls that reading historical and constrains field values to US-ASCII; RFC 8187 ↗'s `filename*=UTF-8''` is the fix. Why a download with a Polish name works in one browser and not another
43	JSON is Unicode, and still has surrogates	08	Measured: `json.dumps("😀")` emits `😀`, `json.loads('"\ud800"')` returns a lone surrogate that cannot be encoded to UTF-8. Plus duplicate keys and the forbidden BOM
44	The characters XML cannot carry	08	XML 1.0 cannot represent `0x01` even escaped. A format that forbids data, which is why database exports fail on control characters. Partly measured already, from the other direction, on Noncharacters and the private use areas: `ElementTree` writes what it then refuses to read, and the `Char` production forbids only 2 of the 66 noncharacters — so forbidden by XML and reserved by Unicode are two different sets. What is still owed is the control-character half and the XML 1.1 comparison
45	The column that counts bytes	10	`ORA-12899` (value too large, in bytes, for a column sized in characters), MySQL's 767/3072-byte index prefix, and `utf8mb3` raising error 1366 on an emoji
46	The client that lies about its encoding	10	Oracle `NLS_LANG`: the client declares an encoding and the server converts to match, so a wrong declaration corrupts on write, invisibly, forever. The classic enterprise mojibake source
47	Two PowerShells, two files	11	Windows PowerShell 5 writes UTF-16LE by default; PowerShell 7 writes BOM-less UTF-8. The same script, two machines, two different files
48	The container has no locales	06	musl ships essentially no locale data, so Alpine behaves differently from Debian on the same code. The container-era version of the BSD/GNU split this library already tracks
49	~~`OsStr`, `Path`, and WTF-8~~	05	Written 2026-09-07 as `OsStr`, `Path`, and WTF-8, on the narrowed scope. The best thing on it is a correction the program made to its own page: `Path::eq` compares components, not bytes, so `a//b == a/b` — drafted as a byte comparison, and the output disagreed in the same fence as the claim. The Windows half is a dated fence read off the std source, because no runner here is a Windows machine
50	~~A token is not a character~~	09	Written 2026-09-08 — buildable after all, by hand-rolling a byte-level BPE and training it inside the program on a corpus small enough to print. No `tiktoken`, no vocabulary file, and no token count attributed to a named model, because nothing here could check one. The measurement that made it worth writing was the one nobody asked for: at a fixed merge budget, adding Polish to the corpus drops the Polish sentence from 28 tokens to 15 and pushes the English one from 8 to 11. A vocabulary is zero-sum, so English is cheap as arithmetic rather than by anyone's decision
51	~~UTF-7, and the seven-bit transport~~	03	Written 2026-09-08, and it was never on this list — which is the point of the row. The library had been using UTF-7 since 2026-09-06 as an attack payload on Two readers, one byte string without ever explaining why the format exists, so the gap was invisible to a backlog organised by topic. The count came out sharper than a hook would have predicted: 2,961 byte strings decode to `<script>`, every one verified, against the one an encoder writes. Two things the page did not set out to find — Rust's std has no UTF-7 and no charset registry at all, and BSD `iconv` will not absorb a trailing `-`, so it cannot round-trip its own output

Landed off the list¶

2026-09-08: A record has to say what it is, how long it is, and whether it arrived. Not item 1–50, and not a stub graduating either — it came from a question Adam asked about a Wikipedia article (Intel HEX ↗) while working on the Tribit project, and the answer turned out to be a page: record type, length and checksum, which are the three fields Tribit's container does not have and which nothing else in this library covered. Chapter 8 went from one page to two, and its shape changed from one project to a project, then the general thing the project is missing.

Worth recording as a pattern rather than as an entry, because it is not the first — Anki: hexadecimal grew out of a kata of Adam's the day before, and the list predicted neither: a question about a real format beats a row in a backlog, because the question already knows why it is being asked. The fifty above are still the ranking for when nobody is asking anything.

2026-09-08, and the pattern held the same day: "Supports Unicode" is a level, not a yes. From "what is different about Unicode regex?" and a link to UTS #18 ↗ — a question about a document this time rather than a format, and the same result. Nothing in the fifty above named it, and what the page found is not what the question implied: the sharpest measurement is not a missing \p{...} but \w, which is defined in the report as an expression over general categories and implemented by almost nobody that way, so one Hindi word is one match, two or three depending on the engine. Row 30 above is narrower for it.

Not on this list, on purpose¶

TOPICS.md marks three sections adjacent: streams and file descriptors (§3), parsers and grammars (§4.2), fonts and rendering (§6). They are real, they touch text, and the lesson belongs in a sibling library or nowhere yet. Two corners of them are core and have been pulled onto this list — items 40 and 41 — which is the point of marking the boundary rather than leaving it to drift.

Also deliberately absent: a second pass over anything already written. When a page here is wrong, that is a correction, not a backlog item; it goes in as a fix the day it is found.

And one item that belonged to a different library. Adam asked for two Perl tutorials to be followed. perlpacktut ↗ became Packing a record on 2026-09-08. The other, perlfaq5 ↗ — file I/O: opening, locking, slurping, $/, in-place edit, temporary files — was not this library's, and four of its questions (flushing, counting lines, why a read-write open wipes the file, renaming reliably) landed the same day in the Python library's Opening a file ↗, where the subject is files rather than encodings. The line between that page and this library's own Opening a file is written at the top of ours — this library owns the codec, that one owns the call — and ours should stay that narrow.

How to work through it¶

One page per sitting, the same as reading them (CONTRIBUTING.md):

python3 tools/run_examples.py --update --only <stem>_py   # then READ the key
python3 tools/check_all.py                                # every gate, CI's order

Then update the row in the chapter README and in ROADMAP.md, and strike the line here. A stub graduates by gaining an examples/ program and losing its notice — the URL never moves.