Roadmap¶
Level: reference
What is written, what is a stub, and what is next — in the order the stubs will be filled, which is the reading order.
Two companion pages, added 2026-09-06. This one tracks the pages that exist; TODO.md ranks the next fifty against the four checkpoints in 00_Start_Here — all four written as of 2026-09-08 — and TOPICS.md is the wider inventory it was drawn from: every term the subject touches, imported from a list Adam pasted and then extended with what that list was missing. Neither has been through the gates that back a lesson; both say so at the top.
Status¶
| Page | Status |
|---|---|
| A byte is eight bits | written, 2026-09-05 |
| Counting in hexadecimal | written, 2026-09-07 — Python, Rust and shell; the odometer with sixteen symbols per wheel, why 99 + 1 and FF + 1 are the same event, and the three things MSB can mean |
| Hex is a shorthand | written, 2026-09-05 |
| Reading a hex dump | written, 2026-09-05 |
| Grouping is a choice | written, 2026-09-07 — Python, Rust and shell; the width you group by is a claim about the unit, cryptii silently switches hex to binary at 5 and 6 bits and never switches back, and a 4-byte grouping "understands" an emoji exactly when its offset divides by four |
| Hex: a number, or a picture of bytes | written, 2026-09-07 — all four languages; one hex string is two objects, four parsers disagree about what a hex digit is, and int(s, 16) succeeding is not a validation |
| Arithmetic has its own width | written, 2026-09-07 — all four languages; a byte stores eight bits and almost nothing computes in eight, C promotes to int with no syntax for it, the type rather than the bits picks arithmetic against logical >>, and >> differs from / on seven of the eight values -8..-1 |
| The bytes do not say which end | written, 2026-09-12 — Python, Rust and shell; endianness had been marked core in TOPICS since the import and had no page, so the question the two lessons before it raise was answered nowhere. One file through six dump commands: three print the file and three print numbers, and a bare hexdump is the one whose byte order nobody chose. Two findings behind it — int.from_bytes has had a default since Python 3.11 and that default is 'big', which is not sys.byteorder on any machine you are likely to own (the correction to hex_number_or_bytes came out of checking it); and the two xxd builds pad a short line differently under -e, so the example normalizes whitespace rather than recording one platform's spacing |
| Which base did you mean? | written, 2026-09-12 — Python, shell and C; place value once for any base, the four notations for naming a base and the field that names none, [ 010 -eq 10 ] true beside [[ 010 -eq 8 ]] true in one shell, the dump's hex offset column that xxd -s will not read back, and inet_aton taking three spellings ipaddress refuses |
| A character is a number | written, 2026-09-05 |
| Rotation is not encryption | written, 2026-09-07 — Python, Rust and shell; ROT13 as arithmetic on the ASCII layout, the keyspace printed in a column, encodings/rot_13.py that str.encode still refuses, and the two things that break when the rotation leaves ASCII |
| Control characters | written, 2026-09-05 — the first page with a C view (examples/*.c, compiled by the runner since the same day) |
| The NUL byte | written, 2026-09-06 — Python, Rust, C and shell; one ordinary character that every UTF-8 validator accepts and almost nothing that carries text will carry, so the same zero byte is a terminator, a binary-file tripwire, the one separator a filename cannot forge, and a ValueError at four different doors |
| Code pages | written, 2026-09-05 — Python, Rust and shell; the agreement matrix is the original bit |
| Unicode code points | written, 2026-09-05 — the neighbourhood map: read U+XXXX as a block plus a house number, and measure the whitespace runs instead of memorising them |
| Writing a code point | written, 2026-09-06 — all four languages; one number and five spellings, and the claim that the shape of an escape says what a language thinks a character IS (scalar value, code unit, or a name) |
| The table has a version | written, 2026-09-06 — Python and Rust; python3 and rustc on one machine were a Unicode release apart, and the page is about which of the three kinds of fact may go in a key. Extended 2026-09-10 with the rules half: नमस्ते is three grapheme clusters to node, Swift and Perl 5.42 and four to both interpreters macOS installs in /usr/bin — and to ripgrep's PCRE2 — because GB9c arrived in UAX #29 for Unicode 15.1 |
| Noncharacters and the private use areas | written, 2026-09-08 — Python, Rust and shell; reserved is not invalid, with both sets generated rather than quoted (66 and 137,468, agreeing across two languages and cross-checked against the Co category). Three findings the backlog hook did not predict: ElementTree writes what it refuses to read, and XML 1.0 forbids only 2 of the 66 — U+FDD0 is a noncharacter its Char production is perfectly happy with; unicodedata.category cannot distinguish a noncharacter from a merely unassigned code point, since both are Cn; and APFS refuses a noncharacter in a filename with EILSEQ, the errno it gives invalid UTF-8, although EF BF BE is well-formed UTF-8 by every other measure — a sharpening of finding 11, and the one claim on the page that is in a dated fence rather than a key |
| Preparing a string | written, 2026-09-06 — Python and Rust; RFC 3454 ↗ stringprep, built on the frozen table the previous page explains, and the reason faß.de is two different domains |
| PRECIS: stringprep, after stringprep | written, 2026-09-08 — Python; the two profiles of RFC 8265 ↗ implemented from the RFC, and the finding that carries the page: U+200B is general category Zs in the frozen 2002 table Python still ships and Cf in a modern one, so PRECIS's own space-mapping rule maps it on one implementation and disallows it on another — the version problem the stringprep freeze existed to prevent, handed straight back |
| A code point is not a character | written, 2026-09-07 — Python and Rust; five rulers over one string, and the decision that only four of them may reach an answer key: the column count is computed from a width table written into the source, because a width class is not a property Unicode freezes and the terminal doing the drawing has its own table anyway. Extended 2026-09-08 with the half the page was asked for and did not yet have: the language crosswalk — .NET ships an enumerator and calls the unit a text element, Swift makes it the default Character, Rust puts it in a crate, Python has nothing — and a dated fence of four real segmenters, which produced two findings. PCRE2 10.45 matches a whole run of regional indicators as one \X where Perl, .NET and Swift pair them up; and one Devanagari conjunct splits 2–2 across the four, because GB9c arrived in Unicode 15.1. So the segmentation rules carry a version the same way the character data does |
| Case is not a per-character operation | written, 2026-09-08 — Python and Rust; three assumptions broken by three languages, and the Rust half is the sharp one: "ΟΔΟΣ".to_lowercase() and the same text through chars().flat_map(char::to_lowercase) give different strings, because a char has no neighbours and cannot know a sigma is word-final. Section 5 says out loud that the case mappings here are recorded as a bet — the frozen table Python ships carries name, category and decomposition, and no case mapping at all, so there is nothing to check them against |
| Where a line may break | written, 2026-09-08 — Python; enough of UAX #14 ↗ hand-rolled to break one Japanese sentence correctly where textwrap orphans the full stop onto its own line, and then the honest half: the same machinery is useless on Thai, where the boundary is a fact about vocabulary. textwrap gets the Thai right at width 4 and wrong at 3, 5 and 6, which is the most useful thing on the page — a test at one width would have passed |
| Confusables and scripts | written, 2026-09-06 — Python and Rust; eight of nine look-alikes survive every normalization form, east_asian_width is a red herring that splits Latin against itself, and the whole-script confusable defeats the detector |
| Logical and visual order | stub — added 2026-09-06; UAX #9 ↗, and why a terminal screenshot is not evidence about a string |
| Unicode in identifiers | written, 2026-09-06 — Python and Rust; NFKC vs NFC on identifiers, the three-row table where they agree once and diverge twice, and rustc's three UTS #39 ↗ lints on by default |
| "Supports Unicode" is a level, not a yes | written, 2026-09-08 — Python only, plus a dated cross-engine fence; UTS #18 ↗ turns the question into fifteen numbered requirements, and re fails five of Level 1's eight — it has no \p{...} at all. The measurement that carries the page is one Hindi word through three engines' \w+: one match, two, or three, and every one exits 0 |
rune is an int32 |
written, 2026-09-10 — Python and Rust, and Go since 2026-09-11 (the runner's first Go examples), plus a dated eleven-decoder census; Go's rune is an alias that checks nothing and .NET's Rune a checked scalar value, and Go writes one U+FFFD per byte where Python and Rust write one per maximal subpart — the two counts checked against each other over all 16,843,008 inputs of up to three bytes, with no exception |
| UTF-8 by hand | written, 2026-09-07 — the checkpoint page; all four languages, and Table 3-7 walked from both ends rather than quoted: 1,112,064 scalar values encoded and 1,112,064 byte sequences decoded, which is what makes it a bijection |
| Validation is a boundary | written, 2026-09-05 — all four languages, and the second C view |
| Overlong sequences | written, 2026-09-05 — all four languages; the shortest-form rule, Table 3-7, and the three formats that break it on purpose |
| UTF-16 and surrogates | written, 2026-09-07 |
| Byte order and the BOM | written, 2026-09-05 — Python, Rust and shell; why U+FFFE makes the mirror proof rather than convention |
| Encode and decode are verbs | written, 2026-09-05 — Python, Rust and shell |
| Mojibake | written, 2026-09-05 — Python and shell; the Rust point is the previous page's |
| Escaping into ASCII | written, 2026-09-06 — all three languages; four schemes that escape four different things, and the layer a field width is measured on |
| Binary to text | written, 2026-09-06 — all four languages; the 24-bit quantum, why base64 has no charset, sixteen spellings of one byte, and the power-of-two line the reference tables do not draw |
| The alphabet is not the encoding | written, 2026-09-07 — Python and shell; four kinds of Base32 that differ by one tr, the sort order base32hex buys, and Crockford's, which pads the other end and so disagrees at every length not divisible by five |
| An encoding is four layers | written, 2026-09-08 — Python and shell; UTR #17's four levels, the seven schemes and which two are compound, and the BOM-less UTF-16 default where the Unicode FAQ says big-endian and CPython says native |
| Normalization | written, 2026-09-06 — Python and Rust; chapter 4's first, and one of the few Unicode table lookups the stability policy lets an answer key hold |
| Bytes that are not text | written, 2026-09-07 — Python and Rust; PEP 383 ↗ mapped byte-by-byte and proved by exhaustion over all 65,536 two-byte strings, the surrogatepass confusion that changes a length with no error, and the JSON escape that writes an arbitrary byte into a filename |
| Encode, decode and errors | written, 2026-09-07 — Python; the third checkpoint. The eight handlers in BOTH directions (two of them raise TypeError on decode), and the finding that decides which to use: every marker the lossy ones insert is a character the input could already have contained, so none of them can be undone or detected |
str vs bytes |
stub — which is which, and why Python will not concatenate them |
| Opening a file | written, 2026-09-07 — Python; the page whose subject is the environment the harness fixes, so every default is read from a child interpreter whose variables are printed beside it. Two values could not be recorded at all: the C locale's name (US-ASCII on macOS against ANSI_X3.4-1968 on glibc, which codecs.lookup().name canonicalises to ascii on both) and sys.getfilesystemencoding(), which macOS hardcodes to UTF-8 where glibc, with UTF-8 Mode off, follows the locale |
| Bytes, hex and int | written, 2026-09-07 — Python; the four conversions and struct, with a PNG header built rather than quoted so its length and CRC are computed. The plan's own hook was stale and is corrected on the page: int.from_bytes stopped requiring byteorder in Python 3.11, so the argument that used to prevent the bug now has a default |
A str in memory |
written, 2026-09-07 — Python; PEP 393 ↗ measured as strides, ratios and thresholds so that not one byte total reaches the answer key. The UTF-8-cache demonstration had to change engine mid-write: sys.audit() fills the cache on CPython 3.14 and not on 3.11–3.13, which looked like a platform split until the same three versions were run on one Linux |
String is bytes that promise UTF-8 |
written, 2026-09-05 — Rust, Python and shell; the one figure in the repo, and valid_up_to() matched against Python's UnicodeDecodeError.start |
char is four bytes |
written, 2026-09-07 — Rust and Python; the fourth checkpoint. Four numbers for one character, and two findings beside them: the surrogate hole that makes char strict is the same hole that makes Option<char> free (4 bytes, against 8 for Option<u32>), and char as u8 compiles, truncates, and is correct for é — so the test a European developer is likeliest to run is the one that hides it |
| From UTF-8, and lossy | written, 2026-09-07 — Rust and Python; valid_up_to() is UnicodeDecodeError.start on every input tried, and error_len() is end - start wherever Rust reports one. The one place they part is the useful one: Rust's None says incomplete, a verdict Python's exception has no field for and only its incremental decoder can give. Both spend the same number of U+FFFD on all six inputs, which is Unicode's maximal-subpart recommendation holding rather than a guarantee |
| Slicing by byte | written, 2026-09-07 — Rust; the panic caught with catch_unwind so the page can show it, and is_char_boundary checked against the outcome on every index — it predicts all of them. The panic text is in a dated fence, not in a key — and the fence had to be rewritten hours after it landed, because a docker sweep of eight rustc versions found the wording changing twice in consecutive releases (1.95 added the start/end prefix, 1.96 dropped the sliced string), which is the rule vindicating itself on the same day. Plus the halfway point of one string that is a boundary and of another that is not |
OsStr, Path, and WTF-8 |
written, 2026-09-07 — Rust; the type as an API, since Bytes that are not text already owns the surrogateescape contrast. Three doors out and only into_string() hands the name back; Cow::Owned as the free answer to can this name be represented; and a correction the program made to its own page — Path::eq compares components, so a//b == a/b, and it was drafted as a byte comparison until the output disagreed. The Windows half is a dated fence read off the std source, because no runner here is a Windows machine |
| Inspecting a file | written, 2026-09-05 — the named-character row is fiction in two dialects, and the ? is the terminal's, not od's |
| The trailing newline | written, 2026-09-06 — shell, Python and Rust; why a two-byte file reports zero lines, and the marker your shell draws that is not in the file |
| A character and its bytes on one line | written, 2026-09-06 — shell and Python; the one-liner, and the three separate newline decisions that let it fit on one row |
Locale and LC_CTYPE |
written, 2026-09-06 — the locale is six independent variables; wc -c and wc -m collapse to one answer under C, LC_CTYPE does not touch sort order, and Python has declined to obey any of it since 3.7 |
| File type is four questions | written, 2026-09-07 — shell, Python and C; the four mechanisms that answer "what type is this", the anatomy of a magic rule, and xdg-mime against file on one unchanged file |
| The first two bytes | written, 2026-09-07 — shell and Python; the kernel compares offset 0 and decodes nothing, a BOM and a CR break a shebang two different ways, and the shell's rescue runs the broken file and exits 0 |
| Binary is a verdict, not a property | written, 2026-09-10 — shell, Python and Rust; c0 ff ee is an error to a UTF-8 decoder and text to file, grep and git, because grep and git decide binary by looking for a NUL — and a file of NULs is valid UTF-8, which is what the kata turns on. On a Mac the name picks the app through a UTI and nothing else: .bin is MacBinary (Archive Utility, and macbinary probe rejects the file), .raw a Panasonic camera image, and an extensionless PNG opens in TextEdit because Launch Services has no magic fallback. Found on the way and folded into the grep page and CONTRIBUTING: by default both greps withhold a line of invalid UTF-8 in a UTF-8 locale, and only GNU says so |
| Terminal hyperlinks, and the URI that is not one | written, 2026-09-06 — shell and Python; a clickable filename is an escape sequence wrapped around ordinary text, and the URI inside it percent-encodes a space and a # but not the accent — which the OSC 8 specification ripgrep's own manual links calls undefined |
printf writes bytes |
written, 2026-09-07 — shell; six ways to write two bytes, the string echo cannot print, and \u20ac giving three answers on three configurations where only one is a euro sign |
iconv |
written, 2026-09-07 — shell and Python; //IGNORE exits 1 on both platforms after doing what it was asked, //TRANSLIT is a per-implementation table, and the yes/no validator accepts a whole family of sequences above U+10FFFF that Python and Rust refuse |
file guesses |
written, 2026-09-07 — shell; five answers rather than four, and file reads 64 KiB — so one byte of padding either way turns a UTF-8 file into iso-8859-1 and then into us-ascii |
| A pipe is not a terminal | written, 2026-09-07 — Python, shell and Rust; a pty and a pipe running the same child, isatty moving exactly one property of sys.stdout, and the backlog's own hook coming out false on POSIX |
| A BOM in a CSV | written, 2026-09-05 — Python and shell; the decision procedure, and the two platform fingerprints |
| CRLF vs LF | written, 2026-09-07 — Python and shell; the byte that cannot appear in its own output, why the totals reconcile while the keys do not, and git reporting a tree clean when every line on disk differs from the blob |
| SAP code pages | written, 2026-09-07 — Python and shell; three numbering catalogues that all look alike, EBCDIC's alphabet in three pieces, the euro twins one byte apart, and the standing rule that every number is verified against the system |
| Mojibake round trip | written, 2026-09-07 — Python and shell; the deciding question asked before the repair, Latin-1 total against cp1252's five holes, and a guard whose false-positive rate is measured rather than assumed |
| Fixed-width byte fields | written, 2026-09-07 — Python and Rust; 9 of 26 field widths split this one value and 17 do not, so whether the naive cut is a bug is a property of the data; error_len() == None is the truncation signature as a value; and two ASCII spaces padding a UTF-16 field are a dagger |
| Packing a record | written, 2026-09-08 — Python, Rust and C; where four languages keep a binary layout, why struct's missing prefix means native sizes and alignment padding rather than no opinion, and the s format packing bytes so a nine-byte cut of a ten-byte name stops being text. Prompted by perlpacktut; a sibling to Fixed-width byte fields — that page is the text record, this one the binary one |
| Windows-1252 vs Latin-1 | written, 2026-09-07 — Python and shell; cp1252 keeps zero of Latin-1's disputed 32 bytes, so it is a replacement rather than a superset, and é cannot name the table while € can |
| Sorting and collation | written, 2026-09-07 — Python; the hook asked for three orders and one list produced five, byte-identical on three C libraries eight years apart, while the punctuation those libraries do disagree about is what glibc 2.28 changed and what silently invalidated PostgreSQL indexes for the two years before PG 13 could warn. Extended 2026-09-08 with the two halves the page named without owning: UTS #10's three levels (primary, secondary, tertiary — reproduced from a toy key, with the DUCET's own weights beside them) and deterministic sorting, where a correct collation still has no repeatable order because it makes ties on purpose |
| From the telegraph to Unicode | written, 2026-09-05 |
| Why UTF-8 won | written, 2026-09-05 |
| Why UTF-16 stayed | written, 2026-09-07 |
| A token is not a character | written, 2026-09-08 — Python; a byte-level BPE trained inside the program, because there is no stdlib tokenizer and no network, and a hand-rolled one teaches the mechanism better than an API call would. Two findings the build produced: the same Polish sentence costs 28 tokens against English's 8 on English-trained merges, and re-training the same 48-merge budget on a corpus half Polish drops Polish to 15 and raises English to 11 — a vocabulary is zero-sum across languages, which is the actual reason English is cheap. Deliberately absent: any token count attributed to a named model, since nothing here could check one |
| UTF-8 everywhere | written, 2026-09-05 |
| Rust strings in practice | written, 2026-09-05 |
| Python text in practice | written, 2026-09-05 |
| "Handles Unicode" is four questions | written, 2026-09-07 |
| Interfaces and storage | written, 2026-09-07 — Python; the library's closing rule, and the map has three groups rather than two: four channels carry the declaration inside the data, JSON needs none because the RFC fixes it, and CSV, fixed-width and a URL query have nowhere to put it and nothing fixing it |
grep on text that is not ASCII |
written, 2026-09-06 — shell and Python; the silent skip, measured, and the binary notice's two streams |
ripgrep — the Rust grep |
written, 2026-09-06 — no rg on either runner, so the session is dated and a Python example checks the rules |
| PCRE2 — the other regex engine | written, 2026-09-06 — shell and Python; the only engine already installed that can match a grapheme cluster, and the price is that a pattern the default engine would have rejected with a paragraph of advice now matches nothing and says nothing |
--pre and -z — decompress, then decode |
written, 2026-09-06 — shell; a stage before everything else, so a UTF-32 file is still unreadable through gzip and --pre is how you finally read it |
find, and filenames that are bytes |
written, 2026-09-06 — shell and Python; cat opens what find -name cannot see |
xargs splits on the wrong things |
written, 2026-09-06 — shell; one apostrophe breaks the pipeline, and -s is a byte budget so the encoding sets the batch count |
sed matches patterns, not bytes |
written, 2026-09-06 — shell; a sequence beats a byte set, and -i has no portable spelling |
awk is three programs |
written, 2026-09-06 — shell, verified identical under BWK awk, mawk and gawk; BWK awk disagrees with itself |
| The shell has no string type | written, 2026-09-08 — shell, byte-identical on bash 3.2 and bash 5.2; ${#var} has three answers and sh is a different program on the two runners, so the cross-shell tables are dated rather than keyed |
cut counts what it is told to count |
written, 2026-09-06 — shell; -c returns four bytes on Ubuntu and five on a Mac |
tr and sort work a byte at a time |
written, 2026-09-06 — shell; deleting é damages the word next door |
diff compares lines, cmp compares bytes |
written, 2026-09-07 — shell and Python; two spellings of café that diff prints identically, and three BSD/GNU splits, one of them a binary heuristic that reads the whole file on a Mac and the first 8 KB on Linux |
split, paste, look and tee |
written, 2026-09-07 — shell and Python; paste -d 'é' becomes two delimiters, split -b leaves two pieces of four that are not text, look misses three words that are in the file, and tee is the control |
touch, : > and install are not three spellings of one command |
written, 2026-09-07 — shell, byte-identical on both platforms; five of the seven idioms truncate the file and one replaces the name, proved with a hard link rather than an inode number — which came back the same on one run and different on the next, on both platforms |
hexdump is a format engine wearing six presets |
written, 2026-09-06 — shell, byte-identical on both platforms; the six presets typed out as -e strings, and the * that turns 128 characters of hex into 4 |
xxd is the dump you can put back |
written, 2026-09-07 — shell; the round trip, the text column it throws away, the offsets it seeks to, and -E reading the same bytes as EBCDIC |
od reads types, not bytes |
written, 2026-09-07 — shell, tidied because it must be; -t as a C type, stacked readings in one pass, and four ways the two implementations disagree |
strings has a printable set, not an encoding |
written, 2026-09-10 — shell and Python; the 95 bytes both builds keep, and the two files they cannot see into (an accented word, UTF-16). Then, in dated fences, everything the builds disagree on: a different default portion of the file on each platform, an -a on the Mac that is not POSIX's, two input paths on one Mac that give different answers, a last byte printed that no table calls printable, and a GNU -U locale that keeps only the first byte of every UTF-8 character |
| Typing a character you cannot type | written, 2026-09-07 — Python; the Compose file and Vim's digraph table parsed rather than described, and they overlap on 611 of the 2,580 characters they can type between them |
uni — the character's name |
written, 2026-09-06 — dated sessions plus a unicodedata example printing the same columns |
uni -h, line by line |
written, 2026-09-10 — Python; the help screen verbatim in dated fences with every keyword linked, and an example that recomputes the columns needing no table. Five places the screen and the program disagree, two of them bugs in uni's data: a width column that swaps N and Na, and a keysym column keyed by the keysym's number instead of the code point |
| The five worth installing | written, 2026-09-06 — each optional tool measured against a machine-checked baseline |
| UTF-7, and the seven-bit transport | written, 2026-09-08 — Python and Rust; the encoding built for a seven-bit mail relay, and the count that killed it: 2,961 legal spellings of <script>, all verified through the decoder, against the one an encoder writes. Rust's std has no UTF-7 at all, which is the finding rather than a gap — and BSD iconv does not absorb a trailing -, so it fails to round-trip its own output, exit 0 both ways |
| Two readers, one byte string | written, 2026-09-06 — Python and shell; the GBK trail byte that eats a backslash, and a UTF-7 payload that is valid UTF-8, valid ASCII and a script tag |
| The check that ran too early | written, 2026-09-06 — Python and Rust; ℀ is one code point that NFKC turns into a path separator, and std has no normalisation at all |
| Two people, one account | written, 2026-09-06 — Python and Rust; four folds, five pairs, and the row where only the uppercase fold merges them |
| The byte that means something to somebody else | written, 2026-09-06 — C, Python and shell; 32 bytes stored and 14 bytes long, and a log line that erases itself |
| What you see is not what runs | written, 2026-09-06 — Python and Rust; rustc denies two of the five lints by default and replaces the override with U+FFFD in its own error |
| The encoding man pages nobody opens | written, 2026-09-07 — shell; the section-5 encoding pages, multibyte(3), and a sweep recipe over a corpus the script builds |
| A page has a date | written, 2026-09-07 — Python; man 5 utf8 is dated 2004 and cites RFC 2279 ↗, and both tables are regenerated from arithmetic |
| What the page does not say | written, 2026-09-07 — Python; //TRANSLIT works and appears in none of the 17,264 man pages on this Mac |
| 15_Hex | written, 2026-09-12 — Python and Rust; 010 Editor's Hex Operations dialog, one page per operation. The dialog as one Python function, and a table that tries every input to find which of Treat Data As, the sign and the Endian toggle can change each operation's bytes: Binary Invert is untouched by all three, only five operations are known to care about the sign, and a rotation ignores the toggle on a Short but not on an Int. Nothing here is measured against the dialog itself; the questions the manual leaves open are listed on the chapter page |
| Assign | written, 2026-09-12 — Python; one operand and four byte patterns, Assign 0xFEFF writes either byte order mark, and Operand Step builds the 256-byte table of which Windows-1252 refuses five |
| Add | written, 2026-09-12 — Python; a carry stops at the edge of a value, so Add 0x10 on F8 00 has three answers, and Add 13 turns café into pnsж |
| Subtract | written, 2026-09-12 — Python; zero minus one is all ones in every type, Subtract 1 is Add 0xFF, and Subtract 0x20 uppercases letters and turns a space into NUL |
| Multiply | written, 2026-09-12 — Python; the low half is the same signed or unsigned, Multiply 2 can write only 128 byte values, and Multiply 171 undoes Multiply 3 where Divide 3 restores 86 bytes of 256 |
| Divide | written, 2026-09-12 — Python; the first operation the sign changes (F0 divided by 2 is 78 or F8), C's rounding against Python's //, and 64 Signed Bytes on which Divide 2 is not Shift Right 1 |
| Negate | written, 2026-09-12 — Python; invert then add one, the same bytes signed or unsigned, 80 as its own negative, and a Float's −0.0 that equals 0.0 and is not the same bytes |
| Modulus | written, 2026-09-12 — Python; Modulus 16 is the low hex digit only when unsigned, and which sign a remainder takes in C, Python and ABAP |
| Set Minimum | written, 2026-09-12 — Python; a floor of 0x20 turns control characters into spaces, changes 32 byte values unsigned against 160 signed, and turns A, tab, B in UTF-16LE into ⁁†⁂ when the values are bytes |
| Set Maximum | written, 2026-09-12 — Python; a ceiling of 7F catches every UTF-8 byte outside ASCII or none of them, disagrees with And 7F on 127 bytes, and an empty range lets the second limit win |
| Swap Bytes | written, 2026-09-12 — Python; UTF-16LE to UTF-16BE in place with its mark, the Endian toggle unable to change it on all 65,536 Shorts, and a range one byte late that misaligns every pair |
| Binary And | written, 2026-09-12 — Python; And 7F is a seven-bit channel, and And DF capitalises café correctly and writes ŁÓDŚ for Łódź, because ź and its capital are not 0x20 apart |
| Binary Or | written, 2026-09-12 — Python; Or 20 lowercases ASCII, turns all 30 two-byte UTF-8 lead bytes into lead bytes for longer characters, and can never be undone |
| Binary Xor | written, 2026-09-12 — Python; Xor twice gives the file back, a one-byte key falls to one known letter, and Operand Step hides repeated bytes without hiding the key |
| Binary Invert | written, 2026-09-12 — Python; the one operation no setting changes, checked on every two-byte input under every type, sign and byte order |
| Shift Left | written, 2026-09-12 — Python; the top bit falls off, Shift Left 1 is Multiply 2, and on a little-endian Short the bits cross rightward on screen |
| Shift Right | written, 2026-09-12 — Python; zeros or copies of the sign bit at the top, both printed, because C leaves the signed case to the implementation and the manual does not say |
| Block Shift Left | written, 2026-09-12 — Python; the range as one number, so Block Shift Left 4 deletes a hex digit, and the manual's sentence read two ways on a little-endian Short |
| Block Shift Right | written, 2026-09-12 — Python; Block Shift Right 8 inserts a NUL and pushes out the last byte, and two candidates for what enters at the top of a signed range |
| Rotate Left | written, 2026-09-12 — Python; nothing is lost, Rotate Left 4 swaps a byte's two hex digits, and Rotate Left 8 on a Short is Swap Bytes |
| Rotate Right | written, 2026-09-12 — Python; Rotate Right 1 turns +1 into −128, and the Endian toggle cannot change a Short's rotation but can change an Int's |
| 14_Resources | written, 2026-09-07 — the resources chapter, not a chapter of the course |
| Anki: hexadecimal | written, 2026-09-07 — 22 cards over Python, Rust, C and bash, each snippet run by the deck's own verify.py; grew out of a seven-question kata of Adam's, one of whose answers it corrects |
| The hard strings | written, 2026-09-08 — Python, Rust and shell; the second resource, and the one you point at your own code. Nineteen strings, one per behaviour, and a sixteen-pair matrix whose per-column counts are generated — which is how the page's own prose got corrected twice while it was being written: casefold() merges six pairs and only three of them are about letter case, and Rust's to_uppercase() merges the Turkish dotless i onto ASCII I where casefold() keeps them apart, so comparing by uppercasing does not merge fewer pairs than a fold, it merges the wrong ones. Carries the one row neither of this library's languages can produce — locale-dependent case mapping — as a dated Java fence |
| The typical words | written, 2026-09-10 — Python; the third resource, and the other half of CAST.md: the words other people's tutorials, test suites and bug reports reach for, each mapped to the one property it is there to show and to the page — and cast member — that shows it here. Two things the measuring turned up: Windows-1258 holds Nguyễn only in a spelling neither NFC nor NFD produces (e-circumflex composed, the tilde after it), and of every codec in Python's alias table only the UTFs and GB18030 can write नमस्ते at all |
| Tribit — the specification | written, 2026-09-05 — the Rust implementation is Adam's project |
| A record has to say what it is, how long it is, and whether it arrived | written, 2026-09-08 — Python, Rust and shell; record type, length and checksum worked on Intel HEX against Intel's own 1988 specification, which turns out to define no line terminator at all — the mark and RECLEN make the stream self-delimiting, and a three-record file with nothing between the records parses |
| RESOURCES.md | written, 2026-09-05 — re-check the links when a page graduates and cites one |
Written out of order, on purpose¶
Chapter 11 was written on 2026-09-06, before chapters 4–7, for the same reason chapters 9 and 10 were: it does not depend on them. Chapter 13 followed on 2026-09-07 for a third reason: it is the only chapter whose subject is on the reader's machine already, so it can be read before anything else and costs nothing to try. It is also the chapter a reader arrives at from outside — somebody whose search did not match, or whose find came up empty, has a concrete problem today and no reason to have read chapter 3 first. Each page states what it needs and links back.
Chapters 9 and 10 were written before chapters 3–7 because they are the two that do not depend on them. 09_History explains the shape of everything the stubs will say, so it makes the remaining pages read as conclusions rather than commandments; 10_Best_Practices is the answer a reader most often arrives wanting, and leaving it until last would have meant a library that could explain every trap and never say what to do. Both link forward into the stubs, so filling those in adds detail under an argument that is already made.
Chapter 3's last two lessons went the same way on 2026-09-05, and for a smaller reason: Encode and decode are verbs and Mojibake are the pair that names the whole problem, and the front page had been describing that problem with nowhere finished to send a reader. Neither needs UTF-8 by hand — they are about the table being an argument, not about the bit-packing inside any one table — so the checkpoint page stayed next after code_pages, and it landed on 2026-09-07.
Deliberately not yet¶
- A C track. Not a fourth language; a short The C view aside where it sharpens the point, compiled and checked like the others. The runner takes
examples/*.csince 2026-09-05; Control characters has the first one (NUL andstrlen) Validation is a boundary the second (the validator nobody writes for you), and Overlong sequences the third (the three comparisons that are the entire shortest-form rule). - A Go view. On the same terms as the C one: not a track, an aside for the pages where what Go itself does is the subject, built and checked like every other example. The runner takes
examples/*.gosince 2026-09-11 —go build, one file, stdlib only, sealed off from anygo.mod— and CI installs the current stable Go on both runners;runeis anint32has the first two. The next candidate is already on the site and still measured by hand: the Go row of "Handles Unicode" is four questions. - Katas. The sibling libraries keep exercises on the page with a compiled solution. Worth adding once chapter 3 is written, because "encode this code point by hand" is the natural first kata and it needs the UTF-8 page to point at. Until then RESOURCES.md points at the outside katas, and the Tribit project is the big one. The Anki deck added on 2026-09-07 is not this, and does not discharge it: a card asks for one fact back in twenty seconds, a kata asks you to build something and hands you a solution to diff against. They fail differently, so the library wants both.
- Tribit version 2 (self-synchronising) and the Rust crate itself. Both are Adam's exercises, listed at the bottom of the spec; when the crate exists, link it from the spec page rather than vendoring it here.
- Polish sections (
## Po polsku). The Rust library carries them on its ownership pages. Relevant here — Latin-2, Windows-1250 and the Polish letters' decomposed forms are all lessons — but the terminology table comes first. - ~~The locale lesson's UTF-8 half.~~ Resolved 2026-09-06. The worry was that a UTF-8 locale might not exist on both runners. It does —
C.UTF-8is present on macOS 26.6 and on Ubuntu, and the example picks one by asking each candidate for itscharmapand keeping the name out of the output. Both examples verified byte-identical on macOS and Linux (Python across 3.11–3.14) before recording. The one thing that genuinely could not be recorded istr, now a dated table on the page and a seventh entry in CONTRIBUTING's BSD/GNU list. file's wording. Differs between versions, so that page will record only--mime-encoding.
Concept backlog — the columns nothing here answers yet¶
Assembled 2026-09-06 by reading uni -h as an inventory rather than a manual. Its -f placeholders are one line per question the Unicode Character Database can answer about a character, and six of them have no page in this library. Each entry below is listed with the fact that makes it a lesson rather than a topic — all of them measured on this Mac the day the list was written, so a page can start from a hook instead of a definition.
- ~~The fifth length: how many columns.~~ Written 2026-09-07 as the graduated A code point is not a character, and the half of this bullet that predicted the method turned out to be wrong in an instructive way.
unicodedata.east_asian_width()is standard library, so the width claim looked machine-checkable — but it is a table lookup with no stability guarantee behind it, and recordingNa W A Nwould have made the key a fact about whichever Python ran it, on a page whose two runners were already a release apart. So the page keeps its width table in its own source, and theeast_asian_widthreading sits in a dated fence instead. The%(width)observation survives intact and is the page's sharpest line:éis ambiguous, which is the table saying the answer depends on a context no string carries — and the family emoji has three defensible column counts, none of which is a property of the string. - ~~Writing a character where only ASCII is allowed.~~ Written 2026-09-06 as Escaping into ASCII, the four schemes and the layering rule; what it did not take is
%(html)/%(xml), the entity forms, which belong with an HTML page this library does not have yet.%(html),%(xml)and%(json)are three answers to one question this library never asks, and it is the question a SAP interface asks every day:é,é,é,%C3%A9,=?utf-8?b?…?=,xn--. The hook isjson.dumps("😀")→"\ud83d\ude00"— a UTF-16 surrogate pair inside a format that is UTF-8 by RFC 8259 ↗, which is UTF-16 and surrogates leaking into a place that has no UTF-16 in it. Percent-encoding is the same lesson upside down:%C5%BCisż's UTF-8 bytes written in hex and nothing else. Fed Interfaces and storage, written 2026-09-07, which took the URL-encoded query string and the JSON escape form and spent its own words on where each channel declares the encoding rather than on how each one escapes. - ~~How you type it.~~ Written 2026-09-06 and 2026-09-07, in two halves that turned out to want different chapters. The source-code half is Writing a code point —
'\N{EURO SIGN}','\u{20AC}', the fixed widths against the braces, and C's universal character name. The keyboard half is Typing a character you cannot type, in11_Toolsbecause the compose key, Vim's digraphs,uni printand a macOS keyboard layout are four tools for one job. What the second one found, and what makes it a lesson rather than a how-to: the Compose file and Vim's table are not two spellings of one thing — 1,884 code points against 1,307, overlapping on 611. - The name is not always a name. The NUL half of this has landed on The NUL byte, which records both
unicodedata.name('\x00')raising andlookup('NULL')succeeding. What is still unwritten is the general rule and its other half:uniprintsNULLbecause the UCD gives control characters no Name at all andNULLis a Name Alias, and Python will not go back the other way from the string uni displays (lookup('LINE FEED (LF)')is aKeyError). And the reason names are frozen is legible in one row: U+FE18 is PRESENTATION FORM FOR VERTICAL RIGHT WHITE LENTICULAR BRAKCET, a typo that can never be corrected because Name is a stability guarantee. Belongs beside The table has a version, which already turns on which facts a key may contain. - Confusables, invisibles and the security page.
%(refs)is the UCD's own cross-reference list (U+2044 FRACTION SLASH points at U+002F and U+2215), and%(props)names the three characters that make text lie: U+200D ZERO WIDTH JOINER (Join Control), U+FE0F VARIATION SELECTOR-16, and U+202E RIGHT-TO-LEFT OVERRIDE (Bidi Control — the Trojan Source ↗ class of bug). Cyrillicаand Latinaare the other half. The library has the tools for this page already —uni identifyis the diagnosis, normalization is the partial cure, and Preparing a string supplies the history: RFC 3454 ↗ prohibited U+202E by number in 2002, nineteen years before Trojan Source was published. The confusables half — one script imitating another, which stringprep explicitly declines to address (§9.1: it will not map look-alikes together, because that needs context a protocol does not have) — is now outlined as Confusables and scripts, so what this bullet still owns is theunicolumns themselves:%(refs)and%(props)as a diagnostic workflow rather than a concept page. - Two databases, not one. uni's names come from the UCD; its emoji search matches CLDR keywords (
uni emoji firefightermatches onfiretruck, a word not in the character's name), and-tone/-genderassemble sequences:👩🏻🚒isU+1F469 U+1F3FB U+200D U+1F692, four code points and one grapheme. That split — character data versus locale data — is also the missing half oftrandsort, since collation order is CLDR's answer and not the UCD's.
One column needs no page of its own and is worth borrowing: %(unicode) prints the release a character was first assigned in (😀 = 6.1, A = 1.1), which is the concrete version of the argument The table has a version makes with two tools disagreeing.