Skip to content

Resources

Level: reference · the outside reading

Every link here was fetched and checked on 2026-09-05 (GET, following redirects; YouTube links through the oEmbed endpoint, which refuses a wrong video id). The Stack Overflow scores quoted below were read the same day through api.stackexchange.com, which is also how that page was fetched — stackoverflow.com itself refuses an automated request. Treat them as a floor: a vote count drifts upward, and the ordering it implies is what matters, not the digits. A link is listed because it teaches something a page here does not, or teaches it better; each entry says which. Anything advocacy-shaped says so. The print books under Books link to Open Library rather than a publisher: O'Reilly's own pages answer an automated GET with 403, and one author's domain has since been sold to a merchandise shop — which is the argument for checking a link by fetching it and reading what comes back, not by trusting the status line.

What is the difference between UTF-8 and Unicode? ↗ — Stack Overflow, 2009, 788 votes on the question and 21 answers. Worth reading, and worth reading three answers of, because they answer three different questions:

  • The accepted answer (679 votes) is the history: ASCII's 7 bits, the eighth bit that every language filled differently (the ISO-8859 family), why one byte could never hold more than one language, and then Unicode as one number per character with UTF-8, UTF-16 and UTF-32 as three ways of writing that number down. It is 02_Characters in six paragraphs.
  • The second answer (531 votes) is the one to do with a pencil: it takes one Chinese character, 汉 at U+6C49, writes its 16 bits, and pushes them through the UTF-8 template table (0xxxxxxx / 110xxxxx 10xxxxxx / 1110xxxx 10xxxxxx 10xxxxxx / 11110xxx …) to get E6 B1 89. That is exactly the exercise on UTF-8 by hand, and it is the single most useful thing in the thread.
  • Jon Skeet's answer (254 votes) is the terminology: Unicode is a coded character set (characters ↔ integers), UTF-8 is an encoding (bytes ↔ characters), and when a platform says "Unicode" as the name of an encoding — .NET's Encoding.Unicode, Java, and SAP's internal form — it means UTF-16, surrogate pairs and all. That confusion is the reason UTF-16 and surrogates exists.

One thing the thread does not settle, because nobody asked: that a code point is still not what a person calls a character. A code point is not a character picks up there.

The two articles every programmer is told to read

What it gives you Note
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets ↗ — Joel Spolsky, 2003 The essay that named the problem: there is no such thing as plain text; a string without an encoding is meaningless. Also the origin of "mojibake" as an English word. Twenty years old; the mechanics are right, the Windows-centric advice is dated.
What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work with Text ↗ — David C. Zentgraf Joel's article rewritten with the bytes shown, and with PHP as the cautionary example. The clearest single explanation of decoding under the wrong table. Best read after chapter 1 here, so the hex means something.

Deeper articles, in a sensible order

What it gives you
A Programmer's Introduction to Unicode ↗ — Nathan Reed The best modern survey: planes, encodings, combining marks, grapheme clusters, normalization, with a diagram of which parts of the code space are actually used. Read after chapter 3.
The Absolute Minimum … in 2023 ↗ — Nikita Prokopov Joel's article redone for today: why len() is a lie in every language, graphemes, and a table of what each language's string length actually counts. Opinionated and right.
It's Not Wrong that "🤦🏼‍♂️".length == 7 ↗ — Henri Sivonen One emoji, and every honest answer to "how long is it": 17 UTF-8 bytes, 7 UTF-16 units, 5 code points, 1 grapheme. The definitive version of the "five answers" table.
Let's Stop Ascribing Meaning to Code Points ↗ — Manish Goregaokar Why Rust gives you chars() and refuses to call them characters. From one of Rust's Unicode maintainers.
Unicode is harder than you think ↗ — Marco Cilloni A tour of the traps with C, C++ and Rust code; good on wchar_t and why it was a mistake.
Pragmatic Unicode ↗ — Ned Batchelder, PyCon 2012 The Python one: the unicode sandwich (decode at the edges, str inside), and five facts of life. Talk video linked from the page.
Dive Into Python 3, chapter 4: Strings ↗ — Mark Pilgrim Still the best gentle Python chapter on str vs bytes, with the "everything is bytes" opening this library shares.
UTF-8 history ↗ — Rob Pike The email from 1992 in which Ken Thompson's UTF-8 design is written up over dinner. Short, and it shows the lead byte announces the length idea being invented.
Hello World, or Καλημέρα κόσμε, or こんにちは 世界 ↗ — Pike & Thompson The Plan 9 paper that first shipped UTF-8. Section 2 is the encoding; the rest is what changed in a whole OS when text stopped being bytes.
UTF-8 and Unicode FAQ for Unix/Linux ↗ — Markus Kuhn Dated, encyclopaedic, and still the reference for locale, LC_CTYPE, and how a terminal decides what to draw. Backs 06_Terminal.
UTF-8 Everywhere ↗ A manifesto: use UTF-8 for storage and interchange, never UTF-16. Advocacy — persuasive, and the argument against wchar_t is correct, but read it as a position.

The standards, when you need the actual rule

Use it for
WHATWG Encoding Standard ↗ The set of encodings every browser actually implements — and a closed one: "User agents must not support any other encodings". A table of every encoding and every label a document may claim, which is where the web-only facts come from: latin1, iso-8859-1 and even ascii are all synonyms for windows-1252, so byte 0x80 decodes as € under all three. The source for Code pages, Windows-1252 vs Latin-1 and file guesses.
RFC 3629: UTF-8 ↗ Six pages. The byte templates, the overlong prohibition, the surrogate prohibition. The only spec in this list short enough to read whole.
RFC 2152: UTF-7 ↗ Ten pages, Informational, May 1997. Set D, Set O and Set B on one screen — read Set O first, because the fact that < and > are optionally direct is the entire security story. It also tells you where it belongs: seven-bit transports such as mail, and UTF-8 everywhere else. Backs UTF-7, and the seven-bit transport.
UTF7Encoding ↗ · SYSLIB0001 ↗ How a platform retires an encoding. Obsolete since .NET 5, still documented through .NET 11, with the three reasons written down: no longer in wide use, forbidden in interchange, and used as an attack vector. The class page adds the fourth — it provides no error detection.
UTS #10: the Unicode Collation Algorithm ↗ Why an alphabetical list is not a property of the text. The three comparison levels — base letters, accents, case — are section 1.1 and Table 2, and they are the vocabulary that makes ordering discussable at all. Appendix A is the one to read even if you skip the rest: four pages separating a stable sort from a deterministic sort from a deterministic comparison, and arguing against the last. The default weight table is allkeys.txt ↗, three weights per character; the per-language tailorings that override it are CLDR ↗. The source for Sorting and collation.
The Unicode Standard, chapter 2: General Structure ↗ Code points vs code units vs encoding forms, stated by the people who defined them. Section 2.5 is the diagram.
…§2.5, Encoding Forms, pinned to 17.0.0 ↗ The same section with an edition number in the URL, which is the form to paste into a bug report or a ticket: versions/latest/ silently follows the standard, so a link that was right last year can be quoting a different sentence this year — the habit The table has a version argues for, applied to a citation rather than to a number. The paragraph itself is the one that says UTF-8, UTF-16 and UTF-32 are three encoding forms of one set of code points and that each is equally legitimate, which is the sentence under every argument in 03_Encodings. Its chapter 3 sibling, §3.9.3 ↗, holds Table 3-6 (the bit distribution) and Table 3-7 (the well-formed byte sequences), which UTF-8 by hand checks by exhaustion rather than quoting.
Unicode FAQ: UTF-8, UTF-16, UTF-32 & BOM ↗ Every BOM question, answered by the consortium.
UTS #18: Unicode Regular Expressions ↗ The document that turns "does it support Unicode" into a number. Level 1 is eight numbered requirements and Level 2 is seven more; Level 3 has been retracted. Annex C is the one to read first — it is where \w, \d and \s are defined as expressions over real properties, which is the definition most engines do not meet. The source for "Supports Unicode" is a level, not a yes.
Unicode FAQ: Private-Use Characters, Noncharacters & Sentinels ↗ The 66 noncharacters and the 137,468 private-use code points, from the consortium — including the answer to "are noncharacters invalid in Unicode strings and UTFs?", which is absolutely not. The source for Noncharacters and the private use areas.
UAX #29: Unicode Text Segmentation ↗ Where one character ends and the next begins, as a numbered list of boundary rules — GB1 to GB999 over the Grapheme_Cluster_Break property. The definition behind the fourth ruler on A code point is not a character, and the annex to open when two libraries give one string two different cluster counts: GB9c, the Indic conjunct rule, arrived in Unicode 15.1, so implementations split by which revision they were built against. Also defines word and sentence boundaries, which nothing here uses yet.
Character encoding in .NET ↗ Microsoft's introduction, and the page that prompted the .NET half of A code point is not a character — read while asking whether this library had a grapheme-cluster page, and noting that .NET calls the unit a text element. Worth it for that specifically: .NET is one of the very few standard libraries shipping an enumerator for the unit (System.Globalization.StringInfo), which is the crosswalk row Python cannot fill. Read its terminology as .NET's own — Encoding.Unicode means UTF-16 there, as Jon Skeet's answer above warns.
Unicode code charts ↗ The PDFs: every block, every glyph, every number.
Unicode Character Encoding Stability Policies ↗ What Unicode promises never to change. Among the promises: a character's code point, name and decomposition, once it is assigned, and the set of noncharacters. Not among them: a letter's General_Category (only Cc, Co and Cs are frozen) or a character's width. The rule that decides which numbers may go in an answer key, on The table has a version, Noncharacters and the private use areas and Normalization.
Wikipedia: UTF-8 ↗ · Mojibake ↗ · Windows-1252 ↗ · Byte order mark ↗ · Baudot code ↗ · ISO/IEC 2022 ↗ · Variable-length quantity ↗ · Base64 ↗ The tables. Wikipedia's UTF-8 page has the template table and the history of how the 6-byte form was cut to 4; the 1252 page has the 32-byte difference; Baudot and 2022 are the ancestors of the Tribit project's CAPS and ESC.
Intel: Hexadecimal Object File Format Specification, Revision A, 1988 ↗ · Wikipedia: Intel HEX ↗ · Wikipedia: SREC ↗ Eleven pages, and the shortest complete format specification in this list — a worked example of framing rather than of encoding. The Wikipedia article is what prompted A record has to say what it is, how long it is, and whether it arrived; the Intel PDF is what every field width, record number and checksum rule on that page was checked against, and it settles two things a summary rounds off: the record mark is not in the checksum, and the format defines no line terminator at all. Motorola's S-record is the same three fields with different choices, which is the useful thing to read beside it.
Danny Cohen: On Holy Wars and a Plea for Peace, IEN 137, 1980 ↗ · 010 Editor: Introduction to Byte Ordering ↗ The two ends of the byte-order question, and both are short. Cohen's eight pages are where big-endian and little-endian enter computing, out of Swift's Lilliput, on an April Fool's date; the argument is that neither order is better and that the cost is only in failing to agree, which is where network byte order comes from. The 010 Editor page is the other end — what the question looks like once it is a control you can click, with LIT / BIG in a status bar and "most tools and the Inspector use this endian setting". Vendor documentation, so a definition rather than a verdict; both are the sources behind The bytes do not say which end.
Microsoft: Code pages ↗ Windows' numbering (1252, 1250, 65001 = UTF-8). SAP's numbering is different and is on SAP code pages.
W3C: Character encodings for beginners ↗ The gentlest official page; the "what is an encoding" many tutorials paraphrase.

When you want to see one implemented

Not reading for now, and not a tutorial — the answer to "how hard is all of this, really?", for after chapter 3 and for the Tribit project.

encoding_rs: a Web-Compatible Character Encoding Library in Rust ↗ — Henri Sivonen, 2018-12-03, twenty-one thousand words on implementing the whole Encoding Standard above, in Rust, for Firefox. It has two halves and only the first is for a learner. Everything through The API Design is about the decisions the Tribit exercises walk straight into: what a decoder returns when the output buffer fills in the middle of a character, why the caller allocates the buffer, why "the input has ended" has to be an explicit flag rather than an empty slice, and what a BOM does to a decoder that has already started. The second half — SIMD, lookup-table compression, benchmarks — is for people optimising one, and can be skipped without loss.

Three things worth taking from it even if you read no further:

  • Age is not safety. uconv, the library it replaced, was written in 1999 and had a buffer overrun found in it in 2016, in code added in 2001.
  • The bugs were in the boring part. The memory-safety problems clustered in the legacy CJK decoders — the encodings nobody thinks about, not UTF-8.
  • Legacy encodings are not history. Sivonen's own bank served him ISO-8859-15, and Japanese news sites still published new articles daily in Shift_JIS. That is the same argument as 07_Real_Data, made about the Web instead of about SAP.

Dated in one place, and usefully so: it puts the Web at "over 90%" UTF-8 while questioning W3Techs' method for counting ISO-8859-1 apart from windows-1252 — a distinction the Encoding Standard says does not exist. W3Techs now says 99.0% (September 2026), so the remaining 1% is the whole reason that library is as large as it is. The crate is encoding_rs on docs.rs ↗; it cannot appear in an example here, since Rust examples in this library are bare rustc with no crates.

Language documentation

Python Rust
Unicode HOWTO ↗ — the official essay; read the Reading and Writing Unicode Data section before Opening a file The Book, 8.2: Storing UTF-8 Encoded Text with Strings ↗ — why s[0] does not compile, in the language's own words
codecs — Standard Encodings ↗ — the table of every codec name and alias (latin_1, cp1252, utf_8_sig, utf_16_le) char ↗ — Unicode scalar value, four bytes, and the is_* / len_utf8 methods
unicodedata ↗ — name(), category(), normalize() str ↗ · String ↗ — from_utf8, from_utf8_lossy, as_bytes, is_char_boundary
Fluent Python, 2nd ed. — example code ↗ — the 04-text-byte folder is the book's Unicode chapter, runnable unicode-segmentation ↗ — the grapheme-cluster crate std deliberately lacks
core::str::validations ↗ — the actual UTF-8 validator behind from_utf8, 300 lines, readable
PEP 540 — UTF-8 Mode ↗ — why sys.flags.utf8_mode is already 1 under the C locale, with nothing set; behind Locale and LC_CTYPE
ABAP keyword documentation ↗ — search code page, xstring, cl_abap_codepage; the reference for every ABAP claim on the 07_Real_Data pages
perlpacktut ↗ — Perl's tutorial for pack/unpack; the template string Python's struct format is a family resemblance to, and what prompted Packing a record. Its companion perlfaq5 ↗ is file I/O rather than encodings

Books

None of these is required, and only the first group is about this library's subject. That group follows the chapters; the three after it are the neighbouring subjects — what happens to a code point after it is decoded, what people do with text once the bytes are settled, and how a machine searches bytes quickly. Each is listed because it teaches something no page here does.

Alongside the chapters

  • Charles Petzold, Code: The Hidden Language of Computer Hardware and Software, 2nd ed. (2022). Chapters on Morse, Braille, bits, bytes and ASCII — the 01_Bits_and_Bytes chapter here, told slowly and beautifully. The companion site codehiddenlanguage.com ↗ has the interactive circuits.
  • Luciano Ramalho, Fluent Python, 2nd ed. (2022), chapter 4 "Unicode Text Versus Bytes". The best Python chapter on the subject: str/bytes, codecs, errors=, BOM, normalization, sorting, and dual-mode APIs. The code is linked above.
  • Jim Blandy, Jason Orendorff & Leonora Tindall, Programming Rust, 2nd ed. (2021), chapter 17 "Strings and Text". char, String, &str, formatting, and a proper section on Unicode — the Rust companion to Ramalho's chapter.
  • Jukka Korpela, Unicode Explained (2006). The thorough one: characters vs glyphs, the properties, the encodings, with more history than you will use. Old, but Unicode's foundations have not moved.
  • Richard Gillam, Unicode Demystified (2002). For when you want to know how normalization and bidi actually work. Reference, not reading.
  • Henry S. Warren Jr., Hacker's Delight, 2nd ed. (2012) ↗. Chapter 1 as an adult: what can be done to a word of bits with no branch and no table. Not a book about text at all, but it is where a validator like the one in the encoding_rs write-up above gets its tricks, and it is the answer to "why would anyone care that a byte is eight bits".

Past where this library stops

  • Yannis Haralambous, Fonts & Encodings (2007) ↗. Carries the chain one step further than any page here: code point → glyph → font file. This library ends when a decoder hands over a code point; this is the thousand pages about what has to happen next before anyone sees a character. The only book that covers the whole chain in one place.
  • Ken Lunde, CJKV Information Processing, 2nd ed. (2009) ↗. The East Asian encodings in full — Shift_JIS, the EUC family, Big5, GB 18030 — and the writing systems that forced them to be shaped that way. 07_Real_Data meets one legacy table at a time and why UTF-8 won spends a paragraph on the Shift_JIS trail-byte overlap; this is the whole family, told by the person who documented it. Worth knowing that these are the encodings where the encoding_rs post above found its memory-safety bugs — not UTF-8.
  • Victor Stinner, Programming with Unicode (free) ↗. The same problem posed in C, Python 2, Python 3 and the Win32 API side by side, by the CPython developer who did that work. The closest thing in print to this library's habit of showing one lesson in several languages — and it covers the platform this library does not, Windows.

Text processing, the next subject along

Where this library asks what the bytes are, these ask what to do with them once that is settled.

Searching bytes, a different subject again

No page here asks how grep or rg is fast. That question starts in these.

Videos

Minutes Why
Characters, Symbols and the Unicode Miracle ↗ — Tom Scott, Computerphile 10 UTF-8's design explained on paper in ten minutes, including why the continuation bytes start with 10. Watch before UTF-8 by hand.
what is a BOM (byte-order-marker) ↗ — Anthony Sottile, anthony explains #560 11 The mark from the encode() call outward, at a REPL: one character through utf-16le and utf-16be, then through a bare utf-32, where the byte count jumps and the extra bytes are the mark. It is the codec-name half of Byte order and the BOM.
Unicode, in friendly terms ↗ — Studying With Alex 20 ASCII → code points → encodings, at exactly the pace of chapters 2 and 3 here.
Plain Text ↗ — Dylan Beattie, GOTO 2023 60 The history from telegraphs to emoji as a talk; the best hour on the subject, and funny. Baudot's shift codes — the Tribit project's CAPS — are in the first fifteen minutes.
Fast Conversion From UTF-8 with C++, DFAs, and SSE Intrinsics ↗ — Bob Steagall, CppCon 2018 60 What it costs to do by machine what UTF-8 by hand does by pencil, in a language that hands you nothing. The first twenty minutes are the encoding itself — the Valid Sequence Example slide is where Validation is a boundary starts, and the three that follow it — Overlong Sequence Example, Boundary Conditions, Finding the Transitions — are Overlong sequences — then it becomes a DFA, then SSE intrinsics. Watchable to the DFA and stoppable there. Code: BobSteagall/CppCon2018 ↗.
Pragmatic Unicode — Ned Batchelder, PyCon 2012 35 Linked from his page ↗ above; the Python talk.

Command-line tools, and which ones you already have

Every example in this library runs on tools that ship with both macOS and Ubuntu, on purpose — a lesson you cannot run is a lesson you have to believe. So the base set below is the whole vocabulary the pages use, and nothing further down is required to follow any of them.

Already on a Mac, no install:

Tool The question it answers
xxd what are the bytes? — and xxd -r -p is the only reverse gear of the three dumpers
od -An -tx1 the same, portably; od -c for octal escapes (never od -a)
hexdump -C the same again; plain hexdump reorders bytes to suit your CPU
wc -c / wc -m / wc -l bytes / characters / newline-terminated lines — three different questions
file --mime-encoding a guess, and an honest one
iconv re-encode, or use -f X -t X as a yes/no validator
cat -v / cat -vet high bits as M-, line ends as $ — the fastest one-line encoding check
strings the readable runs inside a binary
tr -d, sed, tail -c byte surgery when you know which byte — and tr does it a byte at a time whether or not you meant to
piconv Perl's iconv, already installed, with Perl's encoding names and aliases
python3 -c 'import unicodedata; …' the character's name and properties — the column no dumper gives you
man ascii the table, offline, in octal, decimal and hex — ascii(7), and /usr/share/misc/ascii is the same table as a data file you can grep
grep search — and the locale decides what a character is, so use LC_ALL=C on a file you did not write
find walk a directory — and -name is a byte comparison, which is why cat can open what it cannot see
sort / uniq order and deduplicate — byte order in LC_ALL=C, collation order in any other
cut -b slice by byte; cut -c promises characters and delivers bytes unless the locale is a UTF-8 one
diff / cmp are these the same file? — diff compares lines as bytes and cmp compares bytes, so neither of them answers the same text
split / paste / tee move text without reading it — and only tee is guaranteed to hand the next stage exactly what it was given

Worth installing. Nothing in the library depends on these and CI does not have them, so no page records their output — but each one answers a question the base set answers badly or not at all.

Install What it adds
brew install uni "Unicode database query tool for the command-line" — search characters by name, print their properties, the offline answer to what the lookup websites above do. The one to install first, and it has a page.
brew install hexyl Measured against the base toolbox. "Command-line hex viewer" — xxd with colour by byte category, so NUL, ASCII, and high bytes are visible at a glance instead of counted.
brew install recode Measured against the base toolbox. "Convert character set (charsets)" — iconv with a much larger table set and a surface-syntax (recode utf8..latin1) that is easier to type than -f/-t.
brew install uchardet Measured against the base toolbox. "Encoding detector library" — a real detector where file only distinguishes valid-UTF-8 from not. Still a guess; a better-informed one.
brew install dos2unix Measured against the base toolbox. The CRLF repair kit as one command, with unix2dos and a -i flag that reports line-ending counts without changing anything.
brew install icu4c Brings uconv — ICU's converter, which does normalization (-x nfc) as well as transcoding, so it is the one that can answer whether two visually identical strings are the same string. Keg-only: add its bin to your PATH.
brew install coreutils Measured against the base toolbox. The GNU versions, prefixed g — god, gwc, gtr. Worth having for one library-specific reason: this repo documents the places where BSD and GNU tools disagree, and with coreutils installed you can run both sides on the same machine instead of taking CI's word for it. Demonstration below.

The coreutils row is the one to actually act on, so here is the payoff. The named-character row is the sharpest disagreement in the whole toolbox, and until now this repo could only show one half of it per machine:

Measured 2026-09-06 on macOS 25.6, en_US.UTF-8, coreutils 9.x — one file, one machine, two implementations. Not machine-checked; that is the point.
$ printf 'caf\303\251 1\342\202\254\n' > demo.txt

$ od -a demo.txt          # BSD, the one that ships with macOS
0000000    c   a   f   ?   ?  sp   1   ?  82   ?  nl

$ god -a demo.txt         # GNU, from coreutils, same file, same second
0000000   c   a   f   C   )  sp   1   b stx   ,  nl

$ od -An -tx1 demo.txt    # and the row that is actually the file
           63  61  66  c3  a9  20  31  e2  82  ac  0a

GNU invents C, ), b and , — four characters that appear nowhere in the file — by masking the high bit off each byte (c3 & 0x7f is 0x43, which is C). BSD asks isprint() in your locale, says yes, and emits the raw byte, which your terminal then fails to decode and draws as ?. Neither row is the file. The third one is.

bat is the one everyone reaches for that is genuinely not about encodings — a syntax-highlighting cat. rg is not in that category, and this list used to say it was. It is a fast grep, which is why people install it; what you get with it is a search that has no locale, gives the same answer on every platform, and reads a UTF-16 file that grep cannot see. That is a set of encoding decisions, and 11_Tools is where all of them — rg's and grep's and find's — are measured against each other.

The terminal itself

  • Display high-bit characters in Terminal on Mac ↗ — Apple. Terminal's three encoding-adjacent settings: the text-encoding menu, the locale-variables checkbox, and the Control-V input escape. Read it for what the settings are; the advice to change them is written for a world in which UTF-8 was not already the default, and only one of the three is visible to a running program. Which one, and why, is Locale and LC_CTYPE.

Tools for looking a character up

Does
Unicode code converter ↗ — Richard Ishida, W3C Paste anything; see it as code points, UTF-8, UTF-16, escapes in six languages, at once. The tool this library's viewer exercise imitates.
Compart Unicode ↗ One page per character with every encoding, block, and property.
FileFormat.info ↗ The same, older, and the page linked from that Stack Overflow answer. That link is é.
uni ↗ — Martin Tournoij The command-line one, and the only tool in this table that will tell you which Unicode it is answering from — it ships its own copy of the database and uni version prints the release. brew install uni; then uni identify, uni search, uni print. Why that matters is The table has a version.
Awesome Unicode ↗ A curated list of the strange corners: zero-width characters, homoglyphs, the emoji that break things.

Katas — write the real thing

Once the Tribit project is done, the real encoders are the next step, and they are shorter.

The exercise
Rosetta Code: UTF-8 encode and decode ↗ Encode a code point to UTF-8 bytes and back, with the four-row test table. Solutions in ~60 languages to compare yours against — Rust and Python included.
Rosetta Code: Base64 encode data ↗ The 3-bytes-to-4-characters packing that Tribit's layer 3 is the mirror image of.
Codewars: Rust katas matching "utf" ↗ Several small UTF-8 and code-point katas with a test harness; free account needed to submit.
Exercism, Rust and Python tracks — Hexadecimal, Binary, Octal, Bob, Atbash Cipher The base-conversion exercises are chapter 1 as katas. (Exercism's pages refuse automated fetches, so no link here; search the track by name.)

Katas this library will host itself, with compiled solutions, are on the ROADMAP.