The encoding man pages nobody opens¶

Level: 201 · for anyone with a terminal and a question about bytes

One line: Unix ships a man page per character encoding — utf8(5), euc(5), gbk(5), mskanji(5) — plus a hub page that names every multibyte function in libc, and on a normal Mac that is about forty pages of primary material sitting unread under /usr/share/man.

Section 5 is where the encodings live¶

Sections 1, 3 and 8 are the ones people know: commands, library calls, daemons. Section 5 is file formats, and a character encoding is a file format — so BSD, and therefore macOS, documents them there, one page each, in the same register as tar(5) or crontab(5).

man 5 utf8

That page is 40 lines. It gives the byte templates as a table, states the shortest-form rule as a security property in one sentence, and stops. It is a specification, not a tutorial, and it was on the machine the whole time.

Measured 2026-09-07 — macOS 26.6. Not machine-checked: which pages exist is a fact about the machine, and a stock ubuntu:24.04 image ships none of them at all.

$ man -k . | ...   # or: for p in utf8 euc big5 gbk gb2312 gb18030 mskanji utf2; do man -w 5 $p; done

utf8(5)      UTF-8, a transformation format of ISO 10646
utf2(5)      Universal character set Transformation Format encoding of runes
euc(5)       EUC encoding of wide characters
big5(5)      "Big Five"
gb18030(5)   GB 18030 encoding method for Chinese text
gb2312(5)    GB2312 encoding method for Chinese text
gbk(5)       Guojia biaozhun kuozhan (GBK) encoding method for Chinese text
mskanji(5)   Shift-JIS (MS Kanji) encoding for Japanese text

Eight encodings, eight pages. utf2(5) is the same page as utf8(5) under the name UTF-8 was born with in Plan 9, kept so an old reference still resolves.

`multibyte(3)` is the hub, and it is one screen¶

If you only open one, open this one. It defines the two representations the C standard offers — wide characters for holding text in memory, multibyte characters for reading and writing it — and then hands you the entire function table:

	non-restartable	restartable
bytes in one character	`mblen(3)`	`mbrlen(3)`
one character → wide	`mbtowc(3)`	`mbrtowc(3)`
string → wide string	`mbstowcs(3)`	`mbsrtowcs(3)`
wide → one character	`wctomb(3)`	`wcrtomb(3)`
wide string → string	`wcstombs(3)`	`wcsrtombs(3)`

The r is for restartable: those take an mbstate_t you own, so two threads can decode two streams. The others keep the state in a static inside libc, which is why they are the ones you stop using.

It also names, without ceremony, the thing that trips people writing their first decoder — LC_CTYPE decides what a multibyte character is, so setlocale changes the meaning of every function in that table, and shift states are undefined across a call to it. That is Locale and LC_CTYPE stated as an API contract.

Three more worth knowing by name¶

mbrune(3) — the 4.4BSD rune API, with sgetrune and sputrune on its sister page rune(3). The page says two things worth knowing: the whole API is deprecated in favour of C99's wide characters, and three of its functions first appeared in Plan 9, as utfrune, utfrrune and utfutf — Plan 9 being where UTF-8 was born and where the word rune comes from. It is not an ancestor of wchar_t: macOS's headers declare rune_t as wchar_t. Go took the word for its own type and made it an int32 that checks nothing, which is the subject of rune is an int32. Still shipped, still five minutes to read, and it explains a naming choice that otherwise looks arbitrary.

vis(3) — "visually encode characters". A single page containing six invertible escaping formats, selected by flag: VIS_OCTAL, VIS_CSTYLE, VIS_HTTPSTYLE (percent), VIS_MIMESTYLE (=), and two more. Its one-sentence invariant is the definition worth stealing: a unique, invertible representation composed entirely of graphic characters. That is the whole subject of Escaping into ASCII, written as a C API in 1996.

ascii(7) — the table, as a man page. And /usr/share/misc/ascii is the same table as a data file you can grep, which is occasionally exactly what you want.

How to sweep a documentation tree¶

You will not read forty pages to find the one that mentions LC_COLLATE. Build a list of files, then run one grep per term over the list:

find /usr/share/man -type f | sort > /tmp/manfiles.txt
xargs < /tmp/manfiles.txt grep -l -i 'LC_COLLATE' | sed 's|.*/||'

Two pipelines that look like that one return nothing at all and exit 0, which is worse than an error because it reads as an answer. The example builds a corpus of three known pages and runs both:

Verified output of the_encoding_man_pages_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

1. THE CORPUS: THREE PAGES, AND WHICH ONE SAYS WHAT
   utf8.5             The UTF-8 encoding represents UCS-4 characters
   iconv.1            iconv -- codeset conversion utility.  Converts
   banner.1           banner -- print large banner on printer.

2. BUILD THE FILE LIST FIRST, THEN SWEEP IT
$ find man -type f | sort > list.txt; wc -l < list.txt
   3

$ xargs < list.txt grep -l -i 'UTF-8'
   man/man1/iconv.1
   man/man5/utf8.5

   Two of the three.  banner names no encoding, which is the answer we
   wanted: a sweep tells you WHICH pages to open, not what they say.

3. THE SAME QUESTION, ONE TERM AT A TIME
   UTF-8      2 file(s)
   codeset    1 file(s)
   locale     0 file(s)
   octets     1 file(s)

   That loop is the whole method.  Point list.txt at /usr/share/man and
   the same four lines inventory every man page on your machine.

4. THE TRAP: TWO PIPELINES THAT FIND NOTHING AND SAY SO WITH EXIT 0
$ grep -l -i 'UTF-8' $(cat list.txt)     # argv, not stdin
   man/man1/iconv.1
   man/man5/utf8.5
   Correct here -- three files fit in argv.  At 17,000 they do not, and
   the shell fails the whole command rather than the grep: no matches,
   and nothing that looks like an error.  Always build the list, then
   feed it through xargs, which splits it into as many runs as it needs.

$ grep -c -i 'UTF-8' list.txt            # greps the LIST, not the pages
   0
   Zero.  A filename is not its contents, and a list of filenames is a
   text file like any other -- so this reads as a real answer.

grep -l PAT $(cat list.txt) puts every path in argv, and at seventeen thousand files that exceeds ARG_MAX; the shell fails the command before grep starts. grep -c PAT list.txt searches the list — a text file of filenames — and finds nothing in it, correctly. Build the list, then xargs it.

Measured 2026-09-07 — BSD/GNU difference. macOS 26.6 vs ubuntu:24.04.

xargs -a list.txt grep -l PAT     GNU: accepted        BSD: xargs: invalid option -- a
xargs < list.txt grep -l PAT      GNU: accepted        BSD: accepted

xargs -a is a GNU extension. Use the redirect, which is both.

The rest of the machine¶

Man pages are not the only documentation you already have.

Where	What is in it
`/usr/share/misc/ascii`	the ASCII table as a data file, octal plus control-code names
`iconv -l`	every codeset the C library can convert — 215 groups, 873 alias names on this Mac
`locale -a`	every locale installed, and each one names its charmap
`/usr/share/vim/vim*/doc/mbyte.txt`	twelve sections on locale, encoding, terminals and input methods
`info gettext`	nodes Charset conversion, Locale Names, Locale Environment Variables
`man perlunitut` → `perluniintro` → `perlunicode`	a graded Unicode course, installed with Perl, which is installed
`man Encode::Supported`	headings including "Encoding vs. Charset — terminology" and "Microsoft-related naming mess"

iconv -l is the one to try first. It is the single richest keyword source on the machine, and reading it is how you learn that CP819, IBM819, ISO-8859-1 and LATIN1 are one encoding under four names — which is the same lesson the codec probe teaches in Python.

If you are coming from Python or ABAP¶

Python puts the same material in help() and in the module docs: help('codecs') is the analogue of multibyte(3), and encodings.aliases.aliases is iconv -l. The difference is that Python's is introspectable — you can ask the registry a question — where the man page can only be read. That is the subject of What the page does not say.

ABAP has no man pages; the equivalent primary sources are the class documentation for cl_abap_codepage and cl_abap_char_utilities in SE24, and the SAP Notes for a code page rather than a spec for it. The habit transfers exactly: read the class documentation before the blog post, and check the date on whichever you end up trusting. (Not machine-checked — CI cannot run ABAP.)

Try it¶

man 5 utf8, then man 3 multibyte. Ten minutes for both.
iconv -l | wc -l — how many codesets does your machine know? Then find every name for Latin-1: iconv -l | grep -i '8859-1'.
Sweep for a term you have been guessing about: build the file list, then xargs < list grep -l -i 'byte order mark'. Open the first hit.
man -k utf and see how much of the answer is OpenSSL. A keyword search over NAME lines is a blunter instrument than a sweep over page bodies — which is why the recipe above exists.

Practice¶

Forty pages nobody opens. Name the man pages that document character encodings on a Unix system — the per-encoding ones and the two hub pages — and say which section each lives in.

Two of them document the same subject under different names on macOS and Linux; find them, and say why that alone stops most people finding any of it. Then give the four commands that locate these pages without knowing their names, and say which single page to read first.

Answers

Verified output of the_encoding_man_pages_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

THE PAGES THAT EXIST, AND WHERE
   page           where      what it is
   utf8(5)        BSD/macOS  the encoding itself -- and see the date kata
   utf-8(7)       Linux      the same subject, different section AND spelling
   euc(5)         BSD        Extended Unix Code, the CJK family
   gbk(5)         BSD        the Chinese national standard
   mskanji(5)     BSD        Shift-JIS, as Microsoft shipped it
   big5(5)        BSD        traditional Chinese
   multibyte(3)   both       the HUB: every multibyte function in libc
   charsets(7)    Linux      the Linux hub page, a different tour
   locale(1)      both       what your six variables currently are
   iconv(1)       both       and iconv(3), which is a different page

   Note rows 1 and 2. The SAME subject is utf8 in section 5 on a Mac and
   utf-8 in section 7 on Linux -- different name, different section, so
   `man utf8` finds it on one machine and nothing on the other. That is
   the first reason these pages go unread: you cannot guess the name.

HOW TO FIND THEM WITHOUT KNOWING THE NAME
   man -k charset       # apropos: searches the one-line descriptions
   man -k encoding
   man -k multibyte
   man -w utf8          # prints the PATH, or nothing -- the quickest
                        # way to ask 'does this machine have it?'
   ls /usr/share/man/man5 | head

WHY THEY ARE WORTH THE HOUR
   They are PRIMARY sources for the system you are actually on, written
   by the people who shipped the libraries you are calling. Almost
   everything else you will read about encodings is somebody's summary
   of a standard, and a summary cannot tell you what YOUR iconv accepts.

   multibyte(3) is the one to read first if you read only one: it names
   mbrtowc, wcrtomb, mbsrtowcs and the rest in one place, and those
   functions are what every command-line tool on the machine is
   actually calling when it decides what a character is.

AND THE HEALTH WARNING FROM THE PAGE NEXT DOOR
   A man page has a date and cites a standard, and both can be decades
   old while still describing the binary correctly. Read them as
   evidence about YOUR system, not as a description of Unicode.