Skip to content

The man pages, annotated

Level: reference · for anyone who wants the primary documentation of text on their own machine, with every word on it explained

One line: 169 manual pages about text and files, 140 from this Mac and 29 from Ubuntu, dumped verbatim with the date each one carries and the machine it came from, then read in 28 family pages that quote each page, explain its terms, check its claims against both machines, and link the lesson here that owns each idea.

What is in this folder

The encoding man pages nobody opens made the case that a stock Mac ships about forty pages of primary material on this subject and that nobody opens them. This folder opens them, and the hundred and thirty others that turn out to be about text once you ask which pages on the machine decide what a character is. It has three layers.

Layer Where What it is
The dumps raw/macos/, raw/linux/ 169 text files, each exactly what man SECTION NAME prints, one per page. Never edited; regenerated by two scripts
The provenance raw/PROVENANCE-macos.txt, raw/PROVENANCE-linux.txt Which machine, which library version, which command, which day. A man page is documentation of a machine, so the machine is recorded beside it
The family pages the 28 .md files listed below One page per group of related man pages. Each quotes the passages that matter, explains the terms in a table, measures the page's claims on both machines in dated fences, says where the page is stale or silent, and links the lessons

A family page is laid out the same way every time, so you can skip to the part you want: What the pages are for (who wrote it, when, which section, why it matters here), The page, with notes (verbatim excerpts, each followed by commentary), Terms on the page (a table: the word, what it means, where this library explains it), Try it on your machine (commands and what they printed, on macOS 26.6.2 and on ubuntu:24.04), Where the page is dated, and what it does not say, and See also.

To look a page up, find its name in the tables below. The first column links the verbatim dump, the heading links the family page that reads it.

How the dumps were made, and how to check them

dump.sh renders each macOS page with man SECTION NAME | col -bx and writes it to raw/macos/NAME.SECTION.txt. Into a pipe, man on this Mac runs mandoc at 78 columns and marks bold and underline the way a 1970s teletype did, by printing the character, a backspace, and the character again; col -bx removes those backspaces and turns tabs into spaces, and does nothing else. The man(1) family page shows the raw bytes. A page that man finds under another name is dumped under the name you would type: man 1 base64 on this Mac prints the page called bintrans(1), so it is here as base64.1.txt.

dump_linux.sh does the same for the 29 Linux pages in an ubuntu:24.04 container. A stock Ubuntu image ships no man pages at all: its dpkg is configured to drop /usr/share/man on every install, and its /usr/bin/man is a stub that says so. The script builds an image that lifts that exclusion, installs man-db and the manpages packages (release 6.7 of the Linux man-pages project), and renders with the real binary, which the stub leaves at /usr/bin/man.REAL. The experiments in the family pages ran in a second image built on that one with gcc, xxd and python3 added; both Dockerfiles are in the scripts and the provenance files.

The locale decides the bytes, so both scripts pin it. dump.sh runs under LC_ALL=C, and the 140 macOS dumps contain no byte above 0x7f: mandoc writes its special-character escapes as ASCII, EUR for the euro and 'e for an e-acute, and the dash between a page's name and its description as -. Read the same page in a UTF-8 terminal and that dash is , and man 7 mandoc_char gains 1,028 bytes of real glyphs. dump_linux.sh runs under C.UTF-8, the locale a stock Ubuntu ships, so the 29 Linux dumps are the UTF-8 rendering and hold 9,420 bytes above 0x7f: man-db's hyphens, dashes and bullets. Two renderings of one page are a finding in their own right, measured on the man(1) page, which also shows why the pinning matters on the Mac: under C, this Mac's col deletes any byte above 0x7f it is given.

The dumps are verbatim so that they can be checked. On the same machine, bash 13_Documentation/manual_pages/dump.sh followed by git diff --stat 13_Documentation/manual_pages/raw/ should print nothing. On a different machine it will print the pages that differ, which is the point: a man page has a date, and so does the system it describes.

raw/PROVENANCE-macos.txt and raw/PROVENANCE-linux.txt, as written by the two scripts on 2026-09-13
macOS dump: 2026-09-13
  macOS 26.6.2 (x86_64), Darwin 25.6.0
  man: /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/share/man/man5/utf8.5
  rendered with: LC_ALL=C man SECTION NAME | col -bx   (mandoc, 78 columns, ASCII rendering)
  pages: 140 dumped, 0 missing

Linux dump: 2026-09-13
  Ubuntu 24.04.4 LTS in Docker, image enc-man (built from ubuntu:24.04)
  manpages 6.7-2, man-db 2.12.0-4build2, glibc 2.39-0ubuntu8.9
  rendered with: LANG=C.UTF-8 MANWIDTH=78 man.REAL -P cat SECTION NAME | col -bx   (man-db, UTF-8 rendering)
  pages: 29 dumped, 0 missing

Note the first path: man on this Mac serves most section-3 pages from the Command Line Tools SDK, not from /usr/share/man. The SDK is documentation of the headers you compile against, which is the right thing to read, and it is also why a machine without Xcode's tools has fewer of these pages.

Which pages, and why these

The rule was: a page is in if it is about text, files as sequences of bytes, or the tables and settings that turn one into the other. That gave four groups.

  • The encodings themselves. BSD documents each encoding as a file format in section 5, one page each; Linux documents them as essays in section 7 under different names. Both sets are here, and the utf8(5) page puts the two side by side.
  • The C library. Every command-line tool that has an opinion about characters got it from libc, so the pages for the functions that decode (multibyte(3)), classify (ctype(3)), measure (wcwidth(3)), order (strcoll(3)), convert (iconv(3)), escape (vis(3)) and read (stdio(3)) text are here, with the locale pages that give them their meaning.
  • The tools. The commands the 11_Tools chapter measures, read here as pages: what each flag promises, in bytes or in characters, and where the page is silent.
  • The formats. magic(5), tar(5), the uuencode format on bintrans(1), and the format of the manual itself on mandoc_char(7).

Some pages on the machine were left out on purpose. textutil(1), plutil(1), plist(5), core(5) and sh(1) are Apple's, all rights reserved, with no licence that allows copying them into a public repository; read them with man on your own Mac. The Perl documentation (perlunitut, perluniintro, perlunicode, Encode::Supported) is a graded Unicode course in its own right, already recommended on the lesson page, and too large to annotate here. Pages for tools that are not on a stock machine (rg, hexyl, uchardet, recode, dos2unix) belong with The five worth installing.

What the licences allow

Every dump in raw/macos/ except three is a rendering of a BSD-licensed manual page: from the Regents of the University of California (the 1980s and 1990s pages), Tim J. Robbins (the multibyte and CJK pages, 2002 to 2004), the Citrus Project and Gabor Kovesdan (iconv), Henry Spencer (regex, re_format), Kristaps Dzonsons and Ingo Schwarze (mandoc, ISC licence), Tim Kientzle (tar(5), cpio(5)), Ian F. Darwin and Christos Zoulas (file, magic), and the vim project (xxd, under the vim licence). The BSD licence permits redistribution in any form provided the copyright notice, the conditions and the disclaimer are reproduced with the distribution. The rendered pages do not carry those notices; they are comments in the roff sources, at the paths man -w SECTION NAME prints on a Mac and which the provenance file records, and the notice common to them is this:

The BSD notice, as it appears in the roff source of utf8.5 and most of the pages here
Copyright (c) 1993
	The Regents of the University of California.  All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.
3. Neither the name of the University nor the names of its contributors
   may be used to endorse or promote products derived from this software
   without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.

The three exceptions are Apple's own: locale(1) (Darwin, 2004), the xlocale(3) family (newlocale, uselocale, duplocale, querylocale, 2005) and strings(1) from cctools. Those ship in Apple's open-source releases of Libc and cctools under the Apple Public Source License 2.0 ↗, which permits redistribution of unmodified copies with that notice. The 29 Linux pages are from release 6.7 of the Linux man-pages project ↗, whose pages are licensed page by page under the GPL version 2 or later or under the project's own verbatim-copying licence; each page's roff source carries its notice, and the rendered text does not. None of this is legal advice; it is where the notices are.

The families

The encoding itself: utf8(5) and utf-8(7): one encoding, two pages, twenty years apart

Page Also on this page Machine Date on the page What its NAME line says
utf8(5) macOS April 7, 2004 UTF-8, a transformation format of ISO 10646
utf2(5) macOS October 11, 2002 Universal character set Transformation Format encoding of runes
utf-8(7) UTF-8 Linux 2024-03-14 an ASCII compatible multibyte Unicode encoding
unicode(7) Linux 2024-01-28 universal character set

The CJK encodings: euc(5), big5(5), gb2312(5), gbk(5), gb18030(5), mskanji(5)

Page Also on this page Machine Date on the page What its NAME line says
euc(5) macOS November 8, 2003 EUC encoding of wide characters
big5(5) macOS August 7, 2003 "Big Five" encoding for Traditional Chinese text
gb2312(5) macOS November 7, 2003 GB2312 encoding method for Chinese text
gbk(5) macOS August 10, 2003 Guojia biaozhun kuozhan (GBK) encoding method for Chinese text
gb18030(5) macOS August 10, 2003 GB 18030 encoding method for Chinese text
mskanji(5) macOS August 7, 2003 Shift-JIS (MS Kanji) encoding for Japanese text

The code pages, Linux only: charsets(7) and the ISO 8859, Windows and KOI8 pages

Page Also on this page Machine Date on the page What its NAME line says
charsets(7) Linux 2024-01-28 character set standards and internationalization
iso_8859-1(7) Linux 2024-01-28 ISO/IEC 8859-1 character set encoded in octal, decimal, and hexadecimal
iso_8859-2(7) Linux 2024-01-28 ISO/IEC 8859-2 character set encoded in octal, decimal, and hexadecimal
iso_8859-15(7) Linux 2024-01-28 ISO/IEC 8859-15 character set encoded in octal, deci‐ mal, and hexadecimal
iso_8859-16(7) Linux 2024-01-28 ISO/IEC 8859-16 character set encoded in octal, deci‐ mal, and hexadecimal
cp1251(7) Linux 2024-01-28 CP 1251 character set encoded in octal, decimal, and hexa‐ decimal
cp1252(7) Linux 2024-01-28 CP 1252 character set encoded in octal, decimal, and hexa‐ decimal
koi8-r(7) Linux 2022-12-15 Russian character set encoded in octal, decimal, and hexa‐ decimal
koi8-u(7) Linux 2022-12-15 Ukrainian character set encoded in octal, decimal, and hexa‐ decimal
armscii-8(7) Linux 2022-12-15 Armenian character set encoded in octal, decimal, and hexadecimal

The table: ascii(7), twice

Page Also on this page Machine Date on the page What its NAME line says
ascii(7) macOS June 5, 1993 octal, hexadecimal and decimal ASCII character sets
ascii(7) Linux 2024-01-28 ASCII character set encoded in octal, decimal, and hexadeci‐ mal

The C API: decoding and encoding: multibyte(3) and the ten conversion functions

Page Also on this page Machine Date on the page What its NAME line says
multibyte(3) macOS April 8, 2004 multibyte and wide character manipulation functions
mbrtowc(3) mbrtowc_l macOS April 8, 2004 convert a character to a wide-character code (restartable)
wcrtomb(3) wcrtomb_l macOS April 8, 2004 convert a wide-character code to a character (restartable)
mbsrtowcs(3) mbsnrtowcs, mbsnrtowcs_l, mbsrtowcs_l macOS July 21, 2004 convert a character string to a wide-character string (restartable)
wcsrtombs(3) wcsnrtombs, wcsnrtombs_l, wcsrtombs_l macOS July 21, 2004 convert a wide- character string to a character string (restartable)
mbstowcs(3) mbstowcs_l macOS April 8, 2004 convert a character string to a wide-character string
wcstombs(3) wcstombs_l macOS April 8, 2004 convert a wide-character string to a character string
mblen(3) mblen_l macOS April 11, 2004 get number of bytes in a character
mbrlen(3) mbrlen_l macOS April 7, 2004 get number of bytes in a character (restartable)
mbtowc(3) mbtowc_l macOS April 11, 2004 convert a character to a wide-character code
wctomb(3) wctomb_l macOS April 8, 2004 convert a wide-character code to a character
btowc(3) btowc_l, wctob, wctob_l macOS August 3, 2002 convert between wide and single-byte characters
wctob(3) btowc, btowc_l, wctob_l macOS August 3, 2002 convert between wide and single-byte characters
mbsinit(3) mbsinit_l macOS April 8, 2004 determine conversion object status
mbrtowc(3) Linux 2023-10-31 convert a multibyte sequence to a wide character

The C API it replaced: mbrune(3) and rune(3)

Page Also on this page Machine Date on the page What its NAME line says
mbrune(3) mbrrune, mbmb macOS April 19, 1994 multibyte rune support for C
rune(3) setrunelocale, setinvalidrune, sgetrune, sputrune, fgetrune, fungetrune, fputrune macOS October 6, 2002 rune support for C

The C API: classifying and casing: ctype(3), wctype(3) and their families

Page Also on this page Machine Date on the page What its NAME line says
ctype(3) macOS March 30, 2004 digittoint, isalnum, isalpha, isascii, isblank, iscntrl, isdigit, isgraph, ishexnumber, isideogram, islower, isnumber, isphonogram, isprint, ispunct, isrune, isspace, isspecial, isupper, isxdigit, toascii,
isalpha(3) macOS July 17, 2005 alphabetic character test
isprint(3) macOS July 17, 2005 printing character test (space character inclusive)
isascii(3) macOS October 6, 2002 test for ASCII character
toascii(3) macOS June 4, 1993 convert a byte to 7-bit ASCII
tolower(3) tolower_l macOS July 17, 2005 upper case to lower case letter conversion
toupper(3) toupper_l macOS July 17, 2005 lower case to upper case letter conversion
wctype(3) iswctype, iswctype_l, wctype_l macOS March 27, 2004 wide character class functions
iswalpha(3) iswalnum, iswascii, iswblank, iswcntrl, iswdigit, iswgraph, iswhexnumber, iswideogram, iswlower, iswnumber, iswphonogram, iswprint, iswpunct, iswrune, iswspace, iswspecial, iswupper, iswxdigit macOS October 3, 2002 wide
iswctype(3) iswctype_l, wctype, wctype_l macOS March 27, 2004 wide character class functions
towlower(3) towlower_l macOS October 3, 2002 upper case to lower case letter conversion (wide character version)
towupper(3) towupper_l macOS October 3, 2002 lower case to upper case letter conversion (wide character version)
wctrans(3) towctrans, towctrans_l, wctrans_l macOS October 3, 2002 wide character mapping functions
towctrans(3) towctrans_l, wctrans, wctrans_l macOS October 3, 2002 wide character mapping functions

The C API: columns: wcwidth(3) and wcswidth(3)

Page Also on this page Machine Date on the page What its NAME line says
wcwidth(3) wcwidth_l macOS August 17, 2004 number of column positions of a wide-character code
wcswidth(3) wcswidth_l macOS August 20, 2002 number of column positions in wide-character string
wcwidth(3) Linux 2023-10-31 determine columns needed for a wide character

The C API: ordering: strcoll(3), strxfrm(3) and the case-insensitive compares

Page Also on this page Machine Date on the page What its NAME line says
strcoll(3) strcoll_l macOS June 4, 1993 compare strings, according to current collation
strxfrm(3) strxfrm_l macOS June 4, 1993 transform a string under locale
wcscoll(3) wcscoll_l macOS October 4, 2002 compare wide strings according to current collation
wcsxfrm(3) wcsxfrm_l macOS October 4, 2002 transform a wide string under locale
strcasecmp(3) strcasecmp_l, strncasecmp, strncasecmp_l macOS June 9, 1993 compare strings, ignoring case
wcscasecmp(3) macOS March 4, 2009 wcpcpy, wcpncpy, wcscasecmp, wcscat, wcschr, wcscmp, wcscpy, wcscspn, wcsdup, wcslcat, wcslcpy, wcslen, wcsncasecmp, wcsncat, wcsncmp, wcsncpy, wcsnlen, wcspbrk, wcsrchr, wcsspn, wcsstr, wmemchr, wmemcmp, wmemcpy,

The locale: locale(1), setlocale(3), nl_langinfo(3), the xlocale(3) family and environ(7)

Page Also on this page Machine Date on the page What its NAME line says
locale(1) macOS August 27, 2004 display locale settings
setlocale(3) macOS November 21, 2003 natural language formatting for C
localeconv(3) localeconv_l macOS November 21, 2003 natural language formatting for C
nl_langinfo(3) nl_langinfo_l macOS May 3, 2001 language information
xlocale(3) macOS March 11, 2005 Extended locale support.
newlocale(3) macOS March 11, 2005 Create a new locale
uselocale(3) macOS March 11, 2005 Set the per-thread locale
duplocale(3) macOS March 11, 2005 Duplicate existing locale
querylocale(3) macOS March 11, 2005 Get locale name for a specified category
environ(7) macOS April 12, 2003 user environment
environ(7) Linux 2023-10-31 user environment
setlocale(3) Linux 2024-02-25 set the current locale
nl_langinfo(3) nl_langinfo_l Linux 2024-01-28 query language and locale information
locale(1) Linux 2023-10-31 get locale-specific information

The locale's source files, dumped from Linux: locale(5), charmap(5), localedef(1)

Page Also on this page Machine Date on the page What its NAME line says
locale(7) Linux 2024-02-25 description of multilanguage support
locale(5) Linux 2024-01-28 describes a locale definition file
charmap(5) Linux 2023-10-31 character set description file
repertoiremap(5) Linux 2023-10-31 map symbolic character names to Unicode code points
localedef(1) Linux 2023-10-31 compile locale definition files

Converting between tables: iconv(1) and iconv(3), on both machines

Page Also on this page Machine Date on the page What its NAME line says
iconv(1) macOS October 22, 2009 codeset conversion utility
iconv(3) iconv_open, iconv_open_into, iconv_close macOS August 4, 2014 codeset conversion functions
iconvctl(3) macOS November 25, 2009 controlling and diagnostical facility for iconv(3)
iconvlist(3) macOS February 23, 2023 retrieving a list of character encodings supported by iconv(3)
iconv(1) Linux 2024-01-28 convert text from one character encoding to another
iconv(3) Linux 2023-10-31 perform character set conversion
iconv_open(3) Linux 2023-10-31 allocate descriptor for character set conversion

Escaping into ASCII: vis(1), unvis(1), vis(3), unvis(3)

Page Also on this page Machine Date on the page What its NAME line says
vis(1) macOS February 18, 2021 display non-printable characters in a visual format
unvis(1) macOS November 27, 2010 revert a visual representation of data back to original form
vis(3) nvis, strvis, stravis, strnvis, strvisx, strnvisx, strenvisx, svis, snvis, strsvis, strsnvis, strsvisx, strsnvisx, strsenvisx macOS April 22, 2017 visually encode characters
unvis(3) strunvis, strnunvis, strunvisx, strnunvisx macOS March 12, 2011 decode a visual representation of characters

Binary to text: base64(1), uuencode(1), uudecode(1): one page under three names

Page Also on this page Machine Date on the page What its NAME line says
base64(1) bintrans, uuencode, uudecode, b64encode, b64decode macOS January 23, 2024 encode / decode a binary file
uuencode(1) bintrans, uudecode, b64encode, b64decode, base64 macOS January 23, 2024 encode / decode a binary file
uudecode(1) bintrans, uuencode, b64encode, b64decode, base64 macOS January 23, 2024 encode / decode a binary file

Bytes and their order: byteorder(3), swab(3), bitstring(3)

Page Also on this page Machine Date on the page What its NAME line says
byteorder(3) htonl, htons, htonll, ntohl, ntohs, ntohll macOS June 4, 1993 convert values between host and network byte order
swab(3) macOS February 24, 2010 swap adjacent bytes
bitstring(3) bit_alloc, bit_clear, bit_decl, bit_ffs, bit_nclear, bit_nset, bit_set, bitstr_size, bit_test macOS July 19, 1993 bit-string manipulation macros

Streams: stdio(3), fopen(3) and the byte and wide readers

Page Also on this page Machine Date on the page What its NAME line says
stdio(3) macOS March 3, 2009 standard input/output library functions
fopen(3) fdopen, freopen, fmemopen macOS September 1, 2023 stream open functions
fgets(3) gets macOS June 4, 1993 get a line from a stream
fgetln(3) macOS April 19, 1994 get a line from a stream
getline(3) getdelim macOS November 30, 2010 get a line from a stream
fgetws(3) fgetws_l macOS August 6, 2002 get a line of wide characters from a stream
getwc(3) fgetwc, getwchar macOS March 3, 2004 get next wide character from input stream
putwc(3) fputwc, putwchar macOS March 3, 2004 output a wide character to a stream
fwide(3) macOS October 24, 2001 get/set orientation of a stream

Formatting: printf(3), wprintf(3), printf(1), echo(1)

Page Also on this page Machine Date on the page What its NAME line says
printf(3) fprintf, sprintf, snprintf, asprintf, dprintf, vprintf, vfprintf, vsprintf, vsnprintf, vasprintf, vdprintf macOS December 2, 2009 formatted output conversion
wprintf(3) fwprintf, swprintf, vwprintf, vfwprintf, vswprintf macOS July 5, 2003 formatted wide character output conversion
printf(1) macOS July 1, 2020 formatted output
echo(1) macOS April 12, 2003 write arguments to the standard output

Parsing numbers: strtol(3) and strtoul(3)

Page Also on this page Machine Date on the page What its NAME line says
strtol(3) strtoimax, strtoll, strtoq macOS November 28, 2001 convert a string value to a long, long long, intmax_t or quad_t integer
strtoul(3) strtoull, strtoumax, strtouq macOS November 28, 2001 convert a string to an unsigned long, unsigned long long, uintmax_t, or u_quad_t integer

Patterns: re_format(7), regex(3), fnmatch(3), glob(3), grep(1), sed(1)

Page Also on this page Machine Date on the page What its NAME line says
re_format(7) macOS Sept 29, 2011 POSIX 1003.2 regular expressions
regex(3) regcomp, regcomp_l, regerror, regexec, regfree, regncomp, regncomp_l, regnexec, regnwcomp, regnwcomp_l, regnwexec, regwcomp, regwcomp_l, regwexec macOS July 11, 2025 regular-expression library
fnmatch(3) macOS April 7, 2025 test whether a filename or pathname matches a shell-style pattern
glob(3) glob_b, globfree macOS June 23, 2025 generate pathnames matching a pattern
grep(1) egrep, fgrep, rgrep, bzgrep, bzegrep, bzfgrep, zgrep, zegrep, zfgrep macOS November 10, 2021 file pattern searcher
sed(1) macOS December 17, 2024 stream editor
regex(7) Linux 2023-11-01 POSIX.2 regular expressions
glob(7) Linux 2023-10-31 globbing pathnames

Seeing the bytes: hexdump(1), od(1), xxd(1), strings(1)

Page Also on this page Machine Date on the page What its NAME line says
hexdump(1) macOS June 29, 2020 ASCII, decimal, hexadecimal, octal dump
od(1) macOS December 22, 2011 octal, decimal, hex, ASCII dump
xxd(1) macOS May 2024 make a hex dump or do the reverse.
strings(1) macOS June 7, 2016 find the printable strings in a object, or other binary, file

Guessing the type: file(1) and magic(5)

Page Also on this page Machine Date on the page What its NAME line says
file(1) macOS February 5, 2021 determine file type
magic(5) macOS May 9, 2021 file command's magic pattern file

Text tools that count: tr, cut, fold, wc, col and the rest of the column tools

Page Also on this page Machine Date on the page What its NAME line says
tr(1) macOS October 13, 2006 translate characters
cut(1) macOS August 3, 2017 cut out selected portions of each line of a file
fold(1) macOS October 29, 2020 fold long lines for finite width output device
fmt(1) macOS October 29, 2020 simple text formatter
expand(1) unexpand macOS June 6, 2015 expand tabs to spaces, and vice versa
unexpand(1) expand macOS June 6, 2015 expand tabs to spaces, and vice versa
col(1) macOS October 21, 2020 filter reverse line feeds from input
colrm(1) macOS June 23, 2020 remove columns from a file
pr(1) macOS July 3, 2004 print files
ul(1) macOS October 7, 2020 do underlining
rev(1) macOS June 27, 2020 reverse lines of a file
wc(1) macOS April 11, 2020 word, line, character, and byte count

Text tools that cut on lines: cat, head, sort, uniq, paste, xargs and the rest

Page Also on this page Machine Date on the page What its NAME line says
cat(1) macOS January 29, 2013 concatenate and print files
head(1) macOS April 10, 2018 display first lines of a file
tail(1) macOS November 28, 2023 display the last part of a file
nl(1) macOS June 18, 2020 line numbering filter
split(1) macOS May 26, 2023 split a file into pieces
csplit(1) macOS February 6, 2014 split files based on context
tee(1) macOS June 23, 2020 duplicate standard input
look(1) macOS December 29, 2020 display lines beginning with a given string
uniq(1) macOS December 9, 2024 report or filter out repeated lines in a file
sort(1) macOS September 4, 2019 sort or merge records (lines) of text and binary files
comm(1) macOS July 27, 2020 select or reject lines common to two files
join(1) macOS June 20, 2020 relational database operator
paste(1) macOS June 25, 2004 merge corresponding or subsequent lines of files
lam(1) macOS April 7, 2015 laminate files
rs(1) macOS April 7, 2015 reshape a data array
jot(1) macOS September 21, 2019 print sequential or random data
seq(1) macOS June 20, 2020 print sequences of numbers
xargs(1) macOS September 21, 2020 construct argument list(s) and execute utility

Comparing and summing: diff(1), cmp(1), cksum(1), md5(1)

Page Also on this page Machine Date on the page What its NAME line says
diff(1) macOS January 7, 2025 differential file and directory comparator
cmp(1) macOS September 23, 2021 compare two files
cksum(1) sum macOS April 28, 1995 display file checksums and block counts
md5(1) sha1, sha224, sha256, sha384, sha512, md5sum, sha1sum, sha224sum, sha256sum, sha384sum, sha512sum macOS February 13, 2024 calculate a message-digest fingerprint (checksum) for a file

Moving blocks, with six code pages inside: dd(1)

Page Also on this page Machine Date on the page What its NAME line says
dd(1) macOS May 19, 2021 convert and copy a file

The terminal: tty(4) and stty(1)

Page Also on this page Machine Date on the page What its NAME line says
tty(4) macOS August 14, 1992 general terminal interface
stty(1) macOS October 20, 2018 set the options for a terminal device interface

The manual itself: man(1), mandoc(1), mandoc_char(7)

Page Also on this page Machine Date on the page What its NAME line says
man(1) apropos, whatis macOS January 9, 2021 display online manual documentation pages
mandoc(1) macOS August 14, 2021 format manual pages
mandoc_char(7) macOS October 31, 2020 mandoc special characters

Archives: tar(5) and cpio(5)

Page Also on this page Machine Date on the page What its NAME line says
tar(5) macOS December 27, 2016 format of tape archive files
cpio(5) macOS December 23, 2011 format of cpio archive files

What the measurements found

The family pages hold about a hundred dated fences. These are the findings worth knowing before you open any of them, one line each, with the page that measured it.

  • The 2004 utf8(5) table is still what this Mac's iconv implements: it decodes the five- and six-byte spellings the page prints, to numbers no character has; GNU's refuses them; both accept the noncharacter U+FFFE and both refuse an overlong NUL and a surrogate. Python refuses the long forms on both. utf8(5)
  • utf-8(7), dated 2024, still prints six rows and opens with a sentence about a 16-bit code space that has been false since 2001. utf8(5)
  • Neither UTF-8 page mentions the byte order mark, the replacement character, or what to do with bad input. utf8(5)
  • MB_CUR_MAX under a UTF-8 locale is 4 on macOS and 6 on glibc, and MB_LEN_MAX is 6 against 16, so a buffer sized by one platform's header is not sized for the other's. multibyte(3)
  • Under the C locale, macOS decodes every byte and glibc refuses every byte above 0x7f; both are conforming, and no page says which its library does. multibyte(3)
  • On macOS a wchar_t is not a code point outside UTF-8 locales: under ja_JP.eucJP, mbrtowc on c3 a9 returns 0xC3A9, and __STDC_ISO_10646__ is not defined. glibc defines it and promises the opposite. multibyte(3)
  • This Mac's iconv transliterates without being asked: Ł into Latin-1, CP1252 or GBK is L, ź is ´z, is -, all with exit 0 and nothing on stderr, which only the BUGS paragraph of iconvctl(3) admits. The ? its iconv(3) page promises appears for some targets and never for ASCII, where it stops as GNU does. iconv(1)
  • The GNU iconv(1) and iconv_open(3) pages document //TRANSLIT and //IGNORE, the flags What the page does not say found undocumented on the Mac. The GNU iconv(3) page also states the BSD ? fallback that this library measured by hand. iconv(1)
  • iconv -c drops and exits 1 on the Mac, drops and exits 0 on Ubuntu; //TRANSLIT writes 'e for é on the Mac and e on Ubuntu, and the Mac counts its own transliterations as invalid characters. iconv(1)

  • Apple's GB18030 stops at the BMP: its iconv decodes 84 31 A4 37 but refuses 90 30 81 30 (U+10000) and everything above, while GNU decodes the whole four-byte range, and 😀 is 94 39 FC 36. Python's codec on the same Mac agrees with GNU. The CJK pages

  • euc(5)'s recipe for a wchar_t is stale: under ja_JP.eucJP the single-shift byte the page says is first removed is packed into the value (0x8eb1 for SS2), and the page says codesets 2 and 3 where its own definitions mean 3 and 4. The CJK pages
  • Shift-JIS by two names: -f SHIFT_JIS and -f CP932 decode 5c as ¥ and \, 7e as and ~, identically on both machines; MSKANJI, the page's own ENCODING string, is refused by both iconvs and MS_KANJI accepted. The CJK pages
  • The code-page tables check out row for row: Latin-9 differs from Latin-1 in exactly eight rows, cp1252(7) has 27 rows in 0x800x9F and is missing exactly 81 8D 8F 90 9D, which both iconvs and Python refuse, and clearing bit 7 of KOI8-R Привет, мир gives pRIWET, MIR. charsets(7)
  • Neither tar ever wrote a hdrcharset record, and bsdtar lists café.txt from a pax archive in NFD and from a GNU-format archive in NFC, with the same bytes in both headers. A 1960 timestamp gets six different treatments from the two tars' four formats. tar(5) and cpio(5)
  • bitstring(3)'s bit 0 is the least significant bit of byte 0, bit_test returns 32 rather than 1, and Ubuntu has no bitstring.h at all; htonl(0x0A0B0C0D) is 0x0D0C0B0A on both of these x86_64 machines. byteorder(3)
  • glibc's fnmatch gives two answers in a UTF-8 locale: café matches caf? and caf??, a character-wise and a byte-wise reading accepted side by side; macOS gives one answer per locale. Patterns
  • A range is the locale's: [a-z] matches é only under glibc's en_US.UTF-8, [[=e=]] matches it under en_US.UTF-8 on both machines but not under C.utf8, and [[:lower:]] is the only portable spelling. REG_ILLSEQ exists on macOS only. Patterns
  • GNU cmp says byte 4 under C.UTF-8 and char 4 under C; BSD says char always. So the library's finding 18, that both print char, holds only in the C locale. Comparing and summing
  • bintrans(1) misdescribes its own base64: BSD base64 accepts -w as well as -b (finding 14 lists only -b), and rejects --ignore-garbage with exit 64 where the page says it is silently ignored. Binary to text
  • file's encoding guess is code, not magic: -e ascii and -e encoding each switch off half of it, -e soft (no magic at all) leaves it intact, and the page's own TODO says encoding.c is hard-coded. file(1) and magic(5)
  • BSD hexdump's %_u and %_c consult the locale and util-linux's do not; od's page says ** and hexdump's says *, and the newline is nl on one page and lf on the other. Seeing the bytes
  • The 4.4BSD rune API is still there and half-broken: sgetrune decodes c3 a9 to U+00E9 but libc prints a three-line depreciated notice to stderr, and setrunelocale compiles with a deprecated in macOS 10.4 warning and then segfaults. rune_t is wchar_t by typedef. mbrune(3) and rune(3)
  • isalpha(0xE9) is 1 in every non-C locale on the Mac and 0 in every locale on glibc, including C.UTF-8, and toupper(0xE9) is 0xC9 on the Mac and the identity on glibc; the obsolete 4.4BSD extension that lets isalpha take a code point still answers 1 for U+4E2D in 2026. ctype(3)
  • The two wcwidth tables are a version apart: the Mac's is Unicode 15.0, glibc 2.39's is 15.1, both return -1 for every Unicode 16.0 character, and SOFT HYPHEN is -1 on one and 1 on the other. Nothing on either machine pads printf '%-10s' by columns: the shells and both /usr/bin/printf pad by bytes, Python by code points. wcwidth(3)
  • wcscoll(3)'s BUGS paragraph is stale: it says the function falls back to wcscmp, and wcscoll(L"é", L"f") is -1 under en_US.UTF-8 on this Mac. strcasecmp folds 0xC9 and 0xE9 together in every non-C locale on the Mac, including UTF-8 ones, and never on glibc. strcoll(3)
  • strtol under an unsupported base: glibc returns 0 with EINVAL and never writes *endptr, where the Mac sets it to the input; "0b101" is 0 on both under C11 and 5 on glibc under C23, because the header redirects to __isoc23_strtol. Finding 16 reproduced on both. strtol(3)
  • util-linux rev never returns under LC_ALL=C on café (killed by timeout, exit 124) and is fine under C.UTF-8; BSD rev in C reverses the bytes. BSD fmt and BSD col in the C locale delete every byte above 0x7f, so col -b must run under the locale man rendered with. Text tools that count
  • BSD dd conv=ucase in a UTF-8 locale corrupts UTF-8 lead bytes (CAFÉ comes out 63 61 66 e3 89) because it is per-byte toupper; GNU folds ASCII only. conv=ibm is CP1047 on every ASCII byte tested and conv=ebcdic differs from it only at ^ and ~; none of the six tables is CP037, and above 0x7f they are a permutation rather than a code page. dd(1)
  • Bold reaches a pipe as N backspace N on the Mac and not at all on Ubuntu, where man-db strips formatting off a pipe; LC_CTYPE changes man 7 mandoc_char from 0 to 1,040 non-ASCII bytes, and mandoc -T ascii writes \('e as 'e where the page's table says e. The manual itself
  • Canonical-mode erase is per byte until iutf8 is set: Backspace after café hands the program 63 61 66 c3 on both platforms, and both fresh ptys start -iutf8; tty(4), dated 1992, never mentions it. The terminal
  • Smaller splits from the text tools: GNU fold counts bytes even in C.UTF-8 while BSD counts columns; wc -L is bytes on BSD and columns on GNU; uniq -i and paste -d '€' honour the locale on BSD only; look é misses on both machines in every locale on an alphabetically sorted file, exactly as its BUGS section says. Text tools that cut on lines
  • The two C locales are different locales, three pages along: under C, fgetws reads é as U+00C3 U+00A9 on the Mac and fails on the first line on glibc; printf("%ls", L"café") writes 63 61 66 e9 on the Mac and returns -1 with EILSEQ on glibc; putwc(U+00E9) writes e9 on the Mac and a ? with a success return on glibc. Streams, Formatting
  • Stream orientation is enforced only by glibc: after getwc, fgetc on the same stream returns the next byte on the Mac and EOF on glibc. After an EILSEQ, fgetws is stuck for good on both, and the Mac leaves ferror at 0. Streams
  • /usr/bin/vis on this Mac imports no locale function at all, so the MULTIBYTE CHARACTER SUPPORT section of its page describes nothing it does; and strnvisx in a UTF-8 locale writes é as the single byte e9 when the buffer holds a NUL, breaking the page's own invariant. Escaping into ASCII
  • glibc's //TRANSLIT is two tables: ż é € ß become ? ? EUR ss under LC_ALL=C and z e EUR ss under C.UTF-8; the and ß rules are built in, the é and ż rules are lines of translit_combining in /usr/share/i18n/locales. pl_PL's thousands separator is U+00A0 on the Mac and U+202F on glibc. The locale's source files, The locale
  • printf '%d' "'Ł" is 321 under UTF-8 and 197 under C from the BSD binary, GNU, zsh and bash 5.2, and -59, a signed byte, from bash 3.2 in either locale. Formatting The remaining family pages add their own; each one's Try it section is the place to look.

See also