Glossary

Level: reference

Short entries, each with the page that explains it in full. Alphabetical.

Term	In one line	Read more
abstract character repertoire	Level 1 of the Unicode encoding model: which characters exist at all, before any number is assigned. Unicode's answer is why é can be one character or a letter plus a combining mark	An encoding is four layers
alignment	The requirement that a value begin at an address divisible by its alignment — a property of the ABI, never of the file. It is why a struct is usually larger than the sum of its fields, and why a format string that inherits the compiler's rules is not portable. _Alignof(double) is 8 on 64-bit targets and 4 on 32-bit x86	Packing a record
arithmetic shift	A right shift that copies the sign bit into the vacated positions, so a negative value stays negative; the logical shift fills with zeros instead. Which one >> performs is decided by the operand's type, not by its bits — the same eight bits shift to -1 as an i8 and to 31 as a u8	Arithmetic has its own width
ASCII	The 1963 agreement assigning 128 numbers (0–127, seven bits) to characters; the first 128 rows of every table that came after	A character is a number
astral plane	Informal name for any of the 16 supplementary planes — every code point above U+FFFF. Emoji, historic scripts, and the only characters UTF-16 needs a surrogate pair for	Writing a code point
base64	The common binary-to-text encoding: 24 bits re-cut into four 6-bit groups, each indexing a 64-character alphabet. Costs exactly 33% more, encodes bytes and names no charset, and is not canonical — sixteen strings decode to the byte 0x41	Binary to text
base64url	Base64 with - and _ in place of + and / so the result survives a URL or filename; JWT fields also drop the = padding, since . already marks the boundaries	Binary to text
binary file	Not a kind of file but a verdict: a reader's conclusion that the bytes are not text to it — a NUL for grep and git, invalid UTF-8 for a decoder, never anything at all for Latin-1. Nothing in the file records it, and renaming the file does not change it	Binary is a verdict, not a property
binary-to-text encoding	A rewriting of a whole byte stream into printable characters so a channel that carries only text can carry it — base64, base32, hex, Ascii85. Distinct from an escape, which rewrites only the offending characters of text that is already text	Binary to text
bit	One switch, 0 or 1	A byte is eight bits
BMP	The Basic Multilingual Plane, plane 0: U+0000–U+FFFF, every character UTF-16 writes in a single 16-bit unit. Nearly all living scripts are in it	Writing a code point
BOM	Byte order mark, the code point U+FEFF written first in a file so a reader can tell the byte order; EF BB BF in UTF-8, where it marks nothing and still gets written	Byte order and the BOM
byte	Eight bits; 256 patterns; the numbers 0..255; the unit a file is measured in. Has no meaning until a table is applied	A byte is eight bits
CESU-8	UTF-8 applied to UTF-16's surrogate pairs rather than to code points, so a character above U+FFFF takes six bytes instead of four and is not valid UTF-8. Declared obsolete and internal-use-only by the Consortium — and what SAP HANA stores	Why UTF-16 stayed
char	Rust's character type: one Unicode scalar value, four bytes wide in memory whatever it holds, and one to four bytes once encoded into a String. It cannot hold a surrogate, which is why char::from_u32(0xD800) is None rather than a character	char is four bytes
character encoding form	Level 3: code points to code units of a fixed width — 8 bits for UTF-8, 16 for UTF-16, 32 for UTF-32. A form has no byte order; that is the next level down. UTF-16 names a form as well as a scheme	An encoding is four layers
character encoding scheme	Level 4: code units to bytes, which is where byte order is decided. Unicode has seven, and two of them — UTF-16 and UTF-32 — are compound: an optional BOM followed by a simple scheme	An encoding is four layers
character map	All four levels in one operation, characters straight to bytes. What an IANA charset= name actually identifies, so charset=utf-8 in an HTTP header names a character map rather than a scheme	An encoding is four layers
character property	A named fact Unicode records about a code point — its General_Category (Lu, Nd, Mn), its Script, whether it is Alphabetic or White_Space. A regex engine that can reach them writes \p{...}; Python's re cannot, and unicodedata reaches some of them one character at a time	"Supports Unicode" is a level, not a yes
checksum	Arithmetic over a record, stored in the record, so a reader can tell damage from data. Intel HEX takes the two's complement of the sum of the bytes from the length field to the last data byte, which makes the whole record — checksum included — sum to zero. Detects; does not locate, and does not correct	A record has to say what it is, how long it is, and whether it arrived
code page	A 256-entry table agreeing with ASCII on the first 128 and with nobody on the second 128 — Latin-1, Windows-1252, Latin-2, CP437	Code pages
code point	A character's number in Unicode, written U+00E9. A number, not a byte	Unicode code points
code unit	The fixed-width piece an encoding is written in: 8 bits for UTF-8, 16 for UTF-16, 32 for UTF-32. A code point takes one or more of them	UTF-16 and surrogates
coded character set	Level 2: characters to numbers. Unicode's is U+0000–U+10FFFF and nothing at this level says how many bits a number takes	An encoding is four layers
codespace	The range of integers a coded character set spans — for Unicode, U+0000–U+10FFFF, 1,114,112 positions of which 2,048 are surrogates. Older books call it the encoding space; the standard does not	An encoding is four layers
compatibility property	The shorthand character classes every engine ships — \w, \d, \s, \b — which UTS #18 ↗ Annex C defines as expressions over real properties rather than as ASCII ranges. \w is supposed to include \p{gc=Mark}, which is why an engine that gets it wrong breaks any script that writes vowels as marks	"Supports Unicode" is a level, not a yes
collation	The ordering of strings for a reader, as against the ordering of their code points. A table of weights rather than a rule derivable from the text — one per language, since the languages disagree — and the per-language part is a tailoring. Unicode specifies the algorithm (UTS #10 ↗) and ships a default table; the data your program uses comes from the C library, from ICU, or from a database	Sorting and collation
compose key	An X11 key (Multi_key) that starts a lookup: the keys after it are a path through a tree of sequences held in a plain text Compose file. 5,125 of them in the standard en_US.UTF-8 one, and none is a prefix of another, which is why it commits with no Enter	Typing a character you cannot type
continuation byte	Any byte after the first in a multi-byte UTF-8 sequence. It always starts 10, so it is exactly 80–BF, and no lead byte is ever in that range — which is what makes UTF-8 self-synchronising	UTF-8 by hand
control character	ASCII 0–31 and 127: commands to a teletype, not glyphs. TAB (9), LF (10) and CR (13) are the ones still in daily use	Control characters
CRLF	The two-byte Windows line ending, 0D 0A; Unix uses LF alone, 0A	CRLF vs LF
decode	Bytes → code points, under a named table. The inverse of encode	Encode and decode are verbs
deterministic sorting	Three promises sharing two adjectives. A stable sort keeps equal elements in their input order (a property of the algorithm); a deterministic sort returns the same output for the same input (also the algorithm); a deterministic comparison never calls two non-identical strings equal (a property of the comparison). Since a collation exists in order to ignore things, it makes ties on purpose — so a correct order can still be two orders, and only the third of these fixes that	Sorting and collation
digraph	Vim's two-character mnemonic for a character: Ctrl-K then e' gives é. 1,366 of them ship with Vim 9.1, and the table is Vim's own — not a standard, and not the same set the compose key covers	Typing a character you cannot type
encode	Code points → bytes, under a named table	Encode and decode are verbs
encoded-word	The MIME form =?charset?B?payload?= that carries non-ASCII text in an email header — and the only escape in common use that names the charset it wrapped	Escaping into ASCII
encoding	The rule for writing code points as bytes: UTF-8, UTF-16, Latin-1 … Also, loosely, any code page	03_Encodings
endianness	Which byte of a multi-byte number is written first: big-endian (most significant first) or little-endian. A single byte has none; the question exists from the moment a field is two bytes wide, and the file never records the answer — the reader supplies it, and every dump tool has a setting for it	The bytes do not say which end, Byte order and the BOM, Packing a record
escape sequence	A code point written in a source file as ASCII: \u{20AC} in Rust, \u20ac or \N{EURO SIGN} in Python. What goes inside it differs per language — a scalar value, a code point, a UTF-16 code unit — and that is the language telling you what it thinks a character is	Writing a code point
extended grapheme cluster	The variant of grapheme cluster that everything actually implements — UAX #29 ↗ defines a legacy one and an extended one, and only the extended rules hold an emoji ZWJ sequence or a flag together. When a language says "grapheme", it means this	A code point is not a character
field splitting	The shell re-cutting an unquoted expansion into words at every byte listed in IFS, before the command ever sees it — bash calls it word splitting. A run of whitespace separators collapses to one, a run of any other separator does not, so a two-byte character used as a separator produces an empty field between its halves. Quoting the expansion turns it off	The shell has no string type
frame	What a format wraps around the bytes an encoding produced, so a stranger's program can read them back: where a record starts, what kind it is, how long it is, whether it arrived. Independent of the encoding inside it — Intel HEX frames base16, and would frame anything else the same way	A record has to say what it is, how long it is, and whether it arrived
grapheme cluster	What a person calls one character: possibly several code points (e + combining acute; a flag; a family emoji). The fourth of the five rulers, and the one neither Python's nor Rust's standard library ships. How many there are is not a property of the string alone — it depends on which UAX #29 revision your segmenter implements	A code point is not a character
grapheme cluster boundary	Where a cursor may stop, defined by UAX #29 ↗. \X matches one cluster and \b{g} matches a boundary; both are UTS #18 ↗ Level 2, which is why . in most engines matches one code point instead	"Supports Unicode" is a level, not a yes
hex dump	A file shown as offset · hex bytes · ASCII guess, sixteen bytes per line; xxd, od, hexdump -C	Reading a hex dump
hex string	A run of hex digits, which is two different objects and does not say which: a number (leading zeros noise, no width, no byte order) or a byte string (leading zeros are bytes, width is the data, byte order agreed in writing)	Hex: a number, or a picture of bytes
hexadecimal	Base 16; sixteen digits 0–F counted on the ordinary odometer, and binary written four bits per digit, so a byte is always two digits	Counting in hexadecimal · Hex is a shorthand
IFS	The shell variable holding the separators field splitting uses; whitespace by default. It is a set of bytes, not a delimiter — the same distinction that separates sed from tr — and whether the locale reads that set as bytes or as characters decides how many fields you get	The shell has no string type
integer promotion	C's rule that every operand narrower than int is converted to int before an arithmetic operator runs. It is not optional and has no syntax, which is why 255 << 2 on an unsigned char is 1020 as an expression and 252 once stored back	Arithmetic has its own width
keysym	X11's name for a key's meaning — eacute, EuroSign — which is what a Compose file's rules are written in. Most characters have no mnemonic keysym and get a generic one built from the code point instead — U0CA0 for the Kannada ಠ	Typing a character you cannot type
Latin-1	ISO-8859-1. The code page where byte value = code point for 0..255, so decoding under it never fails — which makes it the wrong tool for detecting anything and the right one for undoing mojibake	Code pages
modified Base64	The payload alphabet inside a UTF-7 shift sequence: RFC 2045's base64 without the = pad, the length being recoverable from where the run ends. IMAP4rev1's mailbox names use a second variant again, with & for the shift and , for /	UTF-7, and the seven-bit transport
lead byte	The first byte of a UTF-8 sequence, which counts the sequence's own length in unary: 0 for one byte, 110 for two, 1110 for three, 11110 for four. So a decoder knows how far to read before reading anything else	UTF-8 by hand
MacBinary	The 1985 classic-Mac wrapper that packed a file's two forks and its Finder metadata into one stream, for systems with nowhere to keep them — which is why macOS still maps .bin to an archive and hands it to Archive Utility	Binary is a verdict, not a property
Modified UTF-8	Java's near-UTF-8: U+0000 written as the overlong C0 80, and characters above U+FFFF written as two three-byte surrogates (that half is CESU-8). Used in .class files, DataInput/DataOutput and JNI. Fails a UTF-8 validator, correctly	Overlong sequences
mojibake	Bytes decoded under the wrong table: é is C3 A9 (UTF-8 for é) read as Latin-1	Mojibake
MSB / LSB	Most / least significant bit — or byte, in a byte-order sentence, and the abbreviation is the same for both. Significant means the column is worth more, so it is a fact about the position and not about what is in it: the most significant bit of 0x21 is 0	Counting in hexadecimal
native format	struct's default when no <, >, ! or = prefix is given: the widths, byte order and alignment padding of the C compiler that built this Python. A format that round-trips perfectly on the machine that wrote it and writes a different-length record somewhere else	Packing a record
nibble	Four bits; one hex digit; half a byte	Hex is a shorthand
noncharacter	One of 66 code points permanently reserved and guaranteed never to be assigned: U+FDD0–U+FDEF, and U+FFFE/U+FFFF in each of the 17 planes. Not invalid — every UTF encodes them and a conformant decoder must preserve them — but never legitimate in interchange, which is what makes them usable as internal sentinels. The set is immutable by stability policy	Noncharacters and the private use areas
normalization	Rewriting a string to one canonical code-point sequence (NFC, NFD …) so that two spellings of é compare equal	Normalization
overlong form	A code point written in more bytes than it needs — C0 AF for /, C1 BD for } — which the templates decode fine and every validator rejects, because one character must have exactly one encoding. Forbidden by the shortest in step 2 of encoding by hand. Why C0 and C1 never appear in real UTF-8	Overlong sequences
padding	The dead bytes a compiler inserts between fields to satisfy alignment. Their contents are indeterminate in C, so two records with identical fields can differ byte for byte — breaking memcmp, hashing, and any signature taken over the struct. In a format string you can write it down instead (x in struct), and then it is visible	Packing a record
percent-encoding	%XX per byte, the escape a URL uses — the hex dump with a sign in front. Nothing in the URL records which encoding produced those bytes	Escaping into ASCII
plane	One of Unicode's 17 blocks of 65,536 code points. Plane 0 is the BMP; planes 1–16 are the supplementary, or astral, planes	Writing a code point
private use area	The 137,468 code points whose meaning the standard declines to define, so that a sender and receiver may agree one privately: U+E000–U+F8FF in the BMP, plus all of planes 15 and 16. How the Apple logo and every icon font are addressed. Category Co, no name, no properties, and no font you may assume	Noncharacters and the private use areas
primary, secondary, tertiary	UTS #10's three comparison levels, asked in order and never summed: L1 the base letters, L2 the accents, L3 the case. The first level that answers stops the comparison, which is why an accent difference loses to a base-letter one. Two locales that order a letter differently disagree about which level its difference lands on	Sorting and collation
punycode	The ASCII re-spelling of a domain label (żółw → w-uga1v8h, written xn--w-uga1v8h). Defined over code points, not bytes, so there is no encoding to guess	Escaping into ASCII
printability	Whether a character draws anything. str.isprintable() in Python and char::escape_debug in Rust ask it to decide, independently, whether to print a character as itself or as an escape	Writing a code point
record type	A small number at a fixed offset saying what shape the rest of a record has, so one file can carry several kinds of line and a reader knows which it has before parsing the payload. Paired with a length it is also forward compatibility: a reader can skip a type it has never heard of and carry on	A record has to say what it is, how long it is, and whether it arrived
replacement character	U+FFFD �, what a decoder writes in place of bytes it cannot read, if told to replace rather than raise	Encode, decode and errors
reserved	Set aside by the standard so that nothing may be assigned there — the noncharacters, the surrogates, and the private use areas, for three different reasons. Distinct from unassigned, which is a vacancy that a future release may fill, and from invalid, which is a verdict a particular encoder or format reaches and not a property of the code point	Noncharacters and the private use areas
ROT13	A rotation of the 26 ASCII letters by 13, so the same function encodes and decodes. Not encryption — the shift is in the name, the keyspace has one element, and Python ships it inside the encodings package	Rotation is not encryption
rune	Go's name for one code point: an alias for int32, so it holds -1, a surrogate or a number past U+10FFFF and checks none of them. .NET's Rune is the opposite — a checked scalar value, like Rust's char — and BSD's rune_t is wchar_t under another name	rune is an int32
scalar value	A code point that is not a surrogate: U+0000–U+D7FF and U+E000–U+10FFFF — 1,112,064 of the codespace's 1,114,112 positions. What UTF-8 can encode, and exactly what Rust's char can hold	char is four bytes
Set O	RFC 2152's twenty optional direct characters, < and > among them: a UTF-7 encoder may write them as themselves or escape them, and a decoder must accept both. The one clause that made the format non-canonical, and the reason it was deprecated rather than repaired	UTF-7, and the seven-bit transport
self-synchronising	The property that a byte announces its own role, so a reader dropped at any offset finds the next character boundary within three bytes and needs no state, no lookahead and no reading from the start. UTF-8 has it; UTF-16 has it only once you know where its 2-byte units begin, and a byte stream does not say	UTF-8 by hand
substitution cipher	A cipher that replaces each character using one table fixed for the whole message — Caesar, ROT13, Atbash, affine. The table is the key, and rotation is only the case where the table can be written as a sum	Rotation is not encryption
surrogateescape	Python's error handler for bytes that are not text: each undecodable byte 0x80–0xFF becomes the lone surrogate U+DC00 + byte, and the same handler puts it back. What os.listdir() uses on POSIX, so a filename nobody can decode can still be opened	Bytes that are not text
surrogatepass	The other Python handler, one letter apart and the opposite operation: it encodes the surrogate itself as three bytes (WTF-8), rather than restoring a byte. The Windows filesystem pair	Bytes that are not text
surrogate pair	Two 16-bit UTF-16 units (D800–DBFF then DC00–DFFF) standing for one code point above U+FFFF	UTF-16 and surrogates
text element	.NET's name for a grapheme cluster, and worth knowing because of what comes with it: System.Globalization.StringInfo ships an enumerator for the unit, which very few standard libraries do. string.Length still counts UTF-16 code units, so one woman-firefighter emoji is Length 7 and one text element	A code point is not a character
tofu	The empty box □ shown for a character the FONT has no glyph for. Not an encoding error at all — the bytes decoded correctly, and changing the encoding to chase it is how a working file gets damaged	Mojibake
transfer encoding syntax	A reversible transform of the bytes an encoding scheme produced — base64, quoted-printable, uuencode. Deliberately outside the four levels, because it neither knows nor cares that the bytes were text	An encoding is four layers
WTF-8	UTF-8 extended to hold unpaired surrogates, so UTF-16 data that is not well-formed can round-trip through bytes. Never for interchange; what Rust keeps OsString in on Windows, and what Python's surrogatepass writes	Bytes that are not text
UCS-2	Unicode as a fixed-width 16-bit encoding — one unit per character, nothing above U+FFFF. What "Unicode" meant from 1991 to 1996, what Windows, Java, JavaScript and SAP each built a string type on before surrogates existed — and what the ABAP language still is, which is why it reads a surrogate pair as two characters	Why UTF-16 stayed
Unicode	The one numbering for every character in every script: 1,114,112 code points, 17 planes	Unicode code points
universal character name	C's name for \uXXXX / \UXXXXXXXX. Translated before the source is tokenised, so it works in an identifier — and is forbidden below U+00A0, so it can never smuggle a quote or a semicolon past the parser	Writing a code point
UTF-7	A 1997 transformation format (RFC 2152 ↗) that writes all of Unicode in printable ASCII for seven-bit mail transports: + shifts into modified Base64 over UTF-16BE code units, - shifts out. Its Set O leaves escaping optional, so one string has thousands of legal spellings — obsolete in .NET since 5, and excluded from the web by the Encoding Standard's closed list	UTF-7, and the seven-bit transport
UTF-8	The encoding that writes a code point as 1–4 bytes, leaves ASCII unchanged, and is what nearly every file today is	UTF-8 by hand
UTF-16	The encoding that writes a code point as one or two 16-bit units; Windows, Java, JavaScript, and the system code page of a Unicode SAP system — though the ABAP language itself is the narrower UCS-2	UTF-16 and surrogates
UTI	Uniform Type Identifier: macOS's name for a file type (public.plain-text, com.apple.macbinary-archive), taken from the extension — or, with none, from the execute bit or a classic type code — and never from the content. It decides which app a double-click opens	Binary is a verdict, not a property
UTS #18 level	How Unicode-aware a regex engine is, said as a number instead of a yes — an unrelated sense of level from the four in character encoding form above. Level 1 is eight requirements — hex notation, properties, the compatibility classes, set operations, word and line boundaries, simple case folding, supplementary code points; Level 2 is seven more, including canonical equivalence and grapheme clusters. Level 3 was retracted. Partial conformance is expected: name the level, then name what you are missing	"Supports Unicode" is a level, not a yes
validation	Checking that a run of bytes really follows an encoding's rules. Cheap, done once, at the edge of a program — and what a language remembers about it afterwards is the whole difference between them	Validation is a boundary
Windows-1252	Latin-1 with the 32 bytes 0x80–0x9F reassigned to €, smart quotes and friends; SAP code page 1160	Windows-1252 vs Latin-1
xstring	ABAP's byte-sequence type, displayed in hex; the counterpart of Python's bytes. string and c are characters	Hex is a shorthand