Skip to content

Unicode code points

Level: 101 → 201 · for anyone starting from zero

One line: A code point is a number and U+XXXX is that number in hex — and the skill worth having is not memorising them but reading one as an address: a block, then a house number inside it.

ord('é')          # 233
hex(ord('é'))     # '0xe9'   -> written U+00E9
chr(0x20AC)       # '€'

U+00E9 and 0xE9 are the same number and two different things. One is a code point — Unicode's name for the character. The other is a byte, which means nothing until somebody names a table. Keeping those apart is most of this chapter; how a code point becomes bytes is chapter 3.

Read it as a street address

There are 1,114,112 code points, so nobody holds them in their head. What people who work with text every day actually hold is the map: a couple of dozen block boundaries, nearly all of them round hex numbers, and each block's contents. Then U+20AC is not a number to store — it is currency block, house AC.

That is worth saying out loud because the alternative looks like memorisation and isn't. You will meet U+00A0 and U+2028 and U+000B often enough that they feel like trivia to be drilled. They are not: each one sits at an address you can derive.

The three that pay for themselves

The whitespace run is 9 A B C D. U+0009 TAB, U+000A LF, U+000B VT, U+000C FF, U+000D CR — five consecutive, in order of how far the cursor travels: right a little, down one line, down a block, down a page, back to column 0. And ASCII whitespace is that run minus B, plus 0x20, which turns the VERTICAL TAB oddity ↗ ↗ from a fact into a shape: B is the hole in 9ABCD.

0xA0 is 0x20 | 0x80. NO-BREAK SPACE is SPACE with the top bit set. Same arithmetic as the case bit you already use without thinking — 'A' is 0x41, and 0x41 ^ 0x20 is 0x61, 'a'.

Anything 20-ish is space-ish. U+0020 SPACE, U+00A0 NBSP, U+2000–U+200A the typographic spaces, U+2028/U+2029 the separators, U+202F narrow NBSP. One digit-pair to recognise covers most of the family.

The blocks are the whole trick

Verified output of unicode_code_points_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. U+00E9 IS A NUMBER, WRITTEN IN HEX
------------------------------------------------------------------------
   U+00E9   decimal 233   binary 11101001
   chr(233) = 'é'   ord('é') = 233
   name: LATIN SMALL LETTER E WITH ACUTE

   U+00E9 and 0xE9 are the same NUMBER and different things:
   one is a code point, the other is a byte under one table.
   Which bytes carry U+00E9 is chapter 3's question, not this one.

2. READ IT AS AN ADDRESS: BLOCK, THEN HOUSE NUMBER
------------------------------------------------------------------------
   block     what lives there            example
   U+0000    ASCII                       U+0041   A
   U+0080    Latin-1's high half         U+00A0   NBSP, invisible
   U+0100    Latin Extended-A            U+0104   Ą
   U+0370    Greek                       U+03A9   Ω
   U+0400    Cyrillic                    U+042F   Я
   U+2000    General Punctuation         U+2014   — em dash
   U+20A0    Currency Symbols            U+20AC   €
   U+3000    CJK Symbols/Punctuation     U+3000   ideographic space
   U+4E00    CJK Ideographs              U+4E00   一
   U+1F300   Emoji                       U+1F600  😀

   You do not memorise U+20AC. You memorise that currency starts
   at U+20A0, and read the AC as a house number on that street.

3. THE FIRST 256 CODE POINTS *ARE* LATIN-1 -- WHICH IS THE TRAP
------------------------------------------------------------------------
   bytes 0x00-0xFF decoded as latin-1: 256 of 256 succeed
   0xE9 -> 'é', and ord() of it is 233

   Latin-1 is the identity table: byte N is code point N. So it can
   never raise on decode -- and a mis-tagged file therefore does not
   fail, it just quietly reads as the wrong letters.

4. WHERE POLISH LIVES
------------------------------------------------------------------------
   Ą  U+0104  LATIN CAPITAL LETTER A WITH OGONEK
   Ć  U+0106  LATIN CAPITAL LETTER C WITH ACUTE
   Ę  U+0118  LATIN CAPITAL LETTER E WITH OGONEK
   Ł  U+0141  LATIN CAPITAL LETTER L WITH STROKE
   Ń  U+0143  LATIN CAPITAL LETTER N WITH ACUTE
   Ó  U+00D3  LATIN CAPITAL LETTER O WITH ACUTE
   Ś  U+015A  LATIN CAPITAL LETTER S WITH ACUTE
   Ź  U+0179  LATIN CAPITAL LETTER Z WITH ACUTE
   Ż  U+017B  LATIN CAPITAL LETTER Z WITH DOT ABOVE

   All but Ó are in Latin Extended-A (U+0100-U+017F), because the
   Latin-1 block had no room left. Ó is at U+00D3, inside Latin-1,
   which is why it alone survives a wrong-table read intact.

5. THREE PIECES OF ARITHMETIC, AND FOUR FEWER THINGS TO MEMORISE
------------------------------------------------------------------------
   (a) the whitespace run 9-A-B-C-D, in cursor-travel order
       U+0009  TAB  right a little
       U+000A  LF   down one line
       U+000B  VT   down a block
       U+000C  FF   down a page
       U+000D  CR   back to column 0
       ...and ASCII whitespace is that run MINUS B, PLUS 0x20.

   (b) NBSP is SPACE with the top bit set
       0x20 | 0x80 = 0xA0  -> NO-BREAK SPACE

   (c) the same 0x20 is the case bit you already use
       'A' is 0x41, and 0x41 ^ 0x20 = 0x61 = 'a'

6. SEVENTEEN PLANES, AND THE 2,048 SLOTS THAT ARE NOT CHARACTERS
------------------------------------------------------------------------
   17 planes x 65,536 = 1,114,112 slots
   minus 2,048 surrogates (U+D800-U+DFFF) reserved for UTF-16
   = 1,112,064 usable scalar values

   Python will build a str from a surrogate; Rust's char will not.
   chr(0xD800) is fine here: '\ud800'
   ...until you ask for its bytes: UnicodeEncodeError

Section 3 is the one to sit with. The first 256 code points are Latin-1 — byte N is code point N, the identity table — which is exactly why decoding as Latin-1 never raises, and therefore why a mis-tagged file does not fail loudly but reads as the wrong letters. That is the mojibake mechanism in one line.

Section 4 is the Polish reader's version of the same map. Eight of the nine letters live in Latin Extended-A, U+0100–U+017F, because Latin-1's block was full by the time anyone got to them; Ó alone sits down at U+00D3 inside Latin-1, which is why it is the one letter that survives a wrong-table read intact while the rest turn to soup.

char is one code point, and the type says so

Verified output of unicode_code_points_rs.rs — regenerated by tools/run_examples.py, never hand-typed.

1. A char IS A CODE POINT, AND from_u32 IS THE GATE
------------------------------------------------------------------------
   U+0041    Some('A')
   U+00E9    Some('é')
   U+0104    Some('Ą')
   U+20AC    Some('€')
   U+1F600   Some('😀')
   U+D800    None   <- not a scalar value
   U+110000  None   <- not a scalar value

   The last two are the only two ways to fail: a surrogate,
   and anything past the top of the number line.

2. THE HOLE IN THE MIDDLE: U+D800..=U+DFFF
------------------------------------------------------------------------
   code points in that range rejected by char: 2048
   they exist only so UTF-16 can spell the upper planes,
   which is why they are reserved and never characters.

3. THE COUNT IS DEFINITIONAL, NOT A TABLE LOOKUP
------------------------------------------------------------------------
   0x110000            =   1114112   17 planes of 65,536
   minus the surrogates=      2048
   usable scalars      =   1112064

   and every one of them fits in a char, which is always
   4 bytes wide -- a code point, not a UTF-8 byte.

4. WRITING ONE DOWN IN SOURCE
------------------------------------------------------------------------
   '\u{E9}' is 'é', and as a number 233
   escape_unicode goes the other way: \u{104}
   a char knows its own neighbourhood by arithmetic:
      is_ascii()      false
      is_alphabetic() true
      len_utf8()      2   <- chapter 3's question

5. THE NEIGHBOURHOOD MAP, MEASURED: WHERE WHITESPACE LIVES
------------------------------------------------------------------------
   25 code points, in 10 runs -- not 25 things to learn:

   U+0009..=U+000D     TAB LF VT FF CR -- the run 9-A-B-C-D
   U+0020              SPACE, the first printable code point
   U+0085              NEL, over in the C1 controls (0080-009F)
   U+00A0              NO-BREAK SPACE = 0x20 | 0x80
   U+1680              OGHAM SPACE MARK, first slot of its block
   U+2000..=U+200A     the printer's type case: quads down to a hair
   U+2028..=U+2029     LINE then PARAGRAPH separator, smaller first
   U+202F              NARROW NO-BREAK SPACE
   U+205F              MEDIUM MATHEMATICAL SPACE
   U+3000              IDEOGRAPHIC SPACE, where the CJK zone opens

   Four neighbourhoods carry all of it: the ASCII controls,
   Latin-1's high half, the 2000 punctuation block, and CJK.

   The two exceptions both end in B, and both are traps:
      U+000B  is_whitespace true  is_ascii_whitespace false
      U+200B  is_whitespace false is_ascii_whitespace false
   U+000B VERTICAL TAB is whitespace but not ASCII whitespace.
   U+200B ZERO WIDTH SPACE is named a space and is neither.

Two things worth taking from that. The number line has a hole — U+D800–U+DFFF, 2,048 slots that exist only so UTF-16 can spell the upper planes. Python will hold one in a str and complain only when you ask for bytes; Rust refuses a step earlier, at char::from_u32, so the gap is enforced by the type.

And section 5 is the map measured rather than quoted: every code point Unicode calls whitespace, printed as the runs it forms. Twenty-five characters, ten runs, four neighbourhoods — the ASCII controls, Latin-1's high half, the 2000 punctuation block, and the CJK zone. That is the difference between learning a list and learning a layout.

The 2000 run is worth one more look, because it explains its own last member. It is a printer's type case, shrinking as it goes: en and em quads, en and em spaces, thirds, quarters, sixths, a figure space, a punctuation space, a thin space, a hair space — and then U+200B, zero. It ran the width down to nothing, which is precisely why ZERO WIDTH SPACE has White_Space=no despite the name. It shrank past being a space.

Which gives the last mnemonic, and it is the useful kind because it names both traps at once: the two exceptions both end in B. U+000B VERTICAL TAB is whitespace but not ASCII whitespace; U+200B ZERO WIDTH SPACE is called a space and is neither.

If you are coming from Python or ABAP

Python. ord and chr are code-point functions, not byte functions — ord('é') is 233 whatever encoding the file was saved in, and chr(0x20AC) is '€' on every platform. unicodedata.name is how you ask a number what it is, and unicodedata.lookup('EURO SIGN') is the inverse. The one place Python is looser than Rust is the surrogate range: chr(0xD800) builds a str quite happily and only fails at .encode(), so a lone surrogate can travel some distance through a Python program before anything objects.

ABAP. (Not machine-checked — CI cannot run ABAP.) A string holds 16-bit units — the system code page is UTF-16, though the ABAP language itself is the UCS-2 subset, which SAP glosses as "mainly UTF-16 without surrogates" — so the unit ABAP counts is a code unit, not a code point — which is the same split UTF-16 and surrogates describes, and it means strlen over an emoji returns 2 where Python's len returns 1 and Rust's .chars().count() returns 1. To go from a number to a character, cl_abap_conv_in_ce and the cl_abap_char_utilities constants are the usual doors; the honest summary is that ABAP gives you named constants for the handful of code points it cares about rather than a general chr. Anything involving an SAP code-page number should be verified against the system rather than quoted.

Try it

  • Look up three code points you have met in real files — the em dash a word processor inserted, the € in a price column, the non-breaking space that made a split() behave oddly — and place each one on the map above without looking up its number first.
  • Take a file you suspect is mis-tagged and decode it as latin_1. It will always succeed. Ask what that proves. (Answer: nothing at all — it is the identity table.)
  • Print hex(ord(c)) for every character of your own name. If any of them is above U+007F, that character is where a wrong table would break it.

Practice

Read four addresses. For A, é, 中 and 😀: give the code point in U+ form, name the block it lives in, say whether it is in the BMP or above it, and give its length in UTF-8 and UTF-16 bytes.

One of the four costs two code units in UTF-16. Say which, and what that pair is called.

Answers

Verified output of unicode_code_points_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

A  U+0041
   name        LATIN CAPITAL LETTER A
   block       Basic Latin (ASCII), house 65
   plane       0  (BMP)
   utf-8       41           1 byte(s)
   utf-16      41 00        2 byte(s)

é  U+00E9
   name        LATIN SMALL LETTER E WITH ACUTE
   block       Latin-1 Supplement, house 105
   plane       0  (BMP)
   utf-8       c3 a9        2 byte(s)
   utf-16      e9 00        2 byte(s)

ż  U+017C
   name        LATIN SMALL LETTER Z WITH DOT ABOVE
   block       Latin Extended-A, house 124
   plane       0  (BMP)
   utf-8       c5 bc        2 byte(s)
   utf-16      7c 01        2 byte(s)

中  U+4E2D
   name        CJK UNIFIED IDEOGRAPH-4E2D
   block       CJK Unified Ideographs, house 45
   plane       0  (BMP)
   utf-8       e4 b8 ad     3 byte(s)
   utf-16      2d 4e        2 byte(s)

😀  U+1F600
   name        GRINNING FACE
   block       Emoticons, house 0
   plane       1  (astral -- above U+FFFF)
   utf-8       f0 9f 98 80  4 byte(s)
   utf-16      3d d8 00 de  4 byte(s)  -- a SURROGATE PAIR, two code units

Three things the addresses tell you before any arithmetic.

The number IS the character's identity, and the hex is only how it is
written down: U+0041 and 65 are the same fact. The four digits are a
convention, not a width -- U+1F600 needs five and is not a bigger KIND
of thing, just a house further along.

The block is the useful half of the address. Latin-1 Supplement holds
the accented letters of western Europe because it inherited a code
page's layout; CJK Unified Ideographs is 20,000 houses on one street.
Knowing the street tells you what a neighbour probably is.

And the cost is not the same question as the address. One character is
1 to 4 bytes in UTF-8 and 2 or 4 in UTF-16, so 'how long is this text'
has a different answer per encoding -- and U+1F600 is the case where
UTF-16 needs two code units, which is where surrogates come from.

See also