char is four bytes¶
Level: 101 → 201 · for anyone learning Rust
One line: A char is one Unicode scalar value stored in four bytes, while the same character inside a String takes one to four — so chars().count() and len() disagree, and neither of them counts what a person calls a character.
This is the fourth of the four checkpoints this library was built around, and the disagreement in that sentence is the whole of it. size_of::<char>() is 4. 'é'.len_utf8() is 2. "é".len() is 2. 'é' as u32 is 233. Four numbers, one character, and not one of them is wrong — they answer four different questions, and the only mistake available is to think there is a single number called "the size of a character".
Two widths, and why a language needs both¶
A char is a fixed-width box. Whatever you put in it — A, é, 😀 — occupies four bytes, so an array of them is 4 × n and the tenth one is at offset 40. That is the property arithmetic wants: comparing, ranging, matching and as u32 are all a single machine word away, and chars[9] needs no scan.
UTF-8 is a variable-width encoding. A costs one byte, é two, € three, 😀 four, and the tenth character is wherever the first nine happened to end. That is the property storage and interchange want: English text stays the size it was in ASCII, and no byte of a multi-byte character can be mistaken for the start of another one.
Rust does not choose between them. It uses the fixed box for a character in flight and the variable encoding for a character at rest, and puts the conversion in plain sight: .chars() decodes, .encode_utf8() and pushing a char onto a String encode. So the two numbers that disagree are not a wart. They are the two representations, each doing the job the other is bad at, and the disagreement is where the boundary between them is.
Why four bytes and not three. The largest code point is U+10FFFF, which needs 21 bits. Twenty-one bits round up to 24, and 24 rounds up again — there is no three-byte integer on any machine Rust targets, no register that holds one and no alignment that likes one. So char is a u32 with a hole in it, and the hole is the point.
The hole is what "scalar value" means. char::from_u32 refuses 0xD800 through 0xDFFF: those 2,048 code points are the surrogates, reserved so that UTF-16 can reach past U+FFFF, and they never stand for a character on their own. Unicode scalar value is the official name for "code point, minus those" — and it is the exact set char can hold. Python's chr(0xD800) builds a perfectly ordinary one-character str; Rust's constructor returns None, and there is no way past it in safe code.
And the hole pays rent. Because most of a u32's bit patterns are not valid chars, the compiler stores Option<char>'s None inside one of them: size_of::<Option<char>>() is 4, the same as char, while size_of::<Option<u32>>() is 8. The restriction that makes char feel strict is the same restriction that makes it free to wrap.
In Rust¶
Verified output of char_is_four_bytes_rs.rs — regenerated by tools/run_examples.py, never hand-typed.
1. FOUR NUMBERS FOR ONE CHARACTER, AND ONLY ONE OF THEM IS FOUR
code pt as u32 size_of len_utf8 len_utf16 glyph
U+41 65 4 1 1 A
U+E9 233 4 2 1 é
U+20AC 8364 4 3 1 €
U+1F600 128512 4 4 2 😀
size_of is a property of the TYPE and never moves. len_utf8 is a property
of the VALUE, and it is what the same character costs inside a String.
2. SO THE SAME FOUR CHARACTERS HAVE TWO SIZES
[char; 4] 16 bytes 4 x 4, whatever is in it
String 10 bytes 1 + 2 + 3 + 4, because each one was encoded
41 C3 A9 E2 82 AC F0 9F 98 80
and that String is "Aé€😀"
A char is a fixed-width box. UTF-8 is a variable-width encoding. Putting a
char into a String encodes it; taking one out decodes it. Neither is free.
3. THREE WALKS OVER ONE STRING
"café" len() = 5 bytes
bytes() 63 61 66 C3 A9
chars() c a f é
char_indices() 0:c 1:a 2:f 3:é
Five bytes, four chars. char_indices() gives the BYTE offset of each char,
which is the only index a String will accept — and why the last one is 3,
not 4. Counting characters to build an index is the bug this prevents.
4. THE ONE THING A char CANNOT HOLD
char::from_u32(0x00D7FF) Some — a scalar value
char::from_u32(0x00D800) None — not a scalar value
char::from_u32(0x00DFFF) None — not a scalar value
char::from_u32(0x00E000) Some — a scalar value
char::from_u32(0x10FFFF) Some — a scalar value
char::from_u32(0x110000) None — not a scalar value
The hole from D800 to DFFF is the surrogate range, which exists so UTF-16
can reach past U+FFFF. A char is a Unicode SCALAR VALUE, and that phrase
means exactly 'code point, minus those two thousand and forty-eight'.
char::MAX = '\u{10ffff}', so the widest value is 21 bits — which is why 4 bytes,
not 3: there is no 24-bit integer to be addressed on any machine here.
5. AND THE HOLE PAYS FOR ITSELF
size_of::<char>() 4
size_of::<Option<char>>() 4 <- the None is stored INSIDE the gap
size_of::<u32>() 4
size_of::<Option<u32>>() 8 <- a u32 has no spare bit patterns
Every u32 from 0 to 0xFFFFFFFF is a valid u32, so Option needs a separate
byte to say which case it is (and then 3 more for alignment). A char has
over four billion invalid patterns, so None can be one of them, for free.
The restriction that makes char awkward is the same one that makes it cheap.
6. WRITING ONE, AND READING IT BACK
'\u{1F600}' is 😀, and 128512 == 128512
.escape_unicode() \u{1f600}
'é'.escape_unicode() \u{e9}
b'A' 65 <- a byte literal: u8, not char, no Unicode at all
'A' as u32 65 <- and here the two agree, which is why ASCII hides this
And one cast that compiles and lies. `as u8` on a char TRUNCATES:
U+41 as u32 = 65 as u8 = 65 same value
U+E9 as u32 = 233 as u8 = 233 same value
U+20AC as u32 = 8364 as u8 = 172 TRUNCATED
U+1F600 as u32 = 128512 as u8 = 0 TRUNCATED
Only `as u32` is lossless. u8 keeps the low eight bits and says nothing.
7. NONE OF THESE COUNTS WHAT A PERSON CALLS A CHARACTER
written as bytes chars utf-16 graphemes
len() .count() len_utf16 (by eye)
"café" 5 4 4 4
"cafe\u{301}" 6 5 5 4
"\u{1F600}" 4 1 2 1
Rows 1 and 2 are the same word. One is written with a combining acute, and
chars() counts it as a character of its own — correctly, because it IS a
scalar value. What a cursor moves over is a GRAPHEME CLUSTER, and there is
no method for it in std: the tables move with every Unicode release, so the
unicode-segmentation crate carries them and the standard library does not.
Section 3 is the one to carry away. char_indices() hands you a byte offset with each character — 0, 1, 2, 3 for café, where a character count would have said 0, 1, 2, 3 too and then diverged on the next word. That offset is the only kind of index a String will take, which is why building one by counting characters is the bug that Slicing by byte is entirely about.
Section 6's last block is the cast that compiles and lies. as u32 is lossless for every char; as u8 keeps the low eight bits and tells you nothing, so '€' as u8 is 172 and '😀' as u8 is 0. Both are legal Rust. And 'é' as u8 is 233, which is correct — é really is byte E9 in Latin-1 — so the one case a Polish or French developer is most likely to test is the case that hides the bug.
In Python¶
Python has no character type at all, and arrives at the same three widths from the opposite end.
Verified output of char_is_four_bytes_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. THERE IS NO CHARACTER TYPE TO ASK ABOUT
s = 'café'
s[3] 'é' type = str len = 1
s[3][0] 'é' type = str len = 1
Indexing a str gives a str of length 1, which indexes to itself, forever.
Rust's char is a separate type with a separate size. Python's is a one-character
string, so `size_of::<char>()` has no Python question to be the answer to.
2. WHAT PYTHON HAS INSTEAD, AND IT IS THE SAME NUMBER FROM THE OTHER END
code pt ord() utf-8 bytes/char glyph
U+0041 65 1 1 A
U+00E9 233 2 1 é
U+20AC 8364 3 2 €
U+1F600 128512 4 4 😀
The last column is PEP 393: a str picks ONE width for all of its characters —
1, 2 or 4 bytes — from its widest member. Rust spends 4 bytes on every char and
1 to 4 inside a String; Python spends 1 to 4 per string and the same for every
character in it. Same three widths, opposite thing held fixed.
3. SO ONE CHARACTER CAN QUADRUPLE A STRING
'a' * 1000 len = 1000 1 byte/char 1000 bytes of text
'a' * 999 + '\U0001F600' len = 1000 4 byte/char 4000 bytes of text
Same length, 4x the text. ONE character forced the whole string up to
4 bytes each — there is no mixed-width str. UTF-8 would have spent 1003 bytes
on it; the in-memory form is not UTF-8 and was never trying to be.
(Measured as a per-character difference. A bare sys.getsizeof would add a
header whose size is a fact about this CPython build, not about the text.)
4. THE HOLE RUST LEAVES IN char, PYTHON DOES NOT LEAVE IN str
chr(0x00D7FF) len = 1 .encode('utf-8') -> ed 9f bf
chr(0x00D800) len = 1 .encode('utf-8') -> UnicodeEncodeError
chr(0x00DFFF) len = 1 .encode('utf-8') -> UnicodeEncodeError
chr(0x00E000) len = 1 .encode('utf-8') -> ee 80 80
chr(0x10FFFF) len = 1 .encode('utf-8') -> f4 8f bf bf
Every one of those is a str Python will build, index, slice, sort and use as a
dict key. Two of them cannot be encoded to UTF-8 at all — so the check Rust
makes at char::from_u32 is one Python makes at .encode(), and only there.
That gap is the room PEP 383's surrogateescape lives in.
5. AND NO INDEX INTO A PYTHON STRING IS EVER A BYTE INDEX
s = 'café'
len(s) 4 <- characters
len(s.encode()) 5 <- bytes; Rust's len() is THIS number
s[3] 'é'
s.encode()[3] 195 <- an int, and only half of the character
Rust has char_indices() because its two counts are both needed. Python does not,
because you can only ever have the first one — which is friendlier right up to
the moment you have to write the byte offset into a fixed-width record.
Section 2 is the mirror image, and it is worth stating slowly. Rust fixes the width of a character (4 bytes, always) and lets the width vary along a string. Python — since PEP 393 ↗ — fixes the width of a string and lets it vary between strings: every character in a given str costs 1, 2 or 4 bytes, chosen once from the widest character in it. Same three widths, opposite thing held constant.
Section 3 is what that costs. One 😀 in a thousand-character string quadruples the text, because there is no mixed-width str — and len() still says 1000, so nothing in the program reports it. That is not a defect either; it is the price of Python's s[3] being an O(1) character index, which is exactly the operation Rust refuses to offer.
Section 4 is the checkpoint restated as a boundary. chr(0xD800) gives Python a str it will index, slice, sort and use as a dict key — and refuse to encode. Rust makes that check at the constructor, so the value never exists; Python makes it at .encode(), so the value exists and travels. Neither is wrong, and the gap between the two moments is the room surrogateescape lives in.
If you are coming from Python or ABAP¶
Python. There is no char to port, and that is the whole difficulty. s[3] in Python returns a one-character str; the nearest Rust is s.chars().nth(3), which returns Option<char> and walks the string to get there — O(n), not O(1), because the byte offset of the fourth character is not knowable without decoding the first three. So a Python loop written as for i in range(len(s)): s[i] becomes for c in s.chars() in Rust, and any translation that keeps the index is quadratic. Going the other way, ord() is c as u32 and chr() is char::from_u32, but note the asymmetry: chr() returns a str and cannot fail below 0x110000, while char::from_u32 returns Option and refuses the surrogates. When you port a code-point table, that None is a real case and not a formality. And len() is the trap in both directions — Python's counts characters, Rust's counts bytes, so every length that crosses between the two languages is a bug until you have said which one you meant.
ABAP. (Not machine-checked — CI cannot run ABAP.) ABAP has the fixed-width character type Rust has, and it is narrower: a c field is a run of characters, and on a Unicode system each one is a UTF-16 code unit, not a scalar value. That is the UCS-2 inheritance, and it means 😀 occupies two positions in a c field and strlen( ) counts it as 2 — the len_utf16 column in section 1, arrived at by a different route. So the ABAP number to compare against Rust's chars().count() is not strlen( ); there is no built-in that gives it, because ABAP's character is the code unit and Rust's is the scalar value. cl_abap_conv_out_ce / cl_abap_codepage=>convert_to( ) cross to xstring, and xstrlen( ) on the result is Rust's len(). Two counts, two keywords, already separate — which is more than most languages give you, and one rung short of what this page is about. Verify any specific code-page number against the system rather than a document.
Try it¶
- Run the Rust example, then add a character of your own to
CAST— a Polishż, a CJK ideograph, an emoji you actually use — and predictlen_utf8andlen_utf16before you compile. UTF-8 by hand gives you the first without running anything. - Take the longest single field from a real file of yours — a name column, a subject line — and print
s.len(),s.chars().count()ands.chars().map(char::len_utf16).sum::<usize>(). If any two disagree, that field has been lying to at least one system it passes through. - In Python, run
width_of()over the strings in a dictionary you already have in memory. Find the one string that is holding its whole container at 4 bytes per character, and decide whether you care. - Write
let c: char = 'x';and tryc as u8on the widest character you can type. Then go and look foras u8in a codebase you own; every one of them is either a deliberate byte truncation or a bug, and the two look identical.
Practice¶
Four numbers, one character. For '€' — U+20AC — write down size_of::<char>(), '€'.len_utf8(), '€'.len_utf16() and '€' as u32, before running anything. Then predict size_of::<Option<char>>() and say why it is not five.
Then the harder half. For the string "€1", give len(), chars().count() and the byte offsets char_indices() yields. Say which of those three numbers a fixed-width database column of "2 characters" is actually measuring, and what char::from_u32(0xDC00) returns and why.
Answers
Verified output of char_is_four_bytes_kata_rs.rs — regenerated by tools/run_examples.py, never hand-typed.
THE FOUR NUMBERS FOR '€'
size_of::<char>() 4 the TYPE's width. Never moves, for any value.
'€'.len_utf8() 3 what it costs inside a String
'€'.len_utf16() 1 what it costs in Java, JavaScript or ABAP
'€' as u32 8364 the code point itself, U+20AC
AND THE FIFTH, WHICH IS THE FIRST AGAIN
size_of::<Option<char>>() 4
Not 5, and not 8. A char has more invalid bit patterns than valid ones —
everything above 0x10FFFF, and the surrogate block — so None is stored as
one of them. The tag needs no room of its own. Compare a type with no
spare patterns: size_of::<Option<u32>>() = 8, which is 4 for the value,
1 for the tag and 3 for alignment.
THE STRING "€1"
len() 4 BYTES: 3 for the euro sign, 1 for the digit
chars().count() 2 scalar values
char_indices() 0:€ 3:1
The second character starts at byte 3, not byte 1. Every index Rust will
accept is in the first column; every index a person counts is in the second.
THE COLUMN THAT SAYS "2 CHARACTERS"
Almost always len(), the byte count — 4 here, so "€1" does not fit.
Oracle VARCHAR2(2) defaults to BYTE semantics and needs CHAR spelled out;
MySQL's utf8mb3 sizes in characters but stores three bytes each; a COBOL or
ABAP fixed field is bytes on disk whatever the program calls it. The rule
that survives all of them: a width with no unit is a byte count.
char::from_u32(0xDC00)
None
0xDC00 is a LOW SURROGATE. Surrogates are the 2,048 code points UTF-16
reserves to encode everything above U+FFFF as a pair, so they never stand
for a character alone. A char is a Unicode scalar value, which is defined
as a code point that is not one of these — so the constructor cannot
return one, and returns None rather than a value you would have to check.
For comparison, the code point either side of the block:
0xD7FF Some('\u{d7ff}')
0xDC00 None
0xE000 Some('\u{e000}')
See also¶
Stringis bytes that promise UTF-8 — the container these characters live in, and the promisechars()relies on- Slicing by byte — what happens when an index lands between two bytes of one
char - From UTF-8, and lossy — the three doors that turn bytes into characters
- Unicode code points — where
U+00E9comes from, and what a code point is - A code point is not a character — the five lengths, and why
chars().count()is not the last one - UTF-16 and surrogates — the 2,048 code points
charrefuses, and what they are for - A
strin memory — the Python half of section 2 at length: PEP 393, and whylen()is still O(1) - Meet the
char↗ — the sibling library's tour of the type and its methods - Why a
charis 32 bits wide ↗ — the same four bytes, argued from the language's side