Skip to content

A char is a byte, not a character

Level: 201 · working knowledge

One line: char is the machine's byte, so an accented letter is more than one of them, strlen counts bytes and not letters, toupper changes only the ASCII ones, and the <ctype.h> functions must be handed a value that fits in an unsigned char — because a plain char may be signed, and asking isalpha about a negative number is undefined.

The type is named char, which suggests it holds a character. It holds a byte. On the day UTF-8 arrived that stopped being the same thing: a letter outside ASCII is two, three or four bytes, and each of them is a char. C never learned this, and did not need to — its string functions kept working precisely because they treat text as bytes and never ask what a byte means. That is a strength at the edge of the machine and a trap the moment you assume one char is one letter.

The program

char_is_a_byte_c.c takes the word café — four letters a reader sees — and shows what C sees:

char_is_a_byte_c.c in full — pasted here by tools/run_examples.py from the file CI runs.

/* `char` is the machine's byte, not a letter. A string counts bytes; an
   accented letter costs more than one; `toupper` touches ASCII only; and the
   <ctype.h> functions must be handed a value that fits in unsigned char, or
   the call is undefined. The str/bytes split Python and Rust give you, C does
   not: there is only the byte. */
#include <stdio.h>
#include <string.h>
#include <ctype.h>
#include <limits.h>

int main(void)
{
    const char *s = "caf\xc3\xa9";           /* "cafe" + U+00E9, in UTF-8 */
    size_t len = strlen(s);

    printf("\"cafe-with-accent\": strlen = %zu bytes, a reader counts 4 letters\n", len);
    printf("bytes:            ");
    for (size_t i = 0; i < len; i++)
        printf(" %02x", (unsigned char)s[i]);
    printf("\n");

    /* toupper changes ASCII letters and passes every other byte through -- the
       two accent bytes are untouched, exactly like Python's bytes.upper() and
       Rust's to_ascii_uppercase(). */
    printf("toupper each byte:");
    for (size_t i = 0; i < len; i++)
        printf(" %02x", (unsigned char)toupper((unsigned char)s[i]));
    printf("\n");

    /* The first four bytes are "caf" and then the accent's lead byte, alone.
       C has no idea a character was cut in half: it hands back four bytes and
       asks nothing. Python's decode() and Rust's from_utf8() refuse this. */
    printf("first 4 bytes:    ");
    for (size_t i = 0; i < 4; i++)
        printf(" %02x", (unsigned char)s[i]);
    printf("   (ends inside the accent; C does not notice)\n");

    /* CHAR_BIT is fixed at 8; whether a plain char is signed is not, and it
       differs between this library's two machines. So a byte with the high bit
       set (0xc3) may reach <ctype.h> as a negative int -- undefined unless it
       is cast to unsigned char first, which is why every call above did. */
    printf("a char is %d bits; whether plain char is signed is implementation-defined\n",
           CHAR_BIT);
    unsigned char lead = (unsigned char)s[3];
    printf("byte 0x%02x asked through unsigned char: isalpha = %d, isprint = %d\n",
           lead, isalpha(lead) != 0, isprint(lead) != 0);
    return 0;
}

Verified output of char_is_a_byte_c.c — regenerated by tools/run_examples.py, never hand-typed.

"cafe-with-accent": strlen = 5 bytes, a reader counts 4 letters
bytes:             63 61 66 c3 a9
toupper each byte: 43 41 46 c3 a9
first 4 bytes:     63 61 66 c3   (ends inside the accent; C does not notice)
a char is 8 bits; whether plain char is signed is implementation-defined
byte 0xc3 asked through unsigned char: isalpha = 0, isprint = 0

Five bytes, four letters. strlen returns 5 because the é is c3 a9 in UTF-8 — two bytes, two chars. C has no count of letters to give you; it has bytes, and the only length it knows is the byte length.

toupper touches ASCII and passes the rest through. Upper-casing each byte turns caf into CAF and leaves c3 a9 exactly as they were. toupper works one byte at a time and knows only the ASCII letters, so an accented letter is beyond it — the same limit as Python's bytes.upper() and Rust's to_ascii_uppercase(). Changing the case of café to CAFÉ is not a byte operation, and C's standard library does not offer it — ICU4C does.

The cut lands inside a letter, and C does not notice. The first four bytes are caf and then c3 — the accent's lead byte, alone, with its second half left behind. C hands you four bytes and asks nothing; there is no such thing as a character boundary to it. Python's .decode() and Rust's from_utf8() both refuse this exact four-byte prefix, which is the whole difference between bytes and checked text.

<ctype.h> wants an unsigned char. This is the sharp one. isalpha, isprint, toupper and the rest take an int that must be representable as an unsigned char, or be EOF. A plain char may be signed — and whether it is differs between this library's two machines, since it is unsigned by default on arm64 and signed on x86-64. So a byte with the high bit set, like c3, can arrive at isalpha as a negative number, and asking isalpha about a negative number that is not EOF is undefined behaviour. The cast to unsigned char before every call is not decoration; it is the difference between defined and undefined. In the C locale the byte c3 is not a letter, which is the answer both machines give once the cast makes the question legal.

The same word in Python

Python makes the byte/character split a type: the same word is a str of four characters and a bytes of five. char_is_a_byte_py.py:

char_is_a_byte_py.py in full — pasted here by tools/run_examples.py from the file CI runs.

"""The same word as str and as bytes. Indexing, upper-casing, cutting and
reversing give the C answer on the bytes object and the text answer on the
str -- and the cut that lands inside a character is an error at decode time,
not a silent byte string."""
s = "café"
b = s.encode("utf-8")
print(f"str {s!r}: len {len(s)}, s[3] = {s[3]!r}")
print(f"bytes {b!r}: len {len(b)}, b[3] = {b[3]}")

print(f"s.upper() = {s.upper()!r}")
print(f"b.upper() = {b.upper()!r}  -- ASCII only, like C's toupper")

cut = b[:4]
print(f"b[:4] = {cut!r}")
try:
    cut.decode("utf-8")
except UnicodeDecodeError as e:
    print(f"  .decode() -> UnicodeDecodeError: {e.reason} at byte {e.start}")

print(f"s[::-1] = {s[::-1]!r}")
print(f"b[::-1] = {b[::-1]!r}  -> decode(errors='replace') = {b[::-1].decode(errors='replace')!r}")

Verified output of char_is_a_byte_py.py — regenerated by tools/run_examples.py, never hand-typed.

str 'café': len 4, s[3] = 'é'
bytes b'caf\xc3\xa9': len 5, b[3] = 195
s.upper() = 'CAFÉ'
b.upper() = b'CAF\xc3\xa9'  -- ASCII only, like C's toupper
b[:4] = b'caf\xc3'
  .decode() -> UnicodeDecodeError: unexpected end of data at byte 3
s[::-1] = 'éfac'
b[::-1] = b'\xa9\xc3fac'  -> decode(errors='replace') = '��fac'

Indexing the str gives the é; indexing the bytes gives the number 195. .upper() on the text gives CAFÉ, on the bytes gives the ASCII-only answer C also gave. And the four-byte cut, decoded, raises a UnicodeDecodeError at the exact byte C walked past without a word.

And in Rust

&str is bytes the type has already checked to be UTF-8, so both the byte length and the character count are available, and a cut that lands inside a character is refused rather than returned. char_is_a_byte_rs.rs:

char_is_a_byte_rs.rs in full — pasted here by tools/run_examples.py from the file CI runs.

// `&str` is UTF-8 bytes that the type has checked. Byte length and character
// count are two methods; a cut inside a character is refused by `get`, and the
// ASCII-only case change is a separately named method rather than the default.
fn main() {
    let s = "café";
    let bytes = s.as_bytes();
    println!("\"{s}\": len() = {} bytes, chars().count() = {}", s.len(), s.chars().count());
    println!("bytes: {:02x?}", bytes);

    println!("to_uppercase() = {:?}", s.to_uppercase());
    println!("to_ascii_uppercase() = {:?}  -- the C answer, by name", s.to_ascii_uppercase());

    println!("is_char_boundary(4) = {}", s.is_char_boundary(4));
    println!("s.get(..4) = {:?}, s.get(..3) = {:?}", s.get(..4), s.get(..3));

    let cut = &bytes[..4];
    match std::str::from_utf8(cut) {
        Ok(t) => println!("from_utf8(first 4 bytes) = Ok({t:?})"),
        Err(e) => println!("from_utf8(first 4 bytes) = Err: valid_up_to {}, error_len {:?}",
                           e.valid_up_to(), e.error_len()),
    }

    let reversed: String = s.chars().rev().collect();
    println!("chars().rev() = {reversed:?}");
    let mut rev_bytes = bytes.to_vec();
    rev_bytes.reverse();
    println!("bytes reversed: {}", String::from_utf8_lossy(&rev_bytes));
}

Verified output of char_is_a_byte_rs.rs — regenerated by tools/run_examples.py, never hand-typed.

"café": len() = 5 bytes, chars().count() = 4
bytes: [63, 61, 66, c3, a9]
to_uppercase() = "CAFÉ"
to_ascii_uppercase() = "CAFé"  -- the C answer, by name
is_char_boundary(4) = false
s.get(..4) = None, s.get(..3) = Some("caf")
from_utf8(first 4 bytes) = Err: valid_up_to 3, error_len None
chars().rev() = "éfac"
bytes reversed: ��fac

s.len() is 5 and s.chars().count() is 4 — the two numbers C conflates into one. to_ascii_uppercase() is a separately named method precisely because the ASCII-only behaviour is a choice, not the default. And s.get(..4) returns None where s.get(..3) returns Some("caf"): the boundary is part of the type's promise, so the cut inside the é cannot even be expressed.

If you are coming from another language

ABAP. (Not machine-checked — CI cannot run ABAP.) ABAP's string and c fields are measured in characters, not bytes, and the code page is a system setting rather than a property of the data — closest in spirit to a locale-driven C program, and the reason the encodings library's SAP code pages ↗ page exists. To get at the bytes you convert to an xstring; the byte-versus-character distinction this page is about is the same distinction as string versus xstring.

See also