C or Rust for text¶

Level: 301 · for anyone choosing a language for a job that starts with bytes

One line: C gives you bytes and a locale; Rust gives you bytes and a promise. So C is the language for the edge, where nothing has been decoded yet and the code page may be one only iconv(3) knows, and Rust is the language for everything after the check, because from from_utf8 onwards the type carries the fact that the bytes passed — and the measurement that puts a job on one side or the other is that the same five bytes are five characters, four, or an error in C depending on which libc and which locale the program ran under, while in Rust they are one of two types.

One type, or two¶

C has one type for text and it is not a text type. char is the machine's byte — the standard does not even say whether it has a sign — and a string is a run of them ending at the first zero. No encoding is recorded anywhere, nothing has been validated, and strlen, strcpy and printf("%s") will process any bytes at all. That is a decision rather than an oversight: C's string functions kept working on the day UTF-8 arrived precisely because they never looked. Why UTF-8 won makes the design argument, and UTF-8 by hand has the encoder written in C for the same reason.

Everything in C that can tell a character from a byte lives in the C library's locale machinery: setlocale, and then mbstowcs, mbrtowc, mblen, towupper, wcslen. It is off until you turn it on, it is one global setting for the whole process, its answers come from the libc rather than from the standard, and wchar_t — the type it decodes into — is 32 bits on Unix and 16 on Windows, so even a decoded character is not the same shape everywhere.

Rust has two types, and the second one is the promise. &[u8] is bytes, exactly as char* is. &str is bytes that have been checked to be UTF-8, and in safe code there is no way to hold one that has not been. The check is a function, str::from_utf8, and what it returns on failure is the discovery: how many bytes were fine, and whether the trouble was a character cut short or bytes that will never be text. No locale is consulted because there is none to consult — the case tables are compiled into std — and the operating system's bytes-that-are-not-text arrive as a third type, OsStr, that hands over an &str only when the bytes earn one.

The rest of the page is those two designs asked the same questions, in the order a real job asks them: how long is it, what is it, where does it break, and what is its uppercase.

In C¶

Verified output of c_or_rust_for_text_c.c — regenerated by tools/run_examples.py, never hand-typed.

1. A char IS A BYTE, AND A STRING IS BYTES UP TO THE FIRST ZERO
   char cafe[] = "caf\xc3\xa9";
   sizeof cafe = 6   strlen(cafe) = 5   bytes: 63 61 66 c3 a9
   (signed char)cafe[3] = -61   (unsigned char)cafe[3] = 195
   Six bytes of storage, five of text, and a NUL that is the only
   structure the type has. Nothing says UTF-8: the é is two chars
   because that is how many bytes it takes, and which of those two
   numbers a plain char gives you is the compiler's choice -- the
   standard leaves the sign of char to the implementation.

2. NOTHING KNOWS WHAT THE BYTES MEAN UNTIL YOU SET A LOCALE
   MB_CUR_MAX at startup = 1   (one byte per character: the C locale)
   setlocale(LC_ALL, a UTF-8 locale): obtained   MB_CUR_MAX > 1 now: yes
   mbstowcs(cafe) = 4 wide characters: U+0063 U+0061 U+0066 U+00E9
   Same five bytes, and now they are four characters -- because a
   global setting changed, not because anything about the data did.
   The setting is per process and reaches every thread. And the C
   locale it replaced is not even the same locale on every libc: the
   page's dated fence has this exact mbstowcs call, under LC_ALL=C,
   returning five on macOS and failing on glibc.

3. DISCOVERY IS A LOOP YOU WRITE, AND TWO NEGATIVE NUMBERS
   cafe         U+0063 U+0061 U+0066 U+00E9
   caf c3       U+0063 U+0061 U+0066  | byte 3: (size_t)-2, incomplete
   caf c3 28    U+0063 U+0061 U+0066  | byte 3: (size_t)-1, invalid
   a c0 af      U+0061  | byte 1: (size_t)-1, invalid
   a ed a0 80   U+0061  | byte 1: (size_t)-1, invalid
   mbstowcs(caf c3) == (size_t)-1: yes   errno == EILSEQ: yes
   That is all the one-shot call says: not where it stopped and not
   whether more bytes would have helped. mbrtowc, one character per
   call with an mbstate_t between calls, gives back both. (size_t)-2
   is a valid prefix that ran out -- keep it, read more. (size_t)-1 is
   bytes that will never be text -- skip and resynchronise. Those are
   Rust's error_len() None and Some(n), and Rust hands them to you from
   the first call.

4. CASE: THE BYTE FUNCTION AND THE CHARACTER FUNCTION ARE NOT THE SAME FUNCTION
   toupper over the bytes of cafe: 43 41 46 c3 a9 
   towupper(U+00E9)              : U+00C9
   toupper takes one byte and answered for the three ASCII ones; the
   two bytes of é came back as they went in. towupper takes the decoded
   value, so it can only run after sections 2 and 3 have happened. What
   toupper does with a byte above 0x7F is the locale's business, and
   the page's dated fence shows the two libcs answering differently.

5. WHAT THIS BOUGHT, AND WHAT IT COST
   Bought: the bytes are yours. Nothing copied, validated or refused
   them, so a file of unknown encoding, a Latin-1 record or a filename
   is a char* with no ceremony -- and iconv(3) converts between more
   tables than Rust's std will ever ship.
   Cost: a global setting, a loop, an mbstate_t, and answers that belong
   to whichever libc the program was linked against.

In Rust¶

Verified output of c_or_rust_for_text_rs.rs — regenerated by tools/run_examples.py, never hand-typed.

1. TWO TYPES, AND THE BOUNDARY BETWEEN THEM IS A FUNCTION
   let bytes: &[u8] = b"caf\xc3\xa9";   len 5   63 61 66 c3 a9
   let s: &str = from_utf8(bytes)?         len 5   chars 4   "café"
   bytes[3] = 195   s.as_bytes()[3] = 195   (a u8 is 0..=255; no sign to argue about)
   Same five bytes. One value knows nothing about them; the other exists
   only because from_utf8 said yes, and every method on it -- chars(),
   to_uppercase(), slicing -- is written on that promise.

2. DISCOVERY IS IN THE ERROR, FROM THE FIRST CALL
   63 61 66 c3    Err: valid_up_to 3  error_len None   lossy "caf�"
   63 61 66 c3 28 Err: valid_up_to 3  error_len Some(1)   lossy "caf�("
   61 c0 af       Err: valid_up_to 1  error_len Some(1)   lossy "a��"
   61 ed a0 80    Err: valid_up_to 1  error_len Some(1)   lossy "a���"
   error_len None is a prefix that ran out -- keep it, read more. Some(n)
   is n bytes that will never be text -- skip them. C's mbrtowc says the
   same two things as (size_t)-2 and (size_t)-1, one character per call,
   after a locale has been set and an mbstate_t kept between calls.

3. NO LOCALE ANYWHERE
   "café".to_uppercase() = "CAFÉ"   bytes 5 -> 5   chars 4 -> 4
   "straße".to_uppercase() = "STRASSE"   bytes 7 -> 7   chars 6 -> 7
   "Łódź".to_uppercase() = "ŁÓDŹ"   bytes 7 -> 7   chars 4 -> 4
   The tables are in std, so they are the same on every machine that ran
   this binary. Nothing was set first, so nothing elsewhere in the process
   can have set it differently. It is also why there is no
   to_uppercase_in(locale): the Turkish dotless i is a crate's job.

4. BYTES THAT ARE NOT TEXT HAVE A TYPE TOO
   OsStr::from_bytes(b"caf\xe9.txt") = "caf\xE9.txt"   len 8
   .to_str()          = None
   .to_string_lossy() = "caf�.txt"
   The byte survives, the &str never comes into being, and the lossy view
   says so with U+FFFD. That is C's char* with the decision made visible:
   the place where bytes become text is a method call you can read.

5. WHAT std WILL NOT DO, AND SAYS SO
   String::from_utf16(&[0x63, 0x61, 0x66, 0xE9]) = Ok("café")
   char::from_u32(0xD800)                       = None   (a surrogate is not a char)
   Latin-1 in: (b as char) is the whole decoder, because Latin-1 IS code
   points 0..=255. Windows-1250, Shift_JIS, an EBCDIC page: no table in std
   at all -- encoding_rs is the crate, and iconv is a C library you bind.
   Graphemes, collation, locale-aware case: crates too, and they are good
   precisely because std declined to guess.

The same five bytes on two libcs¶

Everything in the two blocks above is byte-identical on macOS and Ubuntu, because the C program keeps to what the standard promises and the Rust program has nothing that could vary. What the C standard leaves to the implementation is where the two platforms part, and none of it can go in an answer key:

Measured 2026-09-13 — macOS 26.6 (Apple clang 21, libSystem, x86_64) and gcc:14 (Debian, glibc 2.41, x86_64). Not machine-checked: no key can match both.

                                                 macOS (libSystem)               glibc 2.41
LC_ALL=C      mbstowcs on the bytes of café      5: 0063 0061 0066 00C3 00A9     (size_t)-1, errno EILSEQ
LC_ALL=C      mbrtowc at the C3 byte             U+00C3, one byte consumed       (size_t)-1, invalid
UTF-8 locale  MB_CUR_MAX                         4                               6
UTF-8 locale  toupper(0xE2), the lead byte of €  0xC2                            0xE2
UTF-8 locale  toupper(0xF0), the lead byte of 😀 0xD0                            0xF0
UTF-8 locale  toupper(0xB5), which is µ in Latin-1   0x39C  (does not fit a byte)  0xB5
UTF-8 locale  toupper over the 128 bytes 80..FF  changes 32 of them              changes none
UTF-8 locale  toupper over the bytes of €        c2 82 ac                        e2 82 ac

Two things to take from it.

The C locale is not one locale. Under LC_ALL=C, the environment every example in this library runs in, macOS decodes café to five wide characters — its C locale is a single-byte table in which every byte is a character, so C3 becomes U+00C3 and A9 becomes U+00A9 — while glibc's is ASCII and refuses at the first byte above 0x7F. Neither is wrong; the standard says only that a locale called "C" exists. A C program that calls mbstowcs without setting a locale first has therefore not made a portability mistake so much as asked a question with two answers, which is why the program above sets a UTF-8 locale on purpose before it asks, and why its key can hold.

toupper is a byte function, and one libc treats it as something else. ISO C says toupper takes a value representable as unsigned char, or EOF. In a UTF-8 locale glibc leaves every byte above 0x7F alone, which is the only answer that keeps a UTF-8 string a UTF-8 string. macOS answers with the Unicode uppercase of the code point that has the byte's number: 0xE2 is â, whose capital is Â at 0xC2, so toupper over the three bytes of € turns its three-byte lead into a two-byte lead and the result is not UTF-8 at all. For 0xB5, µ, it returns 0x39C, Greek capital mu, a value that will not fit in the byte the caller is about to store it into. The two cast members it happens to spare are é and ż, whose bytes all land on characters that are already capitals or have no case, which is why section 4 of the C program could be recorded and this table could not. It is the lesson of Case is not a per-character operation one layer down: case is not a per-byte operation either, and the function that pretends it is gives you a libc-specific answer.

When to reach for which¶

The job	C	Rust
Sniff a file of unknown encoding — count the high bytes, look for a BOM, try a decode	Natural: the data is already `unsigned char*`, and `iconv(3)` will attempt any table the libc knows — though which bytes a table contains is itself a libc question	Natural too: `&[u8]`, then `from_utf8`. For any table but UTF-8 and UTF-16 you are in a crate
Validate UTF-8	You write it — about thirty lines — or call `iconv` with UTF-8 on both sides	`from_utf8`, with `valid_up_to` and `error_len` in the error. Done
Convert a legacy code page — Windows-1250, CP037, Shift_JIS	`iconv(3)`: inside glibc's libc, and on macOS one `-liconv` away. Every table the platform ships	Nothing in `std`. `encoding_rs` ↗ has the web's forty; an EBCDIC page needs another crate, or a binding to `iconv`
Carry a filename, an argument, an environment value	`char*`, untouched — and nothing stops you printing it as text	`OsStr` / `OsString`, whose `to_str()` is `None` until the bytes earn it
Count or slice characters	`mbrtowc` in a loop after `setlocale`, carrying an `mbstate_t`; or `wchar_t` throughout, at a different width per OS	`chars()`, `char_indices()`, `is_char_boundary()` — no setup, no state
Change case	`toupper` per byte is ASCII-only on one libc and corrupting on the other; `towupper` needs the decode done first	`to_uppercase()` from `std`'s tables; the length may change and the type does not mind
Call a C API	You are already there	`CStr` / `CString`, which check for the interior NUL a C string cannot carry
Parse hostile input — a decoder is where the overflow lives	Every byte index is yours to get wrong, and the legacy East Asian decoders are where the `encoding_rs` write-up found the C++ converters' memory-safety bugs	Bounds are checked, and a slice inside a character panics rather than reading half of one
No libc at all — firmware, a kernel, WebAssembly	`char*` with no `setlocale` to call: bytes and nothing else	`core::str::from_utf8` works without `std`; `alloc` adds `String`

The pattern in the table is the one-line summary: C wins every row where the text is still bytes, and Rust wins every row after the bytes have been checked. The row that decides most real programs is the third. If the data is in a code page the platform knows and Rust's crates do not, C — or a Rust binding to C's iconv — is the pragmatic answer, and the honest survey of which language handles Unicode already gave Rust its one ✗ for exactly that.

Two things the table cannot show. A C program's answers for text belong to the libc, as the dated fence spells out, so "works on my machine" is a weaker claim in C than in Rust for this subject specifically: one source built against musl, glibc and libSystem is three programs where characters are concerned. And Rust's refusals are the feature. The thing that makes &s[0..4] panic is the thing that makes every later function on &str trustworthy, and the place to spend that strictness is the boundary, once — which is the whole of Rust strings in practice.

C++ is C here, with three retired attempts¶

std::string is char* with an owner: bytes, no encoding, size() in bytes, and toupper applied through std::transform has exactly the byte-wise behaviour measured above. C++ has tried three times to say more. std::codecvt_utf8 and std::wstring_convert arrived in C++11, were deprecated in C++17 and are removed in C++26; char8_t and std::u8string (C++20) record in the type that a string is UTF-8, which is the Rust idea, but ship almost no functions that use the fact, so most code casts back to char. What real C++ text code does is link ICU ↗, which is the same conclusion the four-questions page reached for every other language: the sophisticated engine is a library, not a language. (Not machine-checked — this repo compiles C, not C++.)

If you are coming from Python or ABAP¶

Python is the Rust design with the check made optional. bytes is &[u8], str is &str, and .decode() is from_utf8 with a table argument — 117 tables, which is the one thing Rust's std refuses to carry and C's iconv does. Where Python differs from both is errors='surrogateescape': it will let a str carry a byte that is not text, which Rust does with a separate type and C does by never having noticed. And str.upper() is to_uppercase(), not toupper — it never consults the locale, and the byte-wise trap in the fence above has no spelling in Python at all, because bytes.upper() changes ASCII and nothing else, by design.

ABAP (Not machine-checked — CI cannot run ABAP.) xstring is &[u8] and string is &str, and the split runs the same way: XSTRLEN counts bytes of an xstring, STRLEN counts characters of a string, and no function takes either and guesses. Conversion is a class — cl_abap_conv_in_ce and cl_abap_codepage — which is the iconv row: a catalogue of tables the platform maintains, whose code-page numbers should be checked against the system rather than quoted. What ABAP has that neither of these has is that the system fixes the internal encoding once, at installation, so the locale question C asks per process is answered for every program at once — closer to Rust's answer than to C's, reached by administration rather than by type.

Try it¶

Take a real file you know is UTF-8, feed it to the C program's walk in place of cafe, and run it under LC_ALL=C and then under your own locale. Then python3 -c "open('f','rb').read().decode()" on the same file. Note which of the three told you where the first problem was.
Comment out the setlocale call in the C program, build it on a Mac and on a Linux box, and count the wide characters mbstowcs reports for café on each.
On a Mac, change section 4 of the C program to run toupper over the bytes of € under your own locale, and xxd what comes out. Then search any C code base you maintain for toupper( and ask what each call is being handed.
Take the worst filename on your disk — accented, or not valid UTF-8 at all — and read it back through std::fs::read_dir in Rust and readdir in C. Print the length each reports, and say which length it is.

Practice¶

Six bytes, five questions, and the one only a dated fence can answer. The bytes are 61 c3 a9 e2 82 ac — aé€. Before running anything, write down: C's strlen and Rust's len(); Rust's chars().count() and C's mbstowcs in a UTF-8 locale; what each language says about the first four bytes only, 61 c3 a9 e2; the bytes each produces for the uppercase under LC_ALL=C; and, last, what mbstowcs returns under LC_ALL=C with no locale set. Four of the five have one answer on every machine. Say which one does not, and why.

Answers

Verified output of c_or_rust_for_text_kata_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

1. HOW LONG IS IT: SIX AND SIX, AND ONLY ONE SIDE CAN ALSO SAY THREE UNASKED
   C    strlen(s)                     = 6
   Rust s.len()                       = 6
   Rust s.chars().count()             = 3
   C    mbstowcs(s), UTF-8 locale     = 3
   Both count bytes without being told anything. Rust's three came from
   the type; C's three came from a setlocale call that had to run first,
   and before it ran the same call had no portable answer (question 5).

2. THE FIRST FOUR BYTES: A CHARACTER CUT SHORT, SEEN FROM BOTH SIDES
   Rust from_utf8(&b[..4])            : valid_up_to 3, error_len None
   C    mbrtowc on the first 4 bytes  : stopped at byte 3 with (size_t)-2, incomplete
   Same verdict at the same position: three good bytes, then the lead
   byte of € with nothing after it. None and (size_t)-2 both mean
   'read more' -- neither is the answer for bytes that are simply wrong.

3. UPPERCASE UNDER LC_ALL=C
   C    toupper over the bytes        : 41 c3 a9 e2 82 ac
   Rust s.to_uppercase()              = "AÉ€"  bytes 41 c3 89 e2 82 ac
   C changed the one ASCII byte and nothing else: the C locale has no
   opinion above 0x7F on either libc, which is the only reason this
   line could be recorded. Rust changed é to É from its own table, and
   would have printed the same under any environment at all.

4. WHAT EACH ONE NEEDED TO GET HERE
   C    : a locale name that exists on this machine, one global call,
          an mbstate_t, and a loop
   Rust : a slice and a function

5. THE ONE WITH NO ANSWER
   C    mbstowcs(s) under LC_ALL=C     : (not recorded)
   Six on a Mac -- its C locale is a single-byte table where every byte
   is a character -- and (size_t)-1 on glibc, whose C locale is ASCII
   and stops at the first byte above 0x7F. The standard picks neither.
   A key that held either number would be red on the other runner, so
   the page's dated fence holds both and this key holds a sentence.