Four lengths, and which one the other system means¶
Level: 201 · working knowledge
One line: The same string has four different lengths, std will give you two of them cheaply and a third if you ask by name, and every system you exchange text with has already picked one — so "how long is this" is not a question about the string, it is a question about who is asking.
Meet the char establishes that .len() counts bytes and .chars().count() counts scalar values, and that a reader's idea of a character is a third thing std will not count. This page is the part after that: the counts are not trivia, they are a wire format problem. A name that fits a VARCHAR(12) may be rejected by an nvarchar(12), and both may be fine with a limit the user was shown as "12 characters".
The fourth count is the one Rust never puts in front of you. encode_utf16().count() is a method you have to know exists, and it is the number JavaScript, Java, C# and SQL Server mean by "length".
Verified output of four_lengths.rs — regenerated by tools/run_examples.py, never hand-typed.
Round 1 - the four counts std can and cannot give you
sample bytes chars utf-16 clusters
.len() count() units (no std)
--------------------------------------------------
plain ASCII 4 4 4 4
NFC - one char 5 4 4 4
NFD - two chars 6 5 5 4
Polish 10 6 6 6
emoji 4 1 2 1
flag 8 2 4 2
family 18 5 8 1
Four columns, four different numbers for the same seven strings.
Only the first two are O(1) and O(n) in std; the fourth is a crate.
Round 2 - every system you talk to already picked a column
bytes HTTP Content-Length, Postgres octet_length, a VARCHAR sized in bytes
chars Postgres char_length, Python len(), Rust .chars().count()
utf-16 JavaScript .length, Java, C#, SQL Server nvarchar(n)
clusters the person who counted before hitting your character limit
Round 3 - so the same string fits, or does not, depending who asks
Does it fit a limit of 10? (n) = the count that system would use
sample byte column char column nvarchar
plain ASCII yes (4) yes (4) yes (4)
NFC - one char yes (5) yes (4) yes (4)
NFD - two chars yes (6) yes (5) yes (5)
Polish yes (10) yes (6) yes (6)
emoji yes (4) yes (1) yes (2)
flag yes (8) yes (2) yes (4)
family NO (18) yes (5) yes (8)
The Polish row is the one to look at: 10 bytes, 6 chars, 6 utf-16
units. It fits every column here -- and one more letter would
overflow the byte column while the other two still had room.
Round 4 - the UTF-16 column is the one Rust never shows you
emoji "😀" bytes 4 chars 1 utf-16 2
Polish "Zażółć" bytes 10 chars 6 utf-16 6
An emoji is ONE char and TWO utf-16 units -- a surrogate pair.
That is why JavaScript says "\u{1F600}".length === 2, and why a
nvarchar(1) rejects it. encode_utf16() is how you ask for that
number without leaving Rust.
Round 5 - what wc counts, and which method matches
text = "Zażółć gęślą jaźń\n"
wc -c 27 text.len()
wc -m 18 text.chars().count()
wc -w 3 text.split_whitespace().count()
wc -l 1 text.matches('\n').count() -- NEWLINES, not lines
.lines() says 1 too -- they agree here, because the text ends in
a newline. Take it away and they stop agreeing:
"a\nb\n" matches('\n') = 2 .lines() = 2
"a\nb" matches('\n') = 1 .lines() = 2
wc -l is a newline count, and so is matches(). .lines() counts
lines. The two answers differ on exactly the files a person
would call badly formed, which is most files.
The four counts, and who uses which¶
| count | how to ask in Rust | cost | who means this by "length" |
|---|---|---|---|
| bytes | s.len() |
O(1) — it is the len field |
HTTP Content-Length, Postgres octet_length, a VARCHAR sized in bytes, every disk quota |
| chars | s.chars().count() |
O(n) — UTF-8 must be walked | Postgres char_length, Python's len(), Go's utf8.RuneCountInString |
| UTF-16 units | s.encode_utf16().count() |
O(n) | JavaScript .length, Java String.length(), C#, SQL Server nvarchar(n), most JSON schema validators |
| grapheme clusters | not in std |
a crate | the person who counted before they hit your limit |
The third row is the one that causes the arguments, because it is invisible from Rust. An emoji is one char and two UTF-16 units — a surrogate pair — so "😀".length is 2 in a browser and 1 in Python, and a nvarchar(1) refuses it. Nobody is wrong; they are counting different things, and only one of them is counting the thing the user sees.
Why std stops at three¶
std ships UTF-8 correctness — the type will not hold invalid bytes, and every method upholds that. What it does not ship is the Unicode character database: the tables saying which code points combine, which join, which are wide. Grapheme segmentation (UAX #29 ↗) needs those tables, they change with every Unicode release, and putting them in std would tie the standard library's release cadence to Unicode's. So it is unicode-segmentation ↗, and normalizing the two spellings of é is unicode-normalization ↗.
The example above approximates a cluster count in twelve lines, which is enough to show the gap and not enough to be right — it handles combining marks and zero-width joiners and nothing else. That is deliberate: the distance between those twelve lines and the crate is the answer to "why isn't this in std".
The one that bites: truncating to a byte limit¶
Measuring is safe. Cutting is where a byte count turns into a panic, because &s[..n] requires n to be a character boundary and a limit expressed in bytes has no idea where those are. "Łukasiewicz" truncated to 1 byte is not a short name, it is a crash — byte 1 is the second half of Ł.
str::is_char_boundary is the check, str::get is the same question asked politely (returning None instead of panicking), and backing off from the limit until the boundary is found is the whole of the fix. The kata below writes it.
If you are coming from another language¶
Python. len() is the chars column, and it is the only one with a short spelling; bytes need len(s.encode("utf-8")) and UTF-16 needs len(s.encode("utf-16-le")) // 2. Rust inverts exactly that — the cheap answer gets the short name. Neither language is hiding anything, but code translated between them keeps working on ASCII and silently changes meaning at the first accented letter. Python also ships unicodedata, so normalization is in the standard library there and a crate here.
ABAP. Text is UTF-16 internally, so strlen( ) returns the UTF-16 units column — the third row, not the second. A character above U+FFFF counts as 2. xstrlen( ) on an xstring is the bytes column. There is no equivalent of the chars column without walking the string yourself. (Not machine-checked — CI cannot run ABAP.)
JavaScript. .length is the UTF-16 column, [...s].length is the chars column, and Intl.Segmenter is the only one of these four languages with grapheme segmentation built in.
Practice¶
Measure a column four ways, then truncate it without breaking a letter. Take a column of names with Polish, CJK and emoji in it. Print the byte, char and UTF-16 count of each, then pick a limit and report which rows each of the three rules rejects — and count the rows where the rules disagree, because those are the rows that load into one database and bounce off another. Construct at least one value that all three rules answer differently.
Then write truncate_bytes(s, max) -> &str: the longest prefix fitting max UTF-8 bytes that does not split a character. Prove the naive &s[..max] would panic on your data by asking get(..n) at the same indices and showing which return None. Finish by explaining why your own output table is misaligned on the CJK row.
Solution
four_lengths_kata.rs in full — pasted here by tools/run_examples.py from the file CI compiles and runs.
//! K139 — Measure a column four ways, then truncate it without breaking a letter.
//!
//! rustc --edition 2024 four_lengths_kata.rs -o /tmp/flk && /tmp/flk
fn utf16(s: &str) -> usize { s.encode_utf16().count() }
/// Longest prefix of `s` that fits in `max` UTF-8 bytes WITHOUT splitting a
/// character. The whole job: never return a byte index that is not a boundary.
fn truncate_bytes(s: &str, max: usize) -> &str {
if s.len() <= max {
return s;
}
let mut end = max;
while end > 0 && !s.is_char_boundary(end) {
end -= 1;
}
&s[..end]
}
fn main() {
let column = [
"Nowak",
"\u{141}ukasiewicz", // Łukasiewicz
"\u{17B}eromski", // Żeromski
"\u{4E2D}\u{6751}", // 中村
"Zo\u{EB} \u{1F600}", // Zoë + emoji
"\u{1F600}\u{1F600}\u{1F600}\u{1F600}\u{1F600}", // five emoji
];
println!("Round 1 - four measurements of one column");
println!(" {:<16} {:>6} {:>6} {:>7}", "value", "bytes", "chars", "utf-16");
for s in column {
println!(" {:<16} {:>6} {:>6} {:>7}", s, s.len(), s.chars().count(), utf16(s));
}
println!("\nRound 2 - the same limit, three different verdicts");
let limit = 8;
println!(" A column declared as 8 units. Which rows does each rule reject?\n");
println!(" {:<16} {:>10} {:>10} {:>10}", "value", "VARCHAR", "char_length", "nvarchar");
let mut disagreements = 0;
for s in column {
let (b, c, u) = (s.len() <= limit, s.chars().count() <= limit, utf16(s) <= limit);
if !(b == c && c == u) {
disagreements += 1;
}
let mark = |ok: bool| if ok { "ok" } else { "REJECT" };
println!(" {:<16} {:>10} {:>10} {:>10}", s, mark(b), mark(c), mark(u));
}
println!("\n {disagreements} row(s) where the three rules disagree. Every one of them");
println!(" is a row that loads into one database and bounces off another.");
println!(" The five-emoji row splits all three ways: 20 bytes, 5 chars,");
println!(" 10 utf-16 units -- rejected by two rules and accepted by one.");
println!("\nRound 3 - truncating to a byte limit is where it actually breaks");
let max = 6;
for s in column {
let cut = truncate_bytes(s, max);
println!(" {:<16} -> {:?} ({} bytes of {} allowed)", s, cut, cut.len(), max);
}
println!("\n &s[..6] would PANIC on the rows where byte 6 lands inside a letter.");
println!(" is_char_boundary() is the check that makes the cut safe, and the");
println!(" loop backing off from `max` is the whole of the fix.");
println!("\nRound 4 - prove the naive cut really does panic");
let name = "\u{141}ukasiewicz";
for i in [1, 2, 6] {
match name.get(..i) {
Some(p) => println!(" get(..{i}) = Some({p:?})"),
None => println!(" get(..{i}) = None <- &name[..{i}] would panic here"),
}
}
println!("\n get() is the same question asked politely. Byte 1 is inside 'Ł',");
println!(" which is two bytes, so a limit measured in bytes cannot be applied");
println!(" with a slice unless you check first.");
println!("\nRound 5 - the tables above are misaligned, and that is the lesson");
println!(" Look at the CJK row: it does not line up. `{{:<16}}` pads to a");
println!(" width counted in CHARS, and 中 occupies two terminal columns.");
println!(" So the formatter picked a fifth answer to 'how long' -- display");
println!(" width -- and std does not have that one either. Even the code");
println!(" printing this table had to choose a count, and chose wrong.");
}
Verified output of four_lengths_kata.rs — regenerated by tools/run_examples.py, never hand-typed.
Round 1 - four measurements of one column
value bytes chars utf-16
Nowak 5 5 5
Łukasiewicz 12 11 11
Żeromski 9 8 8
中村 6 2 2
Zoë 😀 9 5 6
😀😀😀😀😀 20 5 10
Round 2 - the same limit, three different verdicts
A column declared as 8 units. Which rows does each rule reject?
value VARCHAR char_length nvarchar
Nowak ok ok ok
Łukasiewicz REJECT REJECT REJECT
Żeromski REJECT ok ok
中村 ok ok ok
Zoë 😀 REJECT ok ok
😀😀😀😀😀 REJECT ok REJECT
3 row(s) where the three rules disagree. Every one of them
is a row that loads into one database and bounces off another.
The five-emoji row splits all three ways: 20 bytes, 5 chars,
10 utf-16 units -- rejected by two rules and accepted by one.
Round 3 - truncating to a byte limit is where it actually breaks
Nowak -> "Nowak" (5 bytes of 6 allowed)
Łukasiewicz -> "Łukas" (6 bytes of 6 allowed)
Żeromski -> "Żerom" (6 bytes of 6 allowed)
中村 -> "中村" (6 bytes of 6 allowed)
Zoë 😀 -> "Zoë " (5 bytes of 6 allowed)
😀😀😀😀😀 -> "😀" (4 bytes of 6 allowed)
&s[..6] would PANIC on the rows where byte 6 lands inside a letter.
is_char_boundary() is the check that makes the cut safe, and the
loop backing off from `max` is the whole of the fix.
Round 4 - prove the naive cut really does panic
get(..1) = None <- &name[..1] would panic here
get(..2) = Some("Ł")
get(..6) = Some("Łukas")
get() is the same question asked politely. Byte 1 is inside 'Ł',
which is two bytes, so a limit measured in bytes cannot be applied
with a slice unless you check first.
Round 5 - the tables above are misaligned, and that is the lesson
Look at the CJK row: it does not line up. `{:<16}` pads to a
width counted in CHARS, and 中 occupies two terminal columns.
So the formatter picked a fifth answer to 'how long' -- display
width -- and std does not have that one either. Even the code
printing this table had to choose a count, and chose wrong.
Edit distance, in the unit you choose. Write edit_distance(s1, s2): the fewest single-item inserts, deletes and replacements that turn one string into the other, using the classic table kept to two rows. Make the table generic over the item type, so the same function runs on &[char] and on &[u8]. Then predict both distances for kitten/sitting, café/cafe, żółw/zolw, and café against the same word spelt with a combining accent — the two units agree on the first pair and on none of the others.
// rustc --edition 2024 --test edit_distance.rs -o t && ./t
fn edit_distance(s1: &str, s2: &str) -> usize {
todo!()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_edit_distance() {
assert_eq!(edit_distance("kitten", "sitting"), 3);
assert_eq!(edit_distance("", ""), 0);
assert_eq!(edit_distance("abc", "abc"), 0);
assert_eq!(edit_distance("abc", ""), 3);
assert_eq!(edit_distance("", "abc"), 3);
assert_eq!(edit_distance("intention", "execution"), 5);
}
}
Solution
edit_distance_kata.rs in full — pasted here by tools/run_examples.py from the file CI compiles and runs.
//! Kata solution: Levenshtein distance in two rows — and the same pair of
//! words at different distances in characters and in bytes.
//!
//! rustc --edition 2024 edit_distance_kata.rs -o /tmp/edk && /tmp/edk
//! rustc --edition 2024 --test edit_distance_kata.rs -o /tmp/edkt && /tmp/edkt
fn edit_distance(s1: &str, s2: &str) -> usize {
let a: Vec<char> = s1.chars().collect();
let b: Vec<char> = s2.chars().collect();
levenshtein(&a, &b)
}
/// The classic table, kept to two rows: row i holds the distance from the
/// first i items of `a` to every prefix of `b`. Generic over the unit, so the
/// same function can measure bytes.
fn levenshtein<T: PartialEq>(a: &[T], b: &[T]) -> usize {
let mut prev: Vec<usize> = (0..=b.len()).collect();
let mut cur = vec![0; b.len() + 1];
for (i, x) in a.iter().enumerate() {
cur[0] = i + 1;
for (j, y) in b.iter().enumerate() {
let replace = prev[j] + usize::from(x != y);
let delete = prev[j + 1] + 1;
let insert = cur[j] + 1;
cur[j + 1] = replace.min(delete).min(insert);
}
std::mem::swap(&mut prev, &mut cur);
}
prev[b.len()]
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_edit_distance() {
assert_eq!(edit_distance("kitten", "sitting"), 3);
assert_eq!(edit_distance("", ""), 0);
assert_eq!(edit_distance("abc", "abc"), 0);
assert_eq!(edit_distance("abc", ""), 3);
assert_eq!(edit_distance("", "abc"), 3);
assert_eq!(edit_distance("intention", "execution"), 5);
}
}
fn main() {
println!("1. Adam's six cases");
for (a, b, want) in [
("kitten", "sitting", 3),
("", "", 0),
("abc", "abc", 0),
("abc", "", 3),
("", "abc", 3),
("intention", "execution", 5),
] {
let got = edit_distance(a, b);
assert_eq!(got, want);
println!(" {:<11} {:<11} {got}", format!("{a:?}"), format!("{b:?}"));
}
println!();
println!("2. Which unit is being edited");
println!(" {:<10} {:<14} {:>5} {:>5}", "a", "b", "chars", "bytes");
for (a, b) in [("kitten", "sitting"), ("café", "cafe"), ("żółw", "zolw"), ("caf\u{e9}", "cafe\u{301}")] {
let chars = edit_distance(a, b);
let bytes = levenshtein(a.as_bytes(), b.as_bytes());
println!(" {:<10} {:<14} {chars:>5} {bytes:>5}", format!("{a:?}"), format!("{b:?}"));
}
println!(" ASCII agrees in both units. Here each two-byte letter that differs costs");
println!(" one edit as a char and two as bytes, a replacement plus a deletion:");
println!(" café/cafe is 1 against 2, and żółw/zolw is 3 against 6.");
println!(" The last row is one word spelt two ways, precomposed é against e plus a");
println!(" combining accent: 2 edits in chars, 3 in bytes, and not zero in either");
println!(" until something normalizes both, which std does not do.");
}
Verified output of edit_distance_kata.rs — regenerated by tools/run_examples.py, never hand-typed.
1. Adam's six cases
"kitten" "sitting" 3
"" "" 0
"abc" "abc" 0
"abc" "" 3
"" "abc" 3
"intention" "execution" 5
2. Which unit is being edited
a b chars bytes
"kitten" "sitting" 3 3
"café" "cafe" 1 2
"żółw" "zolw" 3 6
"café" "cafe\u{301}" 2 3
ASCII agrees in both units. Here each two-byte letter that differs costs
one edit as a char and two as bytes, a replacement plus a deletion:
café/cafe is 1 against 2, and żółw/zolw is 3 against 6.
The last row is one word spelt two ways, precomposed é against e plus a
combining accent: 2 edits in chars, 3 in bytes, and not zero in either
until something normalizes both, which std does not do.
See also¶
- STRINGS.md — the map: every string lesson, in reading order
- Meet the
char— what the counts are counting; this page is which one to hand over - String slices — the boundary panic in full, and the
E0502that keeps a view honest - Walking a
String—chars(),char_indices()and the split family - Comparing and sorting text — the other place byte order is not human order
- Counting characters ↗ — the same four counts from Python, where
len()names the second column instead of the first
Po polsku¶
Polski tekst trafia w ten problem od pierwszego słowa, a nie dopiero na emotikonie. "Zażółć" to 10 bajtów, 6 znaków, 6 jednostek UTF-16 — trzy różne liczby dla jednego wyrazu, bo każda litera diakrytyczna zajmuje w UTF-8 dwa bajty. Praktyczny wniosek jest taki, że kolumna VARCHAR(10) liczona w bajtach pomieści sześcioliterowe nazwisko i odrzuci siedmioliterowe, mimo że użytkownikowi obiecano „10 znaków". W SAP-ie ta sama pułapka wygląda inaczej: ABAP trzyma tekst w UTF-16, więc strlen( ) zwraca trzecią kolumnę z tabeli wyżej, a nie drugą — dla polskich liter obie są zresztą równe, i właśnie dlatego błąd ujawnia się dopiero na emotikonie albo na chińskim znaku spoza BMP.
Druga rzecz to obcinanie. &s[..10] w Ruscie panikuje, jeśli bajt 10 wypada w środku litery — a przy polskich nazwiskach wypada tam bardzo często. Nie jest to złośliwość języka, tylko odmowa zgadywania: is_char_boundary( ) pozwala zapytać wcześniej, get( ) zwraca None zamiast panikować, a cała poprawka to cofanie się o bajt, aż trafi się na granicę znaku. Kata wyżej każe to napisać, i warto ją zrobić na własnych danych — jeśli kiedykolwiek importowałeś nazwiska do kolumny o stałej szerokości, ten kod już gdzieś w twoim systemie działa albo powinien.
Szukaj po polsku: długość tekstu w bajtach a w znakach · VARCHAR a NVARCHAR długość · UTF-16 pary zastępcze · obcinanie tekstu UTF-8 bez psucia znaku · rust byte index is not a char boundary