Slicing by byte¶
Level: 201 · working knowledge
One line: &s[0..2] is a byte range, and it panics at run time if either end lands inside a multibyte character — the one string operation Rust cannot check at compile time, and the reason get(0..2) exists.
Everywhere else, Rust's promise about text is kept by the type system: there is no s[0], there is no way to build a &str from arbitrary bytes without a check, and a Vec<u8> cannot pretend to be a String. Slicing is the exception. &s[a..b] compiles for any a and b the type-checker can see are usize, and whether it is legal depends on the value of the string, which is not knowable until the program runs. So the check moves to run time, and the enforcement mechanism moves from a compiler error to a panic.
The safe way, first¶
let s = "café";
s.get(0..4) // None — byte 4 is inside 'é'
s.get(0..3) // Some("caf")
s.is_char_boundary(4) // false — the question the panic asks
get is the whole answer for any range you computed yourself, and it is the same shape as checked_add: the operation that might not work returns an Option instead of trapping. The rest of this page is about why it is needed, what the panic actually says, and how to cut a string to a byte budget without either.
Five byte positions, four characters¶
café is five bytes: 63 61 66 c3 a9. The é is c3 a9, so it occupies byte positions 3 and 4, and only position 3 begins anything. That leaves index 4 pointing into the middle of a character — a place a byte range can name and a string cannot be cut at.
is_char_boundary(i) is the exact question. It is true at 0, 1, 2, 3 and 5 (the end of a string always counts) and false at 4, and that single false is every panic on this page. The rule that generates it is UTF-8's: a byte is a boundary unless it is a continuation byte, 10xxxxxx, 0x80..=0xBF. That is a property of one byte, which is why the check is cheap and why it can be asked before the cut rather than only discovered during it.
In Rust¶
Verified output of slicing_by_byte_rs.rs — regenerated by tools/run_examples.py, never hand-typed.
1. THE RANGE IS BYTES. THE STRING IS CHARACTERS.
"café" len() = 5 bytes, chars().count() = 4
bytes 63 61 66 c3 a9
byte index 0 1 2 3 4
character c a f <-- é -->
There are five byte positions and only four characters, so one index — 4 —
points at the middle of a character. Every bug on this page is that index.
2. ASK BEFORE YOU CUT: is_char_boundary
index boundary? what is there
0 true the start of 'c'
1 true the start of 'a'
2 true the start of 'f'
3 true the start of 'é'
4 false a continuation byte, inside a character
5 true the end of the string
Note that len() itself is a boundary — the end of a string always is, which
is what makes &s[..s.len()] legal and &s[..s.len()-1] a coin toss.
3. THE PREDICTION, CHECKED AGAINST WHAT ACTUALLY HAPPENS
Every &S[0..n], with the answer from section 2 beside the outcome.
expression boundary says outcome value
&S[0..0] true returned ""
&S[0..1] true returned "c"
&S[0..2] true returned "ca"
&S[0..3] true returned "caf"
&S[0..4] false PANICKED -
&S[0..5] true returned "café"
is_char_boundary predicted every row: true
That is the whole relationship. The panic is not a mystery about UTF-8 —
it is is_char_boundary returning false at a moment when nobody asked it.
4. AND ONE OTHER WAY TO PANIC, WHICH IS NOT THIS ONE
&S[0..99] past the end PANICKED
&S[4..3] backwards PANICKED
Same syntax, same panic machinery, different complaint — out of bounds and
start-after-end are ordinary slice errors that &[u8] has too. Only the char
boundary one is about the encoding, and only it survives a correct length.
5. get() ASKS THE SAME QUESTION AND RETURNS AN ANSWER
S.get(0..3) Some("caf")
S.get(0..4) None
S.get(0..5) Some("café")
S.get(0..99) None
Option, not a panic — the same shape as checked_add. Every range that came
from you rather than from std should go through this, and the None case is
a real branch: it means the index was computed by someone counting
characters, or reading a byte budget out of a database column.
6. WHERE A VALID INDEX COMES FROM
Indices that std produced are ALWAYS boundaries. These all are:
find('f') Some(2)
char_indices() [0, 1, 2, 3]
split('a') pieces ["c", "fé"]
An index you computed carries no such guarantee, and the coin lands both
ways on the very same string:
half of "café" index 2 boundary? true
half of the Polish string index 13 boundary? false
Two halfway points, one safe and one not, and nothing in either expression
says which. 'the halfway point', 'the first 20 characters' and 'byte 4' are
the three sources of every panic of this kind.
7. THE FIXED-WIDTH FIELD, DONE PROPERLY
"zażółć gęślą jaźń" 26 bytes, 17 chars
budget bytes chars kept
4 4 3 "zaż"
5 4 3 "zaż"
6 6 4 "zażó"
10 10 6 "zażółć"
20 20 13 "zażółć gęślą "
99 26 17 "zażółć gęślą jaźń"
Never more than the budget, never a split character, and never a panic.
A budget of 5 keeps 4 bytes because byte 5 is inside 'ż' — so the field is
one byte short of full, which is the correct answer and not a rounding
error. Truncating at exactly N bytes instead writes a file that is no
longer text, and nothing downstream will tell you which record did it.
Section 3 is the point of the program. Every &S[0..n] is attempted for real, with the panic caught so the table can be printed, and the is_char_boundary column beside the outcome — the prediction matches the result on every row. The panic is not a separate mechanism that knows something about UTF-8 you do not have access to; it is is_char_boundary returning false at a moment when nobody thought to ask.
Section 4 is the distinction worth keeping. &S[0..99] panics too, and so does a backwards range, but those are ordinary slice errors that &[u8] has as well. Only the boundary panic is about the encoding, and only it can fire on a range whose length you have already checked — which is why a bounds check is not a substitute for get.
Section 7 is the version of this problem that actually reaches production: a field that is N bytes on disk, holding text that is not ASCII. Truncating at exactly N bytes writes a file that is no longer valid UTF-8, and — this is the part that costs a day — nothing complains at the time. The record is written, the job exits 0, and the failure surfaces weeks later in whatever finally tries to decode it. truncate_to_bytes walks back at most three positions to the nearest boundary and returns a field one to three bytes short of full, which is the correct answer.
std ships this, and the method is worth knowing by name: str::floor_char_boundary ↗, stable since Rust 1.91, with ceil_char_boundary as its complement. The four lines above are what it does, written out because the mechanism is the lesson and because they compile on any toolchain. One caveat std states in its own documentation and this page will not improve on: it rounds to a character boundary, so it can still split a grapheme — a scientist emoji truncated to a person, a letter separated from its combining accent. Both halves are valid UTF-8, which is all this operation ever promised.
What the panic says, and what it said last year¶
The message names the byte index, the character, and the byte range that character occupies — three facts that between them tell you what to fix. It is worth reading once. It is not worth depending on, and this page can prove that rather than assert it, because the wording moved twice in two consecutive releases while the page was being written.
&s[0..4] the end of the range lands inside 'é'
1.75.0 .. 1.94.1 byte index 4 is not a char boundary; it is inside 'é' (bytes 3..5) of `café`
1.95.0 end byte index 4 is not a char boundary; it is inside 'é' (bytes 3..5) of `café`
1.96.1 .. end byte index 4 is not a char boundary; it is inside 'é' (bytes 3..5 of string)
&s[0..99] a different complaint — the range is simply too long
1.75.0 .. 1.94.1 byte index 99 is out of bounds of `café`
1.95.0 end byte index 99 is out of bounds of `café`
1.96.1 .. end byte index 99 is out of bounds for string of length 5
Two changes, one release apart, and each one takes something different away.
1.95 added the start / end prefix. A range has two ends and only one of them is reported, so the prefix tells you which — genuinely useful, and absent from every toolchain older than that. Any advice that says "read whether it says start or end" is advice about a recent compiler.
1.96 removed your string from the message. The older wording ended of `café`, quoting the actual value that was being sliced; the newer one says only of string and, for the out-of-bounds case, its length. That is the change with a consequence beyond readability: on a toolchain before 1.96, a panic in a request handler copies the data it was handling into the panic message, and from there into whatever collects your logs or crash reports. If you are pinned to an older compiler and slicing anything a user typed, that is worth knowing about.
And this is exactly why the generated block above records PANICKED and the is_char_boundary column instead of the sentence. Had the key held the message, this page would have passed CI on the machine that wrote it and failed on any runner a few releases behind or ahead — the failure CONTRIBUTING.md calls out for interpreter diagnostics, met here in a compiler, and found by asking eight builds the same question rather than by asking one twice.
The parts that have not moved across all eight are the parts worth teaching: it is a panic and not a Result; it names the offending byte index; it distinguishes a boundary violation from an out-of-bounds range; and is_char_boundary predicts which one you are about to get.
catch_unwind is not a try/catch¶
The example above catches its own panics so it can print a table instead of dying at its fourth line. That is a legitimate use — the same one a test harness makes — and it is very nearly the only one.
std::panic::catch_unwind exists so a panic does not cross an FFI boundary (which is undefined behaviour) and so one worker thread's failure does not take a server down. It is not error handling: it cannot catch a panic in a binary built with panic = "abort", the payload it hands back is a Box<dyn Any> you have to downcast and guess at, and a caught panic may leave data in a partially-updated state that no type says anything about. Using it to make &s[..n] safe would be writing an exception handler in a language that deliberately has none, when get returns the same information as a value.
The example also installs an empty panic hook — panic::set_hook(Box::new(|_| {})) — because the default one writes to stderr, and stderr is not what an answer key records. Without it the run still passes and prints six unpredictable lines of backtrace advice into the log. That is the second reason the message above is in a dated fence: it does not arrive on the same stream as the rest of the program's output, so it could not have been captured even if the wording were stable.
If you are coming from Python or ABAP¶
Python. s[0:4] slices by character, so the operation on this page has no Python failure mode at all — you can slice any string at any index and get a string. That is the difference to hold onto, because it also means the two languages' slices are not translations of each other: Python's s[0:4] is Rust's s.chars().take(4).collect::<String>(), and Rust's &s[0..4] is Python's s.encode()[0:4] — which returns bytes, and which will happily hand you b'caf\xc3', half a character, with no complaint. So Python has the same bug; it just files it under bytes and lets it through. If you have ever truncated a name field to fit and later found a UnicodeDecodeError in a downstream job, that is this page's panic, arriving late and somewhere else. And when a Python slice does need to be a byte slice, str has nothing to check the boundary with — the nearest is decoding with errors='ignore' and losing the evidence, or an incremental decoder, which is the other page.
ABAP. (Not machine-checked — CI cannot run ABAP.) ABAP's offset/length notation field+3(2) counts characters on a string or c field, so it behaves like Python's — but a Unicode ABAP system's character is a UTF-16 code unit, so field+0(1) on a string beginning with an emoji hands you half a surrogate pair, which is the same bug one encoding along. The genuinely byte-indexed case is xstring, where xfield+3(2) really is bytes and nothing checks that the piece you took is a whole character; a cl_abap_conv_in_ce on the result raises cx_sy_conversion_codepage, which is the panic arriving as an exception at the next conversion rather than at the cut. The habit that transfers is Rust's: work out the boundary before you cut, not after — and treat a length in a DDIC field as bytes unless the data element says otherwise. Verify any specific code-page number against the system.
Try it¶
- Take the longest non-ASCII value in a real table of yours and truncate it at the column's byte width with
&s[..n]. Note the index it panics at, then runis_char_boundaryover every index near it and see the run offalses. - Find a place in code you own where an index into a string was computed — a
len()/2, afindresult plus a constant, a width from configuration. Change it togetand see whether theNonebranch you now have to write is one you can actually answer. - Run
truncate_to_bytesover a list of your own strings at the width of a field you have to fill, and count how many come back short. That count is how many records the naive version would have corrupted. - Write
&s[..s.len() - 1]on a string ending in a non-ASCII character and predict the outcome before running it. Then do the same on one ending in ASCII. Both are the same expression.
Practice¶
Where can you cut "żółw"? It is Polish for turtle, and four characters. Write down its byte length first, then every index from 0 to that length for which is_char_boundary is true — before running anything.
Then say what each of these does, and why the last two differ: &s[0..2], s.get(0..2), &s[..s.len()], &s[..s.len() - 1]. Finish with the practical one: to fit this word into a field of five bytes, how many characters survive, and what does the naive &s[..5] do instead?
Answers
Verified output of slicing_by_byte_kata_rs.rs — regenerated by tools/run_examples.py, never hand-typed.
THE STRING "żółw"
chars ż ó ł w
bytes c5 bc c3 b3 c5 82 77
len() 7 bytes for 4 characters
Three of the four letters cost two bytes; only 'w' is one.
WHERE YOU MAY CUT
boundaries [0, 2, 4, 6, 7]
not boundaries [1, 3, 5]
Five legal indices for a seven-byte string. The odd ones below 6 are all
inside a two-byte letter, and 7 is the end, which always counts.
THE FOUR EXPRESSIONS
&s[0..2] ż
s.get(0..2) Some("ż")
&s[..s.len()] żółw
&s[..s.len() - 1] żół
&s[0..2] byte 2 is the START of 'ó', so this is legal — it is
the whole of 'ż' and nothing else. Legal by luck: the
same expression on "café" would have been fine too, and
on a string starting with a three-byte character it is not.
s.get(0..2) the same answer, wrapped in Some. On a bad index it is
None instead of a panic, which is the entire difference.
&s[..s.len()] always legal, for every string. The end of a string is a
boundary by definition, so this can never fail.
&s[..s.len() - 1] works HERE and is the one that can fail. 'w' is one
byte, so len()-1 lands on a boundary; on a string ending
in any non-ASCII character the same expression panics.
AND THAT LAST PAIR IS THE LESSON
len()-1 is 6 here, and is_char_boundary(6) is true.
"żółw" ends in 'w', a one-byte character, so cutting one byte off the end
removes exactly one character and is safe. Reverse the word and it is not:
"włóż" len 7 boundary at len-1? false
Same expression, same four letters, different answer — because the question
was never about the string's length. It was about its LAST CHARACTER, and
the expression does not mention that.
FIVE BYTES OF FIELD
truncate_to_bytes("żółw", 5) = "żó" 4 bytes, 2 characters
Two characters survive, in four bytes. Byte 5 is inside 'ł', so the honest
answer is to stop at 4 and leave the field one byte short.
The naive &s[..5] does not get an answer at all:
&s[..5] PANICS — byte 5 is not a char boundary
And the version people actually write, in a language with no such check,
is the byte slice — which does not panic and does not work either:
&s.as_bytes()[..5] = c5 bc c3 b3 c5 valid UTF-8? false
Five bytes written to a five-byte field, no error anywhere, and the file is
no longer text. That is the bug this page exists to make loud.
See also¶
Stringis bytes that promise UTF-8 — why the promise has to be re-checked at a cutcharis four bytes — wherechar_indices()gets the indices that are always safe- From UTF-8, and lossy —
valid_up_to(), the other byte offset that is guaranteed to be a boundary - Fixed-width byte fields — the same truncation, as a data problem rather than a language one
- UTF-8 by hand — why a continuation byte is recognisable on sight, which is what makes the check cheap
- Rust strings in practice — "never invent a byte index", in a checklist
- String slices ↗ — the sibling library on slices as borrowed views; this page is only about where the cut may fall