A file is bytes; a `String` is a promise¶

Level: 201 · working knowledge

One line: A File reads and writes [u8], never str — so on the way out a &str is widened to bytes for free, on the way in the UTF-8 promise is checked once at the door and read_to_string refuses with InvalidData rather than hand you a lie, and include_str! moves that same check to compile time.

String vs &str says what the two text types promise; Six kinds of string says the bottom row of that family — Vec<u8> and &[u8] — promises nothing. A file is that bottom row. Everything on this page is what happens when text crosses the line between the two, in either direction, and the program below crosses it both ways inside a scratch directory it makes and removes (how that stays deterministic).

Verified output of a_file_is_bytes.rs — regenerated by tools/run_examples.py, never hand-typed.

1. On the way out, three spellings of the same seven bytes
   "Łódź" is 4 chars and 7 bytes: [C5, 81, C3, B3, 64, C5, BA]
   write_all(b"..")       [C5, 81, C3, B3, 64, C5, BA]
   write_all(as_bytes())  [C5, 81, C3, B3, 64, C5, BA]
   fs::write(path, word)  [C5, 81, C3, B3, 64, C5, BA]
   write_all(word) with no as_bytes() is E0308: the file wants &[u8], and
   a &str is not one until you say so. The transcript is on the page.

2. write! into a String and write! into a File are two traits
   String  "Łódź = 7 bytes"
   File    "Łódź = 7 bytes"
   Same macro, same text. fmt::Write returns a fmt::Error a String never
   produces, so it is .unwrap(); io::Write returns an io::Error, so it is ?.

3. On the way in, the promise is checked at the door
   read_to_string   Err(InvalidData): stream did not contain valid UTF-8
   fs::read         Ok([A3, F3, 64, BC])  -- no promise, so nothing to check
   from_utf8        Err: invalid utf-8 sequence of 1 bytes from index 0
   from_utf8_lossy  "��d�"
   Three letters replaced and one kept: d is 0x64 in every code page there is.
   the same word saved as UTF-8 reads back: "Łódź"

4. A failed check leaves the String you passed in untouched
   read_to_string into "kept:" from the Latin-2 file  Err(InvalidData), buffer "kept:"
   and from the UTF-8 file it appends:                "kept:Łódź"

5. include_str! moves the check to compile time
   this program's own source, embedded; its first line:
   "//! A file is bytes; a String is a promise about bytes. On the way out the"
   include_bytes! of the same file has the same length: true
   No file is opened at run time. Had the file been Latin-2, this program
   would not have compiled -- that transcript is on the page too.

6. Scratch directory removed: true

On the way out, widening is free¶

write_all takes &[u8], and section 1 writes the same seven bytes three ways: a byte string b"…" with the bytes spelled out (the literal prefixes page has the whole family), as_bytes() on the &str, and fs::write ↗, which takes impl AsRef<[u8]> — str implements it, so the &str goes in as it is. No copy and no check: a &str already is valid bytes, so widening it to "just bytes" costs nothing. That is the same direction Six kinds of string calls free and silent; the checked direction is the next section but one.

Leave the as_bytes() out and rustc says what the book says — don't forget the b in front — and offers to type it for you:

rustc 1.98.0 on bytes_out.rs, which calls file.write_all with a plain string literal — abridged: the note pointing into std is dropped

error[E0308]: mismatched types
    --> bytes_out.rs:6:20
     |
   6 |     file.write_all("first line\n")?;
     |          --------- ^^^^^^^^^^^^^^ expected `&[u8]`, found `&str`
     |          |
     |          arguments to this method are incorrect
     |
     = note: expected reference `&[u8]`
                found reference `&'static str`
help: consider adding a leading `b`
     |
   6 |     file.write_all(b"first line\n")?;
     |                    +

Two `write!`s, two traits¶

Section 2. write!(s, …) into a String and write!(f, …) into a File are the same macro, and it expands to a write_fmt call that two different traits provide: std::fmt::Write ↗ for anything that collects text, std::io::Write ↗ for anything that takes bytes. A String implements the first and a File the second, and the error types differ — fmt::Error, which a String never produces, against io::Error, which a disk can — so one call ends in .unwrap() and the other in ?. Both traits can be in scope at once, as they are in the example; nothing collides, because no std type implements both. Import neither and the message is the one A trait must be in scope is about, with the right use line in the help:

rustc 1.98.0 on no_trait.rs, which calls writeln! on a File with no use line — abridged: the two notes pointing into std are dropped

error[E0599]: cannot write into `File`
    --> no_trait.rs:5:14
     |
   5 |     writeln!(file, "first line")?;
     |              ^^^^
     |
     = help: items from traits can only be used if the trait is in scope
help: trait `Write` which provides `write_fmt` is implemented but not in scope; perhaps you want to import it
     |
   1 + use std::io::Write;
     |

The help says Write, and there are two of those. Import std::fmt::Write because it was the first one in the completion list and the error is unchanged — same code, same help, still asking for the io one — because a File has no fmt::Write to find. Building a String is the fmt side of the same macro, where the import trap runs the other way.

On the way in, the promise is checked once¶

Section 3 writes Łódź the way a Polish file from the 1990s would hold it — four ISO-8859-2 bytes, A3 F3 64 BC — and reads it back three ways:

fs::read_to_string ↗ refuses: InvalidData, stream did not contain valid UTF-8. It will not hand you a String that breaks the promise every method on String relies on.
fs::read ↗ hands the four bytes back as a Vec<u8>, because bytes promise nothing and there is nothing to check.
String::from_utf8 is the same check as a function you call yourself, and its error says where it failed — index 0, the Ł. String::from_utf8_lossy is the recovery, and it replaces what it cannot read with U+FFFD: three of the four letters here, all but the d, because ASCII is the one range every code page agrees on.

The check runs once, at the door. After it, .chars(), .len(), slicing and every str method trust the bytes without looking again — which is why the check cannot be skipped, and the whole subject of Validation is a boundary ↗ in the encodings library, which reads core's validator line by line. Section 4 is the other half of the contract: when the check fails, the String you passed to read_to_string is left exactly as it was ↗ — no half-decoded prefix — and when it succeeds the text is appended to it, not assigned.

Where the bytes came from is the encodings library's territory — Code pages ↗ is where ISO-8859-2 is explained, and SAP code pages ↗ is where it turns up as code page 1401. The kata below decodes it.

`include_str!` moves the check to compile time¶

include_str! ↗ reads a file while rustc runs and embeds its contents in the binary as a &'static str; section 5 embeds the example's own source and prints its first line. Nothing is opened at run time, which is why the macro returns a &str with no Result around it: a file that is not there, or is not UTF-8, is a build error instead of a runtime one.

rustc 1.98.0 on embed_latin2.rs, which include_str!s a file holding the four Latin-2 bytes above

error: `lodz_latin2.txt` wasn't a utf-8 file
 --> lodz_latin2.txt:0:1
  |
0 |
  | ^ byte `163` is not valid utf-8

error: aborting due to 1 previous error

163 is 0xA3, the Ł. include_bytes! ↗ is the same embedding with no promise attached — a &'static [u8; N] — and would have accepted the file.

The price is the binary. The book's demonstration is Dracula from Project Gutenberg, at 999 KB embedded against 166 KB read at run time; measured here with a one-million-byte text file and a program that counts its lines, so the optimizer cannot drop the text as unused:

measured 2026-09-12, rustc 1.98.0, macOS 26.6 — big.txt is 1,000,000 bytes, 18,182 lines

                                    include_str!("big.txt")   fs::read_to_string("big.txt")   difference
rustc --edition 2024                          1,467,680 bytes                   475,840 bytes      991,840
rustc --edition 2024 -O                       1,458,672 bytes                   460,440 bytes      998,232

The difference is the file, give or take the bookkeeping. One measurement worth knowing before you try to reproduce this: a program that only prints text.len() shows no difference under -O, because the length is a compile-time constant and the unused bytes are dropped — the text has to be used for it to be kept.

How this example stays deterministic¶

Same recipe as Opening a file: everything is written inside a directory created under std::env::temp_dir() and named with the process id, no path is ever printed, and the directory is removed at the end. The three error strings on this page — stream did not contain valid UTF-8, invalid utf-8 sequence of 1 bytes from index 0, and the InvalidData kind — come from std, not from the operating system, so they read the same everywhere; the key was diffed against a Linux run (rust:1.98-slim in Docker) all the same.

If you are coming from another language¶

Python. Python's open() defaults to text mode, which means the decoding this page does by hand happens inside read() — and the failure has a name of its own:

Python	Rust
`open(p, "rb").read()`	`fs::read(p)`	bytes, no check, no failure to decode
`open(p, encoding="utf-8").read()`	`fs::read_to_string(p)`	`UnicodeDecodeError` is `Err(InvalidData)`; both refuse rather than guess
`open(p, encoding="utf-8", errors="replace").read()`	`String::from_utf8_lossy(&fs::read(p)?)`	U+FFFD in place of what could not be read
`open(p, encoding="iso-8859-2").read()`	the kata's table, by hand	Python ships every code page; `std` ships UTF-8 and nothing else
`f.write("Łódź")` in text mode	`f.write_all("Łódź".as_bytes())`	Python encodes for you with the file's `encoding`; Rust never encodes, because a `File` has no encoding to use
`open(p).read()` with no `encoding=`	(no equivalent)	the machine's locale decides — the sibling library's What the bet is on ↗; Rust has no default to bet on, which is the point of this page

What changes: in Python the encoding is a property of the open file object, so read() and write() transcode silently and the bet is placed once, at open(). In Rust the file has no encoding, read_to_string is UTF-8 by definition, and anything else is a decode you write or a crate you add (The string crates lists encoding_rs). Nothing is placed for you; nothing is placed silently.

ABAP. The same split, with the encoding as a clause on the open — and the code page the kata decodes by hand has a number in SAP's table:

ABAP	Rust
`OPEN DATASET dset FOR INPUT IN BINARY MODE.` + `READ DATASET dset INTO xstr.`	`fs::read(p)`	bytes into an `xstring`; nothing converted
`OPEN DATASET dset FOR INPUT IN TEXT MODE ENCODING UTF-8.` + `READ DATASET dset INTO str.`	`fs::read_to_string(p)`	converted from UTF-8 into the system's UTF-16 on the way in; a bad byte raises `CX_SY_CONVERSION_CODEPAGE` where Rust returns `InvalidData`
`OPEN DATASET dset FOR INPUT IN LEGACY TEXT MODE CODE PAGE '1401'.`	the kata's table	1401 is ISO-8859-2 in SAP's code-page table — the decoder is a clause, not a function you write
`cl_abap_conv_codepage=>create_in( )->convert( xstr )`	`String::from_utf8(bytes)`	the same check as a standalone call, on bytes you already hold
`TRANSFER str TO dset.` in text mode	`write_all(str.as_bytes())`	ABAP encodes on the way out with the `ENCODING` clause; Rust writes the UTF-8 that a `str` already is

What changes: ABAP text is UTF-16 inside the system, so every text-mode READ DATASET converts, and the conversion's source encoding is whatever the OPEN said — get the clause wrong and the letters come out wrong with no error at all, exactly as open(p, encoding=…) does in Python. Rust's String is UTF-8 inside and out, so a UTF-8 file converts nothing and a Latin-2 file converts nowhere until you say how. (Not machine-checked — CI cannot run ABAP; the clause semantics are from the 7.58 keyword documentation for OPEN DATASET ↗.)

Practice¶

Read a Latin-2 file properly. Write read_text(path) -> io::Result<(String, &'static str)> that tries read_to_string first and, on InvalidData only, reads the bytes and decodes them as ISO-8859-2 — ASCII maps to itself, the eighteen Polish letters go through a table of (u8, char) pairs, and any other high byte becomes U+FFFD. Return which path was taken. Write Łódź to one file as UTF-8 and to another as the four bytes A3 F3 64 BC, read both, and compare what from_utf8_lossy would have made of the second.

Make the mistake first: match on is_err() instead of on the kind, call it on a file that does not exist, and work out what the caller would have seen.

Then write the two bytes C3 B3 to a file and read it. In Latin-2 they are Ă and ł; in UTF-8 they are one letter, ó, and your function cannot tell. Say what that means for a design that guesses.

Solution

a_file_is_bytes_kata.rs in full — pasted here by tools/run_examples.py from the file CI compiles and runs.

//! Kata solution: read a file that may be ISO-8859-2 -- InvalidData as the
//! signal, an eighteen-letter table as the decoder, and the two bytes that
//! pass as UTF-8 by accident.
//!
//!   rustc --edition 2024 a_file_is_bytes_kata.rs -o /tmp/afibk && /tmp/afibk

use std::fs;
use std::io::{self, ErrorKind};
use std::path::{Path, PathBuf};

fn scratch_dir() -> io::Result<PathBuf> {
    let dir = std::env::temp_dir().join(format!("a_file_is_bytes_kata_{}", std::process::id()));
    fs::create_dir_all(&dir)?;
    Ok(dir)
}

/// The nine Polish letters ISO-8859-2 places above 0x7F, in both cases.
const LATIN2_POLISH: [(u8, char); 18] = [
    (0xA1, 'Ą'), (0xB1, 'ą'), (0xC6, 'Ć'), (0xE6, 'ć'), (0xCA, 'Ę'), (0xEA, 'ę'),
    (0xA3, 'Ł'), (0xB3, 'ł'), (0xD1, 'Ń'), (0xF1, 'ń'), (0xD3, 'Ó'), (0xF3, 'ó'),
    (0xA6, 'Ś'), (0xB6, 'ś'), (0xAC, 'Ź'), (0xBC, 'ź'), (0xAF, 'Ż'), (0xBF, 'ż'),
];

/// ASCII maps to itself, the eighteen letters go through the table, and any
/// other high byte becomes U+FFFD -- the same honesty as from_utf8_lossy, one
/// byte at a time, because a single-byte code page has no sequences to lose.
fn latin2_to_string(bytes: &[u8]) -> String {
    bytes
        .iter()
        .map(|&b| match LATIN2_POLISH.iter().find(|(code, _)| *code == b) {
            _ if b < 0x80 => b as char,
            Some(&(_, ch)) => ch,
            None => '\u{FFFD}',
        })
        .collect()
}

/// UTF-8 first; on InvalidData, and only then, decode as Latin-2. Every other
/// error -- the file is missing, the disk failed -- goes back to the caller.
fn read_text(path: &Path) -> io::Result<(String, &'static str)> {
    match fs::read_to_string(path) {
        Ok(text) => Ok((text, "utf-8")),
        Err(e) if e.kind() == ErrorKind::InvalidData => Ok((latin2_to_string(&fs::read(path)?), "latin-2")),
        Err(e) => Err(e),
    }
}

fn replaced(s: &str) -> usize {
    s.chars().filter(|&c| c == '\u{FFFD}').count()
}

fn main() -> io::Result<()> {
    let dir = scratch_dir()?;
    let word = "Łódź";

    println!("1. Two files, one word, two encodings");
    let utf8 = dir.join("utf8.txt");
    let latin2 = dir.join("latin2.txt");
    fs::write(&utf8, word)?;
    fs::write(&latin2, [0xA3u8, 0xF3, 0x64, 0xBC])?;
    for (name, path) in [("utf8.txt", &utf8), ("latin2.txt", &latin2)] {
        let (text, how) = read_text(path)?;
        println!("   {name:<11} {} bytes -> {text:?} via {how}", fs::metadata(path)?.len());
    }
    println!("   from_utf8_lossy on the Latin-2 bytes would have replaced {} of 4;",
        replaced(&String::from_utf8_lossy(&fs::read(&latin2)?)));
    println!("   the table replaced {}, because it knows what the bytes meant.", replaced(&read_text(&latin2)?.0));

    println!("\n2. A high byte the table does not know is still reported, not invented");
    let odd = dir.join("odd.txt");
    fs::write(&odd, [0x4C, 0xF3, 0x64, 0xFF])?; // 0xFF is a dot-above in Latin-2; not in our table
    let (text, how) = read_text(&odd)?;
    println!("   {text:?} via {how}, {} byte replaced", replaced(&text));

    println!("\n3. The mistake worth making first: matching on is_err() instead of the kind");
    let missing = dir.join("missing.txt");
    println!("   read_text(missing)  {}", match read_text(&missing) {
        Ok((text, how)) => format!("Ok({text:?} via {how})"),
        Err(e) => format!("Err({:?})", e.kind()),
    });
    println!("   With is_err() the missing file would have gone down the Latin-2 branch,");
    println!("   fs::read would have failed a second time, and the caller would see an");
    println!("   error about decoding a file that was never there.");

    println!("\n4. The signal is a heuristic, and here is the file that fools it");
    let accident = dir.join("accident.txt");
    fs::write(&accident, [0xC3u8, 0xB3])?; // Latin-2 for the two letters below
    let (text, how) = read_text(&accident)?;
    println!("   C3 B3 read as {text:?} via {how}");
    println!("   In Latin-2 that is Ă then ł -- the table, which knows only Polish letters,");
    println!("   gives {:?} -- and in UTF-8 it is one letter. The UTF-8 reading wins", latin2_to_string(&[0xC3, 0xB3]));
    println!("   because it is tried first and it succeeds.");
    println!("   Nothing in the bytes says which was meant. A file format that names its");
    println!("   encoding -- or a caller that does -- is the fix; guessing is not.");

    fs::remove_dir_all(&dir)?;
    Ok(())
}

Verified output of a_file_is_bytes_kata.rs — regenerated by tools/run_examples.py, never hand-typed.

1. Two files, one word, two encodings
   utf8.txt    7 bytes -> "Łódź" via utf-8
   latin2.txt  4 bytes -> "Łódź" via latin-2
   from_utf8_lossy on the Latin-2 bytes would have replaced 3 of 4;
   the table replaced 0, because it knows what the bytes meant.

2. A high byte the table does not know is still reported, not invented
   "Lód�" via latin-2, 1 byte replaced

3. The mistake worth making first: matching on is_err() instead of the kind
   read_text(missing)  Err(NotFound)
   With is_err() the missing file would have gone down the Latin-2 branch,
   fs::read would have failed a second time, and the caller would see an
   error about decoding a file that was never there.

4. The signal is a heuristic, and here is the file that fools it
   C3 B3 read as "ó" via utf-8
   In Latin-2 that is Ă then ł -- the table, which knows only Polish letters,
   gives "�ł" -- and in UTF-8 it is one letter. The UTF-8 reading wins
   because it is tried first and it succeeds.
   Nothing in the bytes says which was meant. A file format that names its
   encoding -- or a caller that does -- is the fix; guessing is not.

Po polsku¶

Plik to bajty, a String to obietnica o bajtach — że są poprawnym UTF-8. Cała ta strona opisuje, co się dzieje, gdy tekst przekracza tę granicę. W stronę pliku przejście jest darmowe: write_all bierze &[u8], a &str to już są poprawne bajty, więc as_bytes() niczego nie kopiuje ani nie sprawdza; zapomnij o nim, a kompilator sam zaproponuje literał bajtowy b"…" (błąd E0308, z podpowiedzią consider adding a leading b). W stronę programu obietnica jest sprawdzana raz, przy drzwiach: read_to_string na pliku zapisanym w ISO-8859-2 — a tak wyglądają polskie pliki z lat dziewięćdziesiątych, Łódź to cztery bajty A3 F3 64 BC — odmawia z InvalidData i zostawia twój String nietknięty, fs::read oddaje bajty bez sprawdzania, bo bajty niczego nie obiecują, a from_utf8_lossy podstawia U+FFFD za trzy z czterech liter — przeżywa tylko d, bo ASCII jest jedynym zakresem, co do którego zgadzają się wszystkie strony kodowe.

Dwie rzeczy, których angielski czytelnik może nie zauważyć nigdy. Po pierwsze, write! do String i write! do File to jedno makro, ale dwie różne cechy (traits) — std::fmt::Write dla tekstu i std::io::Write dla bajtów — i zaimportowanie tej niewłaściwej daje dokładnie ten sam błąd E0599 co brak importu. Po drugie, include_str! wczytuje plik w trakcie kompilacji i wkleja go do pliku wykonywalnego, więc plik w Latin-2 nie skompiluje się w ogóle (wasn't a utf-8 file), a plik wykonywalny rośnie mniej więcej o jego rozmiar — zmierzone powyżej na milionie bajtów. W ABAP-ie odpowiednikiem tej decyzji jest klauzula IN TEXT MODE ENCODING … albo IN BINARY MODE na OPEN DATASET, a kodowanie ISO-8859-2 ma tam numer strony kodowej 1401; w Ruscie plik nie ma żadnego kodowania, read_to_string z definicji znaczy UTF-8, a wszystko inne to dekoder, który piszesz sam — jak w zadaniu na końcu strony — albo crate, który dodajesz.

Szukaj po polsku: zapis tekstu do pliku w Ruscie · odczyt pliku w kodowaniu ISO-8859-2 · literał bajtowy · rust read_to_string InvalidData · rust include_str! binary size

A file is bytes; a String is a promise¶