Testing with trybuild

Level: 201 · working knowledge

One line: trybuild compiles small programs against your crate, t.pass expecting each to build and run, t.compile_fail expecting each to be refused with exactly the error recorded in a .stderr file beside it, so the messages your users see are under test too.

The test

This workspace is the facade and derive from The re-export pattern, with the two crates named tested_describe and tested_describe_derive, and the generated paths changed to match: tested_describe holds the trait, and tested_describe_derive refuses anything but a struct with named fields. The UI tests live in the facade, because a test program has to use the derive the way a user does:

demo/tested_describe/Cargo.toml

[package]
name = "tested_describe"
version = "0.1.0"
edition = "2024"
publish = false

[dependencies]
tested_describe_derive = { path = "../tested_describe_derive" }

[dev-dependencies]
trybuild = "1"

demo/tested_describe/tests/ui.rs

//! Every file in `tests/ui/`, compiled against this crate as a program of its own.

#[test]
fn ui() {
    let t = trybuild::TestCases::new();
    t.pass("tests/ui/pass_*.rs");
    t.compile_fail("tests/ui/fail_*.rs");
}

Each file under tests/ui/ is a whole program. One should compile and run:

demo/tested_describe/tests/ui/pass_named_fields.rs

use tested_describe::Describe;

#[derive(Describe)]
struct Point {
    x: i32,
    y: i32,
}

fn main() {
    let p = Point { x: 1, y: 2 };
    assert_eq!(p.describe(), "Point: x = 1, y = 2");
}

Two should not, and each has the error it must produce recorded beside it:

demo/tested_describe/tests/ui/fail_enum.rs

use tested_describe::Describe;

#[derive(Describe)]
enum Direction {
    Up,
    Down,
}

fn main() {}

demo/tested_describe/tests/ui/fail_enum.stderr

error: Describe needs a struct
 --> tests/ui/fail_enum.rs:4:6
  |
4 | enum Direction {
  |      ^^^^^^^^^

demo/tested_describe/tests/ui/fail_tuple_struct.rs

use tested_describe::Describe;

#[derive(Describe)]
struct Meters(f64);

fn main() {}

demo/tested_describe/tests/ui/fail_tuple_struct.stderr

error: Describe needs named fields
 --> tests/ui/fail_tuple_struct.rs:4:8
  |
4 | struct Meters(f64);
  |        ^^^^^^

Verified output of cargo test -q -p tested_describe --test ui — declared in cargo_runs.toml and regenerated by tools/run_cargo_demos.py, never hand-typed.

test tests/ui/pass_named_fields.rs [should pass] ... ok
test tests/ui/fail_enum.rs [should fail to compile] ... ok
test tests/ui/fail_tuple_struct.rs [should fail to compile] ... ok

• One #[test], three cases. trybuild runs its cases when the TestCases value is dropped ↗, at the end of ui(), and libtest sees one test. The globs pass_*.rs and fail_*.rs pick up new cases without editing ui.rs.
• Every case is built as a real program. trybuild generates a package called tested_describe-tests, in tests/trybuild/tested_describe under the target directory ↗, with one binary per case. It depends on tested_describe, and on everything tested_describe depends on, dev-dependencies included ↗, trybuild itself excepted, so a case can use tested_describe_derive directly too. A pass case is built and run, so the assert_eq! in pass_named_fields.rs is checked too.
• What the key leaves out. The key is trybuild's report on stderr. That build prints Cargo's progress lines, Compiling tested_describe-tests v0.0.0 (/…) and Finished … in 0.30s, which hold absolute paths and timings and appear only when something was rebuilt; the declared run deletes them. libtest's own running 1 test summary goes to stdout and is not in the key.

Where a .stderr file comes from

Not from typing it. Run the tests once with the variable set:

TRYBUILD=overwrite cargo test -p tested_describe --test ui

and trybuild writes each compile_fail case's output to its .stderr file. Without the variable, a case with no .stderr file gets its output written to a wip/ directory beside Cargo.toml instead, and the test fails ↗ until you move the file into place. Both workflows are in the crate's documentation ↗. Read the diff before committing, as with any recorded key.

The file is not the compiler's raw output. trybuild normalizes it, line by line, so it holds on another machine:

• Paths become relative. rustc was compiling a file outside the generated package, so it printed an absolute path; trybuild strips the crate's directory ↗, leaving tests/ui/fail_enum.rs:4:6. A path into the standard library becomes $RUST/… ↗, one into the Cargo registry $CARGO/… ↗, and line numbers outside the test file are blanked.
• Cargo's closing lines are gone. error: could not compile … and For more information about this error, try rustc --explain … are dropped ↗; compare the cargo build transcripts on The re-export pattern, which keep both.
• Trailing whitespace is trimmed.

The message text and the ^^^^ under the name are left alone. They are what the test is for.

A regression, recorded

tests/ui_stale/fail_enum.stderr was recorded with TRYBUILD=overwrite while the derive still said Describe can only be derived for a struct. The message was then changed to Describe needs a struct, and this file was kept as it was:

demo/tested_describe/tests/ui_stale.rs

//! One failing input, held to a `.stderr` file recorded before the derive's
//! message was reworded.

#[test]
fn ui_stale() {
    let t = trybuild::TestCases::new();
    t.compile_fail("tests/ui_stale/fail_enum.rs");
}

demo/tested_describe/tests/ui_stale/fail_enum.stderr

error: Describe can only be derived for a struct
 --> tests/ui_stale/fail_enum.rs:4:6
  |
4 | enum Direction {
  |      ^^^^^^^^^

Verified output of cargo test -q -p tested_describe --test ui_stale, which fails on purpose — declared in cargo_runs.toml and regenerated by tools/run_cargo_demos.py, never hand-typed.

test tests/ui_stale/fail_enum.rs ... mismatch

EXPECTED:
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error: Describe can only be derived for a struct
 --> tests/ui_stale/fail_enum.rs:4:6
  |
4 | enum Direction {
  |      ^^^^^^^^^
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈

ACTUAL OUTPUT:
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error: Describe needs a struct
 --> tests/ui_stale/fail_enum.rs:4:6
  |
4 | enum Direction {
  |      ^^^^^^^^^
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
note: If the actual output is the correct output you can bless it by rerunning
      your test with the environment variable TRYBUILD=overwrite



error: test failed, to rerun pass `-p tested_describe --test ui_stale`

EXPECTED is the file, ACTUAL OUTPUT is today's compiler output after normalization, and the case fails on the one line that differs. The same filter as above removed Cargo's progress lines. The last line is Cargo's own, because the test binary panicked; the panic message itself, 1 of 1 tests failed, is on stdout.

A mismatch means one of two things, and trybuild cannot tell which. If the change was a regression, fix the macro. If it was intended, as here, bless the new output by running with TRYBUILD=overwrite again and commit the updated .stderr. A new compiler release that rewords a note is the same kind of mismatch, which is one reason the library pins its toolchain.

Unit tests and UI tests

The two halves test different things, and a derive wants both:

	Unit test of expand	trybuild UI test
Written against	proc_macro2 tokens, in #[cfg(test)] beside the logic	a whole program, as a user writes it
Can check	the generated tokens, and an error's message	whether the output compiles and runs, and the error exactly as rustc prints it, with its location and underline
Cannot check	whether the generated code compiles, or where an error points	anything that does not show in a build
Runs	the logic as a function, with no compiler involved	a Cargo build of the generated package

proc-macro2 makes it testable is the first column: its refuses_an_enum test asserts an error's text. The UI test above also pins down that rustc underlines Direction, the span the derive chose, and not enum.

If you are coming from another language

• Java. Google's compile-testing library runs javac in-process on source strings, with an annotation processor attached, and asserts .failed() and .hadErrorContaining("…"). That is compile_fail with a substring match. trybuild compares the whole normalized message, location and underline included, against a file, so any change shows up in review as a diff of that file.
• Python. The nearest habit is snapshot testing, with syrupy or pytest-snapshot, whose --snapshot-update does what TRYBUILD=overwrite does. The thing being snapshotted is what differs: a Python snapshot holds a value a function returned, and a .stderr file holds what a compiler said about a program.
• C and C++. Compilers test their own diagnostics this way: clang's test suite compiles a file with -verify, and // expected-error {{…}} comments beside the offending line say which errors must appear. That harness belongs to the compiler. trybuild gives the same check to a library, such as a derive whose errors are part of its interface, with one dev-dependency.

See also

• proc-macro2 makes it testable — the unit-test half
• The re-export pattern — the facade and derive tested here
• Errors: from panic! to syn::Error — the messages worth testing
• Where a test goes — why tests/ui.rs is an integration test
• The trybuild crate ↗

Po polsku

trybuild kompiluje małe programy korzystające z twojego crate'a: przypadki z t.pass(…) muszą się zbudować i uruchomić, a przypadki z t.compile_fail(…) — zostać odrzucone z dokładnie takim błędem, jaki zapisano w pliku .stderr obok. Dzięki temu komunikaty, które widzi użytkownik makra, też są pod testem. Plik .stderr generuje się przez TRYBUILD=overwrite cargo test, a trybuild go normalizuje: ścieżki stają się względne, a końcowe linie Cargo (error: could not compile …) znikają, więc plik jest ten sam na każdej maszynie.

Gdy komunikat makra się zmieni, test pokazuje EXPECTED i ACTUAL OUTPUT — to regresja albo zamierzona zmiana, którą zatwierdza się ponownym uruchomieniem z TRYBUILD=overwrite. Testy jednostkowe funkcji expand na proc_macro2 sprawdzają wygenerowane tokeny; trybuild sprawdza, czy kod się kompiluje i gdzie kompilator wskazuje błąd.

Szukaj po polsku: testowanie komunikatów błędów makr w Ruscie · trybuild compile_fail · TRYBUILD=overwrite · rust ui test proc macro · trybuild stderr mismatch