Output:`¶

Level: 201 · anyone who has copied an example from package documentation and found it no longer compiled

One line: func ExampleOrder_Total() with an // Output: comment is compiled and run by go test, and what it prints is compared with the comment: a wrong comment fails with got: and want:, // Unordered output: compares the lines as a set, and an Example with no Output comment is compiled but never run.

This library has one rule above the others: a page never types what a program prints — it marks the spot, and tools/run_examples.py fills it from a real run (CONTRIBUTING.md, "Output is generated, never typed"). An Example function is that rule inside go test. order.Total sums an order's lines; order.PrintUnits prints one line per item from a map. The examples live in example_test.go, in the external package order_test, and import the package the way a client would.

Verified output of an_example_test_checks_its_output_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go mod init example
$ go test -v
=== RUN   ExampleOrder_Total
--- PASS: ExampleOrder_Total
=== RUN   ExampleOrder_Total_wrongComment
--- FAIL: ExampleOrder_Total_wrongComment
got:
2500
want:
2400
=== RUN   ExampleOrder_PrintUnits
--- PASS: ExampleOrder_PrintUnits
FAIL
exit status 1
FAIL    example
exit status 1
$ go test -list .
ExampleOrder_Total
ExampleOrder_Total_wrongComment
ExampleOrder_PrintUnits
ok      example
exit status 0

Reading the output¶

Run, and compared. --- FAIL: ExampleOrder_Total_wrongComment followed by got: 2500 and want: 2400: go test ran the function, captured its standard output, and compared it with the text after // Output:, ignoring leading and trailing space (testing, "Examples" ↗). The comment was wrong — two crates at 1250 are 2500 — and the test says so, on every run. An example that stops compiling fails the build of the whole test binary.

// Unordered output: for a map. PrintUnits ranges over a map[string]int, and the spec leaves the iteration order over a map unspecified (For statements with range clause ↗) — an // Output: for it would pass on some runs and fail on others. With // Unordered output: the lines are compared as a set, and ExampleOrder_PrintUnits passes whatever order the map chose. It is the Example-test form of the rule in Deterministic about nondeterminism: force the order or compare without it.

No Output comment: compiled, never run. ExampleOrder_PrintUnits_notRun appears nowhere in the -v transcript, and go test -list . does not name it. It is still compiled, so an example that cannot be run in a test — it dials a server, prints the time — still cannot drift from the API. The list shows the three runnable examples in source order, and -run selects among them like any test.

The external package. package order_test in a file next to package order is a separate package that can only see what order exports (go help test ↗). An example written there reads as a client's code, order.Order{...}, which is what a reader of the documentation wants to copy.

The name says what is documented¶

The name of an Example function is how go doc and pkg.go.dev know where to show it: Example for the package, ExampleTotal for a function, ExampleOrder for a type, ExampleOrder_Total for a method, and ExampleOrder_Total_discounted for a second example of the same method, where the suffix begins with a lowercase letter (testing, "Examples" ↗). go vet's tests analyzer checks the names against the package, and go test runs that analyzer before building:

Verified output of an_example_test_naming_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go mod init example
$ go vet
# example_test
# [example_test]
./example_test.go:16:1: ExampleOrder_Total_Discounted has malformed example suffix: Discounted
./example_test.go:22:1: ExampleInvoice refers to unknown identifier: Invoice
./example_test.go:28:1: ExampleOrder_Ship refers to unknown field or method: Order.Ship
./example_test.go:34:1: ExampleOrder should be niladic
exit status 1
$ go test
# example_test
# [example_test]
./example_test.go:16:1: ExampleOrder_Total_Discounted has malformed example suffix: Discounted
./example_test.go:22:1: ExampleInvoice refers to unknown identifier: Invoice
./example_test.go:28:1: ExampleOrder_Ship refers to unknown field or method: Order.Ship
./example_test.go:34:1: ExampleOrder should be niladic
FAIL    example [build failed]
exit status 1

ExampleOrder_Total_discounted passed the check; a capital D turned the same suffix into a method that does not exist. An Example takes no parameters and returns nothing — should be niladic — and one for an identifier the package lacks is refused too. On pkg.go.dev each example is rendered under the identifier it names, with the output shown and a button that runs it in the playground; Testable Examples in Go ↗ walks through the rendering. The other analyzers go test runs are on the vet page.

an_example_test_checks_its_output_sh.sh and an_example_test_naming_sh.sh

an_example_test_checks_its_output_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# An Example function with an "// Output:" comment is compiled and run by go
# test, and its printed output is compared with the comment. A wrong comment
# fails with a got/want report. "// Unordered output:" compares the lines as
# a set, which is what a range over a map needs. An Example with no Output
# comment is compiled but never run, and -list does not name it.
set -u

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir" || exit 1
export GOTOOLCHAIN=local

cat >order.go <<'GO'
// Package order totals a customer's order.
package order

import "fmt"

// Line is one line of an order: an item, how many, and the unit price in cents.
type Line struct {
    Item  string
    Qty   int
    Cents int
}

// Order is what one customer asked for.
type Order struct{ Lines []Line }

// Total is the order's value in cents.
func (o Order) Total() int {
    sum := 0
    for _, l := range o.Lines {
        sum += l.Qty * l.Cents
    }
    return sum
}

// Units counts the units of each item over all of the order's lines.
func (o Order) Units() map[string]int {
    units := map[string]int{}
    for _, l := range o.Lines {
        units[l.Item] += l.Qty
    }
    return units
}

// PrintUnits prints one line per item, in map order -- which is no order.
func (o Order) PrintUnits() {
    for item, n := range o.Units() {
        fmt.Printf("%s x%d\n", item, n)
    }
}
GO

cat >example_test.go <<'GO'
package order_test

import (
    "fmt"

    "example"
)

func ExampleOrder_Total() {
    o := order.Order{Lines: []order.Line{{"crate", 2, 1250}, {"pallet", 1, 9900}}}
    fmt.Println(o.Total())
    // Output: 12400
}

// The comment is wrong: two crates at 1250 come to 2500.
func ExampleOrder_Total_wrongComment() {
    o := order.Order{Lines: []order.Line{{"crate", 2, 1250}}}
    fmt.Println(o.Total())
    // Output: 2400
}

func ExampleOrder_PrintUnits() {
    o := order.Order{Lines: []order.Line{{"crate", 2, 1250}, {"pallet", 1, 9900}, {"crate", 3, 1250}}}
    o.PrintUnits()
    // Unordered output:
    // pallet x1
    // crate x5
}

// No Output comment: compiled, shown in the documentation, never run.
func ExampleOrder_PrintUnits_notRun() {
    o := order.Order{Lines: []order.Line{{"crate", 2, 1250}}}
    o.PrintUnits()
}
GO

say() { printf '$ %s\n' "$*"; }

tidy() {
    sed -E 's/ \([0-9.]+s\)$//; s/^(ok|FAIL)([[:space:]]+example)[[:space:]]+[0-9.]+s$/\1\2/' test.txt
}

say 'go mod init example'
go mod init example 2>/dev/null || exit 1

say 'go test -v'
go test -v >test.txt 2>&1
status=$?
tidy
echo "exit status $status"

say 'go test -list .'
go test -list . >test.txt 2>&1
status=$?
tidy
echo "exit status $status"

an_example_test_naming_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# The name of an Example function says what it documents: ExampleT_M for a
# method, ExampleT_M_suffix for a second example, and the suffix starts with a
# lowercase letter. go vet's "tests" analyzer checks the names against the
# package, and go test runs that analyzer before it builds the test binary.
set -u

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir" || exit 1
export GOTOOLCHAIN=local

cat >order.go <<'GO'
package order

// Order is what one customer asked for.
type Order struct{ Cents int }

// Total is the order's value in cents.
func (o Order) Total() int { return o.Cents }
GO

cat >example_test.go <<'GO'
package order_test

import (
    "fmt"

    "example"
)

// A lowercase suffix is fine: this is a second example of Order.Total.
func ExampleOrder_Total_discounted() {
    fmt.Println(order.Order{Cents: 1000}.Total() * 9 / 10)
    // Output: 900
}

// An uppercase suffix is read as a method name -- and Order has no Discounted.
func ExampleOrder_Total_Discounted() {
    fmt.Println(order.Order{Cents: 1000}.Total())
    // Output: 1000
}

// Nothing in the package is called Invoice.
func ExampleInvoice() {
    fmt.Println("invoice")
    // Output: invoice
}

// Order has no method Ship.
func ExampleOrder_Ship() {
    fmt.Println("shipped")
    // Output: shipped
}

// An example takes no parameters.
func ExampleOrder(o order.Order) {
    fmt.Println(o.Total())
    // Output: 0
}
GO

say() { printf '$ %s\n' "$*"; }

say 'go mod init example'
go mod init example 2>/dev/null || exit 1

say 'go vet'
go vet 2>&1
echo "exit status $?"

say 'go test'
go test 2>&1 | sed -E 's/^(ok|FAIL)([[:space:]]+example)[[:space:]]+[0-9.]+s$/\1\2/'
echo "exit status ${PIPESTATUS[0]}"

What to do¶

Write the example you would want to copy from the documentation, and give it an // Output: comment, so that it is run on every go test and cannot rot. Print what the reader would look at — the result, not a debugging trace.
// Unordered output: when the order is not yours to promise — a map, a set of goroutines' results — and otherwise force the order before printing, exactly as a page in this library does.
Put examples in package order_test, in a file whose name ends _test.go, so that they read as client code and can import anything without an import cycle.
Leave the Output comment off an example that cannot run in a test, and know that it is then documentation only: go test -list will not name it and nothing checks what it prints.

Practice¶

Rebuild the mechanism by hand: write checkExample(name, run, want, unordered) that runs a function with os.Stdout redirected to a pipe, reads back what it printed, trims the surrounding space and compares it with want — sorting the lines of both first when unordered is set — and prints --- PASS or --- FAIL with got: and want:. Check three examples against an invoice held in a map: a right total, a wrong total, and the unordered lines.

Solution

an_example_test_checks_its_output_kata_go.go in full — pasted here by tools/run_examples.py from the file CI runs.

// Kata: the mechanism behind an Example test, by hand. checkExample runs a
// function with os.Stdout redirected to a pipe, reads back what it printed,
// trims the surrounding space and compares it with the expected text -- and
// for an unordered example compares the sorted lines instead. A mismatch is
// reported as got/want, the way go test reports it.
//
//  go build an_example_test_checks_its_output_kata_go.go && ./an_example_test_checks_its_output_kata_go
package main

import (
    "fmt"
    "io"
    "os"
    "sort"
    "strings"
)

type invoice struct{ lines map[string]int }

func (inv invoice) printLines() {
    for item, cents := range inv.lines {
        fmt.Printf("%s %d\n", item, cents)
    }
}

func (inv invoice) total() int {
    sum := 0
    for _, cents := range inv.lines {
        sum += cents
    }
    return sum
}

// capture returns what run printed to os.Stdout.
func capture(run func()) string {
    saved := os.Stdout
    r, w, err := os.Pipe()
    if err != nil {
        panic(err)
    }
    os.Stdout = w
    run()
    w.Close()
    os.Stdout = saved
    printed, _ := io.ReadAll(r)
    return string(printed)
}

func sortedLines(s string) string {
    lines := strings.Split(s, "\n")
    sort.Strings(lines)
    return strings.Join(lines, "\n")
}

func checkExample(name string, run func(), want string, unordered bool) {
    got := strings.TrimSpace(capture(run))
    want = strings.TrimSpace(want)
    if unordered {
        got, want = sortedLines(got), sortedLines(want)
    }
    if got == want {
        fmt.Printf("--- PASS: %s\n", name)
        return
    }
    fmt.Printf("--- FAIL: %s\ngot:\n%s\nwant:\n%s\n", name, got, want)
}

func main() {
    inv := invoice{lines: map[string]int{"crates": 2500, "pallet": 9900, "delivery": 1500}}
    checkExample("ExampleInvoice_total", func() { fmt.Println(inv.total()) }, "13900", false)
    checkExample("ExampleInvoice_total_wrong", func() { fmt.Println(inv.total()) }, "13000", false)
    checkExample("ExampleInvoice_printLines", inv.printLines, "pallet 9900\ncrates 2500\ndelivery 1500\n", true)
}

Verified output of an_example_test_checks_its_output_kata_go.go — regenerated by tools/run_examples.py, never hand-typed.

--- PASS: ExampleInvoice_total
--- FAIL: ExampleInvoice_total_wrong
got:
13900
want:
13000
--- PASS: ExampleInvoice_printLines

In other languages¶

The Rust library's The example that is a test ↗ — the closest cousin: every code block in a /// doc comment is compiled and run by cargo test. A Rust doc test checks itself with assert_eq! inside the block rather than by comparing printed output, and it is attached to the item by position, not by a naming rule.
Python's doctest ↗ compares a REPL transcript pasted into a docstring — >>> total() followed by the expected output — with what the expressions print, which is exactly // Output: with the prompt kept.

Sources¶

testing, "Examples" ↗ — the naming rules, // Output:, // Unordered output:, and that an example without an output comment is compiled but not executed.
Testable Examples in Go ↗ — Andrew Gerrand, the Go blog, 2015: examples as tests and how they are rendered in the documentation.
go help test ↗ — the _test external test package.
go vet's tests analyzer ↗ — malformed names and example suffixes.
The Go specification, For statements with range clause ↗ — the iteration order over a map is not specified.
Alan A. A. Donovan and Brian W. Kernighan, The Go Programming Language (Addison-Wesley, 2015), chapter 11, "Testing" — 11.6, "Example Functions", and 11.2.4, "External Test Packages".
Jon Bodner, Learning Go, 2nd ed. (O'Reilly, 2024), chapter 15, "Writing Tests" — "Testing Your Public API".

An Example test runs its // Output:¶