Coverage and the flags that choose tests¶
Level: 201 · anyone who runs go test ./... and reads only the last line
One line: go test -cover prints the share of statements the tests executed — exact for fixed code and fixed tests — and go tool cover -func breaks it down per function; -run and -skip match a regular expression per /-separated level of a test's name; (cached) appears only when go test is given a package argument, and -count=1 switches it off; -shuffle=on prints a seed that changes every run, and -shuffle=1 repeats one order; testing.Short() reports -short.
ParseQuantity and Plural are tested by two cases at first, and by four after a second test file is added. The driver records the percentage, the per-function table, the first lines of the profile, and the percentage again.
Verified output of coverage_and_the_run_flag_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
$ go mod init example
$ go test -cover
PASS
coverage: 60.0% of statements
ok example
exit status 0
$ go test -coverprofile=cover.out >/dev/null && go tool cover -func=cover.out
example/quantity.go:11: ParseQuantity 85.7%
example/quantity.go:24: Plural 0.0%
total: (statements) 60.0%
exit status 0
$ head -n 3 cover.out
mode: set
example/quantity.go:11.51,13.22 2 1
example/quantity.go:13.22,15.3 1 1
$ go test -cover # with more_test.go added
PASS
coverage: 100.0% of statements
ok example
exit status 0
Reading the output¶
coverage: 60.0% of statements — of the ten statements in the package, the two cases executed six: Plural was never called, and ParseQuantity's strconv.Atoi error branch never taken. The figure is a property of the code and the tests, not of the run, so it is a key; the Docker check found the same 60.0% on Linux. In local directory mode, plain go test, the line stands on its own; with a package argument, go test -cover ., it is appended to the ok line (measured, not shown).
go tool cover -func names the functions. ParseQuantity 85.7% and Plural 0.0%, then total: (statements) 60.0%. The profile itself, cover.out, is a list of blocks — file, start line and column, end line and column, the number of statements, and how many times the block ran — under a mode: line; set records whether a block ran, count how often, and atomic counts safely under -race (cmd/cover ↗). go tool cover -html=cover.out paints the source from the same file.
Coverage is a map of what was not tested, not a score. 100.0% of statements after more_test.go says only that every statement ran. A TestPlural that called Plural and compared nothing would produce the same figure — the first bullet of What no tool catches. The cover story ↗ ends on the same caution.
-run and -skip, per level¶
Verified output of coverage_run_and_skip_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
$ go mod init example
$ go test -v -run TestParse
=== RUN TestParseQuantity
=== RUN TestParseQuantity/crates
=== RUN TestParseQuantity/empty
=== RUN TestParseQuantity/negative
=== RUN TestParseUnit
=== RUN TestParseUnit/plural
=== RUN TestParseUnit/empty
PASS
ok example
exit status 0
$ go test -v -run '/empty'
=== RUN TestParseQuantity
=== RUN TestParseQuantity/empty
=== RUN TestParseUnit
=== RUN TestParseUnit/empty
=== RUN TestTotal
PASS
ok example
exit status 0
$ go test -v -run 'TestParseQuantity/^(crates|negative)$'
=== RUN TestParseQuantity
=== RUN TestParseQuantity/crates
=== RUN TestParseQuantity/negative
PASS
ok example
exit status 0
$ go test -v -skip '/empty'
=== RUN TestParseQuantity
=== RUN TestParseQuantity/crates
=== RUN TestParseQuantity/negative
=== RUN TestParseUnit
=== RUN TestParseUnit/plural
=== RUN TestTotal
PASS
ok example
exit status 0
$ go test -v -skip 'empty' # no slash: matched against the top level only
=== RUN TestParseQuantity
=== RUN TestParseQuantity/crates
=== RUN TestParseQuantity/empty
=== RUN TestParseQuantity/negative
=== RUN TestParseUnit
=== RUN TestParseUnit/plural
=== RUN TestParseUnit/empty
=== RUN TestTotal
PASS
ok example
exit status 0
$ go test -run TestInvoice
testing: warning: no tests to run
PASS
ok example
exit status 0
-run TestParse matches the top-level names that contain TestParse — both TestParseQuantity and TestParseUnit, with all of their subtests — and not TestTotal. Patterns are unanchored regular expressions (Testing flags ↗).
-run '/empty' has an empty first element, which matches every top-level test; the second element selects subtests called empty. TestTotal ran too: it has no subtests for the second element to exclude, and go help testflag says that possible parents of matches are run so that their subtests can be looked for. Anchor the elements to be exact: 'TestParseQuantity/^(crates|negative)$'.
-skip '/empty' removes those subtests and keeps everything else; -skip was added in Go 1.20 (release notes ↗). -skip 'empty', with no slash, is matched against the top level only — no test is called empty — and skipped nothing. That silent no-op is the mistake to know about: a pattern that never matches is not an error.
-run TestInvoice: no tests to run, exit 0. A -run that selects nothing is a warning on the first line of the output, and the package still reports ok. A CI job that misspells a test name goes green.
-count=1, the cache, and -failfast¶
Verified output of coverage_count_and_the_cache_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
$ go mod init example
$ go test .
ok example
exit status 0
$ go test . # the same package, the same tests: served from the cache
ok example (cached)
exit status 0
$ go test -count=1 .
ok example
exit status 0
$ go test # local directory mode: never cached
PASS
ok example
exit status 0
$ go test -v -run Fails
=== RUN TestFirstFails
--- FAIL: TestFirstFails
=== RUN TestSecondFails
--- FAIL: TestSecondFails
FAIL
FAIL example
exit status 1
$ go test -v -failfast -run Fails
=== RUN TestFirstFails
--- FAIL: TestFirstFails
FAIL
FAIL example
exit status 1
(cached) on the second go test .. In package list mode — any go test with a package argument, . or ./... — a passing result is cached, keyed on the test binary and the flags, and the next identical run prints (cached) in place of a time (go help test, caching ↗). Only a short list of flags is cacheable — -run, -v, -short, -failfast, -cpu, -parallel and a few more — and -count is not on it, so -count=1 forces a real run, and so would -count=100. Plain go test, local directory mode, never caches, which is why the fourth line shows PASS and ok with no (cached). The cache also watches the files and environment variables a test reads, so a test that opens testdata/ is rerun when that changes.
-failfast stops at the first failure. Two failing tests without it, one with it. It stops the package's tests after the first failure; other packages still run.
-shuffle and -short¶
Verified output of coverage_shuffle_and_short_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
$ go mod init example
$ go test -v -shuffle=1 -run 'Crates|Pallets|Bins'
-test.shuffle 1
=== RUN TestCrates
=== RUN TestPallets
=== RUN TestBins
ok example
exit status 0
$ go test -v -shuffle=7 -run 'Crates|Pallets|Bins'
-test.shuffle 7
=== RUN TestPallets
=== RUN TestBins
=== RUN TestCrates
ok example
exit status 0
$ go test -v -run TestFullImport
=== RUN TestFullImport
stock_test.go:13: importing every delivery note
--- PASS: TestFullImport
PASS
ok example
exit status 0
$ go test -v -short -run TestFullImport
=== RUN TestFullImport
stock_test.go:11: the full import takes minutes; skipped under -short
--- SKIP: TestFullImport
PASS
ok example
exit status 0
-shuffle=1 prints -test.shuffle 1 and one order; -shuffle=7 another. The seed drives a math/rand source, whose sequence is fixed for a fixed seed, so a seed reproduces an order on any machine (Go 1.17 release notes ↗). -shuffle=on picks the time in nanoseconds as the seed and prints it, so that a failure can be reproduced with -shuffle=<that seed>:
1790206819370744000: TestCrates TestBins TestPallets
1790206821056712000: TestPallets TestCrates TestBins
1790206823458345000: TestCrates TestBins TestPallets
1790206825682666000: TestPallets TestBins TestCrates
1790206827537114000: TestBins TestCrates TestPallets
Five runs, five seeds, four orders — bash demo/shuffle_seeds.sh from this folder repeats it. A test that passes in source order and fails under some seed depends on a test before it; the seed is the way to find which.
-short is a flag the test reads. testing.Short() is true when -short was given, and nothing else changes: the test itself must call t.Skip, and then it is reported as --- SKIP with the reason, and the package is still ok (testing.Short ↗). Without -short the full import runs.
-json. go test -json prints one JSON event per line — run, output, pass, fail, with the package and test names and a timestamp — for tooling to read (cmd/test2json ↗); the timestamps keep it out of these keys.
coverage_and_the_run_flag_sh.sh, coverage_run_and_skip_sh.sh, coverage_count_and_the_cache_sh.sh and coverage_shuffle_and_short_sh.sh
coverage_and_the_run_flag_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.
#!/usr/bin/env bash
# go test -cover prints the share of statements the tests ran, and
# -coverprofile plus go tool cover -func breaks it down per function. Both
# are fixed for fixed code and fixed tests, so they are the key. A third test
# case reaches the last untested branch and the figure rises to 100%.
set -u
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir" || exit 1
export GOTOOLCHAIN=local
cat >quantity.go <<'GO'
// Package stock parses the quantities on a delivery note.
package stock
import (
"errors"
"strconv"
"strings"
)
// ParseQuantity reads "12 crates" into its number and its unit.
func ParseQuantity(s string) (int, string, error) {
fields := strings.Fields(s)
if len(fields) != 2 {
return 0, "", errors.New("want a number and a unit")
}
n, err := strconv.Atoi(fields[0])
if err != nil {
return 0, "", err
}
return n, fields[1], nil
}
// Plural returns the unit's plural for a count.
func Plural(n int, unit string) string {
if n == 1 {
return unit
}
return unit + "s"
}
GO
cat >quantity_test.go <<'GO'
package stock
import "testing"
func TestParseQuantity(t *testing.T) {
cases := []struct {
name, in string
n int
unit string
wantErr bool
}{
{"crates", "12 crates", 12, "crates", false},
{"empty", "", 0, "", true},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
n, unit, err := ParseQuantity(c.in)
if (err != nil) != c.wantErr {
t.Fatalf("ParseQuantity(%q): err = %v, want an error: %t", c.in, err, c.wantErr)
}
if n != c.n || unit != c.unit {
t.Errorf("ParseQuantity(%q) = %d, %q; want %d, %q", c.in, n, unit, c.n, c.unit)
}
})
}
}
GO
say() { printf '$ %s\n' "$*"; }
tidy() {
sed -E 's/ \([0-9.]+s\)$//; s/^(ok|FAIL)([[:space:]]+example)[[:space:]]+[0-9.]+s$/\1\2/' test.txt
}
say 'go mod init example'
go mod init example 2>/dev/null || exit 1
say 'go test -cover'
go test -cover >test.txt 2>&1
status=$?
tidy
echo "exit status $status"
say 'go test -coverprofile=cover.out >/dev/null && go tool cover -func=cover.out'
go test -coverprofile=cover.out >/dev/null 2>&1 && go tool cover -func=cover.out
echo "exit status $?"
say 'head -n 3 cover.out'
head -n 3 cover.out
cat >more_test.go <<'GO'
package stock
import "testing"
func TestParseQuantityNotANumber(t *testing.T) {
if _, _, err := ParseQuantity("many crates"); err == nil {
t.Error("want an error for a quantity that is not a number")
}
}
func TestPlural(t *testing.T) {
if got := Plural(1, "crate"); got != "crate" {
t.Errorf("Plural(1) = %q", got)
}
if got := Plural(3, "crate"); got != "crates" {
t.Errorf("Plural(3) = %q", got)
}
}
GO
say 'go test -cover # with more_test.go added'
go test -cover >test.txt 2>&1
status=$?
tidy
echo "exit status $status"
coverage_run_and_skip_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.
#!/usr/bin/env bash
# -run and -skip take a regular expression per level of the test name,
# split on "/": -run TestParse matches the top-level tests whose names
# contain TestParse, -run '/empty' matches subtests called empty under any
# test, and -skip '/empty' removes them. A -run that matches nothing is a
# warning, not a failure: go test still exits 0.
set -u
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir" || exit 1
export GOTOOLCHAIN=local
cat >stock_test.go <<'GO'
package stock
import "testing"
func TestParseQuantity(t *testing.T) {
for _, name := range []string{"crates", "empty", "negative"} {
t.Run(name, func(t *testing.T) {})
}
}
func TestParseUnit(t *testing.T) {
for _, name := range []string{"plural", "empty"} {
t.Run(name, func(t *testing.T) {})
}
}
func TestTotal(t *testing.T) {}
GO
say() { printf '$ %s\n' "$*"; }
# Only the "=== RUN" lines: which tests ran.
ran() {
grep -E '^(=== RUN|testing:|ok|FAIL|PASS)' test.txt | sed -E 's/^(ok|FAIL)([[:space:]]+example)[[:space:]]+[0-9.]+s$/\1\2/'
}
say 'go mod init example'
go mod init example 2>/dev/null || exit 1
say 'go test -v -run TestParse'
go test -v -run TestParse >test.txt 2>&1
status=$?
ran
echo "exit status $status"
say "go test -v -run '/empty'"
go test -v -run '/empty' >test.txt 2>&1
status=$?
ran
echo "exit status $status"
say "go test -v -run 'TestParseQuantity/^(crates|negative)$'"
go test -v -run 'TestParseQuantity/^(crates|negative)$' >test.txt 2>&1
status=$?
ran
echo "exit status $status"
say "go test -v -skip '/empty'"
go test -v -skip '/empty' >test.txt 2>&1
status=$?
ran
echo "exit status $status"
say "go test -v -skip 'empty' # no slash: matched against the top level only"
go test -v -skip 'empty' >test.txt 2>&1
status=$?
ran
echo "exit status $status"
say 'go test -run TestInvoice'
go test -run TestInvoice >test.txt 2>&1
status=$?
ran
echo "exit status $status"
coverage_count_and_the_cache_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.
#!/usr/bin/env bash
# In package list mode -- go test with a package argument, here "." -- a
# passing result is cached, and the second run prints "(cached)" instead of
# a time. -count=1 asks for a real run. In local directory mode, plain
# "go test", nothing is cached. -failfast stops at the first failing test.
set -u
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir" || exit 1
export GOTOOLCHAIN=local
cat >stock_test.go <<'GO'
package stock
import "testing"
func TestCrates(t *testing.T) {}
func TestPallets(t *testing.T) {}
GO
say() { printf '$ %s\n' "$*"; }
# Keep "(cached)"; drop a time.
tidy() {
sed -E 's/^(ok|FAIL)([[:space:]]+example)[[:space:]]+[0-9.]+s$/\1\2/; s/ \([0-9.]+s\)$//' test.txt
}
say 'go mod init example'
go mod init example 2>/dev/null || exit 1
say 'go test .'
go test . >test.txt 2>&1
status=$?
tidy
echo "exit status $status"
say 'go test . # the same package, the same tests: served from the cache'
go test . >test.txt 2>&1
status=$?
tidy
echo "exit status $status"
say 'go test -count=1 .'
go test -count=1 . >test.txt 2>&1
status=$?
tidy
echo "exit status $status"
say 'go test # local directory mode: never cached'
go test >test.txt 2>&1
status=$?
tidy
echo "exit status $status"
cat >failing_test.go <<'GO'
package stock
import "testing"
func TestFirstFails(t *testing.T) { t.Error("the first failure") }
func TestSecondFails(t *testing.T) { t.Error("the second failure") }
GO
say "go test -v -run Fails"
go test -v -run Fails >test.txt 2>&1
status=$?
tidy | grep -E '^(=== RUN|--- |FAIL|ok)'
echo "exit status $status"
say "go test -v -failfast -run Fails"
go test -v -failfast -run Fails >test.txt 2>&1
status=$?
tidy | grep -E '^(=== RUN|--- |FAIL|ok)'
echo "exit status $status"
coverage_shuffle_and_short_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.
#!/usr/bin/env bash
# -shuffle=on runs the tests in a random order and prints the seed it used,
# which changes every run; -shuffle=<seed> repeats one order, so a fixed
# seed is what a key can hold. testing.Short reports the -short flag, and a
# test that calls t.Skip under it is reported as SKIP, not PASS.
set -u
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir" || exit 1
export GOTOOLCHAIN=local
cat >stock_test.go <<'GO'
package stock
import "testing"
func TestCrates(t *testing.T) {}
func TestPallets(t *testing.T) {}
func TestBins(t *testing.T) {}
func TestFullImport(t *testing.T) {
if testing.Short() {
t.Skip("the full import takes minutes; skipped under -short")
}
t.Log("importing every delivery note")
}
GO
say() { printf '$ %s\n' "$*"; }
tidy() {
sed -E 's/ \([0-9.]+s\)$//; s/^(ok|FAIL)([[:space:]]+example)[[:space:]]+[0-9.]+s$/\1\2/' test.txt
}
say 'go mod init example'
go mod init example 2>/dev/null || exit 1
say "go test -v -shuffle=1 -run 'Crates|Pallets|Bins'"
go test -v -shuffle=1 -run 'Crates|Pallets|Bins' >test.txt 2>&1
status=$?
tidy | grep -E '^(-test.shuffle|=== RUN|ok|FAIL)'
echo "exit status $status"
say "go test -v -shuffle=7 -run 'Crates|Pallets|Bins'"
go test -v -shuffle=7 -run 'Crates|Pallets|Bins' >test.txt 2>&1
status=$?
tidy | grep -E '^(-test.shuffle|=== RUN|ok|FAIL)'
echo "exit status $status"
say 'go test -v -run TestFullImport'
go test -v -run TestFullImport >test.txt 2>&1
status=$?
tidy
echo "exit status $status"
say 'go test -v -short -run TestFullImport'
go test -v -short -run TestFullImport >test.txt 2>&1
status=$?
tidy
echo "exit status $status"
What to do¶
- Read
go tool cover -funcfor the functions at 0%, not the total for a number to raise. A function nobody calls in a test is either untested or dead. -run 'TestName/^case$'while fixing one case, and check the=== RUNlines once — a pattern that matches nothing is a warning and exit 0, not a failure.-count=1when the result must be fresh — after changing an environment variable or a file the test does not open itself — andgo test .or./...the rest of the time, so that the cache does its work.- Run
-shuffle=onin CI and keep the seed from a failing run; reproduce with-shuffle=<seed>. - Guard slow tests with
testing.Short()and rungo test -shortlocally, with the full suite in CI — and never guard the only test of a package, which makes-shorta run of nothing.
Practice¶
Take ParseQuantity's coverage from two test cases to 100%: record a profile with -coverprofile, read go tool cover -func to find the branch the two cases miss, add the one case that reaches it in a second test file, and print the table again.
Solution
coverage_and_the_run_flag_kata_go.go in full — pasted here by tools/run_examples.py from the file CI runs.
// Kata: take ParseQuantity's coverage from two test cases to 100% by adding
// the case that reaches the last untested branch, and prove it with go tool
// cover -func before and after. The program writes the module to a
// temporary directory and prints both per-function tables.
//
// go build coverage_and_the_run_flag_kata_go.go && ./coverage_and_the_run_flag_kata_go
package main
import (
"fmt"
"os"
"os/exec"
"path/filepath"
)
const source = `package stock
import (
"errors"
"strconv"
"strings"
)
func ParseQuantity(s string) (int, string, error) {
fields := strings.Fields(s)
if len(fields) != 2 {
return 0, "", errors.New("want a number and a unit")
}
n, err := strconv.Atoi(fields[0])
if err != nil {
return 0, "", err
}
return n, fields[1], nil
}
`
const test = `package stock
import "testing"
func TestParseQuantity(t *testing.T) {
if n, unit, err := ParseQuantity("12 crates"); n != 12 || unit != "crates" || err != nil {
t.Errorf("ParseQuantity(\"12 crates\") = %d, %q, %v", n, unit, err)
}
if _, _, err := ParseQuantity(""); err == nil {
t.Error("want an error for an empty quantity")
}
}
`
const moreTest = `package stock
import "testing"
func TestParseQuantityNotANumber(t *testing.T) {
if _, _, err := ParseQuantity("many crates"); err == nil {
t.Error("want an error for a quantity that is not a number")
}
}
`
func main() {
dir, err := os.MkdirTemp("", "kata")
if err != nil {
panic(err)
}
defer os.RemoveAll(dir)
must(os.WriteFile(filepath.Join(dir, "quantity.go"), []byte(source), 0o644))
must(os.WriteFile(filepath.Join(dir, "quantity_test.go"), []byte(test), 0o644))
run(dir, "go", "mod", "init", "example")
coverage(dir)
fmt.Println("$ cat >more_test.go # the case that reaches the Atoi error branch")
must(os.WriteFile(filepath.Join(dir, "more_test.go"), []byte(moreTest), 0o644))
coverage(dir)
}
// coverage records a profile and prints the per-function table.
func coverage(dir string) {
run(dir, "go", "test", "-coverprofile=cover.out")
fmt.Println("$ go tool cover -func=cover.out")
cmd := exec.Command("go", "tool", "cover", "-func=cover.out")
cmd.Dir = dir
out, err := cmd.CombinedOutput()
fmt.Print(string(out))
must(err)
}
// run runs a command whose output is not part of the key.
func run(dir string, name string, args ...string) {
cmd := exec.Command(name, args...)
cmd.Dir = dir
if out, err := cmd.CombinedOutput(); err != nil {
panic(string(out))
}
}
func must(err error) {
if err != nil {
panic(err)
}
}
Verified output of coverage_and_the_run_flag_kata_go.go — regenerated by tools/run_examples.py, never hand-typed.
In other languages¶
- The Rust library's How
cargo testruns your tests ↗ —cargo test namefilters by substring and--skipremoves, with no per-level/;--test-threads=1fixes an order where Go has-shuffleto unfix one; coverage is not incargoat all and needscargo-llvm-covortarpaulin. - Python's
unittestcommand line ↗ has-kfor a substring pattern and-ffor fail-fast; coverage is the separatecoverage.py↗, which reports per file and per line rather than per function.
Sources¶
- Testing flags ↗ —
-run,-skip,-count,-cover,-coverprofile,-failfast,-shuffle,-short,-json, and how-run's elements are matched. go help test↗ — local directory mode against package list mode, and the test cache.cmd/cover↗ — the profile format and-func/-html; The cover story ↗ — Rob Pike, the Go blog, 2013: how coverage is instrumented and what it does not tell you; Coverage profiling support for integration tests ↗ —go build -coverfor a whole binary, since Go 1.20.testing.Short↗,cmd/test2json↗.- Release notes: Go 1.10 ↗ (the test cache,
-failfast,-json), Go 1.17 ↗ (-shuffle), Go 1.20 ↗ (-skip). - Alan A. A. Donovan and Brian W. Kernighan, The Go Programming Language (Addison-Wesley, 2015), chapter 11, "Testing" — 11.1, "The go test Tool", and 11.3, "Coverage".
- Jon Bodner, Learning Go, 2nd ed. (O'Reilly, 2024), chapter 15, "Writing Tests" — "Caching Test Results" and "Checking Your Code Coverage".
- Miki Tebeka, Effective Go Recipes (Pragmatic Bookshelf, 2024), chapter 13, "Testing Your Code" — recipe 65, "Conditionally Running Continuous Integration Tests".