Skip to content

Parallel subtests wait, Cleanup runs last

Level: 301 · anyone whose parallel subtests all saw the last row of the table, or whose test left files behind

One line: A subtest that calls t.Parallel() pauses until the parent's function has returned, then runs with its siblings, up to -parallel of them at once; t.Cleanup functions run after the test and all of its subtests, last registered first; t.TempDir is removed by one of them; t.Context is cancelled just before they run; and a test that calls both t.Setenv and t.Parallel panics.

Four warehouse subtests call t.Parallel(). The parent logs how many of them are running the moment its loop is done, and its Cleanup logs the highest number that ran at once. The first test makes the four wait for each other at a barrier, so it can only pass when all four run together — under -parallel 4. The second has no barrier and runs under -parallel 1. The order in which paused subtests resume changes between runs, so the driver prints the transcript verbatim up to the parent's last log line and, after it, counts and sorts.

Verified output of t_parallel_subtests_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go mod init example
$ go test -v -parallel 4 -run 'TestWarehousesTogether$'
=== RUN   TestWarehousesTogether
=== RUN   TestWarehousesTogether/north
=== PAUSE TestWarehousesTogether/north
=== RUN   TestWarehousesTogether/south
=== PAUSE TestWarehousesTogether/south
=== RUN   TestWarehousesTogether/east
=== PAUSE TestWarehousesTogether/east
=== RUN   TestWarehousesTogether/west
=== PAUSE TestWarehousesTogether/west
=== NAME  TestWarehousesTogether
    warehouse_test.go:41: parent: the loop is done; subtests running right now: 0
=== CONT lines: 4
    warehouse_test.go:38: checking east
    warehouse_test.go:38: checking north
    warehouse_test.go:38: checking south
    warehouse_test.go:38: checking west
    warehouse_test.go:42: parent's Cleanup: most subtests running at once: 4
    --- PASS: TestWarehousesTogether/east
    --- PASS: TestWarehousesTogether/north
    --- PASS: TestWarehousesTogether/south
    --- PASS: TestWarehousesTogether/west
--- PASS: TestWarehousesTogether
PASS
ok      example
exit status 0
$ go test -v -parallel 1 -run 'TestWarehousesOneAtATime$'
=== PAUSE lines: 4
=== CONT lines: 4
    warehouse_test.go:57: parent's Cleanup: most subtests running at once: 1
PASS
ok      example
exit status 0
$ go test -v -cpu 1,2 -run 'TestWarehousesOneAtATime$'
the test ran 2 times, under the name TestWarehousesOneAtATime
exit status 0

Reading the output

=== PAUSE, then the parent finishes, then === CONT. Each t.Parallel() prints === PAUSE and blocks; subtests running right now: 0 is the parent's own log line, printed after its loop and before any subtest has resumed. Parallel subtests do not run until the parent test function returns (T.Parallel ↗) — the parent then waits for them, so a --- PASS for the parent still means every subtest passed. The === NAME line marks the output switching back to the parent after the subtests' lines.

With -parallel 4, all four ran at once. most subtests running at once: 4 is the parent's Cleanup reading a counter that each subtest incremented on entry. The barrier is a sync.WaitGroup sized to the table: every subtest calls Done and then Wait, so none gets past it until all four have started. Under -parallel 1 the same barrier would wait forever — the flag caps how many parallel tests run at once, and Wait would be holding the one slot. -parallel defaults to GOMAXPROCS (Testing flags ↗), so a test that needs n subtests together should say -parallel n rather than assume the machine has the cores.

With -parallel 1, the peak is 1. Same === PAUSE and === CONT counts, one subtest in its body at a time. The transcript looks the same; only the counter tells the two apart — which is why the parallelism of a test is not something to read off go test -v.

-cpu 1,2 runs the test once per value. Two --- PASS lines, both named TestWarehousesOneAtATime: the test is run with GOMAXPROCS set to each value in turn, and, unlike a benchmark, a test's name does not gain a -2 suffix. The purpose is to run scheduling-sensitive tests under several settings; how -cpu and -count shake out a rare interleaving is the Concurrency library's stress-test page ↗.

Cleanup, TempDir and Context

Verified output of t_cleanup_and_tempdir_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go mod init example
$ go test -v
=== RUN   TestCleanupOrder
=== RUN   TestCleanupOrder/child
    warehouse_test.go:14: child's body
    warehouse_test.go:13: child's cleanup
=== NAME  TestCleanupOrder
    warehouse_test.go:16: parent's body ends
    warehouse_test.go:11: cleanup 2: registered second
    warehouse_test.go:10: cleanup 1: registered first, runs last
--- PASS: TestCleanupOrder
    --- PASS: TestCleanupOrder/child
=== RUN   TestTempDir
    warehouse_test.go:29: in Cleanup, the TempDir still exists: true
--- PASS: TestTempDir
=== RUN   TestTempDirIsGone
    warehouse_test.go:36: after TestTempDir, its directory exists: false
--- PASS: TestTempDirIsGone
=== RUN   TestContextEndsBeforeCleanup
    warehouse_test.go:42: in the body, ctx.Err() = <nil>
    warehouse_test.go:41: in Cleanup, ctx.Err() = context canceled
--- PASS: TestContextEndsBeforeCleanup
PASS
ok      example
exit status 0

Last registered, first run. cleanup 2 printed before cleanup 1, after parent's body ends. Cleanup registers a function to be called when the test and all its subtests complete, and the functions run in last-added-first order (T.Cleanup ↗) — a defer for the test, with one difference that matters: a defer in the test function runs when that function returns, before parallel subtests have run; a Cleanup waits for them.

The child's cleanup ran before the parent's body ended. child's cleanup sits between child's body and parent's body ends: a sequential subtest completes when its t.Run returns, and its cleanups run then. A parallel subtest's would run when it finishes, after the parent's function.

t.TempDir exists inside Cleanup and is gone by the next test. TempDir creates a directory and registers its removal with Cleanup (T.TempDir ↗); the test's own Cleanup, registered later, ran first and found it still there. TestTempDirIsGone looked for the same path and did not find it. Each call to TempDir in one test returns a new directory, so a test can have several.

t.Context is cancelled just before the cleanups. <nil> in the body, context canceled in Cleanup. T.Context, added in Go 1.24, returns a context that is cancelled just before the Cleanup-registered functions are called (T.Context ↗) — so a goroutine the test started with that context is told to stop before the test's resources are removed. synctest.Wait instead of a sleep derives its cancellable context from it.

t.Setenv and t.Parallel do not mix

Verified output of t_parallel_refuses_setenv_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go mod init example
$ go test -v -run 'TestRegion$|TestRegionAfter'
=== RUN   TestRegion
    region_test.go:10: WAREHOUSE_REGION = north
--- PASS: TestRegion
=== RUN   TestRegionAfter
    region_test.go:14: after TestRegion, WAREHOUSE_REGION = ""
--- PASS: TestRegionAfter
PASS
ok      example
exit status 0
$ go test -v -run TestRegionParallel
=== RUN   TestRegionParallel
=== PAUSE TestRegionParallel
=== CONT  TestRegionParallel
--- FAIL: TestRegionParallel
panic: testing: test using t.Setenv or t.Chdir can not use t.Parallel [recovered, repanicked]
exit status 2
FAIL    example
exit status 1

Set for the test, restored after it. TestRegionAfter sees "": Setenv calls os.Setenv and registers the old value's restoration with Cleanup (T.Setenv ↗). The environment is one per process, which is the whole reason for the next line.

In a parallel test it panics. panic: testing: test using t.Setenv or t.Chdir can not use t.Parallel — and the same message in the other order, Setenv first and then Parallel (measured, not shown). A parallel test shares the process with whichever tests are running alongside, and an environment variable set for one of them would be seen by all. t.Chdir, added in Go 1.24, is refused for the same reason. The [recovered, repanicked] suffix is how the Go 1.25 runtime prints a panic that a deferred function recovered and raised again (Go 1.25 release notes ↗) — the testing package's own handler, which marks the test failed and re-panics.

t_parallel_subtests_sh.sh, t_cleanup_and_tempdir_sh.sh and t_parallel_refuses_setenv_sh.sh

t_parallel_subtests_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# Subtests that call t.Parallel pause until the parent's function has
# returned, then run together, up to -parallel of them at once. The order in
# which they resume changes from run to run, so this script prints the lines
# up to the parent's last log line verbatim, then counts and sorts the rest.
set -u

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir" || exit 1
export GOTOOLCHAIN=local

cat >warehouse_test.go <<'GO'
package warehouse

import (
    "sync"
    "sync/atomic"
    "testing"
)

// running counts subtests inside their body; peak is the most seen at once.
var running, peak atomic.Int32

func enter() {
    n := running.Add(1)
    for {
        p := peak.Load()
        if n <= p || peak.CompareAndSwap(p, n) {
            return
        }
    }
}

func leave() { running.Add(-1) }

var names = []string{"north", "south", "east", "west"}

// The four subtests wait for each other at a barrier, so this test can only
// pass when all four are running at the same time: go test -parallel 4.
func TestWarehousesTogether(t *testing.T) {
    var allStarted sync.WaitGroup
    allStarted.Add(len(names))
    for _, name := range names {
        t.Run(name, func(t *testing.T) {
            t.Parallel()
            enter()
            defer leave()
            allStarted.Done()
            allStarted.Wait() // returns only once all four subtests are running
            t.Log("checking", name)
        })
    }
    t.Log("parent: the loop is done; subtests running right now:", running.Load())
    t.Cleanup(func() { t.Log("parent's Cleanup: most subtests running at once:", peak.Load()) })
}

// No barrier: under -parallel 1 the subtests still pause and resume, one
// at a time.
func TestWarehousesOneAtATime(t *testing.T) {
    peak.Store(0)
    for _, name := range names {
        t.Run(name, func(t *testing.T) {
            t.Parallel()
            enter()
            defer leave()
            t.Log("checking", name)
        })
    }
    t.Cleanup(func() { t.Log("parent's Cleanup: most subtests running at once:", peak.Load()) })
}
GO

say() { printf '$ %s\n' "$*"; }

# The transcript, with what varies between runs made fixed: the lines up to
# the parent's log line are in a fixed order; after it the subtests resume in
# any order, so those lines are counted or sorted.
report() {
    sed -n '1,/parent: the loop/p' test.txt
    echo "=== CONT lines: $(grep -c '^=== CONT' test.txt)"
    grep 'checking' test.txt | sort
    grep "parent's Cleanup" test.txt
    grep -E '^ *--- ' test.txt | sed -E 's/ \([0-9.]+s\)$//' | sort
    tail -n 2 test.txt | sed -E 's/^(ok|FAIL)([[:space:]]+example)[[:space:]]+[0-9.]+s$/\1\2/'
}

say 'go mod init example'
go mod init example 2>/dev/null || exit 1

say "go test -v -parallel 4 -run 'TestWarehousesTogether$'"
go test -v -parallel 4 -run 'TestWarehousesTogether$' >test.txt 2>&1
status=$?
report
echo "exit status $status"

say "go test -v -parallel 1 -run 'TestWarehousesOneAtATime$'"
go test -v -parallel 1 -run 'TestWarehousesOneAtATime$' >test.txt 2>&1
status=$?
echo "=== PAUSE lines: $(grep -c '^=== PAUSE' test.txt)"
echo "=== CONT lines: $(grep -c '^=== CONT' test.txt)"
grep "parent's Cleanup" test.txt
tail -n 2 test.txt | sed -E 's/^(ok|FAIL)([[:space:]]+example)[[:space:]]+[0-9.]+s$/\1\2/'
echo "exit status $status"

say "go test -v -cpu 1,2 -run 'TestWarehousesOneAtATime$'"
go test -v -cpu 1,2 -run 'TestWarehousesOneAtATime$' >test.txt 2>&1
status=$?
echo "the test ran $(grep -c '^--- PASS: TestWarehousesOneAtATime' test.txt) times, under the name $(grep '^--- PASS' test.txt | cut -d' ' -f3 | sort -u | tr '\n' ' ' | sed 's/ $//')"
echo "exit status $status"

t_cleanup_and_tempdir_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# t.Cleanup functions run after the test and its subtests, last registered
# first. t.TempDir is removed by one of them, so it still exists inside a
# Cleanup and is gone by the next test. t.Context is cancelled just before
# the Cleanup functions run. None of this depends on timing, so the whole
# -v transcript is the key, minus the durations.
set -u

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir" || exit 1
export GOTOOLCHAIN=local

cat >warehouse_test.go <<'GO'
package warehouse

import (
    "os"
    "path/filepath"
    "testing"
)

func TestCleanupOrder(t *testing.T) {
    t.Cleanup(func() { t.Log("cleanup 1: registered first, runs last") })
    t.Cleanup(func() { t.Log("cleanup 2: registered second") })
    t.Run("child", func(t *testing.T) {
        t.Cleanup(func() { t.Log("child's cleanup") })
        t.Log("child's body")
    })
    t.Log("parent's body ends")
}

var manifestDir string

func TestTempDir(t *testing.T) {
    dir := t.TempDir()
    manifest := filepath.Join(dir, "manifest.csv")
    if err := os.WriteFile(manifest, []byte("WH-1,12\n"), 0o644); err != nil {
        t.Fatal(err)
    }
    t.Cleanup(func() {
        _, err := os.Stat(dir)
        t.Log("in Cleanup, the TempDir still exists:", err == nil)
    })
    manifestDir = dir
}

func TestTempDirIsGone(t *testing.T) {
    _, err := os.Stat(manifestDir)
    t.Log("after TestTempDir, its directory exists:", err == nil)
}

func TestContextEndsBeforeCleanup(t *testing.T) {
    ctx := t.Context()
    t.Cleanup(func() { t.Log("in Cleanup, ctx.Err() =", ctx.Err()) })
    t.Log("in the body, ctx.Err() =", ctx.Err())
}
GO

say() { printf '$ %s\n' "$*"; }

tidy() {
    sed -E 's/ \([0-9.]+s\)$//; s/^(ok|FAIL)([[:space:]]+example)[[:space:]]+[0-9.]+s$/\1\2/' test.txt
}

say 'go mod init example'
go mod init example 2>/dev/null || exit 1

say 'go test -v'
go test -v >test.txt 2>&1
status=$?
tidy
echo "exit status $status"

t_parallel_refuses_setenv_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# t.Setenv sets a process-wide variable and restores it in a Cleanup, which
# only makes sense while no other test runs: a test that calls both t.Setenv
# and t.Parallel panics, in either order. The panic's goroutine trace holds
# paths and addresses, so only its first line is printed.
set -u

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir" || exit 1
export GOTOOLCHAIN=local
unset WAREHOUSE_REGION

cat >region_test.go <<'GO'
package warehouse

import (
    "os"
    "testing"
)

func TestRegion(t *testing.T) {
    t.Setenv("WAREHOUSE_REGION", "north")
    t.Log("WAREHOUSE_REGION =", os.Getenv("WAREHOUSE_REGION"))
}

func TestRegionAfter(t *testing.T) {
    t.Logf("after TestRegion, WAREHOUSE_REGION = %q", os.Getenv("WAREHOUSE_REGION"))
}

func TestRegionParallel(t *testing.T) {
    t.Parallel()
    t.Setenv("WAREHOUSE_REGION", "north")
}
GO

say() { printf '$ %s\n' "$*"; }

tidy() {
    sed -E 's/ \([0-9.]+s\)$//; s/^(ok|FAIL)([[:space:]]+example)[[:space:]]+[0-9.]+s$/\1\2/' test.txt
}

say 'go mod init example'
go mod init example 2>/dev/null || exit 1

say "go test -v -run 'TestRegion$|TestRegionAfter'"
go test -v -run 'TestRegion$|TestRegionAfter' >test.txt 2>&1
status=$?
tidy
echo "exit status $status"

say 'go test -v -run TestRegionParallel'
go test -v -run TestRegionParallel >test.txt 2>&1
status=$?
tidy | grep -E '^(=== |--- |panic:|FAIL|exit status)'
echo "exit status $status"

What to do

  • Call t.Parallel() first thing in the subtest, and only for subtests that share nothing — no package-level variable, no environment, no working directory, no fixed port. Run the suite with -race once after adding it: The race detector sees what the subtests share.
  • t.Cleanup for anything a subtest may still be using; defer only for what the test function itself is done with when it returns.
  • t.TempDir() instead of os.MkdirTemp — no name to invent, no removal to remember, and a test that fails leaves nothing behind.
  • t.Setenv and t.Chdir in sequential tests only. A test that needs both an environment variable and parallelism should take the value as a parameter instead of reading the environment.
  • Pass -parallel n explicitly when a test's correctness depends on n running together, and expect the default, GOMAXPROCS, to differ between a laptop and a CI runner.

Practice

Write a test with three parallel subtests, one per region, each of which makes its own t.TempDir() and records its name and its directory for the parent under a mutex. In the parent's t.Cleanup, print the names sorted and count how many of the three directories no longer exist. Run it with -parallel 3 and print only the lines whose order cannot vary.

Solution

t_parallel_and_t_cleanup_kata_go.go in full — pasted here by tools/run_examples.py from the file CI runs.

// Kata: three parallel subtests, each with its own t.TempDir, that record
// their names for the parent; the parent's Cleanup -- which runs after all
// three -- prints the names sorted and checks that every directory is gone.
// The program writes the test to a temporary module, runs it, and prints
// only the lines whose order cannot vary.
//
//  go build t_parallel_and_t_cleanup_kata_go.go && ./t_parallel_and_t_cleanup_kata_go
package main

import (
    "fmt"
    "os"
    "os/exec"
    "path/filepath"
    "regexp"
    "strings"
)

const test = `package warehouse

import (
    "os"
    "sort"
    "sync"
    "testing"
)

func TestRegions(t *testing.T) {
    var mu sync.Mutex
    var finished []string
    var dirs []string
    for _, region := range []string{"north", "south", "east"} {
        t.Run(region, func(t *testing.T) {
            t.Parallel()
            dir := t.TempDir()
            mu.Lock()
            finished = append(finished, region)
            dirs = append(dirs, dir)
            mu.Unlock()
        })
    }
    t.Cleanup(func() {
        sort.Strings(finished)
        t.Log("kata: subtests finished:", finished)
        gone := 0
        for _, dir := range dirs {
            if _, err := os.Stat(dir); os.IsNotExist(err) {
                gone++
            }
        }
        t.Logf("kata: temp dirs removed: %d of %d", gone, len(dirs))
    })
}
`

var keep = regexp.MustCompile(`^(=== RUN|=== PAUSE|--- PASS|PASS|ok)|kata:`)

func main() {
    dir, err := os.MkdirTemp("", "kata")
    if err != nil {
        panic(err)
    }
    defer os.RemoveAll(dir)
    must(os.WriteFile(filepath.Join(dir, "regions_test.go"), []byte(test), 0o644))
    init := exec.Command("go", "mod", "init", "example")
    init.Dir = dir
    must(init.Run())

    fmt.Println("$ go test -v -parallel 3 -run TestRegions")
    cmd := exec.Command("go", "test", "-v", "-parallel", "3", "-run", "TestRegions")
    cmd.Dir = dir
    out, _ := cmd.CombinedOutput()
    for _, line := range strings.Split(string(out), "\n") {
        if keep.MatchString(line) {
            line = regexp.MustCompile(` \([0-9.]+s\)$|\t[0-9.]+s$`).ReplaceAllString(line, "")
            if strings.HasPrefix(line, "--- PASS") {
                fmt.Println(line) // the parent's line; its subtests' lines are indented
                continue
            }
            fmt.Println(strings.TrimSpace(line))
        }
    }
    fmt.Println("exit status", cmd.ProcessState.ExitCode())
}

func must(err error) {
    if err != nil {
        panic(err)
    }
}

Verified output of t_parallel_and_t_cleanup_kata_go.go — regenerated by tools/run_examples.py, never hand-typed.

$ go test -v -parallel 3 -run TestRegions
=== RUN   TestRegions
=== RUN   TestRegions/north
=== PAUSE TestRegions/north
=== RUN   TestRegions/south
=== PAUSE TestRegions/south
=== RUN   TestRegions/east
=== PAUSE TestRegions/east
regions_test.go:26: kata: subtests finished: [east north south]
regions_test.go:33: kata: temp dirs removed: 3 of 3
--- PASS: TestRegions
PASS
ok      example
exit status 0

In other languages

  • The Rust library's How cargo test runs your tests ↗ — every Rust test is a thread and they run in parallel by default, --test-threads=1 serializes them, and std::env::set_var is unsafe in edition 2024 because of exactly the hazard that makes t.Setenv refuse t.Parallel. Go's default is the other way round: sequential unless a test opts in.
  • Python's unittest.TestCase.addCleanup ↗ runs its functions in last-in, first-out order, like t.Cleanup; pytest's tmp_path fixture ↗ is t.TempDir. Neither runtime runs tests in parallel without a plugin.
  • The Concurrency library's Can a test wait an hour in a millisecond? ↗ — the synctest bubble that this library's chapter 07 uses, next to the injected clocks the other languages need.

Sources