Skip to content

A panic ends the whole program

Level: 201 · anyone who has put a recover in main and expected it to protect the program

One line: A panic that reaches the top of any goroutine's stack ends the whole process with exit status 2, however healthy main is; a deferred recover stops a panic only in the goroutine that is panicking, so the one in main never sees a worker's.

A bug in a worker

Verified output of panic_in_a_worker_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go build prices.go
$ ./prices
main:   starting the worker
worker: deferred call runs as the panic unwinds
panic: runtime error: index out of range [3] with length 3
exit status 2

Reading the output

sumPrices goes one index too far, and indexing a slice out of range is a run-time panic ↗. The panic happened in the worker goroutine. What ran and what did not:

  • The worker's deferred call ran. A panic stops the function it happens in and runs deferred calls on its way up the panicking goroutine's stack — the spec's Handling panics ↗.
  • Nothing more of main ran. main had nothing left to do but receive from totals. Its deferred Println never printed, and neither did its deferred recover: the spec says that when the unwinding reaches the top-level function of the goroutine that panicked, the program is terminated and the panic reported. Terminated then and there, with no turn for main's deferred calls.
  • Exit status 2. That is what the runtime documentation ↗ says an unrecovered panic exits with by default, after a stack trace of the current goroutine. The script keeps the first line of that report and drops the trace, which names a goroutine number and addresses that change between runs.

Even if main's deferred function had run, its recover would have returned nil: recover returns the panic's value only when called directly by a deferred function in a goroutine that is panicking, and main's goroutine was not.

panic_in_a_worker_sh.sh

panic_in_a_worker_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# A panic in a goroutine that main started. main is healthy -- it is waiting
# for a total -- and it has a deferred recover, yet the whole process exits
# with status 2. The stack trace under the panic line names a goroutine number
# and addresses that change between runs, so the script keeps stdout, the
# `panic:` line from stderr, and the exit status.
set -u
export GOTOOLCHAIN=local

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT

cat >"$dir/prices.go" <<'GO'
package main

import "fmt"

func sumPrices(prices []int) (cents int) {
    for i := 0; i <= len(prices); i++ { // one step too far
        cents += prices[i]
    }
    return cents
}

func main() {
    defer func() {
        if r := recover(); r != nil {
            fmt.Println("main:   recovered:", r)
        }
    }()
    defer fmt.Println("main:   deferred call")

    totals := make(chan int)
    fmt.Println("main:   starting the worker")
    go func() {
        defer fmt.Println("worker: deferred call runs as the panic unwinds")
        totals <- sumPrices([]int{1999, 450, 1200})
    }()

    fmt.Println("main:   total", <-totals)
}
GO

say() { printf '$ %s\n' "$*"; }

cd "$dir" || exit 1
say 'go build prices.go'
go build prices.go || exit 1

say './prices'
./prices 2>stderr.txt
status=$?
grep '^panic: ' stderr.txt
echo "exit status $status"

Recover where the panic is

Verified output of panic_recovered_in_the_worker_go.go — regenerated by tools/run_examples.py, never hand-typed.

main: received err: worker panicked: runtime error: index out of range [3] with length 3
main: still running; returning normally

Reading the output

The same off-by-one, and this time the program lives. The worker defers its own recover, at the top of its function and so before anything in it can panic. When sumPrices panics, the unwinding reaches that deferred function, recover returns the panic's value, and the worker turns it into an error and sends it on results — the same road a result takes in A goroutine has no handle. main receives an error, not a crash, and returns normally.

The error still says panicked. An index out of range is a bug, and recovering from it keeps the other goroutines alive; it does not make the total correct.

Two places in the standard library make the opposite choices for you. net/http recovers a panic in a request handler, logs a stack trace, and closes that connection, on the assumption that the damage stayed within one request (http.Handler ↗). And WaitGroup.Go ↗ says the function passed to it must not panic, so a function that might has to recover inside itself.

panic_recovered_in_the_worker_go.go

panic_recovered_in_the_worker_go.go in full — pasted here by tools/run_examples.py from the file CI runs.

// The recover that works is the one in the goroutine that panicked. Deferred
// at the top of the worker, it turns the panic into an error, and the error
// travels back on the channel like any other result.
//
//  go run panic_recovered_in_the_worker_go.go
package main

import "fmt"

type total struct {
    cents int
    err   error
}

func sumPrices(prices []int) (cents int) {
    for i := 0; i <= len(prices); i++ { // one step too far
        cents += prices[i]
    }
    return cents
}

func main() {
    results := make(chan total)
    go func() {
        defer func() {
            if r := recover(); r != nil {
                results <- total{err: fmt.Errorf("worker panicked: %v", r)}
            }
        }()
        results <- total{cents: sumPrices([]int{1999, 450, 1200})}
    }()

    r := <-results
    fmt.Println("main: received err:", r.err)
    fmt.Println("main: still running; returning normally")
}

What to do

  • Treat a panic in any goroutine as a crash of the program, because it is one. main being fine does not help.
  • When a goroutine must survive a panic — a request handler, a job runner, one worker of many — defer the recover at the top of that goroutine's function, and send the value back as an error, like any other result. With several workers, that is one recover per worker: A worker pool.
  • Do not put a recover in main for the goroutines' sake. It can only stop a panic in main's own goroutine.
  • Recovering is not fixing. Report the recovered value loudly; if it is a bug, the program's answer is still wrong.
  • If one failure should stop the others, send the error and cancel the rest: First error cancels the rest.

In other languages

  • The Rust library's Spawning a thread ↗: a panicking Rust thread does not take the process with it. join returns Err holding the panic's payload, and main carries on. Go has no join to hand a panic to, so an unrecovered one has nowhere to go but the whole process.
  • The Concurrency library's Getting a result back ↗: C++'s get() and Python's result() rethrow the exception on the waiting thread, and Java's get() throws an ExecutionException with it as the cause. In Go, the recover-and-send above is the version you write yourself.
  • The Concurrency library's 01 Threads chapter plans a lesson on this very question, A failure nobody is waiting for.

Sources