Skip to content

All goroutines are asleep

Level: 201 · anyone who has met fatal error: all goroutines are asleep - deadlock!, and anyone whose program hung without it

One line: When every goroutine is blocked, the Go runtime stops the program with fatal error: all goroutines are asleep - deadlock! and exit status 2 — but only then: one goroutine that is still sleeping, or waiting on a timer, keeps the detector quiet while the rest of the program stays stuck.

When the runtime sees it

The script runs two small programs. In lonely_send, main sends on an unbuffered channel that nothing will ever receive from. In sleeper, a packer waits for a shipping label, a shipper waits for a box, and main waits for the shipment — while a fourth goroutine, which is not part of the deadlock, sleeps for 2 s and returns. From stderr, the script keeps the fatal error line and, for each goroutine in the dump that follows it, the goroutine's state and the function it was in, with its number replaced by N.

Verified output of all_goroutines_are_asleep_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go build lonely_send.go
$ ./lonely_send
main: sending an order that nothing will receive
fatal error: all goroutines are asleep - deadlock!
goroutine N [chan send] in main.main
exit status 2
$ go build sleeper.go
$ ./sleeper
main: waiting for a shipment
sleeper: awake after 2 s, returning
fatal error: all goroutines are asleep - deadlock!
goroutine N [chan receive] in main.main
goroutine N [chan receive] in main.packer
goroutine N [chan receive] in main.shipper
exit status 2

lonely_send was stopped at once. Its only goroutine was parked in [chan send] — the state from An unbuffered send waits for a receiver — so every goroutine was asleep, and the runtime ended the program with exit status 2. The line after the send never printed.

sleeper was stopped only after the sleeper had returned. Three goroutines were deadlocked from the start, and yet the sleeper's line is in the output: the runtime let the program run on until the last goroutine that could still wake up had woken and finished. Only then came the fatal error, listing the three that were left, each parked in [chan receive].

The rule is in the runtime's source. The check is checkdead, in proc.go ↗. It returns without a word if any thread is still busy. Otherwise it confirms that every goroutine is waiting, and then, before reporting anything, looks at the timers each processor holds — and returns again if there is one. A goroutine in time.Sleep is waiting on a timer, so one sleeper is enough.

When it does not

Verified output of all_goroutines_are_asleep_go.go — regenerated by tools/run_examples.py, never hand-typed.

main: nothing shipped after 2 s, and the runtime reported no deadlock
packer:  parked in [chan receive]
shipper: parked in [chan receive]

The same packer and shipper, deadlocked in the same way. This time main waits in a select whose other case is time.After(2 * time.Second) ↗. That timer is pending the whole time, so the runtime reports nothing. When it fires, main gives up and reads the goroutine dump itself: the packer and the shipper are both still parked in [chan receive], and they are still there when main returns and the program exits with status 0.

A timeout at least ends the wait. Put a goroutine that sleeps in a loop in its place — a ticker, a periodic flush — and by the rule above the runtime would never report this deadlock, and the program would hang with no message at all.

all_goroutines_are_asleep_sh.sh and all_goroutines_are_asleep_go.go

all_goroutines_are_asleep_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# The runtime's deadlock detector, twice.
#
# lonely_send: main sends on an unbuffered channel that nothing will ever
# receive from. The runtime stops the program with a fatal error, exit status 2.
#
# sleeper: a packer and a shipper wait for each other forever, main waits for
# the shipment, and one more goroutine sleeps for 2 s and returns. The runtime
# reports nothing while the sleeper sleeps; its line is printed, and only then
# does the fatal error come.
#
# stderr also carries goroutine numbers, a temporary path and code offsets,
# which vary. This script keeps the fatal error line and, for each goroutine,
# its state and the function it is in, with its number replaced by N.
set -u
export GOTOOLCHAIN=local

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT

cat >"$dir/lonely_send.go" <<'GO'
package main

import "fmt"

func main() {
    orders := make(chan string)
    fmt.Println("main: sending an order that nothing will receive")
    orders <- "order-17"
    fmt.Println("main: never printed")
}
GO

cat >"$dir/sleeper.go" <<'GO'
package main

import (
    "fmt"
    "time"
)

func main() {
    boxes := make(chan string)
    labels := make(chan string)
    shipped := make(chan string)

    fmt.Println("main: waiting for a shipment")
    go packer(labels, boxes)
    go shipper(boxes, labels, shipped)
    go sleeper()
    fmt.Println("main: shipped", <-shipped)
}

// packer will not pack a box until it has a shipping label.
func packer(labels <-chan string, boxes chan<- string) {
    label := <-labels
    boxes <- "box with " + label
}

// shipper will not print a label until it has a box.
func shipper(boxes <-chan string, labels chan<- string, shipped chan<- string) {
    box := <-boxes
    labels <- "label-1"
    shipped <- box
}

// sleeper is not part of the deadlock: it sleeps for 2 s and returns.
func sleeper() {
    time.Sleep(2 * time.Second)
    fmt.Println("sleeper: awake after 2 s, returning")
}
GO

say() { printf '$ %s\n' "$*"; }

# run NAME: build NAME.go, run it, and print its stdout, the stable part of its
# stderr, and its exit status.
run() {
    say "go build $1.go"
    (cd "$dir" && go build "$1.go") || exit 1
    say "./$1"
    (cd "$dir" && "./$1" 2>stderr.txt)
    local status=$?
    awk '
        /^fatal error:/ { print }
        /^goroutine [0-9]+ \[/ {
            state = $0
            sub(/^goroutine [0-9]+ /, "goroutine N ", state)
            sub(/:$/, "", state)
            getline frame
            sub(/\(.*/, "", frame)
            print state " in " frame
        }
    ' "$dir/stderr.txt"
    echo "exit status $status"
}

run lonely_send
run sleeper

all_goroutines_are_asleep_go.go in full — pasted here by tools/run_examples.py from the file CI runs.

// The runtime reports a deadlock only when every goroutine is blocked. Here a
// packer and a shipper wait for each other forever, but main is waiting on a
// timer as well, so the runtime reports nothing. After 2 s main gives up and
// asks the runtime where the two are stuck.
//
//  go build all_goroutines_are_asleep_go.go && ./all_goroutines_are_asleep_go
package main

import (
    "fmt"
    "runtime"
    "strings"
    "time"
)

func main() {
    boxes := make(chan string)
    labels := make(chan string)
    shipped := make(chan string)

    go packer(labels, boxes)
    go shipper(boxes, labels, shipped)

    select {
    case parcel := <-shipped:
        fmt.Println("main: shipped", parcel)
    case <-time.After(2 * time.Second):
        fmt.Println("main: nothing shipped after 2 s, and the runtime reported no deadlock")
    }
    fmt.Printf("packer:  parked in [%s]\n", waitingOnChannel("main.packer"))
    fmt.Printf("shipper: parked in [%s]\n", waitingOnChannel("main.shipper"))
}

// packer will not pack a box until it has a shipping label.
func packer(labels <-chan string, boxes chan<- string) {
    label := <-labels
    boxes <- "box with " + label
}

// shipper will not print a label until it has a box.
func shipper(boxes <-chan string, labels chan<- string, shipped chan<- string) {
    box := <-boxes
    labels <- "label-1"
    shipped <- box
}

// waitingOnChannel returns the runtime's own word for why the goroutine running
// fn is parked on a channel: "chan send" or "chan receive". It reads the same
// goroutine dump that a crash prints, and polls until that goroutine is parked
// on a channel; the sleep between polls only keeps the loop from spinning.
func waitingOnChannel(fn string) string {
    buf := make([]byte, 1<<16)
    for {
        n := runtime.Stack(buf, true)
        for _, g := range strings.Split(string(buf[:n]), "\n\n") {
            header, frames, _ := strings.Cut(g, "\n")
            if !strings.Contains("\n"+frames, "\n"+fn+"(") {
                continue
            }
            _, state, _ := strings.Cut(header, "[") // "goroutine 7 [chan receive]:"
            state, _, _ = strings.Cut(state, "]")
            state, _, _ = strings.Cut(state, ",") // a long wait reads "chan receive, 2 minutes"
            if strings.HasPrefix(state, "chan ") {
                return state
            }
        }
        time.Sleep(time.Millisecond)
    }
}

What to do

  • Read the whole dump, not only its first line. Each goroutine's bracketed state says what it is waiting for, and the frames under it say where. Two goroutines each waiting to receive what the other would send is this page's deadlock.
  • For a program that hangs with no message, ask for the same dump. A SIGQUIT — Ctrl-\ in a terminal — makes a Go program exit with a stack dump (os/signal ↗).
  • Give every wait that might never end a way out: a timeout, as main has here — A timeout is a channel — or a cancellation, Cancel reaches every goroutine. A timeout gets the waiter out; it does not end the goroutines stuck in the deadlock, which leak.
  • In tests, use a detector that timers do not fool. Inside a testing/synctest ↗ bubble, when every goroutine in the bubble is durably blocked, time jumps ahead to the next moment that would unblock one; if there is none, synctest.Test panics with a deadlock. See synctest makes time virtual.

In other languages

  • The Rust library's Channels ↗ — Rust has no such detector: that page's for x in rx, waiting on a Sender nobody dropped, hangs with no message at all, and the page uses recv_timeout to show the hang without hanging, as main uses time.After here.
  • The Concurrency library's chapter 03, When locks go wrong, which will put deadlock and lock order to all of its languages, is planned.

Sources