A leaked goroutine never ends¶
Level: 201 · anyone who has put a timeout around a goroutine's answer
One line: A goroutine blocked on a send that nobody will receive, because the caller timed out and left, stays blocked until the program ends; give its channel room for the answer, or make the send a select with <-ctx.Done(), and it finishes.
The shape is everywhere: start a goroutine to fetch something, then wait for its answer or for the context, whichever comes first. The caller is fine either way. The goroutine may not be. A send on an unbuffered channel waits until a receiver is ready (An unbuffered send waits for a receiver), and once the caller has returned, no receiver is coming.
The program makes the answer late on purpose. Each supplier goroutine waits for a channel that main closes only after every caller has given up, so the answer is late on every run and on every machine, not just on a slow one. Three callers try each of three versions of the fetch, and after the suppliers have answered, the program counts the goroutines that are still there.
Verified output of a_leaked_goroutine_never_ends_go.go — regenerated by tools/run_examples.py, never hand-typed.
unbuffered channel:
callers: 3 × context deadline exceeded
goroutines left behind: 3
buffered channel of 1:
callers: 3 × context deadline exceeded
goroutines left behind: 0
send in a select with ctx.Done():
callers: 3 × context deadline exceeded
goroutines left behind: 0
main returns with 4 goroutines running:
3 × [chan send] in main.fetchUnbuffered.func1
Reading the output¶
Unbuffered channel. All three callers got context deadline exceeded and returned. Three goroutines were left behind: each supplier got its answer and then blocked, sending a price nobody would take. runtime.NumGoroutine ↗ returns the number of goroutines that currently exist. The program reads it once the count is back down to what it was before the callers started, or after two seconds, whichever comes first, because a goroutine that is on its way out can still be counted for a moment. For this version the count never came back down, and after two seconds it was still three higher.
Buffered channel of 1. The same callers and the same timeouts. make(chan int, 1) gives the send somewhere to put the price without a receiver (A buffered channel is a bounded queue), so each supplier sends, returns, and leaves behind a channel that nothing refers to, which the garbage collector takes like any other value.
Send in a select with ctx.Done(). The channel is unbuffered again, but the supplier sends inside a select with a second case, <-ctx.Done(). The caller's context was done by the time it left, so the send case can never proceed and the Done case can, and the goroutine returns.
main returns with 4 goroutines running: main and the three from the first version. runtime.Stack ↗, asked for all goroutines, writes each one's stack after the caller's own, in the form a panic prints; the program reads the wait state and the function from each and counts them. All three are in [chan send], in the function literal inside fetchUnbuffered. None of this is reported as an error. The runtime's deadlock report needs every goroutine blocked, and main never was (All goroutines are asleep); and when main returned, the program ended without waiting for them (main does not wait). In a server, main does not return, and each leaked goroutine keeps its stack and everything it refers to for as long as the server runs. Each is small (Goroutines are cheap), which is why a few per request can go unnoticed.
a_leaked_goroutine_never_ends_go.go
a_leaked_goroutine_never_ends_go.go in full — pasted here by tools/run_examples.py from the file CI runs.
// A goroutine blocked on a send that nobody will ever receive never ends. Its
// caller timed out and left; the goroutine it started is still there.
//
// go build a_leaked_goroutine_never_ends_go.go && ./a_leaked_goroutine_never_ends_go
package main
import (
"context"
"fmt"
"runtime"
"sort"
"strings"
"time"
)
// Each fetch asks a supplier for a price on a goroutine of its own and waits
// for the answer or for ctx, whichever comes first. The supplier answers only
// when late is closed, and main closes it after every caller has given up: so
// the answer is late on every run, not just on a slow machine.
// fetchUnbuffered leaks: once the caller has gone, the send waits forever.
func fetchUnbuffered(ctx context.Context, late <-chan struct{}) (int, error) {
prices := make(chan int)
go func() {
<-late
prices <- 1299
}()
select {
case p := <-prices:
return p, nil
case <-ctx.Done():
return 0, ctx.Err()
}
}
// fetchBuffered does not: the send has room for one value and never waits.
func fetchBuffered(ctx context.Context, late <-chan struct{}) (int, error) {
prices := make(chan int, 1)
go func() {
<-late
prices <- 1299
}()
select {
case p := <-prices:
return p, nil
case <-ctx.Done():
return 0, ctx.Err()
}
}
// fetchSelect does not either: the goroutine gives up when the caller has.
func fetchSelect(ctx context.Context, late <-chan struct{}) (int, error) {
prices := make(chan int)
go func() {
<-late
select {
case prices <- 1299:
case <-ctx.Done():
}
}()
select {
case p := <-prices:
return p, nil
case <-ctx.Done():
return 0, ctx.Err()
}
}
func main() {
try("unbuffered channel", fetchUnbuffered)
try("buffered channel of 1", fetchBuffered)
try("send in a select with ctx.Done()", fetchSelect)
fmt.Println("main returns with", runtime.NumGoroutine(), "goroutines running:")
for _, line := range blockedGoroutines() {
fmt.Println(" " + line)
}
}
// try runs three callers with a 10ms timeout each, lets the suppliers answer,
// and counts the goroutines that are still there afterwards.
func try(name string, fetch func(context.Context, <-chan struct{}) (int, error)) {
before := runtime.NumGoroutine()
late := make(chan struct{})
errs := map[string]int{}
for range 3 {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Millisecond)
_, err := fetch(ctx, late)
cancel()
errs[fmt.Sprint(err)]++
}
close(late) // the suppliers answer now, and nobody is listening
fmt.Println(name + ":")
for err, n := range errs {
fmt.Printf(" callers: %d × %s\n", n, err)
}
fmt.Println(" goroutines left behind:", settle(before)-before)
}
// settle waits until the goroutine count is back down to want, and gives up
// after two seconds. A goroutine that has finished its work can still be
// counted for a moment while it exits, so a single reading could be one high.
func settle(want int) int {
deadline := time.Now().Add(2 * time.Second)
for runtime.NumGoroutine() > want && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
return runtime.NumGoroutine()
}
// blockedGoroutines reads every goroutine's stack, as a panic would print it,
// and counts the others by their wait state and the function they are in.
func blockedGoroutines() []string {
buf := make([]byte, 1<<20)
traces := strings.Split(string(buf[:runtime.Stack(buf, true)]), "\n\n")
counts := map[string]int{}
for _, trace := range traces[1:] { // traces[0] is main itself
lines := strings.Split(trace, "\n")
state := lines[0][strings.Index(lines[0], "[") : strings.Index(lines[0], "]")+1]
function := lines[1][:strings.LastIndex(lines[1], "(")]
counts[state+" in "+function]++
}
var out []string
for where, n := range counts {
out = append(out, fmt.Sprintf("%d × %s", n, where))
}
sort.Strings(out)
return out
}
Without the gate¶
In the program above, every answer is late. In real code an answer is late only sometimes, so how many goroutines leak depends on timing. demo/leak_race.go starts 1000 callers at once, each with a 5ms timeout, against a supplier that takes 5ms and sends on an unbuffered channel, and prints how many goroutines are still there a second later. Each number below is one run, sorted smallest first:
740 745 747 749 757 757 760 762 763 765
768 771 771 778 794 797 798 804 819 911
No run leaked none of its goroutines, none leaked all of them, and hardly two runs agreed. To count on your own machine, run bash demo/tally.sh from this folder, or bash demo/tally.sh 100 for a hundred runs. Your numbers will not match these.
What to do¶
- Give a goroutine whose answer may be abandoned a channel with room for everything it will send:
make(chan T, 1)for a single result. - Or make each send a
selectwith a<-ctx.Done()case, when the goroutine sends more than once and a buffer cannot hold it all. - Look for the shape in review: a
selectthat can return onctx.Done()or a timer while a goroutine it started still has to send to it. - Count goroutines after a test has settled, the way
settledoes here. Go 1.26 added an experimentalgoroutineleakprofile, enabled at build time withGOEXPERIMENT=goroutineleakprofile, which uses the garbage collector to find goroutines blocked on something that no runnable goroutine can reach (Go 1.26 release notes ↗). (This library runs Go 1.25, so the profile is not machine-checked here.)
In other languages¶
- The Concurrency library's Who waits when main returns? ↗ shows that Go does not wait for goroutines when
mainreturns, which is why the three here vanish without a word at exit. The same page shows Python waiting for every ordinary thread, so an ordinary Python thread blocked forever keeps the program from exiting at all: the same leak, showing up as a hang at the end. - The Rust library's Spawning a thread ↗ starts the threads; Rust's channels also know when the receiving end has gone.
Sender::send↗ never blocks and fails once the receiver has been dropped, and the boundedSyncSender::send↗ may return an error once the receiver has disconnected. A Go channel has no such notion: a send waits until the channel can take it ↗, whether or not anything is left to receive. - The Concurrency library has no page on leaks yet; cancellation, the other half of this problem, is in its planned chapter 06, Async.
Sources¶
- The Go Programming Language Specification: Send statements ↗ — a send blocks until it can proceed; on an unbuffered channel, until a receiver is ready.
runtime.NumGoroutine↗ andruntime.Stack↗.context.WithTimeout↗, and theDonemethod ofContext↗ with itsselectexample.- Go Concurrency Patterns: Pipelines and cancellation ↗ — Sameer Ajmani, the Go blog, 2014: a goroutine hung on a send is a resource leak, because goroutines are not garbage collected and have to exit on their own.
- Go 1.26 release notes: goroutine leak profile ↗.
- Katherine Cox-Buday, Concurrency in Go (O'Reilly, 2017), chapter 4, "Concurrency Patterns in Go", section "Preventing Goroutine Leaks".
- Jon Bodner, Learning Go, 2nd ed. (O'Reilly, 2024), chapter 12, "Concurrency in Go", section "Always Clean Up Your Goroutines".