Skip to content

Deferred calls run last in, first out

Level: 201 · anyone who defers more than one cleanup, or wonders which ways out of a function run them

One line: Deferred calls run when the surrounding function returns, in the reverse of the order they were deferred — and they run while a panic unwinds the function and when runtime.Goexit ends the goroutine, but never after os.Exit.

postInvoice takes three things in order — a lock, a file, a transaction — and defers the release of each right after taking it. Then a function that panics with two defers pending, recovered by its caller. Last, a worker goroutine that calls runtime.Goexit with a deferred call pending, while main waits on a WaitGroup.

Verified output of deferred_calls_run_last_in_first_out_go.go — regenerated by tools/run_examples.py, never hand-typed.

postInvoice: lock the ledger
postInvoice: open the invoice file
postInvoice: begin the transaction
postInvoice: returning
postInvoice: end the transaction
postInvoice: close the invoice file
postInvoice: unlock the ledger

panicking: close the invoice file
panicking: unlock the ledger
caller: recovered: invoice 18 has no customer

worker: deferred call ran after Goexit
main: the worker ended; main goes on

Reading the output

Released in reverse: transaction, file, lock. The spec ↗ says deferred functions are executed immediately before the surrounding function returns, in the reverse order they were deferred. Reverse is the right order for things that depend on each other: the transaction that needed the file ends before the file closes, and the file closes before the lock that guarded it is released.

A panic runs them too. postInvoiceThatPanics never reaches a return, and both deferred lines printed anyway, in reverse, before the caller's recover saw the value. The spec's Handling panics ↗: a panic terminates the function and runs its deferred functions, then does the same for the caller, on up the goroutine's stack. That is what makes defer mu.Unlock() safe — A mutex guards a counter — where an Unlock written at the end of the function would be skipped by a panic in between.

runtime.Goexit ran the worker's deferred call and ended only the worker. Goexit ↗ terminates the goroutine that calls it after running all deferred calls; no other goroutine is affected, and main went on once wg.Wait returned. testing's t.FailNow and t.Fatal end a test this way, which is why they must be called from the test's own goroutine.

The ways out that are not a return

Verified output of deferred_calls_on_panic_exit_and_goexit_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go build panics.go
$ ./panics
main: posting invoice 18
deferred: unlock the ledger
panic: invoice 18 has no customer
exit status 2
$ go build exits.go
$ ./exits
main: posting invoice 18, then os.Exit(3)
exit status 3
$ go build goexit.go
$ ./goexit
main: calling runtime.Goexit
deferred: unlock the ledger
fatal error: no goroutines (main called runtime.Goexit) - deadlock!
exit status 2

A panic that nothing recovers: the deferred line, then the panic line, then exit status 2. The deferred calls run first, all the way up the stack; then the runtime prints the panic and exits with status 2 (runtime ↗, under GOTRACEBACK). A panic ends the whole program shows the same from a goroutine that is not main.

os.Exit(3): no deferred line, exit status 3. os.Exit ↗ ends the process at once; its documentation says deferred functions are not run. Whatever status it is given is the program's. log.Fatal is Print followed by os.Exit(1) (log.Fatal ↗), so it skips them too.

runtime.Goexit from main: the deferred line, then a fatal error, exit status 2. Goexit ran main's deferred call and ended main's goroutine — but main had not returned, so the program did not end with it. With no goroutine left, the runtime stops with no goroutines (main called runtime.Goexit) - deadlock!, the case the Goexit documentation ↗ describes: since func main has not returned, the program continues execution of other goroutines, and if all other goroutines exit, the program crashes.

What a defer costs

Since Go 1.14, most deferred calls cost about as much as calling the function directly: the release notes ↗ say the release improves the performance of most uses of defer to incur almost zero overhead compared to calling the deferred function directly. The mechanism is open-coded defers — the compiler places the deferred call inline at every return and keeps a small bitmap for the panic path — described in the design document ↗, which also sets the limits: at most eight defers in the function, none of them in a loop, and the product of return statements and defers at most fifteen. Outside those limits the compiler falls back to a record of the deferred call on the stack, or, when it cannot know how many will run, on the heap.

The compiler will say which it chose. -gcflags=-d=defer prints one line per defer statement:

Verified output of deferred_calls_open_coded_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go build -gcflags=-d=defer kinds.go
# command-line-arguments
./kinds.go:14:2: open-coded defer
./kinds.go:15:2: open-coded defer
./kinds.go:21:2: stack-allocated defer
./kinds.go:22:2: stack-allocated defer
./kinds.go:23:2: stack-allocated defer
./kinds.go:24:2: stack-allocated defer
./kinds.go:25:2: stack-allocated defer
./kinds.go:26:2: stack-allocated defer
./kinds.go:27:2: stack-allocated defer
./kinds.go:28:2: stack-allocated defer
./kinds.go:29:2: stack-allocated defer
./kinds.go:39:3: heap-allocated defer
exit status 0

twoDefers (lines 14 and 15) is open-coded; the nine in nineDefers are one over the limit and go on the stack; the one in the loop of deferInLoop (line 39) goes on the heap — the case A defer in a loop runs at function end is about. How many nanoseconds each kind costs is a benchmark question this page does not settle; the design document's own measurements are the place to look. (Not machine-checked here.)

deferred_calls_run_last_in_first_out_go.go, deferred_calls_on_panic_exit_and_goexit_sh.sh and deferred_calls_open_coded_sh.sh

deferred_calls_run_last_in_first_out_go.go in full — pasted here by tools/run_examples.py from the file CI runs.

// Deferred calls run when the surrounding function returns, in the reverse of
// the order they were deferred: the last resource taken is the first released.
// They also run while a panic unwinds the function, and when runtime.Goexit
// ends the goroutine. os.Exit runs none of them; the driver on the page shows
// that, because os.Exit ends the program.
//
//  go build deferred_calls_run_last_in_first_out_go.go && ./deferred_calls_run_last_in_first_out_go
package main

import (
    "fmt"
    "runtime"
    "sync"
)

func postInvoice() {
    fmt.Println("postInvoice: lock the ledger")
    defer fmt.Println("postInvoice: unlock the ledger")
    fmt.Println("postInvoice: open the invoice file")
    defer fmt.Println("postInvoice: close the invoice file")
    fmt.Println("postInvoice: begin the transaction")
    defer fmt.Println("postInvoice: end the transaction")
    fmt.Println("postInvoice: returning")
}

func postInvoiceThatPanics() {
    defer fmt.Println("panicking: unlock the ledger")
    defer fmt.Println("panicking: close the invoice file")
    panic("invoice 18 has no customer")
}

func main() {
    postInvoice()

    fmt.Println()
    func() {
        defer func() { fmt.Println("caller: recovered:", recover()) }()
        postInvoiceThatPanics()
    }()

    fmt.Println()
    var wg sync.WaitGroup
    wg.Go(func() {
        defer fmt.Println("worker: deferred call ran after Goexit")
        runtime.Goexit()
    })
    wg.Wait()
    fmt.Println("main: the worker ended; main goes on")
}

deferred_calls_on_panic_exit_and_goexit_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# Three ways out of main that are not a return, and what each does with the
# deferred calls: a panic runs them and exits 2; os.Exit runs none and exits
# with the status it was given; runtime.Goexit from main runs them, and then
# the runtime stops the program because main's goroutine is gone. The
# goroutine traces on stderr carry addresses that change between runs, so
# only the first line of each program's stderr is kept.
set -u
export GOTOOLCHAIN=local

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT

cat >"$dir/panics.go" <<'GO'
package main

import "fmt"

func main() {
    defer fmt.Println("deferred: unlock the ledger")
    fmt.Println("main: posting invoice 18")
    panic("invoice 18 has no customer")
}
GO

cat >"$dir/exits.go" <<'GO'
package main

import (
    "fmt"
    "os"
)

func main() {
    defer fmt.Println("deferred: unlock the ledger")
    fmt.Println("main: posting invoice 18, then os.Exit(3)")
    os.Exit(3)
}
GO

cat >"$dir/goexit.go" <<'GO'
package main

import (
    "fmt"
    "runtime"
)

func main() {
    defer fmt.Println("deferred: unlock the ledger")
    fmt.Println("main: calling runtime.Goexit")
    runtime.Goexit()
}
GO

say() { printf '$ %s\n' "$*"; }
cd "$dir" || exit 1

for program in panics exits goexit; do
    say "go build $program.go"
    go build "$program.go" || exit 1
    say "./$program"
    "./$program" 2>stderr.txt
    status=$?
    head -n 1 stderr.txt
    echo "exit status $status"
done

deferred_calls_open_coded_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# What the compiler does with each defer statement, in its own words. The
# debug flag -d=defer prints one line per defer. Two defers in a plain
# function are open-coded -- the call is placed inline at every return, the
# fast path since Go 1.14. Nine of them are over the limit of eight and fall
# back to a record on the stack. A defer inside a loop needs a record on the
# heap, because how many will run is not known until run time.
set -u
export GOTOOLCHAIN=local

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT

cat >"$dir/kinds.go" <<'GO'
package main

import (
    "fmt"
    "os"
)

// twoDefers: at most eight defers and none in a loop -> open-coded.
func twoDefers() {
    f, err := os.CreateTemp("", "orders-*.csv")
    if err != nil {
        return
    }
    defer os.Remove(f.Name())
    defer f.Close()
    fmt.Fprintln(f, "order-1")
}

// nineDefers: one over the open-coding limit -> a record on the stack.
func nineDefers() {
    defer fmt.Println("step 1")
    defer fmt.Println("step 2")
    defer fmt.Println("step 3")
    defer fmt.Println("step 4")
    defer fmt.Println("step 5")
    defer fmt.Println("step 6")
    defer fmt.Println("step 7")
    defer fmt.Println("step 8")
    defer fmt.Println("step 9")
}

// deferInLoop: how many defers run is decided at run time -> records on the heap.
func deferInLoop(paths []string) {
    for _, path := range paths {
        f, err := os.Open(path)
        if err != nil {
            return
        }
        defer f.Close()
    }
}

func main() {
    twoDefers()
    nineDefers()
    deferInLoop(nil)
}
GO

say() { printf '$ %s\n' "$*"; }
cd "$dir" || exit 1

say 'go build -gcflags=-d=defer kinds.go'
go build -gcflags=-d=defer kinds.go 2>&1
echo "exit status $?"

What to do

  • Defer the release right after the acquire, on the next line, so every way out of the function — return, panic, Goexit — releases it, and the releases run in the reverse of the acquires.
  • Do not count on deferred calls after os.Exit or log.Fatal. Return an error to main and let main choose the exit status after everything else has run — the Rust library's main returns Result ↗ makes the same argument.
  • Keep Goexit out of main, and out of ordinary code: testing uses it, and a library that calls it ends its caller's goroutine without a word.
  • Stop worrying about the cost of a defer that is not in a loop: since Go 1.14 it is open-coded. A defer in a loop is the one that is both slow and usually wrong.

Practice

Write runJob(steps []string, failAt string) that, for each step, prints do <step> and defers undo <step>, and panics at the step named failAt. Run it once with no failure and once failing at the last step, recovering in the caller, and show that the undos run in reverse both times — this is the one loop where a defer per iteration is what you want.

Solution

deferred_calls_run_last_in_first_out_kata_go.go in full — pasted here by tools/run_examples.py from the file CI runs.

// Kata: run the steps of a job and defer an undo for each one, so that the
// undos run in reverse — after a normal finish, and while a panic unwinds.
// This is the one loop in which a defer per iteration is what you want.
package main

import "fmt"

func runJob(steps []string, failAt string) {
    for _, step := range steps {
        fmt.Println("  do  ", step)
        defer fmt.Println("  undo", step)
        if step == failAt {
            panic(step + ": printer offline")
        }
    }
    fmt.Println("  all steps done")
}

func main() {
    steps := []string{"reserve stock", "charge card", "print label"}

    fmt.Println("job 1: every step succeeds")
    runJob(steps, "")

    fmt.Println("job 2: the last step fails")
    func() {
        defer func() { fmt.Println("  recovered:", recover()) }()
        runJob(steps, "print label")
    }()
}

Verified output of deferred_calls_run_last_in_first_out_kata_go.go — regenerated by tools/run_examples.py, never hand-typed.

job 1: every step succeeds
  do   reserve stock
  do   charge card
  do   print label
  all steps done
  undo print label
  undo charge card
  undo reserve stock
job 2: the last step fails
  do   reserve stock
  do   charge card
  do   print label
  undo print label
  undo charge card
  undo reserve stock
  recovered: print label: printer offline

In other languages

  • Rust drops locals at scope end in reverse declaration order — the same last in, first out — as the Rust library's Scope is about names, not values ↗ shows, with the trap that struct fields drop in declaration order. std::process::exit ↗ runs no destructors, as os.Exit runs no deferred calls.
  • Python: a finally: block runs on return and on an exception, but not after os._exit ↗, and not when the default handler kills the process on Ctrl-C — the Python library's Ctrl-C is a signal ↗ compares three endings of which only two run the finally.
  • C++ destroys automatic objects in the reverse order of their construction; std::exit ↗ does not unwind the stack, so those destructors do not run.

Sources