Memory: what go vet and gofmt catch¶

Level: 301 · anyone who wants the tools to find a copied lock or a stale pointer before a run does

One line: Two go vet analyzers watch this chapter's mistakes — copylocks for a struct with a sync.Mutex passed by value, unsafeptr for a uintptr turned back into a pointer — and -gcflags=-m is the lens for everything about where a value lives; the leaks that no tool sees are the ones a slice, a goroutine or a pool keeps alive on purpose.

copylocks: a struct with a Mutex is copied¶

An Account holds a sync.Mutex. The bad package copies it three ways — a value receiver, a value parameter, an assignment — and the good one uses pointers:

Verified output of memory_vet_copylocks_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ cd bad && go mod init example && go vet .
# example
# [example]
./account.go:14:9: Balance passes lock by value: example.Account contains sync.Mutex
./account.go:20:17: snapshot passes lock by value: example.Account contains sync.Mutex
./account.go:20:43: return copies lock value: example.Account contains sync.Mutex
./account.go:24:12: assignment copies lock value to backup: example.Account contains sync.Mutex
./account.go:25:42: call of snapshot copies lock value: example.Account contains sync.Mutex
exit status 1
$ cd good && go mod init example && go vet .
exit status 0

The warning means that the copy has its own mutex, in whatever state the original's was at the moment of copying, and locks nothing that the original locks. Balance on a value receiver locks a throwaway copy on every call, so two goroutines calling it are not serialized at all; A mutex guards a counter is what the lock was meant to do.

The good version gives every method a pointer receiver, passes *Account, and copies the number rather than the struct. It is silent under vet: exit status 0.

When the tool is right, and when it is noise. Right almost always: a copied lock is a bug that compiles, runs and only shows under load. The one legitimate copy is at construction, before anyone could hold the lock, and vet allows a composite literal for that reason. The check also covers sync.WaitGroup, sync.Once, sync.Pool and atomic types, and a struct that embeds any of them.

unsafeptr: a uintptr turned back into a pointer¶

The bad file stores uintptr(unsafe.Pointer(&o)) + offset in a variable and converts the variable back; the good file does the arithmetic inside the conversion, and once more with unsafe.Add:

Verified output of memory_vet_unsafeptr_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ cd bad && go mod init example && go vet .
# example
# [example]
./second_field.go:16:18: possible misuse of unsafe.Pointer
exit status 1
$ cd good && go mod init example && go vet .
exit status 0

The warning means that between the statement that made the integer and the one that uses it, the object may have been moved — a stack copy — or freed, because a uintptr neither keeps it alive nor gets updated. unsafe.Pointer ↗ lists the valid patterns; the third is exactly the one-expression form the good file uses.

The good version converts unsafe.Pointer(uintptr(p) + off) in one expression, or calls unsafe.Add(p, off), which is the same thing with the rule built in. Vet prints nothing.

When the tool is right, and when it is noise. The analyzer looks for the syntactic pattern unsafe.Pointer(x) where x is a uintptr that is not itself an expression of the allowed shapes, so a value that was a uintptr all along — a syscall result, say — is flagged too, and there the fix is a comment and a //nolint-style suppression only outside the standard tools. When the flagged code is yours and the integer came from a pointer, the tool is right.

-gcflags=-m: not a linter, but the lens¶

The flag prints one line per decision the compiler made about where a value lives and what it inlined. This file was written to produce every kind of line:

Verified output of memory_vet_gcflags_m_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go build -gcflags=-m registry.go 2>&1 | grep -v "^#"
./registry.go:11:6: can inline register
./registry.go:16:6: can inline idOf
./registry.go:21:6: can inline notesOf
./registry.go:26:6: can inline double
./registry.go:30:6: can inline main
./registry.go:32:10: inlining call to register
./registry.go:33:9: inlining call to idOf
./registry.go:33:24: inlining call to notesOf
./registry.go:33:35: inlining call to double
./registry.go:11:15: leaking param: o
./registry.go:12:19: append escapes to heap
./registry.go:16:11: o does not escape
./registry.go:21:14: leaking param: o to result ~r0 level=1
./registry.go:31:7: &Order{...} escapes to heap
./registry.go:31:36: []string{...} escapes to heap
./registry.go:32:10: append escapes to heap
./registry.go:34:9: "nothing registered" escapes to heap
exit status 0

What each wording means:

Line	Meaning
`can inline register`	The function is small enough to be inlined into callers.
`inlining call to register`	This call site was inlined; escape decisions at the site are made in the caller's frame.
`leaking param: o`	The pointer parameter is stored somewhere that outlives the call: the caller's object must be on the heap.
`leaking param: o to result ~r0 level=1`	The parameter leaks only through the result — one dereference deep here — so the caller's object escapes only if the result does.
`o does not escape`	Nothing the function does with `o` keeps it: the caller may keep the object in its frame.
`&Order{...} escapes to heap`	A value allocated at this site goes on the heap, and the next `-m=2` level says why (`flow: {heap} ← ...`).
`moved to heap: n`	A named local is allocated on the heap because its address outlives the frame.
`append escapes to heap`	The slice grown by this `append` gets a heap backing array.
`"nothing registered" escapes to heap`	A constant passed to `panic`, an interface parameter: the value is boxed.

-m -m or -m=2 prints the reasoning behind each verdict, and -gcflags=all=-m covers dependencies too. Escape analysis decides stack or heap reads a full transcript; The compiler says what it inlines reads the inlining half.

The three drivers

memory_vet_copylocks_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# go vet's copylocks analyzer: a struct that holds a sync.Mutex must not be
# copied, because the copy has its own lock state and guards nothing. The
# bad file copies an Account three ways; the good file uses pointers.
set -u
export GOTOOLCHAIN=local

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
mkdir "$dir/bad" "$dir/good"

cat >"$dir/bad/account.go" <<'GO'
package main

import (
    "fmt"
    "sync"
)

type Account struct {
    mu    sync.Mutex
    cents int64
}

// Balance has a value receiver: every call locks a copy of the mutex.
func (a Account) Balance() int64 {
    a.mu.Lock()
    defer a.mu.Unlock()
    return a.cents
}

func snapshot(a Account) Account { return a }

func main() {
    var savings Account
    backup := savings
    fmt.Println(savings.Balance(), snapshot(savings), backup.cents)
}
GO

cat >"$dir/good/account.go" <<'GO'
package main

import (
    "fmt"
    "sync"
)

type Account struct {
    mu    sync.Mutex
    cents int64
}

// Balance has a pointer receiver: the one mutex is locked.
func (a *Account) Balance() int64 {
    a.mu.Lock()
    defer a.mu.Unlock()
    return a.cents
}

// snapshot copies the number, not the struct that holds the lock.
func snapshot(a *Account) int64 { return a.Balance() }

func main() {
    savings := &Account{}
    fmt.Println(savings.Balance(), snapshot(savings))
}
GO

say() { printf '$ %s\n' "$*"; }

for pkg in bad good; do
    say "cd $pkg && go mod init example && go vet ."
    (cd "$dir/$pkg" && go mod init example >/dev/null 2>&1 && go vet . 2>&1)
    echo "exit status $?"
done

memory_vet_unsafeptr_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# go vet's unsafeptr analyzer: a uintptr that was a pointer is an integer
# the collector does not know about, so converting it back is flagged. The
# good file does the arithmetic in one expression, which is a pattern the
# unsafe.Pointer rules allow, and with unsafe.Add.
set -u
export GOTOOLCHAIN=local

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
mkdir "$dir/bad" "$dir/good"

cat >"$dir/bad/second_field.go" <<'GO'
package main

import (
    "fmt"
    "unsafe"
)

type Order struct {
    ID  int64
    Qty int64
}

func main() {
    o := Order{ID: 1, Qty: 2}
    addr := uintptr(unsafe.Pointer(&o)) + unsafe.Offsetof(o.Qty)
    qty := (*int64)(unsafe.Pointer(addr))
    fmt.Println(*qty)
}
GO

cat >"$dir/good/second_field.go" <<'GO'
package main

import (
    "fmt"
    "unsafe"
)

type Order struct {
    ID  int64
    Qty int64
}

func main() {
    o := Order{ID: 1, Qty: 2}
    qty := (*int64)(unsafe.Pointer(uintptr(unsafe.Pointer(&o)) + unsafe.Offsetof(o.Qty)))
    viaAdd := (*int64)(unsafe.Add(unsafe.Pointer(&o), unsafe.Offsetof(o.Qty)))
    fmt.Println(*qty, *viaAdd)
}
GO

say() { printf '$ %s\n' "$*"; }

for pkg in bad good; do
    say "cd $pkg && go mod init example && go vet ."
    (cd "$dir/$pkg" && go mod init example >/dev/null 2>&1 && go vet . 2>&1)
    echo "exit status $?"
done

memory_vet_gcflags_m_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# -gcflags=-m is not a linter, but it is the tool that says where each value
# went and which calls were inlined. This file is built to show every kind
# of line the flag prints, so the page can say what each wording means.
set -u
export GOTOOLCHAIN=local

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT

cat >"$dir/registry.go" <<'GO'
package main

type Order struct {
    ID    int
    Notes []string
}

var registry []*Order

// register keeps o: the parameter leaks to the heap.
func register(o *Order) {
    registry = append(registry, o)
}

// idOf reads through o and keeps nothing: o does not escape.
func idOf(o *Order) int {
    return o.ID
}

// notesOf returns what o points at, not o itself: the content leaks.
func notesOf(o *Order) []string {
    return o.Notes
}

// double is small enough to inline.
func double(qty int) int {
    return qty * 2
}

func main() {
    o := &Order{ID: 1, Notes: []string{"rush"}}
    register(o)
    if idOf(o)+len(notesOf(o))+double(2) == 0 {
        panic("nothing registered")
    }
}
GO

say() { printf '$ %s\n' "$*"; }

say 'go build -gcflags=-m registry.go 2>&1 | grep -v "^#"'
(cd "$dir" && go build -gcflags=-m registry.go 2>&1) | grep -v '^#'
echo "exit status ${PIPESTATUS[0]}"

What no tool catches¶

A small slice of a big read pins the whole backing array. magic := whole[:2] keeps the 64 MiB that whole was, because the slice header points into it. Vet, the compiler and staticcheck all see correct code. The program below reads 64 MiB, keeps two bytes of it, and measures the heap after a collection with the slice and then with a copy:

Verified output of memory_vet_pinned_backing_array_go.go — regenerated by tools/run_examples.py, never hand-typed.

heap before the read is under 8 MiB: true
kept whole[:2]: len 2  cap 67108864
  heap after collection is over 32 MiB: true
kept a copy:    len 2  cap 2
  heap after collection is under 8 MiB: true

The fix is copy into a slice of the size that is needed — slices.Clip and s[:2:2] change the capacity but still point at the same array, so they free nothing.

A goroutine that never ends keeps everything it can reach. Its stack, the buffer it was filling, the connection it was reading: A leaked goroutine never ends shows the goroutine, and the memory follows it. No tool counts goroutines for you; runtime.NumGoroutine in a test does.
A sync.Pool that hands out buffers of every size grows to the largest. A pool caches whatever is Put back, so a request that once needed 100 MB leaves a 100 MB buffer for the next 1 KB request to draw. sync.Pool is on The sync package's Planned list; until then, the sync.Pool documentation ↗ and the fmt package's own pool are the references.
A map that is only ever added to is a cache without eviction, and the collector cannot tell a cache from a leak: everything the map reaches is live. The GC guide ↗ points at the heap profile as the tool that names the map.

staticcheck ↗ and golangci-lint ↗ add checks beyond go vet — SA6002 on passing non-pointer values to a sync.Pool, SA4006 on unused values — and are not run here.

memory_vet_pinned_backing_array_go.go

memory_vet_pinned_backing_array_go.go in full — pasted here by tools/run_examples.py from the file CI runs.

// A two-byte slice of a 64 MiB read keeps the whole 64 MiB alive: the slice
// header points into the big backing array, and the collector keeps what
// is pointed at. No tool flags this; a copy the size of what is needed is
// the fix. Measured with HeapAlloc after a forced collection.
//
//  go build memory_vet_pinned_backing_array_go.go && ./memory_vet_pinned_backing_array_go
package main

import (
    "fmt"
    "runtime"
)

func heapAllocMiB() uint64 {
    runtime.GC()
    var m runtime.MemStats
    runtime.ReadMemStats(&m)
    return m.HeapAlloc >> 20
}

var magic []byte

func main() {
    fmt.Println("heap before the read is under 8 MiB:", heapAllocMiB() < 8)

    whole := make([]byte, 64<<20) // the whole file, read in one go
    whole[0], whole[1] = 'P', 'K'
    magic = whole[:2] // the two bytes the program actually keeps
    whole = nil
    fmt.Println("kept whole[:2]: len", len(magic), " cap", cap(magic))
    fmt.Println("  heap after collection is over 32 MiB:", heapAllocMiB() > 32)

    copied := make([]byte, len(magic))
    copy(copied, magic)
    magic = copied
    fmt.Println("kept a copy:    len", len(magic), " cap", cap(magic))
    fmt.Println("  heap after collection is under 8 MiB:", heapAllocMiB() < 8)
}

Sources¶

go vet ↗ and the analyzers copylocks ↗ and unsafeptr ↗.
unsafe.Pointer ↗ — the valid patterns the analyzer checks for.
cmd/compile ↗ — the -m flag under "Compiler Directives" and the flag list.
A Guide to the Go Garbage Collector — "Identifying costs" ↗.