Skip to content

Reflection and unsafe: what go vet and gofmt catch

Level: 301 · anyone running go vet on code that imports reflect, unsafe or encoding/json

One line: Three analyzers matter here — unsafeptr for a uintptr turned back into a pointer, structtag for a tag the reflect parser cannot read, and unmarshal for a decoder handed a value instead of a pointer — plus the compiler's -d=checkptr instrumentation that -race turns on; gofmt has nothing to say, and the mistakes that matter most are the ones no tool sees.

unsafeptr: possible misuse of unsafe.Pointer

A handle type stores an address as a uintptr and converts it back later; a second program keeps to the documented forms and is wrong anyway.

Verified output of reflection_vet_unsafeptr_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go mod init example
$ go vet ./flagged
# example/flagged
# [example/flagged]
flagged/main.go:14:24: possible misuse of unsafe.Pointer
exit status 1
$ go vet ./clean
exit status 0
reflection_vet_unsafeptr_sh.sh

reflection_vet_unsafeptr_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# unsafeptr flags a conversion from a uintptr variable to unsafe.Pointer. It
# judges the form of the expression, not what happens at run time: the second
# program keeps to the documented forms and is vet-clean, although one of its
# pointers is past the end of its allocation and the other is a uintptr that
# reaches a system call from a variable.
set -u
export GOTOOLCHAIN=local

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir" || exit 1

say() { printf '$ %s\n' "$*"; }

mkdir flagged clean

cat >flagged/main.go <<'GO'
package main

import (
    "fmt"
    "unsafe"
)

// A handle that stores an address as an integer.
type handle struct{ addr uintptr }

func main() {
    readings := new([4]float64)
    h := handle{addr: uintptr(unsafe.Pointer(readings))}
    back := (*[4]float64)(unsafe.Pointer(h.addr))
    fmt.Println(back[0])
}
GO

cat >clean/main.go <<'GO'
package main

import (
    "fmt"
    "syscall"
    "unsafe"
)

func main() {
    buf := make([]byte, 8)
    // Same expression, so vet accepts it -- but it points one past the end.
    end := unsafe.Pointer(uintptr(unsafe.Pointer(&buf[0])) + uintptr(len(buf)))
    // A uintptr that reaches the system call from a variable, not from the call's argument list.
    addr := uintptr(unsafe.Pointer(&buf[0]))
    _, _, errno := syscall.Syscall(syscall.SYS_GETPID, addr, 0, 0)
    fmt.Println(end != nil, errno)
}
GO

say 'go mod init example'
go mod init example >/dev/null 2>&1 || exit 1

say 'go vet ./flagged'
go vet ./flagged 2>&1
echo "exit status $?"

say 'go vet ./clean'
go vet ./clean 2>&1
echo "exit status $?"

The warning means — unsafe.Pointer(h.addr) converts a uintptr that is not part of one of the valid forms (arithmetic in the same expression, a syscall argument, reflect.Value.Pointer); the collector does not know h.addr is an address, so the object may have been freed or moved (the uintptr round trip).

The good version — keep the unsafe.Pointer, which the collector understands:

type handle struct{ p unsafe.Pointer }

h := handle{p: unsafe.Pointer(readings)}
back := (*[4]float64)(h.p)

When the tool is right, and when it is noise — right every time the uintptr outlives the expression, which is the only way a uintptr variable ever reaches Pointer. Noise only for code that is deliberately holding an address the collector must not follow — memory from mmap or C — and that code should convert through unsafe.Pointer(uintptr(...)) in one expression with a comment. The second program is the limit of what vet can know: end is one past the allocation and addr reaches a system call from a variable, both invalid by the docs' rules, both vet-clean, because vet judges the form of an expression and not where the pointer lands.

structtag: bad syntax for struct tag value

Five tags the compiler accepts and the reflect parser cannot read, or reads wrongly.

Verified output of reflection_vet_structtag_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go mod init example
$ go build -o customer ./bad
exit status 0
$ go vet ./bad
# example/bad
# [example/bad]
bad/main.go:9:2: struct field tag `json:name` not compatible with reflect.StructTag.Get: bad syntax for struct tag value
bad/main.go:10:2: struct field tag `json: "email"` not compatible with reflect.StructTag.Get: bad syntax for struct tag value
bad/main.go:11:2: struct field tag `json:"phone",xml:"phone"` not compatible with reflect.StructTag.Get: key:"value" pairs not separated by spaces
bad/main.go:12:2: struct field Contact repeats json tag "phone" also at main.go:11
bad/main.go:13:2: struct field balance has json tag but is not exported
exit status 1
$ go vet ./good
exit status 0
$ go run ./good
{"name":"Ada","email":"","phone":"","contact":""} <nil>
exit status 0
reflection_vet_structtag_sh.sh

reflection_vet_structtag_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# structtag checks the key:"value" form that reflect.StructTag.Get parses, a
# json or xml name used twice in one struct, and a json tag on a field that
# the encoder can never see. The compiler accepts all five tags: a tag is a
# string. The corrected struct passes vet with exit status 0.
set -u
export GOTOOLCHAIN=local

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir" || exit 1

say() { printf '$ %s\n' "$*"; }

mkdir bad good

cat >bad/main.go <<'GO'
package main

import (
    "encoding/json"
    "fmt"
)

type Customer struct {
    Name    string `json:name`
    Email   string `json: "email"`
    Phone   string `json:"phone",xml:"phone"`
    Contact string `json:"phone"`
    balance int    `json:"balance"`
}

func main() {
    data, err := json.Marshal(Customer{Name: "Ada", balance: 10})
    fmt.Println(string(data), err)
}
GO

cat >good/main.go <<'GO'
package main

import (
    "encoding/json"
    "fmt"
)

type Customer struct {
    Name    string `json:"name"`
    Email   string `json:"email"`
    Phone   string `json:"phone" xml:"phone"`
    Contact string `json:"contact"`
    balance int
}

func main() {
    data, err := json.Marshal(Customer{Name: "Ada", balance: 10})
    fmt.Println(string(data), err)
}
GO

say 'go mod init example'
go mod init example >/dev/null 2>&1 || exit 1

say 'go build -o customer ./bad'
go build -o customer ./bad 2>&1
echo "exit status $?"

say 'go vet ./bad'
go vet ./bad 2>&1
echo "exit status $?"

say 'go vet ./good'
go vet ./good 2>&1
echo "exit status $?"

say 'go run ./good'
go run ./good
echo "exit status $?"

The warning means — reflect.StructTag.Get parses key:"value" pairs separated by spaces; json:name has no quotes, json: "email" has a space after the colon, and json:"phone",xml:"phone" uses a comma. Each of those tags returns "" to Get, and encoding/json silently uses the field name. The fourth finding is a real name used twice, so two fields would encode under one key; the fifth is a json tag on an unexported field that the encoder can never see.

The good version — the good program in the driver: quotes, no spaces inside a pair, spaces between pairs, unique names, and no tag on balance. It builds, vets with exit status 0 and marshals every exported field.

When the tool is right, and when it is noise — right on all five; the compiler will never help, because a tag is a string. Noise for a tag in a format of your own that does not use key:"value" pairs — then Get is not the reader, and vet cannot know that. A json tag on an unexported field can be deliberate documentation, but it does nothing.

unmarshal: call of Unmarshal passes non-pointer as second argument

json.Unmarshal into a Customer rather than a *Customer.

Verified output of reflection_vet_unmarshal_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go mod init example
$ go vet .
# example
# [example]
./main.go:14:23: call of Unmarshal passes non-pointer as second argument
exit status 1
$ go run .
name "", err: json: Unmarshal(non-pointer main.Customer)
exit status 0
reflection_vet_unmarshal_sh.sh

reflection_vet_unmarshal_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# unmarshal: json.Unmarshal into a value rather than a pointer to it. The
# call compiles, and at run time it returns an error instead of filling the
# struct; vet reports it before the program runs.
set -u
export GOTOOLCHAIN=local

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir" || exit 1

say() { printf '$ %s\n' "$*"; }

cat >main.go <<'GO'
package main

import (
    "encoding/json"
    "fmt"
)

type Customer struct {
    Name string `json:"name"`
}

func main() {
    var c Customer
    err := json.Unmarshal([]byte(`{"name": "Ada"}`), c)
    fmt.Printf("name %q, err: %v\n", c.Name, err)
}
GO

say 'go mod init example'
go mod init example >/dev/null 2>&1 || exit 1

say 'go vet .'
go vet . 2>&1
echo "exit status $?"

say 'go run .'
go run .
echo "exit status $?"

The warning means — the decoder receives a copy of c and could only fill the copy; it refuses at run time with json: Unmarshal(non-pointer main.Customer) and c stays empty. vet sees it before the program runs. The same rule, from reflect's side, is reflect sets only addressable values.

The good version

err := json.Unmarshal([]byte(`{"name": "Ada"}`), &c)

When the tool is right, and when it is noise — right every time: there is no way to decode into a value. The same analyzer covers encoding/xml, encoding/gob and encoding/asn1.

-d=checkptr: the run-time check behind -race

Not a vet analyzer but a compiler flag, and the check that catches what unsafeptr can only suspect. The driver on the unsafe.Pointer lesson runs the flagged program plain, under -gcflags=all=-d=checkptr, and under -race:

Verified output of unsafe_pointer_checkptr_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go mod init example
$ go vet ./...
# example
# [example]
./roundtrip.go:23:25: possible misuse of unsafe.Pointer
exit status 1
$ go build -o roundtrip . && ./roundtrip
first reading through the round trip: {1 20.5}
exit status 0
$ go build -gcflags=all=-d=checkptr -o roundtrip_checkptr . && ./roundtrip_checkptr
fatal error: checkptr: pointer arithmetic result points to invalid allocation
exit status 2
$ go build -race -o roundtrip_race . && ./roundtrip_race
fatal error: checkptr: pointer arithmetic result points to invalid allocation
exit status 2

The warning means — the instrumented conversion produced a pointer into a heap allocation that none of the expression's inputs pointed into, which is the signature of an address that went through an integer. Since Go 1.14 ↗ -race and -msan enable it, so a go test -race run has this check for free. It allows an unaligned conversion when the target type holds no pointers — a *uint64 at an odd address passes, and the comment in checkptr.go ↗ says why — and it does not instrument unsafe.Add. (Those two limits are not machine-checked here.)

gofmt

gofmt has no opinion on reflect or unsafe beyond formatting; it will align the backticked tags in a struct, which is the moment to notice one that does not look like the others.

What no tool catches

  • Unexported fields skipped by json.Marshal. A struct whose fields lost their capital letters encodes as {} with a nil error — the last JSON line of reflect reads struct tags. structtag reports a json tag on an unexported field, but a field with no tag is silent.
  • A pointer past the end, and a uintptr that never comes back. The clean program above computes end, one past buf, in the approved one-expression form, and hands addr to syscall.Syscall from a variable; both are invalid by the unsafe docs ↗, and both are vet-clean, because vet judges the form of an expression and not where the pointer lands. checkptr catches the first when buf is on the heap; unsafe.Add(p, len(buf)), the same overrun with a nicer name, passes both tools. (The checkptr results are not machine-checked here.)
  • Reflection on a nil interface. reflect.TypeOf(nil) returns a nil Type, and the next method call is a nil-pointer dereference; reflect.ValueOf(nil) returns the zero Value, on which Kind is Invalid and Type panics — the compiler errors page records both. A function that takes any and reflects on it needs an IsValid check first.
  • A string over bytes that are then reused. unsafe.String(unsafe.SliceData(buf), n) returned from a function that reuses buf on the next call is a string whose contents change — the rule on unsafe.Pointer reinterprets memory, and nothing in the toolchain checks it.

staticcheck ↗ adds checks such as SA1019 for the deprecated reflect.SliceHeader and StringHeader; golangci-lint ↗ bundles it. Neither runs in this library.

Sources