Skip to content

unsafe.Pointer reinterprets memory

Level: 301 · anyone who needs the bits of a float, a struct's field by offset, or a []byte as a string without the copy

One line: A conversion changes a value; unsafe.Pointer changes only what type the bytes are read as, so *(*uint64)(unsafe.Pointer(&f)) is math.Float64bits(f) bit for bit, unsafe.String and unsafe.Slice view a []byte as a string and back for zero allocations — and an address kept as a uintptr and converted back later is the mistake that go vet flags and -d=checkptr turns into a fatal error.

The program does the reinterpretations the unsafe docs ↗ allow: a float64 read as a uint64 and a uint64 read as a float64, next to the conversion float64(…) that computes a new value instead; an element and a field reached by offset with unsafe.Add and with uintptr arithmetic in one expression; and the Go 1.20 functions that make a string over a []byte's memory. It ends by counting allocations: the copying conversions against the views.

Verified output of unsafe_pointer_reinterprets_memory_go.go — regenerated by tools/run_examples.py, never hand-typed.

1.5 reinterpreted as uint64: 0x3ff8000000000000
math.Float64bits(1.5):      0x3ff8000000000000, equal true
0x7ff8000000000001 reinterpreted as float64: NaN, IsNaN true
float64(0x40) converted instead: 64
unsafe.Add one Sizeof on from &readings[0]: {2 21}
uintptr + Offsetof(Value) into readings[2]: 19.75
unsafe.String over 64 bytes: "abcdefgh"... len 64
unsafe.Slice of that string: same first byte as data: true
string(data) copies:         same first byte as data: false
allocations per call, string(data):                      1
allocations per call, unsafe.String(SliceData, len):     0
allocations per call, []byte(text):                      1
allocations per call, unsafe.Slice(StringData, len):     0

Reading the output

Pattern 1: same bytes, different type. *(*uint64)(unsafe.Pointer(&f)) prints 0x3ff8000000000000 for 1.5, and so does math.Float64bits(f) — because that is how Float64bits is written ↗, and the unsafe docs use it as the example of their first valid pattern: a *T1 converted to Pointer converted to *T2, provided T2 is no larger than T1 and the two share an equivalent memory layout. The other direction gives a NaN from a bit pattern no arithmetic produces; and the line below it, float64(0x40), is an ordinary conversion producing 64 — a new value, not the same bytes.

Pattern 3: an offset inside an allocation. unsafe.Add(p, n) (Go 1.17 ↗) moves a Pointer by n bytes; one Sizeof(readings[0]) on from &readings[0] is readings[1]. The older spelling, unsafe.Pointer(uintptr(p) + offset), is valid only when the conversion to uintptr, the arithmetic and the conversion back are in one expression — here it reads readings[2].Value through Offsetof. The result must stay inside the original allocation: the docs list one-past-the-end as invalid, unlike C. Sizeof, Alignof and Offsetof themselves are measured in Struct layout and padding.

unsafe.String and unsafe.Slice share the memory. unsafe.String(unsafe.SliceData(data), len(data)) is a string whose bytes are data's bytes, and unsafe.Slice(unsafe.StringData(text), len(text)) goes back: the program checks that the view's first byte and data's first byte have the same address, and that string(data)'s does not. The four functions arrived in Go 1.20 ↗ and replaced the deprecated SliceHeader and StringHeader. This is what strings.Builder.String ↗ does to return its buffer without a copy.

Why the bytes must then be left alone. The docs for unsafe.String ↗ put it in one sentence: since Go strings are immutable, the bytes passed to String must not be modified as long as the returned string value exists. The compiler and the runtime assume a string never changes — a map key computed from it, a comparison already made, a substring already taken — and a write to data after the view exists breaks all of them. The program does not demonstrate what happens, because a page cannot record undefined behaviour; it shows the pattern and the rule. The same applies to unsafe.Slice(unsafe.StringData(s), …): a string literal's bytes may live in read-only memory, and writing through the slice is a crash on some platforms and silent corruption on others. (Not machine-checked here.)

Zero allocations against one. string(data) and []byte(text) each allocate once per call, the copy; the views allocate nothing. That is the whole reason to use them, and the whole risk: a copy is yours, a view is a promise.

The uintptr round trip, and the two tools that see it

A uintptr is an integer. The docs' pattern 2 allows converting a Pointer to one — to print it — and says plainly that the conversion back is not valid in general: the garbage collector will not update the integer if the object moves, and the integer does not keep the object alive. The driver stores an address in a uintptr variable, converts it back, and runs the program three ways:

Verified output of unsafe_pointer_checkptr_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ go mod init example
$ go vet ./...
# example
# [example]
./roundtrip.go:23:25: possible misuse of unsafe.Pointer
exit status 1
$ go build -o roundtrip . && ./roundtrip
first reading through the round trip: {1 20.5}
exit status 0
$ go build -gcflags=all=-d=checkptr -o roundtrip_checkptr . && ./roundtrip_checkptr
fatal error: checkptr: pointer arithmetic result points to invalid allocation
exit status 2
$ go build -race -o roundtrip_race . && ./roundtrip_race
fatal error: checkptr: pointer arithmetic result points to invalid allocation
exit status 2

go vet's unsafeptr analyzer reports possible misuse of unsafe.Pointer at the conversion and exits 1; the plain build runs and prints the reading, because nothing moved this time. Built with -gcflags=all=-d=checkptr, the same conversion is fatal error: checkptr: pointer arithmetic result points to invalid allocation and exit status 2 — the instrumentation added in Go 1.14 ↗ checks at run time that a Pointer produced from a uintptr points into a live allocation it was derived from. The last block shows why it matters: -race enables checkptr too, so a test suite that runs the race detector already runs this check. What checkptr does not catch, and what vet does not either, is on the vet page.

unsafe_pointer_reinterprets_memory_go.go and unsafe_pointer_checkptr_sh.sh

unsafe_pointer_reinterprets_memory_go.go in full — pasted here by tools/run_examples.py from the file CI runs.

// unsafe.Pointer converts between pointer types without converting the bytes
// they point at: *(*uint64)(unsafe.Pointer(&f)) is the bit pattern of the
// float64 f, which is exactly how math.Float64bits is written. unsafe.String
// and unsafe.Slice reinterpret a []byte as a string and back without a copy.
//
//  go build unsafe_pointer_reinterprets_memory_go.go && ./unsafe_pointer_reinterprets_memory_go
package main

import (
    "fmt"
    "math"
    "testing"
    "unsafe"
)

type Reading struct {
    Station uint16
    Value   float64
}

var (
    sinkString string
    sinkBytes  []byte
)

func main() {
    // Pattern 1: *T1 -> Pointer -> *T2, the bytes untouched.
    f := 1.5
    bits := *(*uint64)(unsafe.Pointer(&f))
    fmt.Printf("1.5 reinterpreted as uint64: %#016x\n", bits)
    fmt.Printf("math.Float64bits(1.5):      %#016x, equal %t\n", math.Float64bits(f), bits == math.Float64bits(f))
    quiet := uint64(0x7ff8000000000001)
    asFloat := *(*float64)(unsafe.Pointer(&quiet))
    fmt.Printf("%#016x reinterpreted as float64: %v, IsNaN %t\n", quiet, asFloat, math.IsNaN(asFloat))
    fmt.Printf("float64(%#x) converted instead: %v\n", uint64(0x40), float64(uint64(0x40)))

    // Pattern 3: through uintptr with arithmetic, in one expression -- or unsafe.Add.
    readings := [3]Reading{{1, 20.5}, {2, 21}, {3, 19.75}}
    second := (*Reading)(unsafe.Add(unsafe.Pointer(&readings[0]), unsafe.Sizeof(readings[0])))
    fmt.Println("unsafe.Add one Sizeof on from &readings[0]:", *second)
    third := *(*float64)(unsafe.Pointer(uintptr(unsafe.Pointer(&readings[2])) + unsafe.Offsetof(readings[2].Value)))
    fmt.Println("uintptr + Offsetof(Value) into readings[2]:", third)

    // Zero-copy []byte <-> string (Go 1.20).
    data := make([]byte, 64)
    for i := range data {
        data[i] = 'a' + byte(i%26)
    }
    text := unsafe.String(unsafe.SliceData(data), len(data))
    fmt.Printf("unsafe.String over 64 bytes: %q... len %d\n", text[:8], len(text))
    view := unsafe.Slice(unsafe.StringData(text), len(text))
    fmt.Println("unsafe.Slice of that string: same first byte as data:", &view[0] == &data[0])
    fmt.Println("string(data) copies:         same first byte as data:", unsafe.StringData(string(data)) == &data[0])

    // What the copy costs, and what the reinterpretation saves.
    fmt.Println("allocations per call, string(data):                     ", testing.AllocsPerRun(100, func() { sinkString = string(data) }))
    fmt.Println("allocations per call, unsafe.String(SliceData, len):    ", testing.AllocsPerRun(100, func() { sinkString = unsafe.String(unsafe.SliceData(data), len(data)) }))
    fmt.Println("allocations per call, []byte(text):                     ", testing.AllocsPerRun(100, func() { sinkBytes = []byte(text) }))
    fmt.Println("allocations per call, unsafe.Slice(StringData, len):    ", testing.AllocsPerRun(100, func() { sinkBytes = unsafe.Slice(unsafe.StringData(text), len(text)) }))
}

unsafe_pointer_checkptr_sh.sh in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env bash
# The uintptr round trip: an address kept as an integer and converted back to
# a pointer later. go vet's unsafeptr analyzer flags the conversion; the
# program still builds and runs. Built with -d=checkptr -- which -race turns on
# as well -- the same conversion is a fatal error at run time. The stderr also
# carries a goroutine trace with a temporary path, so only the fatal error
# line is printed.
set -u
export GOTOOLCHAIN=local

dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir" || exit 1

cat >roundtrip.go <<'GO'
package main

import (
    "fmt"
    "runtime"
    "unsafe"
)

type Reading struct {
    Station uint16
    Value   float64
}

//go:noinline
func newReadings() *[4]Reading {
    return &[4]Reading{{1, 20.5}, {2, 21}, {3, 19.75}, {4, 22}}
}

func main() {
    readings := newReadings()
    addr := uintptr(unsafe.Pointer(readings)) // an integer, not a reference
    runtime.GC()
    again := (*[4]Reading)(unsafe.Pointer(addr)) // the mistake: uintptr -> Pointer from a variable
    fmt.Println("first reading through the round trip:", again[0])
    runtime.KeepAlive(readings)
}
GO

say() { printf '$ %s\n' "$*"; }

say 'go mod init example'
go mod init example >/dev/null 2>&1 || exit 1

say 'go vet ./...'
go vet ./... 2>&1
echo "exit status $?"

say 'go build -o roundtrip . && ./roundtrip'
go build -o roundtrip . && ./roundtrip
echo "exit status $?"

say 'go build -gcflags=all=-d=checkptr -o roundtrip_checkptr . && ./roundtrip_checkptr'
go build -gcflags=all=-d=checkptr -o roundtrip_checkptr . || exit 1
./roundtrip_checkptr 2>stderr.txt
status=$?
grep '^fatal error:' stderr.txt
echo "exit status $status"

# -race needs cgo, and on Linux a C compiler; without them this stops here.
say 'go build -race -o roundtrip_race . && ./roundtrip_race'
go build -race -o roundtrip_race . || exit 1
./roundtrip_race 2>stderr.txt
status=$?
grep '^fatal error:' stderr.txt
echo "exit status $status"

The six valid patterns

The unsafe.Pointer docs ↗ list the only uses they promise to keep working, and say that code using other patterns is likely invalid today or to become invalid. Paraphrased:

  1. *T1 → Pointer → *T2, when T2 is no larger than T1 and the layouts are equivalent (Float64bits).
  2. Pointer → uintptr, to print or compare the address — and not back.
  3. Pointer → uintptr → arithmetic → Pointer, all in one expression, staying inside the same allocation; not one past the end, and not from a nil pointer.
  4. uintptr(unsafe.Pointer(p)) written directly in the argument list of syscall.Syscall, where the compiler keeps the object alive for the call.
  5. The uintptr returned by reflect.Value.Pointer or UnsafeAddr, converted to Pointer in the same expression.
  6. The fields of reflect.SliceHeader and StringHeader — deprecated since unsafe.Slice and unsafe.String exist.

go vet recognises the forms of 3, 4 and 5 and flags a uintptr that reaches Pointer any other way; it cannot see whether pattern 3 stayed inside the allocation.

What to do

  • Use math.Float64bits, binary.LittleEndian and encoding/binary before unsafe. They are the same reinterpretations with the package name as a warning label, and they are inlined.
  • Never store a uintptr. Convert, use, forget, in one expression. If you want a handle to an object, keep the unsafe.Pointer — it is a real pointer the collector understands.
  • Treat an unsafe.String as borrowed. Do not write to the bytes while the string exists, and do not return the string from a function that will reuse the buffer.
  • Run tests with -race, which turns on checkptr and makes the uintptr round trip a fatal error where vet only suspects it.
  • Read the unsafe package docs once a year: the list of valid patterns is the contract, and SliceHeader shows that it changes.

Practice

Write bitsOf(f float32) uint32 and fromBits(b uint32) float32 with pattern 1 and check them against math.Float32bits and math.Float32frombits for 1, -2.5, 0 and 3e38. Then write view(b []byte) string with unsafe.String and show with testing.AllocsPerRun that it allocates nothing where string(b) allocates once, and print the rule a caller of view has to keep.

Solution

unsafe_pointer_reinterprets_memory_kata_go.go in full — pasted here by tools/run_examples.py from the file CI runs.

// Kata: float32 bits through unsafe.Pointer, checked against math.Float32bits
// and back through math.Float32frombits, and a zero-copy view of a []byte as
// a string whose allocation count is measured against the copying conversion.
//
//  go build unsafe_pointer_reinterprets_memory_kata_go.go && ./unsafe_pointer_reinterprets_memory_kata_go
package main

import (
    "fmt"
    "math"
    "testing"
    "unsafe"
)

var sink string

func bitsOf(f float32) uint32   { return *(*uint32)(unsafe.Pointer(&f)) }
func fromBits(b uint32) float32 { return *(*float32)(unsafe.Pointer(&b)) }
func view(b []byte) string      { return unsafe.String(unsafe.SliceData(b), len(b)) }
func copyOf(b []byte) string    { return string(b) }
func allocs(f func()) float64   { return testing.AllocsPerRun(200, f) }
func check(f float32, b uint32) bool {
    return b == math.Float32bits(f) && fromBits(b) == math.Float32frombits(b)
}

func main() {
    for _, f := range []float32{1, -2.5, 0, 3e38} {
        b := bitsOf(f)
        fmt.Printf("%-8v bits %#08x  agrees with math: %t\n", f, b, check(f, b))
    }
    line := make([]byte, 48)
    for i := range line {
        line[i] = "0123456789"[i%10]
    }
    fmt.Println("view(line) == copyOf(line):", view(line) == copyOf(line))
    fmt.Println("allocations per call, copyOf:", allocs(func() { sink = copyOf(line) }))
    fmt.Println("allocations per call, view:  ", allocs(func() { sink = view(line) }))
    fmt.Println("rule: a view is valid only while the bytes are neither modified nor reused")
}

Verified output of unsafe_pointer_reinterprets_memory_kata_go.go — regenerated by tools/run_examples.py, never hand-typed.

1        bits 0x3f800000  agrees with math: true
-2.5     bits 0xc0200000  agrees with math: true
0        bits 0x00000000  agrees with math: true
3e+38    bits 0x7f61b1e6  agrees with math: true
view(line) == copyOf(line): true
allocations per call, copyOf: 1
allocations per call, view:   0
rule: a view is valid only while the bytes are neither modified nor reused

In other languages

  • Rust — Raw pointers: *const T and *mut T ↗ are unsafe.Pointer with a type still attached, and mem::transmute ↗ is pattern 1 as a single call that the compiler checks for equal size; What unsafe turns off ↗ lists the five powers an unsafe block grants, where Go has no block and the package name is the only marker. f32::to_bits is Float32bits, and a &str over a Vec<u8> is std::str::from_utf8_unchecked, with the same must-not-mutate rule enforced by the borrow checker instead of by a sentence in the docs.
  • C — the C library's A record on the wire ↗ writes a struct's memory to a file and counts the padding bytes and byte order that go with it; Byte order on the wire ↗ is why copying an integer's memory is machine-specific. C's cast through void * is unsafe.Pointer without pattern rules, and memcpy into a differently typed object is the sanctioned type pun.
  • C++ has std::bit_cast ↗ since C++20: pattern 1 with the size check done by the compiler.

Sources