Skip to content

A record on the wire

Level: 201 → 301 · for anyone reading or writing a binary interface

One line: A struct is a layout in memory, not a layout on the wire — the compiler leaves gaps between the fields so each is aligned, and it stores the multi-byte fields in the machine's byte order — so writing a struct's memory straight to a file sends padding bytes and an endianness the reader cannot rely on; the wire format is the fields, in order, each serialized by hand, and the byte count proves the difference.

The tempting shortcut is fwrite(&record, sizeof record, 1, file) — write the struct's memory and be done. It is fast, it round-trips on the machine that wrote it, and it is not a format. Two things travel with those bytes that you did not choose: the padding the compiler inserted for alignment, and the byte order of the machine. A reader on a different compiler, or a different CPU, gets neither back. The fix costs a few lines and makes the layout something you can write on paper.

The gaps you did not write

record_on_the_wire_c.c declares a three-field struct, shows that it is bigger than its fields, and serializes it into a frame with no gaps:

record_on_the_wire_c.c in full — pasted here by tools/run_examples.py from the file CI runs.

/* A struct is a layout in memory, not a layout on the wire. The compiler is
   free to leave gaps between the fields so each is aligned, and it stores the
   multi-byte fields in the machine's byte order -- so writing the struct's
   memory straight to a file sends padding and an endianness no reader can rely
   on. The wire format is the fields, in order, each serialised by hand. */
#include <stdio.h>
#include <stddef.h>
#include <stdint.h>
#include <string.h>

struct order {
    uint8_t  type;      /* 1 byte  */
    uint32_t id;        /* 4 bytes */
    uint16_t qty;       /* 2 bytes */
};

static void dump(const char *label, const void *p, size_t n)
{
    const unsigned char *b = p;
    printf("%-32s", label);
    for (size_t i = 0; i < n; i++)
        printf(" %02x", b[i]);
    printf("\n");
}

int main(void)
{
    printf("fields add up to %zu bytes, but sizeof(struct order) = %zu\n",
           sizeof(uint8_t) + sizeof(uint32_t) + sizeof(uint16_t),
           sizeof(struct order));
    printf("the gaps: offsetof type=%zu id=%zu qty=%zu\n",
           offsetof(struct order, type),
           offsetof(struct order, id),
           offsetof(struct order, qty));

    struct order o = { .type = 7, .id = 0x01020304u, .qty = 1000 };

    /* Serialise the fields into a 7-byte frame, big-endian, no padding. */
    unsigned char wire[7];
    wire[0] = o.type;
    wire[1] = o.id >> 24; wire[2] = o.id >> 16; wire[3] = o.id >> 8; wire[4] = o.id;
    wire[5] = o.qty >> 8; wire[6] = o.qty;
    dump("on the wire (7 bytes):", wire, sizeof wire);

    /* Read it back, again by hand. */
    struct order r;
    r.type = wire[0];
    r.id = (uint32_t)wire[1] << 24 | (uint32_t)wire[2] << 16
         | (uint32_t)wire[3] << 8  | wire[4];
    r.qty = (uint16_t)(wire[5] << 8 | wire[6]);
    printf("read back: type=%u id=0x%08x qty=%u, round trip ok: %s\n",
           r.type, r.id, r.qty,
           (r.type == o.type && r.id == o.id && r.qty == o.qty) ? "yes" : "no");
    return 0;
}

Verified output of record_on_the_wire_c.c — regenerated by tools/run_examples.py, never hand-typed.

fields add up to 7 bytes, but sizeof(struct order) = 12
the gaps: offsetof type=0 id=4 qty=8
on the wire (7 bytes):           07 01 02 03 04 03 e8
read back: type=7 id=0x01020304 qty=1000, round trip ok: yes

The fields are 1, 4 and 2 bytes — seven in all — but sizeof(struct order) is 12. The compiler put the uint32_t at offset 4, not 1, so that it is aligned to a four-byte boundary, and rounded the whole struct up so that an array of them keeps every id aligned. Those gaps are real bytes with no defined contents, and fwrite-ing the struct would send all twelve, three of them uninitialized. The 12 here is what both of this library's machines produce; a different ABI could choose different padding, which is the deeper reason not to send it — the number is not even stable across compilers, let alone across the wire.

The wire format is the seven bytes you actually meant: type, then id most-significant-byte-first, then qty, laid out by hand into a char array. 07 01 02 03 04 03 e8 — one byte of type, four of id big-endian, two of quantity — and reading it back by hand returns the original values. No padding, no machine order, nothing the reader has to guess.

Python's struct is this, as a format string

Python's struct module is exactly the by-hand serialization, written as a compact format string instead of shifts. record_on_the_wire_py.py:

record_on_the_wire_py.py in full — pasted here by tools/run_examples.py from the file CI runs.

"""Python's struct module IS the by-hand serialisation, written as a format
string. '>' is big-endian and pins down both the order and the absence of
padding; the codes B, I, H are the field widths. Native mode ('@') brings the
gaps back, which is exactly what you do not want on the wire."""
import struct

wire = struct.pack(">BIH", 7, 0x01020304, 1000)
print(f"struct.pack('>BIH', 7, 0x01020304, 1000) = {wire.hex(' ')}  ({len(wire)} bytes)")
print(f"struct.calcsize('>BIH') = {struct.calcsize('>BIH')}  -- packed, no padding")
print(f"struct.calcsize('@BIH') = {struct.calcsize('@BIH')}  -- native, with padding")

back = struct.unpack(">BIH", wire)
print(f"struct.unpack('>BIH', ...) = {back}")
print(f"round trip ok: {back == (7, 0x01020304, 1000)}")

Verified output of record_on_the_wire_py.py — regenerated by tools/run_examples.py, never hand-typed.

struct.pack('>BIH', 7, 0x01020304, 1000) = 07 01 02 03 04 03 e8  (7 bytes)
struct.calcsize('>BIH') = 7  -- packed, no padding
struct.calcsize('@BIH') = 10  -- native, with padding
struct.unpack('>BIH', ...) = (7, 16909060, 1000)
round trip ok: True

struct.pack(">BIH", ...) produces the same seven bytes as the C: > pins the order to big-endian and turns off padding, and B, I, H are the field widths. The proof is calcsize: ">BIH" is 7 bytes, and "@BIH" — native mode — is 10, because native mode brings the alignment gaps back. That gap between 7 and 10 is the same padding C's sizeof showed as the gap between 7 and 12, and it is exactly what you are turning off when you write > instead of @.

If you are coming from another language

Rust. A plain struct has no guaranteed layout at all unless you ask for one with #[repr(C)] or #[repr(packed)], and idiomatic serialization goes through serde with a format crate — the derive teaches the type to describe itself and the crate decides the bytes, which is the serde split ↗. The to_be_bytes family from the previous lesson is the by-hand path when you want the bytes exact.

ABAP. (Not machine-checked — CI cannot run ABAP.) A structure written to a file or a stream is serialized field by field, and EXPORT ... TO DATA BUFFER uses a defined internal format rather than raw memory precisely so that another system can read it back — the same reason this page does not fwrite the struct. The encodings library's packing a record ↗ has the ABAP column beside the C.

See also