Skip to content

pack and unpack speak Perl's template language

Level: 201 · anyone reading or writing a binary format

One line: Array#pack and String#unpack take Perl's template letters — N and n big-endian, V and v little-endian, a, C, U — and produce the same bytes Perl does; Ruby adds letters Perl's pack lacks, such as m for Base64, and labels each result BINARY, UTF-8 or US-ASCII according to its letters.

Verified output of pack_speaks_perl_rb.rb — regenerated by tools/run_examples.py, never hand-typed.

  [1, 258].pack("Nn")                000000010102
  [1, 258].pack("Vv")                010000000201
  ["hi", 7].pack("a4C")              6869000007
  "\x00\x01\x01\x02".unpack("n*")    [1, 258]
  "caf\u{e9}".unpack("C*")           [99, 97, 102, 195, 169]
  "caf\u{e9}".unpack("U*")           [99, 97, 102, 233]

A letter Perl's pack does not have:
  ["hi"].pack("m0")                  "aGk="
  "aGk=".unpack1("m0")               "hi"

What each result is labelled:
  [1].pack("N")                      ASCII-8BIT
  [233].pack("U")                    UTF-8
  ["hi"].pack("m0")                  US-ASCII

Reading the output

  • N and n write a 32-bit and a 16-bit unsigned integer, most significant byte first — network order. V and v write them least significant byte first.
  • a4 writes a string into four bytes, padding with NULs; C writes one byte.
  • C* and U* unpack the same string differently: C reads bytes, so é is 195, 169; U reads UTF-8 characters, so é is 233.
  • m0 is Base64 without line breaks.
  • The label depends on the letters. Integers in bytes are BINARY, U output is UTF-8, and Base64 is US-ASCII — which is why a pack result can be joined to text without an error or not, depending on what it holds.

The same templates in Perl

Verified output of pack_speaks_perl_pl.pl — regenerated by tools/run_examples.py, never hand-typed.

  pack('Nn', 1, 258)               000000010102
  pack('Vv', 1, 258)               010000000201
  pack('a4C', 'hi', 7)             6869000007
  unpack('n*', "\x00\x01\x01\x02") 1, 258
  pack('m0', 'hi')                 Invalid type 'm' in pack

Byte for byte the same, and Perl's pack has no m — Perl does Base64 with the MIME::Base64 module instead.

And Python's struct

Verified output of pack_speaks_perl_py.py — regenerated by tools/run_examples.py, never hand-typed.

  struct.pack('>IH', 1, 258)         000000010102
  struct.pack('<IH', 1, 258)         010000000201
  struct.pack('4sB', b'hi', 7)       6869000007
  struct.unpack('>2H', ...)          (1, 258)
  base64.b64encode(b'hi')            b'aGk='

The same bytes again, described differently: a prefix for byte order (> or <) and then I and H for the sizes. Base64 is its own module there too.

The full list of Ruby's letters is on the packed data page ↗; Perl's are under pack ↗. The C library's Byte order on the wire ↗ and the Encodings library's Packing a record ↗ are the same bytes seen from C and from a hex dump.