Reading a hex dump¶

Level: 101 · for anyone starting from zero

One line: A hex dump is three columns — where you are, what the bytes are, what they would look like as ASCII — and reading one is the skill every later lesson uses to check a claim about text against the bytes that are actually there.

The three columns¶

00000000: 4869 2074 6865 7265 0a                   Hi there.
│         │                                        │
│         │                                        the same bytes read as ASCII; a dot for anything unprintable
│         sixteen bytes per line, two hex digits each, xxd pairs them
the offset: how many bytes came before this line, in hex

That is xxd. Every hex-dump tool prints the same three things, and the differences are cosmetic: xxd pairs the bytes, hexdump -C spaces them singly and frames the text in |, od -An -tx1 -c puts the characters on a second row under the hex. Learn to read one and you can read all of them.

One thing the shell example does that you would not: it pipes od through a small tidy helper. BSD od (macOS) and GNU od (Linux) lay their columns out differently — a blank address column, padding, and the character row aligned one column apart — and this library records every example's output on both, so the helper re-prints the fields four wide to make the two agree. The bytes are untouched; only the whitespace is.

The offset column is in hex, like everything else. The second line of a dump starts at 00000010 — that is sixteen, not ten. The right-hand column is a guess: the tool applies the ASCII agreement to each byte and prints a dot where that agreement has nothing to say. A dot does not mean "garbage"; it means "not ASCII, ask a different reader".

In the terminal¶

Verified output of reading_a_hex_dump_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

1. xxd: offset | sixteen bytes in hex, paired | the same bytes as ASCII

$ printf 'Hi there\n' | xxd
00000000: 4869 2074 6865 7265 0a                   Hi there.

2. A LONGER INPUT: the offset column counts bytes, in hex, sixteen per line

$ printf 'The quick brown fox jumps over it\n' | xxd
00000000: 5468 6520 7175 6963 6b20 6272 6f77 6e20  The quick brown 
00000010: 666f 7820 6a75 6d70 7320 6f76 6572 2069  fox jumps over i
00000020: 740a                                     t.

3. xxd -g1 ungroups the pairs; -c 8 changes the line width

$ printf 'Hi there\n' | xxd -g1 -c 8
00000000: 48 69 20 74 68 65 72 65  Hi there
00000008: 0a                       .

4. od: the POSIX tool. -An drops the offset, -tx1 = hex bytes, -c = as characters
   (a tab instead of the space, so -c has something to show for every byte)

$ printf 'Hi\tthere\n' | od -An -tx1 -c | tidy
  48  69  09  74  68  65  72  65  0a
   H   i  \t   t   h   e   r   e  \n

5. hexdump -C: the third classic; same three columns, |ascii| framed

$ printf 'Hi there\n' | hexdump -C
00000000  48 69 20 74 68 65 72 65  0a                       |Hi there.|
00000009

6. NON-ASCII BYTES: the right-hand column gives up and prints a dot

$ printf 'caf\xc3\xa9\n' | xxd
00000000: 6361 66c3 a90a                           caf...

$ printf 'caf\xc3\xa9\n' | od -An -tx1 -c | tidy
  63  61  66  c3  a9  0a
   c   a   f 303 251  \n
   Four letters, five bytes plus the newline: c a f, then TWO bytes c3 a9 for the e-acute.
   od -c shows those two as octal (303 251) because it has no character to show.

Section 6 is the first sight of the thing this library is about. café is four letters and the dump shows five bytes before the newline: 63 61 66 for caf, then c3 a9 for the é. The right-hand column prints .. for those two, and od -c prints their octal values 303 251, because neither tool has a single character to show for a byte above 127. Why é is two bytes, and why exactly those two, is chapter 3. For now the lesson is only: the number of letters and the number of bytes are different questions, and the dump answers the second one.

In Python¶

Writing the tool yourself takes ten lines, and afterwards the columns are no longer magic.

reading_a_hex_dump_py.py in full — pasted here by tools/run_examples.py from the file CI runs.

#!/usr/bin/env python3
"""A ten-line xxd, so the columns stop being magic.

Run:  python3 reading_a_hex_dump_py.py
"""


def dump(data: bytes, width: int = 16) -> str:
    """Offset | hex bytes | printable ASCII, the way xxd and hexdump lay it out."""
    lines = []
    for offset in range(0, len(data), width):
        chunk = data[offset : offset + width]
        hexes = " ".join(f"{b:02x}" for b in chunk)
        text = "".join(chr(b) if 32 <= b < 127 else "." for b in chunk)
        lines.append(f"{offset:08x}  {hexes:<{width * 3 - 1}}  |{text}|")
    return "\n".join(lines)


def main() -> None:
    print("1. THE SAME THREE COLUMNS, BUILT BY HAND")
    print(dump(b"Hi there\n"))
    print()

    print("2. OFFSETS ARE BYTE POSITIONS, IN HEX, SO LINE TWO STARTS AT 0x10 = 16")
    print(dump(b"The quick brown fox jumps over it\n"))
    print()

    print("3. WHAT 'PRINTABLE' MEANS: 32..126 get a character, everything else a dot")
    print(dump(b"tab\there\nnew line\x00null\x7fdel"))
    print()

    print("4. THE SHORTCUT: bytes.hex(' ') is the middle column on its own")
    word = "café".encode("utf-8")
    print(f"   {word!r}")
    print(f"   {word.hex(' ')}")
    print(f"   len('café') = {len('café')} characters, len(word) = {len(word)} bytes")
    print()

    print("5. HOW MANY BYTES IS THIS TEXT?  Count the hex pairs, or ask.")
    for s in ("Hi", "Hi there\n", "café", "naïve façade"):
        print(f"   {s!r:<16} {len(s.encode('utf-8')):>2} bytes   {s.encode('utf-8').hex(' ')}")


if __name__ == "__main__":
    main()

Verified output of reading_a_hex_dump_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. THE SAME THREE COLUMNS, BUILT BY HAND
00000000  48 69 20 74 68 65 72 65 0a                       |Hi there.|

2. OFFSETS ARE BYTE POSITIONS, IN HEX, SO LINE TWO STARTS AT 0x10 = 16
00000000  54 68 65 20 71 75 69 63 6b 20 62 72 6f 77 6e 20  |The quick brown |
00000010  66 6f 78 20 6a 75 6d 70 73 20 6f 76 65 72 20 69  |fox jumps over i|
00000020  74 0a                                            |t.|

3. WHAT 'PRINTABLE' MEANS: 32..126 get a character, everything else a dot
00000000  74 61 62 09 68 65 72 65 0a 6e 65 77 20 6c 69 6e  |tab.here.new lin|
00000010  65 00 6e 75 6c 6c 7f 64 65 6c                    |e.null.del|

4. THE SHORTCUT: bytes.hex(' ') is the middle column on its own
   b'caf\xc3\xa9'
   63 61 66 c3 a9
   len('café') = 4 characters, len(word) = 5 bytes

5. HOW MANY BYTES IS THIS TEXT?  Count the hex pairs, or ask.
   'Hi'              2 bytes   48 69
   'Hi there\n'      9 bytes   48 69 20 74 68 65 72 65 0a
   'café'            5 bytes   63 61 66 c3 a9
   'naïve façade'   14 bytes   6e 61 c3 af 76 65 20 66 61 c3 a7 61 64 65

The dump function is xxd with the pairing removed: slice the bytes sixteen at a time, print the offset padded to eight hex digits, print each byte as {b:02x}, and print the byte as a character only if it is in the printable range 32..126. Section 3 shows why that range: a tab, a newline, a NUL and a DEL are all bytes, and none of them can be drawn.

Which tool to reach for¶

Tool	Reach for it when	Note
`xxd`	you want the default view, or to go back (`xxd -r`)	ships with vim; `-b` for binary, `-g1` to unpair, and `-r` ignores the text column on the way back
`od -An -tx1 -c`	you are on a machine with nothing else	POSIX, always present; `-c` shows octal for high bytes (set `LC_ALL=C`), and skip `-a` — it makes names up
`hexdump -C`	you want the text column framed in bars, or somebody else has to read the dump	macOS always, Linux only with `bsdextrautils`; plain `hexdump` swaps every pair, and `-C` is the one view that is identical on both platforms
`bytes.hex(' ')` ↗	you are already in Python	just the middle column, one line

If you are coming from Python or ABAP¶

Python. bytes.hex(' ') is the dump's middle column; bytes.fromhex() is xxd -r. The dump function above is worth keeping in a scratch file — it is what you will paste in the moment a CSV from Excel does something strange, because print(repr(data)) shows Python's interpretation of the bytes and a dump shows the bytes.

ABAP. The debugger's view of an xstring is a hex dump — the middle column only, no offsets and no ASCII guess. The habit that transfers is the same one: when a string looks wrong after a file read or an RFC call, convert it to xstring with cl_abap_codepage=>convert_to( ) and look at the bytes before deciding whose fault it is. Nine times out of ten the bytes are fine and the reader applied the wrong agreement. (Not machine-checked — CI cannot run ABAP.)

Try it¶

cd 01_Bits_and_Bytes/reading_a_hex_dump/examples
bash reading_a_hex_dump_sh.sh
python3 reading_a_hex_dump_py.py

Then dump something real: xxd ~/.zshrc | head, or any file on your disk. Find the newlines (0a). Find the first byte the right-hand column shows as a dot, and note its value.

Practice¶

Read the columns. Build a file whose bytes you chose:

printf 'Hi\tthere\n\303\251\007ok\n' > f
xxd f

From the dump alone, answer four things: where the newlines are, the offset and value of the first byte the right-hand column draws as a dot, how many bytes the letter é takes and what the text column does with them, and what the byte at offset 0c is. Then say which of the three columns is the file.

Answers

Verified output of reading_a_hex_dump_kata_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ xxd f
   00000000: 4869 0974 6865 7265 0ac3 a907 6f6b 0a    Hi.there....ok.

1. WHERE ARE THE NEWLINES?
   Offsets 09 and 0f, both 0a. The right-hand column draws each as a dot,
   which is the column telling you it has nothing to draw rather than the
   file containing a dot.

2. THE FIRST BYTE SHOWN AS A DOT
   0x09 -- at offset 02, the tab.
   It is a real byte, 09, and it is the reason the words on that line are
   spaced the way they are. The dot is the dump's placeholder for every
   byte outside 20..7e, so a dot never means 'a dot'.

3. HOW MANY BYTES DOES THE LETTER MAKE?
   Two: c3 a9 at offsets 0a and 0b, one letter e-acute. The text column
   draws two dots for them, side by side, because it decodes nothing --
   it tests each byte against ASCII on its own. A dump cannot show you a
   character; it can only show you the bytes one is spelled with.

4. THE BYTE THAT IS NOT A LETTER AND NOT A NEWLINE
   07 at offset 0c, BEL. Printed to a terminal it would make a sound and
   move the cursor nowhere -- which is exactly why cat is not a way to
   look at a file and xxd is.

THE THREE COLUMNS, RESTATED
   left       the offset, in hex, of the first byte on the line
   middle     the bytes, and this is the file
   right      a guess: ASCII where it can, a dot where it cannot
   Only the middle column is the file. The left one is counting and the
   right one is a courtesy that has already thrown information away.