Skip to content

Bytes, hex and int

Level: 101 → 201 · for Python programmers

One line: Four conversions — bytes.hex, bytes.fromhex, int.from_bytes, int.to_bytes — are the whole toolkit for reading a binary format by hand, and struct is the same four with a template.

b = "é".encode("utf-8")

b.hex()                             # 'c3a9'
bytes.fromhex("c3a9")               # b'\xc3\xa9'
int.from_bytes(b, "big")            # 50089
(50089).to_bytes(2, "big")          # b'\xc3\xa9'

Two of those four move bytes to a picture of the same bytes. Two move bytes to a number. Telling them apart is the whole lesson.

This is chapter 1, in Python

Hex: a number, or a picture of bytes makes the distinction that everything here rests on: the same run of hex digits is two different objects. As a number, width is nothing and a leading zero is noise — 0x1e240 and 0x0001e240 are the same integer. As a byte string, width is the data and a leading zero is a NUL byte you must write.

Python gives you a separate pair of functions for each reading, which is a kindness other languages do not extend:

you have you want the call
bytes their hex picture b.hex()
a hex picture bytes bytes.fromhex(s)
bytes a number int.from_bytes(b, order)
a number bytes of a given width n.to_bytes(width, order)

Reach for int(s, 16) when you meant bytes.fromhex(s) and you have silently thrown the width away. Grouping is a choice covers hex()'s separator arguments — including the rule that the sign of the group width is the direction — and this page does not repeat it.

The argument that stopped being compulsory

int.from_bytes was added in Python 3.2 with no default for byteorder. You could not call it without answering the question, which is the rarer kind of API: one that prevents a bug rather than documenting it.

Python 3.11 gave it one ↗, and the default is 'big':

int.from_bytes(b"\xc3\xa9")         # 50089 on 3.11+; TypeError on 3.10 and earlier

So the guard is gone and the question is not. Two bytes have two readings, the bytes do not say which one you meant, and now neither does your code — a reader cannot tell a deliberate big-endian from a forgotten argument. Write the byte order down even where the default would have done.

Measured 2026-09-07, and it is a version boundary rather than a platform one:

                            3.10.21     3.11.16   3.12.14   3.13.15   3.14.7
int.from_bytes(b, 'big')      50089       50089     50089     50089    50089
int.from_bytes(b)         TypeError       50089     50089     50089    50089

The 3.10 cell is the whole history in one word, and its message names the thing that went missing: from_bytes() missing required argument 'byteorder' (pos 2). That wording is CPython's and belongs in this fence rather than in an answer key, but the class is the point — a call that could not be made at all is now a call that quietly picks for you.

The program

Verified output of bytes_hex_and_int_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. BYTES AND THEIR PICTURE
------------------------------------------------------------------------
   bytes.hex() and bytes.fromhex() are inverses, and neither one is a
   number: they move between some bytes and a written-down picture of
   the same bytes, digit for digit, in order.

     EACUTE = 'e-acute'.encode('utf-8')       b'\xc3\xa9'
     EACUTE.hex()                             'c3a9'
     bytes.fromhex('c3a9') == EACUTE          True

   The separator argument groups the picture without changing it, and
   the SIGN of the group width is the direction: positive counts from
   the RIGHT, negative from the left. On an odd number of bytes that
   decides where the short group lands, which the grouping page works
   through in full:

     b.hex(' ')           00 01 e2 40 ff   one byte at a time
     b.hex(' ', 2)        00 01e2 40ff     short group first
     b.hex(' ', -2)       0001 e240 ff     short group last

   Same five bytes every time. Only the spaces moved, and the spaces
   are not in the data.

2. BYTES AND A NUMBER
------------------------------------------------------------------------
   int.from_bytes() reads bytes as one integer, and the byte order is
   the whole question. Two bytes, two readings:

     int.from_bytes(EACUTE, 'big')            50089
     int.from_bytes(EACUTE, 'little')         43459

   Neither is right. They are answers to different questions, and the
   bytes do not say which one you meant -- that is what a format spec
   is for.

   Going back the other way, the LENGTH is the other half of the same
   point. One number, four widths:

     (123456).to_bytes(3, 'big')              01 e2 40
     (123456).to_bytes(4, 'big')              00 01 e2 40
     (123456).to_bytes(8, 'big')              00 00 00 00 00 01 e2 40
     (123456).to_bytes(2, 'big')              OverflowError

   The leading zeros are not decoration. As a NUMBER 123456 has no
   width; as a field in a file it has exactly the width the format
   says, and the zeros are bytes that must be written.

3. THE ARGUMENT THAT STOPPED BEING COMPULSORY
------------------------------------------------------------------------
   From Python 3.2 to 3.10, byteorder had no default. You could not
   call int.from_bytes without answering the question, which is the
   kind of API that prevents a bug rather than documenting one.

   Python 3.11 gave it one, and the default is 'big':

     int.from_bytes(EACUTE)                   50089
     ...same as 'big':                        True
     ...same as 'little':                     False
     (255).to_bytes()                         ff

   On Python 3.10 and earlier the first line is a TypeError. So the
   guard is gone, the question is not, and it is now yours to
   remember: write the byte order down even where the default would
   have done, because the reader of your code cannot tell a deliberate
   big-endian from a forgotten argument.

4. NEGATIVE NUMBERS ARE ONE MORE ARGUMENT
------------------------------------------------------------------------
   signed=True is two's complement, spelled as a keyword:

     (-1).to_bytes(4, 'big', signed=True)     ff ff ff ff
     (-2).to_bytes(4, 'big', signed=True)     ff ff ff fe
     (1).to_bytes(4, 'big', signed=True)      00 00 00 01

   And the same four bytes read both ways -- which is why a format
   spec has to say signed or unsigned, and why getting it wrong gives
   you a number instead of an error:

     int.from_bytes(ff, 'big')                4294967295
     int.from_bytes(ff, 'big', signed=True)   -1

     (-1).to_bytes(4, 'big')                  OverflowError
     Unsigned refuses a negative, which is the one place the
     conversion does check up on you.

5. struct IS THE SAME FOUR WITH A TEMPLATE
------------------------------------------------------------------------
   The first character of a struct format is the byte order, and it is
   the first thing to read in any format string you meet:

     hex(value)                               0x1e240

     format   packed bytes               meaning
     <I       40 e2 01 00                little-endian, 4 bytes
     >I       00 01 e2 40                big-endian, 4 bytes
     <H       struct.error               the value does not fit
     >Q       00 00 00 00 00 01 e2 40    big-endian, 8 bytes

     struct.calcsize('<I')                    4
     struct.calcsize('<HH')                   4
     struct.unpack('>I', pack('>I', v))       True

   Leave the byte-order character off and you get NATIVE order plus
   native alignment padding, which is a different size on a different
   machine. In a file format that is always a bug:

     struct.calcsize('=HI')  (no padding)     6
     struct.calcsize('HI')   (native, padded) 8
     struct.calcsize('<HI')  (explicit)       6

   The middle row is two bytes of padding you did not ask for and
   cannot see. Which order 'native' means is sys.byteorder, and it is
   a fact about the machine rather than about your data -- which is
   why it is named here rather than printed: an answer key that
   recorded it would be recording the runner.

6. A REAL HEADER, BUILT AND THEN READ BACK
------------------------------------------------------------------------
   The first 16 bytes of a PNG. Everything below is computed here --
   the length, the CRC -- so it is a real header rather than a quoted
   one:

     89 50 4e 47 0d 0a 1a 0a
     00 00 00 0d 49 48 44 52

   Field by field:

     bytes          field                      value
     89 50 4e 47..  PNG signature              8 bytes, fixed
     00 00 00 0d    chunk length (>I)          13
     49 48 44 52    chunk type (4 ASCII)       IHDR
     00 00 00 01    width (>I)                 1
     00 00 00 01    height (>I)                1
     08             bit depth (B)              8
     02             colour type (B)            2
     90 77 53 de    CRC-32 of type+body        checks: True

   PNG is big-endian throughout, which is why every length above is
   '>I'. Read it as '<I' and the 13-byte header announces itself as
   218103808 bytes:

     struct.unpack('>I', png[8:12])           13
     struct.unpack('<I', png[8:12])           218103808

7. AND THE SIGNATURE IS AN ENCODING TEST
------------------------------------------------------------------------
   Those first eight bytes are not a magic number somebody liked. Six
   of the eight are there to catch a file that was TRANSFORMED in
   transit, and two of them are this library's subject exactly:

     89       high bit set: dies if the path is 7-bit clean
     50       P
     4e       N
     47       G
     0d       CR: gone if LF was translated to CRLF
     0a       LF: gone if CRLF was translated to LF
     1a       DOS end-of-file, so TYPE stops here
     0a       LF again, catching the reverse translation

   A PNG that went through an FTP client in text mode arrives with its
   CR/LF pair rewritten and fails on byte five, before any decoder has
   to guess what went wrong. That is a file format defending itself
   against the line-ending problem -- the same one that turns a CSV
   into a column of blank rows.

The PNG header is not an arbitrary example

The last two sections decode the first bytes of a PNG, and the file is built in the program — length computed, CRC computed with zlib.crc32 — so it is a real header rather than a quoted one.

It earns its place here because its signature is an encoding test. Six of those eight bytes exist to catch a file that was transformed in transit, and two of them are this library's subject exactly: the 0D 0A pair is there so that a client which "helpfully" translated line endings destroys the signature and the file fails immediately, instead of arriving as a plausible-looking image that no decoder can read. The trailing 0A catches the reverse translation, and the leading 89 — high bit set — dies on a 7-bit-clean path.

That is a file format defending itself against the line-ending problem, designed in by people who had been bitten. Most formats do not, which is why the rest of this library exists.

Base64 is a different thing in the same drawer

b.hex() and base64.b64encode(b) both turn bytes into text you can put in a JSON field, and they are not the same kind of operation as .decode('utf-8'). Hex and Base64 are text encodings of arbitrary bytes — they never fail, they never need to know what the bytes mean, and they cost you size (hex doubles; Base64 is four characters per three bytes). A character encoding is the opposite: it does need to know what the bytes mean, and it can refuse.

That distinction, the six-bits-per-character arithmetic, and the padding rules are Binary to text. For a hex dump specifically, prefer bytes.hex() over base64.b16encode — same encoding, and b16decode refuses lowercase.

If you are coming from Python or ABAP

Python. Three habits. First, struct's byte-order character is not optional in practice: leave it off and you get native order and native alignment padding, so struct.calcsize('HI') is 8 where '<HI' is 6 — a format string that works on your machine and produces a differently-sized record on someone else's. Always lead with <, >, or ! (which is >, spelled "network order"). Second, int.to_bytes raises OverflowError rather than truncating, which is the one place these conversions check up on you — and signed=True is required for a negative, in both directions. Third, for anything with more than a few fields, struct.Struct('<IIH') compiles the format once and is both faster and self-documenting; memoryview lets you slice a large buffer without copying it, which matters exactly when you are parsing a file big enough to care.

ABAP. (Not machine-checked — CI cannot run ABAP.) The type is xstring for a variable-length byte string and x / raw for fixed-length, and the hex picture is not a conversion at all — an xstring written into a character field is already its own hex representation, which is why xstring values print as C3A9 with no function call. The genuine conversions are cl_abap_conv_in_ce / cl_abap_conv_out_ce between xstring and string (newer systems: cl_abap_codepage), and there you must name the code page, exactly as Python's .decode() takes an encoding. The number half needs no call at all, because a plain assignment is from_bytes with the byte order welded shut: moving an x or xstring into an i reads its last four bytes as a signed big-endian number ↗, zero-filled on the left when there are fewer — int.from_bytes(b[-4:].rjust(4, b'\0'), 'big', signed=True) in Python's terms — so '0010' is 16, 'FFFFFFFF' is -1, and a two-byte 'FFFF' is 65535, because the fill is zeros rather than the sign. int8 does the same with eight. Bytes in front of the last four are ignored rather than refused, and a little-endian field has to be byte-reversed before the assignment, because nothing in the statement lets you say otherwise. The assignment back the other way writes big-endian too ↗, and truncates on the left where to_bytes would raise OverflowError. The application server's own order is a separate fact — sys.byteorder is spelled cl_abap_char_utilities=>endian ↗ — and it shows through exactly where struct's native order does, once you stop converting and start reinterpreting: ASSIGN … CASTING lays a type over the bytes where they lie, and SAP warns that the result can depend on the platform ↗, because the internal byte order becomes visible in the cast. So 00 00 00 0D is 13 through an assignment on every server and 218103808 through a cast on a little-endian one — the PNG header's two readings from above, with no > or < in sight to say which one you wrote. Slice fields out with +off(len), which makes the width explicit, and assign them rather than cast them. Bit work is GET BIT / SET BIT and the BIT-AND / BIT-OR / BIT-XOR / BIT-NOT operators ↗, on x and xstring only. Verify any code-page number you find in a document against the system that will run the job.

Try it

  • Take a binary file you actually have — a PNG, a ZIP, a .class, a Parquet footer — and head -c 16 file | xxd, then decode those bytes field by field against the format's spec with struct.unpack. Getting > versus < wrong is the fastest way to learn which one the format uses.
  • python3 -c "import sys; print(open(sys.argv[1],'rb').read(8).hex(' '))" yourfile against a dozen files of different types. Most formats announce themselves in the first four bytes, and file guesses is doing exactly this with a bigger table.
  • Round-trip a negative number: pick one, to_bytes(4, 'big', signed=True), then read it back without signed=True. The number you get is the bug that reaches a database column.
  • git grep -n "int(.*, *16)" in a project you maintain, and check each hit for whether the width mattered. The ones parsing a fixed-width field are the ones to change.

Practice

Six bytes, and a format spec you have not been given. The bytes are 00 00 00 0d 49 48. Predict bytes.fromhex round-tripping, then: int.from_bytes of the first four as big-endian and as little-endian, what the last two are as ASCII, and which of struct.unpack('>I', ...) and struct.unpack('<I', ...) on the first four gives the smaller number.

Then the one that is not arithmetic: int.from_bytes(b'\xff\xff\xff\xff') and the same call with signed=True give two answers, both correct. Say what a format spec must state for only one of them to be right, and what happens if it does not.

Answers

Verified output of bytes_hex_and_int_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

SIX BYTES
   the bytes                              00 00 00 0d 49 48
   RAW.hex()                              '0000000d4948'
   bytes.fromhex(RAW.hex()) == RAW        True

   The round trip is exact because neither call is arithmetic.
   fromhex() also skips ASCII spaces, so a dump pasted straight out
   of xxd goes back in:
   bytes.fromhex('00 00 00 0d') == HEAD   True

THE FIRST FOUR AS A NUMBER
   int.from_bytes(HEAD, 'big')            13
   int.from_bytes(HEAD, 'little')         218103808

   struct.unpack('>I', HEAD)[0]           13
   struct.unpack('<I', HEAD)[0]           218103808
   '>I' agrees with 'big':                True
   the smaller answer is:                 >I / big

   Big-endian reads the significant byte first, so three leading
   zeros mean a small number. Little-endian reads them last, where
   they are the TOP three bytes -- so the same four bytes become
   thirteen or two hundred and eighteen million, and nothing about
   the bytes prefers either reading.

THE LAST TWO AS TEXT
   TAIL                                   b'IH'
   TAIL.decode('ascii')                   'IH'
   ...and as a number, 'big':             18760

   Both readings are available and only one is intended. These six
   bytes are the start of a PNG's IHDR chunk: a big-endian length
   followed by a four-byte ASCII type, of which we have two letters.
   A length field and a name field, side by side, distinguishable
   only by a spec.

FOUR BYTES THAT ARE TWO NUMBERS
   int.from_bytes(FF, 'big')              4294967295
   int.from_bytes(FF, 'big', signed=True) -1
   both round-trip to the same bytes:     True

   A format spec has to state THREE things before four bytes are one
   number: the width, the byte order, and whether it is signed.
   Miss any one and the bytes still decode -- to a different value,
   with no error and no clue. The signed case is the meanest of the
   three, because the two answers agree for every value below 2^31
   and diverge only once the top bit is set: a counter that has been
   correct for years starts reporting -1 the day it crosses over.

   2147483647 both ways agree:            True
   2147483648 both ways agree:            False

See also