Text and binary are both bytes¶

Level: 101 → 201 · for anyone starting from zero

One line: A file and a socket carry bytes and nothing else, so "text" is not a kind of file — it is a promise that the bytes are printable ones and a newline you agreed on, and "binary" is the absence of that promise; the number 1000 makes the split concrete, because as text it is four printable bytes and as an integer it is four different bytes, one of them a NUL, sharing nothing in common.

Programmers say "a text file" and "a binary file" as if they were two kinds of thing. They are not. There is one kind of thing — a sequence of bytes — and "text" is a claim about which bytes are in it: values in a printable range, broken into lines by an agreed newline. Nothing in the file enforces the claim. A tool that shows you text is applying the claim; a tool that shows you hex has dropped it. Seeing that clearly is the start of every wire-format decision in this chapter.

The same number, two ways¶

text_and_binary_c.c writes 1000 as text and as a 4-byte integer, and dumps the bytes of each:

text_and_binary_c.c in full — pasted here by tools/run_examples.py from the file CI runs.

/* A file and a socket carry bytes; nothing else. "Text" is a promise about
   which bytes -- printable ones, and a newline you agree on -- and "binary" is
   the absence of that promise. The number 1000 shows the split: four bytes as
   text, four as a raw integer, and they share not one byte in common. */
#include <stdio.h>
#include <string.h>
#include <stdint.h>

static void dump(const char *label, const void *p, size_t n)
{
    const unsigned char *b = p;
    printf("%-28s", label);
    for (size_t i = 0; i < n; i++)
        printf(" %02x", b[i]);
    printf("\n");
}

int main(void)
{
    /* As text: the digits '1' '0' '0' '0', every byte in the printable range. */
    char text[16];
    int len = snprintf(text, sizeof text, "%d", 1000);
    printf("1000 as text \"%s\": %d bytes\n", text, len);
    dump("  bytes:", text, (size_t)len);

    /* As a 4-byte integer, big-endian (the order agreed on the wire). One of
       the bytes is 0x00 -- a NUL, which ends a C string and stops text tools. */
    uint32_t n = 1000;
    unsigned char raw[4] = { n >> 24, n >> 16, n >> 8, n };
    printf("1000 as a big-endian int32: 4 bytes\n");
    dump("  bytes:", raw, sizeof raw);
    printf("  a NUL (00) is in there: %s -- so this is not text\n",
           memchr(raw, 0, sizeof raw) ? "yes" : "no");

    /* A newline is one byte, 0x0a. That is the only structure a text file has;
       CR+LF is two bytes, and which one a line ends with is a convention, not
       a property of the file. */
    printf("'\\n' is one byte: %02x   \"\\r\\n\" is two: ", (unsigned char)'\n');
    dump("", "\r\n", 2);
    return 0;
}

Verified output of text_and_binary_c.c — regenerated by tools/run_examples.py, never hand-typed.

1000 as text "1000": 4 bytes
  bytes:                     31 30 30 30
1000 as a big-endian int32: 4 bytes
  bytes:                     00 00 03 e8
  a NUL (00) is in there: yes -- so this is not text
'\n' is one byte: 0a   "\r\n" is two:                              0d 0a

As text, 1000 is 31 30 30 30 — the ASCII codes for the digits 1 0 0 0, four bytes, every one of them in the printable range, and every one self-describing: a reader that knows ASCII can read them without being told the width. As a big-endian 4-byte integer it is 00 00 03 e8 — the same value, no byte in common with the text form, and one of the bytes is 00. That NUL is the tell. It ends a C string, and it is why running a binary file through a text tool truncates it at the first zero: the tool is applying the text promise to bytes that never made it.

The newline is the only structure a text file has, and it is one byte, 0a. \r\n is two bytes, 0d 0a, and which one ends a line is a convention between writer and reader — not a property the bytes carry. Two machines that disagree about it produce the crlf problems the encodings library documents ↗, and that is before any character above ASCII is involved.

Python makes the split a type¶

Python will not let text and bytes mix by accident: str goes through a codec on the way out, and bytes goes out verbatim. text_and_binary_py.py:

text_and_binary_py.py in full — pasted here by tools/run_examples.py from the file CI runs.

"""Python makes the split a type: text goes through str and a codec, bytes go
out as they are. open() in "w" mode encodes for you; "wb" does not, and hands
back exactly the bytes you wrote. The same number, both ways, shares no byte."""
n = 1000
as_text = str(n).encode("utf-8")
as_int = n.to_bytes(4, "big")
print(f"1000 as text: {as_text!r}  ({len(as_text)} bytes)")
print(f"1000 as int32 big-endian: {as_int!r}  ({len(as_int)} bytes)")
print(f"a NUL byte in the integer form: {(0 in as_int)}")

# "text mode" is str + an encoding; "binary mode" is bytes, verbatim.
import io
text_file = io.StringIO()
text_file.write("1000")
print(f'open(..., "w").write(\"1000\") stored the str {text_file.getvalue()!r}')

binary_file = io.BytesIO()
binary_file.write(as_int)
print(f'open(..., "wb").write(int) stored the bytes {binary_file.getvalue()!r}')

# The one byte a text file agrees on -- and the two-byte version Windows uses.
print(rf"newline '\n' is {chr(10).encode()!r}; Windows '\r\n' is {chr(13)+chr(10)!r}"
      f" = {(chr(13)+chr(10)).encode()!r}")

Verified output of text_and_binary_py.py — regenerated by tools/run_examples.py, never hand-typed.

1000 as text: b'1000'  (4 bytes)
1000 as int32 big-endian: b'\x00\x00\x03\xe8'  (4 bytes)
a NUL byte in the integer form: True
open(..., "w").write("1000") stored the str '1000'
open(..., "wb").write(int) stored the bytes b'\x00\x00\x03\xe8'
newline '\n' is b'\n'; Windows '\r\n' is '\r\n' = b'\r\n'

str(1000).encode() is b'1000'; (1000).to_bytes(4, "big") is b'\x00\x00\x03\xe8'. Opening a file in "w" mode is the text promise made explicit — it takes a str and encodes it — and "wb" mode is bytes, verbatim, which is the mode you use the moment the promise does not hold. C has the same distinction on Windows, where a file opened in text mode translates \n to \r\n on the way out; on Unix, C's text and binary modes are identical, which is a large part of why the difference is invisible to so many programmers until a file crosses an operating system.

If you are coming from another language¶

Rust. &str is UTF-8 by type and &[u8] is bytes, the same split as Python's, checked at compile time — String::from_utf8 is the door between them, and it returns an error rather than a guess. The encodings library's Rust chapter ↗ is the full treatment.

ABAP. (Not machine-checked — CI cannot run ABAP.) string is text and xstring is bytes, and the conversion between them names a code page — the promise, stated. Reading a file IN TEXT MODE versus IN BINARY MODE is the same choice C makes, with the same newline translation attached to text mode.

Text and binary are both bytes¶

The same number, two ways¶

Python makes the split a type¶

If you are coming from another language¶

See also¶