Skip to content

str is not bytes

Level: 101 · for anyone starting from zero

One line: Python 3 has two types for "a sequence of characters" and refuses to mix them, so the question "is this text or is this data?" has to be answered at the moment the value is created rather than at the moment it breaks.

Most languages let you drift. C hands you char * and leaves the meaning to you; ABAP has string and xstring but converts between them with a helper you can call almost anywhere. Python 3 made a harder choice: str holds code points, bytes holds numbers 0–255, and the only ways across are .encode() and .decode(). Nothing is implicit. That is why porting Python 2 code hurt, and it is also why a Python 3 program that runs on your machine tends to run in Warsaw too.

The practical consequence is that len() answers a different question depending on which type you are holding — and for anything outside ASCII, the two answers differ.

Verified output of str_is_not_bytes_py.py — regenerated by tools/run_examples.py, never hand-typed.

1. TWO TYPES, ONE WORD
     'Łódź'                   str    len = 4
     b'\xc5\x81\xc3\xb3d\xc5\xba' bytes  len = 7

     Same word. Different length. len() is not one question.

2. WHAT len() COUNTS
     str   -> code points
     bytes -> bytes

     Ł  U+0141  2 byte(s)  C5 81
     ó  U+00F3  2 byte(s)  C3 B3
     d  U+0064  1 byte(s)  64
     ź  U+017A  2 byte(s)  C5 BA

3. INDEXING GIVES YOU DIFFERENT THINGS
     TEXT[0] = 'Ł'        a one-character str
     DATA[0] = 197        an int -- the number 0-255
     DATA[0:1] = b'\xc5'  slicing bytes gives bytes

     Iterating bytes gives ints, not one-byte bytes. This is the
     single most surprising line in the whole type.

4. THE BOUNDARY PYTHON REFUSES TO CROSS
     "a" + b"b"                                   -> TypeError: can only concatenate str (not "bytes") to str
     "Łódź" == b"\xc5\x81\xc3\xb3d\xc5\xba"       -> False
     b"x".startswith("x")                         -> TypeError: startswith first arg must be bytes or a tuple of bytes, not str

     The middle one is the trap: it does not raise, it just says False.
     A str is never equal to a bytes, however identical they look.

5. THE ONLY TWO DOORS BETWEEN THEM
     TEXT.encode('utf-8')  -> b'\xc5\x81\xc3\xb3d\xc5\xba'
     DATA.decode('utf-8')  -> 'Łódź'
     round trip is exact:     True

     .encode() is only on str. .decode() is only on bytes. The method
     you can reach tells you which type you are holding.

Three things in that run are worth pausing on.

len() is not one question. Łódź is four characters and seven bytes. Neither number is wrong; they answer "how many characters?" and "how much disk?". Counting characters shows there are two more answers after these.

Indexing bytes gives you an int. DATA[0] is 197, not b'\xc5'. This catches everyone once: iterating a bytes yields numbers, and the only way to get a one-byte bytes back is to slice, DATA[0:1]. It follows from bytes being a sequence of numbers, which is what it says on the tin, but it reads as a surprise every time.

The comparison does not raise — it returns False. Concatenating str and bytes is a TypeError, loud and immediate. Comparing them is quiet: "Łódź" == b"..." is simply False, always, no matter what the bytes hold. A str is never equal to a bytes. This is the one that reaches production, usually as a config value read in binary mode that silently never matches.

If you are coming from ABAP

str is string and bytes is xstring, and the mapping is closer than you might expect: ABAP also refuses to compare them directly, and also makes you name a code page to convert (cl_abap_conv_codepage=>create_out( ) and friends). What ABAP leaves to habit and Python enforces is the default: an ABAP program that never thinks about code pages inherits the system's, whereas Python 3 has no default conversion at all — you cannot accidentally get one. (Not machine-checked — CI cannot run ABAP. Verify any specific code-page number against the system rather than trusting a page.)

Coming from C, the difference is starker still: char * is bytes with no str anywhere in the language, which is why the Unicode-aware C in this repo's sibling ↗ has to do by hand what .decode() does in one call.

Try it

  1. Predict len("😀") and len("😀".encode("utf-8")) before you run them. Then try "😀"[0] — does it give you the emoji or half of it? (Python's answer differs from JavaScript's here, and the reason is in UTF-16 and surrogates ↗.)
  2. Read a file two ways — open(path) and open(path, "rb") — and compare the len() of each result. When are they equal?
  3. Find the bug: if user_input == b"quit": where user_input came from input(). What happens, and why is there no traceback?

Practice

Eight expressions, one word, two types. Take Łódź held both ways:

TEXT = "Łódź"
DATA = TEXT.encode("utf-8")

For each of the eight, write down the type of the result and its value — or, if it raises, the name of the exception. No running anything first.

  1. len(TEXT)
  2. len(DATA)
  3. DATA[0]
  4. DATA[0:1]
  5. TEXT == DATA
  6. TEXT[0] + DATA[0:1]
  7. b"x".startswith("x")
  8. DATA.decode() == TEXT

Then the question the eight are really for: which one is the dangerous one — and why is it not one of the two that raise?

Answers

Verified output of str_is_not_bytes_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.

     TEXT = 'Łódź'          DATA = TEXT.encode('utf-8')

     expression               type     value      note
     ----------------------------------------------------------------------------
     len(TEXT)                int      4          characters, and the word has four
     len(DATA)                int      7          bytes, and UTF-8 spent two on each of Ł ó ź
     DATA[0]                  int      197        indexing bytes gives a NUMBER
     DATA[0:1]                bytes    b'\xc5'    slicing bytes gives bytes -- the only way back to one byte
     TEXT == DATA             bool     False      the quiet one
     TEXT[0] + DATA[0:1]      raises   TypeError  the loud one
     b'x'.startswith('x')     raises   TypeError  even the method arguments are typed
     DATA.decode() == TEXT    bool     True       the round trip is exact

     Two of the eight raise. The other six hand back a value, and
     one of those six is wrong -- which is the whole lesson.

     WHICH IS THE DANGEROUS ONE?
     Not the TypeErrors. An exception is a bug that has already been
     found. It is line 5: TEXT == DATA is False, always, for every
     text and every bytes -- and False is a perfectly good answer that
     a program will act on. A str is never equal to a bytes, however
     identical the two look on screen.

     TEXT == DATA                 False
     TEXT == DATA.decode()        True   <- what you meant
     TEXT.encode() == DATA        True   <- or this

     The fix is not a cast. It is deciding, at the boundary where the
     value arrived, which of the two types this program holds it in.

See also