String literals¶
Level: 201 · for Python programmers
One line: Everything a literal does happens at compile time — the prefix picks the type and the escaping rules, and an escape Python does not recognise is kept rather than rejected, which is why re.search('\bfoo\b', 'a foo b') finds nothing and never says why.
A literal looks like the least interesting thing in the language: you type some characters between quotes and get those characters. But there are nine legal prefixes, five ways to spell one code point, two numeric escapes with different widths, and one rule — the fate of an unknown escape — that quietly decides whether your regular expressions work.
It is worth separating two things that get taught together. The prefix answers three independent questions: what type comes out (b gives bytes), whether backslashes mean anything (r says no), and whether the braces are code (f says yes). The escapes are a separate grammar underneath, and it is not the same grammar for str and for bytes — because three of the escapes name a code point, and a byte does not have one.
Verified output of string_literals_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. NINE PREFIXES, AND THE ONES THAT DO NOT EXIST
prefix type what it changes
--------------------------------------------------------------
'' (none) str nothing -- this is the plain form
'r' str backslash sequences are NOT interpreted
'b' bytes the result is bytes; source must be ASCII
'f' str braces are expressions, evaluated at run time
'u' str nothing at all -- a no-op kept for 2-to-3 ports
'rb' bytes both of the above
'rf' str raw and f, which do compose
'br' bytes both, spelled the other way round
'fr' str the same, other order
9 valid, 8 rejected: bf bu fb fu ru ub uf ur
Read the rejected list as two rules. 'u' combines with nothing --
it is a compatibility no-op and was never meant to be useful. And
'b' and 'f' cannot meet: an f-string is built at run time out of
str pieces, and there is no such thing as an f-bytes literal.
Case does not matter: 8 of 8 uppercase spellings compile,
B'x' and Rb'x' and BR'x' among them.
2. FIVE LITERAL SPELLINGS OF ONE NUMBER, AND ONE THAT IS NOT PYTHON
how written value len
------------------------------------------------------------------------
the character itself 'é' é 1
hex escape, 2 digits '\xe9' é 1
hex escape, 4 digits '\u00e9' é 1
hex escape, 8 digits '\U000000e9' é 1
by name '\N{LATIN SMALL LETTER E WITH ACUTE}' é 1
built at run time chr(0xE9) é 1
all six are the same object value: True
U+00E9 is a sixth notation and it is NOT Python syntax. It is the
Unicode standard's way of NAMING the code point; 0xE9 is an integer
literal that happens to equal it; and 'é' is source text. Same
number, three jobs -- and only two of the three are code.
3. TWO NUMERIC ESCAPES, TWO DIFFERENT WIDTHS
literal value len why
------------------------------------------------------------------
'\x41' 'A' 1 exactly two hex digits, always
'\xA1A' '¡A' 2 the A is data -- \x stopped after two
'\101' 'A' 1 octal, and 101 octal is 65
'\1010' 'A0' 2 octal took three digits, then '0' is data
'\10' '\x08' 1 octal will take one, two or three
'\0' '\x00' 1 NUL -- and it is octal zero, not a special case
Python has one fixed-width numeric escape and one greedy one, in
the same grammar. '\x' can never swallow a following hex digit;
'\NNN' can, and does. If you write octal at all, write all three
digits -- which is the same advice other languages have to give
about their \x, because theirs is the greedy one.
4. AN UNKNOWN ESCAPE IS KEPT, NOT REJECTED -- AND THAT IS THE TRAP
source value backslash survived?
------------------------------------------------
'\\d' '\\d' True
'\\w' '\\w' True
'\\s' '\\s' True
'\\b' '\x08' False
'\\A' '\\A' True
'\\Z' '\\Z' True
'\\n' '\n' False
'\\t' '\t' False
Five of those eight are regex syntax that arrives at re intact
purely because Python did not recognise them. '\b' is the
exception, and it is the one every real pattern uses:
cooked pattern '\x08foo\x08' on 'a foo b' -> None
raw pattern '\\bfoo\\b' on 'a foo b' -> (2, 5)
The cooked pattern asked for a BACKSPACE character on both sides
of 'foo' and found nothing. It did not raise; it just never
matches. This is the whole reason regex patterns are written r''
-- not style, and not about the backslashes you can see.
Since Python 3.12 an unrecognised escape is a SyntaxWarning, and
it is documented as becoming an error in a future release. Until
then '\d' works and '\b' silently does not.
5. RAW DOES NOT MEAN 'NO BACKSLASH RULES'
r'\"' is '\\"', length 2
-- the backslash still escaped the quote, so the string did not
end there. It just also stayed in the result.
a raw literal ending in ONE backslash -> SyntaxError
So a raw string cannot end in an odd number of backslashes, which
is why r'C:\Users\' does not compile and every Windows path
example ends one character short of where you wanted it.
And a code-point escape is not one either: r'\u0041' is '\\u0041', 6 characters.
6. A bytes LITERAL IS ASCII-ONLY, AND HAS A SMALLER ESCAPE SET
escape in a str in a bytes
--------------------------------------------------------------
'\\n' '\n' b'\n'
'\\x41' 'A' b'A'
'\\101' 'A' b'A'
'\\u0041' 'A' b'\\u0041'
'\\U00000041' 'A' b'\\U00000041'
'\\N{LATIN CAPITAL LETTER A}' 'A' b'\\N{LATIN CAPITAL LETTER A}'
b'e-acute' written directly -> SyntaxError
The three that survive as text are the three that name a CODE
POINT, and a byte does not have one. So they are not escapes
inside a bytes literal at all -- they stay as a backslash and a
letter, and since 3.12 they warn. A bytes literal is ASCII-only
no matter what encoding the source file declares, because it is
spelling out bytes and the source encoding is not one of them.
7. TWO STRINGS SIDE BY SIDE ARE ONE STRING
'spam' 'eggs' -> 'spameggs', length 8
Adjacent literals are concatenated by the COMPILER, so there is
no run-time cost and no '+' -- which is how a long literal gets
wrapped over several lines inside brackets.
a list with one comma missing -> ['spam', 'eggsbeans', 'toast']
len(menu) is 3, and nobody typed a concatenation.
That is the cost of the feature: a missing comma in a list of
strings is not a syntax error, it is a shorter list. Linters look
for it precisely because the language cannot.
(No answer above depends on which Python you ran it under: this
grammar has been this shape since 3.6, and the file prints the
same bytes on 3.11, 3.12, 3.13 and 3.14.)
What the run shows¶
Nine prefixes exist and eight two-letter combinations do not, and the rejected list is more informative than the accepted one. u combines with nothing at all: it is a no-op kept so Python 2 code could be ported without touching every literal, and it means exactly nothing today. And b and f cannot meet — there is no f-bytes literal, because an f-string is built at run time by joining str pieces, so the thing it produces is a str by construction. If you need an interpolated bytes, you build the str and encode it, or you use % on bytes (the format page has the reason % survived for exactly this).
One code point, five literal spellings and a sixth notation that is not Python. 'é', '\xe9', '\u00e9', '\U000000e9' and '\N{LATIN SMALL LETTER E WITH ACUTE}' are one character each and compare equal; chr(0xE9) builds the same thing at run time. U+00E9 is the sixth, and it is the Unicode standard's way of naming the code point — it is not syntax in any language. 0xE9 is an integer literal that happens to equal the same number. Same number, three jobs, and only two of the three are code. (This is the answer to "I thought Unicode started with U+ and here is an example with 0x": both are correct, in documents that are doing different things.)
Two numeric escapes, two different widths, in the same grammar. \xNN is exactly two hex digits and can never take a third, so '\xA1A' is two characters — ¡ and a literal A. \NNN is octal and takes one, two or three digits, greedily, so '\1010' is 'A0': the escape swallowed three digits and the fourth became data. If you write octal at all, write all three digits. And '\0' is not a special NUL escape — it is octal zero, which is why '\00' and '\000' are the same character.
An unrecognised escape is kept, not rejected — and that is the trap. '\d' is a backslash followed by d, two characters, because Python looked at \d, did not know it, and left it alone. Five of the eight regex escapes in the run survive that way, which means a pattern written in a plain string usually works. '\b' is the exception, and it is the one every real pattern uses: \b is a Python escape — backspace, U+0008 — so '\bfoo\b' asks the regex engine for a backspace character on either side of foo, matches nothing, and raises nothing. That is the actual reason regex patterns are written r''. Not style, and not about the backslashes you can see — about the one escape in the middle of the list that Python happens to claim. Since 3.12 an unrecognised escape is a SyntaxWarning and the docs say it will become an error; until then, '\d' works and '\b' silently does not.
Raw does not mean "no backslash rules". In a raw literal the backslash is still consumed for the purpose of finding the closing quote, it just also stays in the result: r"\"" is two characters, a backslash and a quote. The consequence is the famous one — a raw string cannot end in an odd number of backslashes, so r'C:\Users\' does not compile and every Windows-path example in every tutorial stops one character short. \u is not an escape in a raw string either: r'\u0041' is six characters.
A bytes literal is ASCII-only and has a smaller escape set. b'é' written directly is a SyntaxError regardless of what encoding the source file declares, because the literal is spelling out bytes and the source encoding is not one of them. And \u, \U and \N{} — the three escapes that name a code point — are simply not escapes inside a bytes literal: b'\u0041' is six bytes, a backslash and u0041. That is the type boundary showing up in the grammar rather than in a method: str is not bytes says a byte has no character; here the compiler says a byte has no code point either.
And two literals side by side are one literal. "spam" "eggs" is 'spameggs', joined by the compiler, with no + and no run-time cost — which is how a long string gets wrapped across lines inside brackets. The price is in the run's last block: a missing comma in a list of strings is not a syntax error, it is a shorter list. ['spam', 'eggs' 'beans', 'toast'] has three elements and nobody typed a concatenation. Linters look for it because the language cannot.
One note on the program itself: it builds several of its literals at run time with eval() rather than writing them in the file. That is not indirection for its own sake — escape processing happens when the file is compiled, so a file containing '\d' would emit its SyntaxWarning at import, before any of the lesson ran, and on 3.11 it would emit nothing at all. Building the source text out of chr(92) is the only way to show an invalid escape without the page's own file containing one, and it is a small demonstration of the point: the escape is not a run-time feature.
The Rust and C view¶
The interesting question across languages is how an escape knows where to stop, and there are three different answers to it — one of which is Python's, twice.
question Python Rust C (clang 21)
----------------------------------------------------------------------------------------------------------
how wide is a hex escape \xNN — exactly 2 \xNN — exactly 2 \x… — UNBOUNDED
'\x41' then 'A' '¡A' from '\xA1A' (2) "AA" from "\x41A" (2) "\x41A" does not compile
"hex escape out of range"
a hex escape above 0x7F '\xe9' is fine "\xff" does NOT compile fine (it is a byte)
— use \u{ff} in a str
naming a code point \u 4 digits, \U 8 digits \u{…} — braced, 1 to 6 \u 4 digits, \U 8
octal \NNN — 1 to 3, greedy none at all \NNN — 1 to 3, greedy
an unknown escape kept; SyntaxWarning 3.12+ compile error warning (clang), UB
raw strings r"…" — backslash still r"…" and r#"…"# none (C++11 has R"(…)")
ends the literal — r"C:\Users\" COMPILES
adjacent literals join yes, at compile time no — syntax error yes, at compile time
Rust's \u{…} is the design that removes the question. Braces delimit the escape, so it needs no fixed width and cannot be greedy — one to six digits, unambiguous either way. Python fixed the width of \x and got the same safety for two digits; C left \x unbounded and gets "hex escape sequence out of range" on "\x41A", which is a diagnostic about the value when the actual problem is that the compiler could not tell where you meant to stop. C#, from the docs that prompted this page, is in the same family: its \x takes one to four digits, so \xA1A is ਚ rather than ¡A, and the documentation carries a warning telling you to always write four.
Two rows go the other way and are worth not glossing over. Python's raw string is the weaker one: r'C:\Users\' is a SyntaxError in Python and r"C:\Users\" compiles in Rust, because Rust's raw string does not treat the backslash as special at all, not even for finding the closing quote — and r#"…"# lets you nest quotes by counting hashes. Second, Rust rejects an unknown escape at compile time, so the regex trap in section 4 cannot exist there: "\d" does not compile, and a Rust regex crate takes the raw string or nothing.
The sibling library owns the Rust side of this: raw strings and escapes ↗. And the four-language treatment of the escape itself — the same code point written five ways, and what the shape of an escape says about what a language thinks a character is — is writing a code point ↗ in the encodings library.
If you are coming from ABAP¶
The shape of the comparison is unusual here: ABAP has essentially no escape grammar, so almost everything above is new rather than different. A quote inside a literal is doubled rather than backslashed, and the choice you make at the quote mark is a type choice, not an escaping one — '…' is type c with trailing blanks trimmed, `…` is type string with them kept, and |…| is a string template. There is no \n: a newline comes from cl_abap_char_utilities=>newline, which is why that class exists at all and why it is the first thing anyone learns about ABAP text. Coming to Python, the two habits to change are that the quote style here does not change the type ('x' and "x" are identical, and the choice is only about which quote you need inside), and that a backslash now means something — so a Windows path or a regex pasted from somewhere else is no longer inert text. Coming the other way, the habit to keep is that ABAP made you name the control character; '\x1c' in Python is a number, and a named constant beside it is worth the line. String templates do have three escapes of their own (\|, \\, \{) — verify the exact list on your own system, since template syntax is 7.02+ and has grown. (Not machine-checked — CI cannot run ABAP.)
Try it¶
- Take a Windows path and write it four ways: plain with doubled backslashes, raw, with forward slashes, and via
pathlib. Then try to writeC:\Users\as a raw literal and work out from section 5 exactly why the error message is about the quote and not about the backslash. - Section 4 shows five regex escapes surviving and one not. Find the others: loop over the ASCII letters, build
chr(92) + letterat run time, and print every letter Python claims. How many are there, and which of them are also regex syntax? '\N{...}'needs the Unicode name table at compile time. Find a character added in a recent Unicode version and try its name on two Pythons built against different editions — this is the table has a version ↗ reaching all the way into the parser.- Implicit concatenation costs nothing at run time. Prove it: compile
"spam" "eggs"and"spam" + "eggs"withcompile()and comparedis.dison both. Which one has an instruction the other does not — and does the answer change if the two pieces are variables? b'\u0041'is six bytes. Write the code that turns'\u0041'into the one byteb'A', and then say which of the two directions in that sentence needed an encoding argument and why.
Practice¶
Eight literals: how long is each one? Write down len() and the value for each. Every answer is one or two characters except where a backslash survived.
'\xA1A''\1010''\d''\b'r'\u0041'b'\u0041''spam' 'eggs''\0'
Then: one of the eight is the reason every regular expression in Python is written r''. Which one, what does the cooked version of '\bfoo\b' actually ask the engine for, and what does re.search return when you give it that?
Answers
Verified output of string_literals_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.
written len value why
------------------------------------------------------------------------------
'\xA1A' 2 '¡A' \x is exactly two digits, always
'\1010' 2 'A0' octal is greedy: it took three
'\d' 2 '\\d' unknown escape -- the backslash is KEPT
'\b' 1 '\x08' known escape -- BACKSPACE, U+0008
r'\u0041' 6 '\\u0041' raw: not an escape at all
b'\u0041' 6 b'\\u0041' bytes: a byte has no code point
'spam' 'eggs' 8 'spameggs' adjacent literals join at compile time
'\0' 1 '\x00' octal zero, not a special NUL escape
Two escapes, two widths, one grammar. Python has exactly one
fixed-width numeric escape and one greedy one, and the greedy
one is the one nobody writes on purpose. If you write octal at
all, write all three digits.
WHICH ONE BREAKS A REGULAR EXPRESSION?
pattern written '\bfoo\b' subject 'a foo b'
cooked -> '\x08foo\x08' re.search -> None
raw -> '\\bfoo\\b' re.search -> (2, 5)
'\d', '\w', '\s', '\A' and '\Z' all reach re intact --
purely because Python did not recognise them and left the
backslash alone. '\b' is the exception, and it is the one
every real pattern uses: Python claims it for BACKSPACE, so the
cooked pattern asked for a backspace character on both sides of
'foo'. It found nothing and it raised nothing. That is the
actual reason regex patterns are written r'' -- not style, and
not about the backslashes you can see.
THE ONE THAT IS NOT A LITERAL AT ALL
Line 7. 'spam' 'eggs' is one string, joined by the COMPILER,
with no '+' and no run-time cost. The price is what a missing
comma does to a list:
['spam', 'eggs' 'beans', 'toast'] -> ['spam', 'eggsbeans', 'toast']
len is 3, and nobody typed a concatenation. Not a syntax
error -- a shorter list. Linters look for it because the
language cannot.
FIVE SPELLINGS OF ONE CODE POINT
'é'
'\xe9'
'\u00e9'
'\U000000e9'
'\N{LATIN SMALL LETTER E WITH ACUTE}'
all five are the same one-character string: True
U+00E9 is a sixth notation and it is NOT Python syntax -- it is
the Unicode standard NAMING the code point. Same number, three
jobs, and only two of the three are code.
See also¶
stris notbytes— the boundary that section 6 shows the compiler enforcing- The format mini-language — what the
fprefix does after the parsing is over repris notstr— the other direction: turning a value back into something that looks like a literal- Making a
bytesobject — the four constructors, for when a literal will not do - What to write next — this page closes #3 and #32
- The crosswalk — which idea lives in which library
- String and bytes literals ↗ — the grammar this page is unpacking, in the language reference rather than the library docs
- Writing a code point ↗ — the same escapes in four languages, and what their shape reveals