strip is a set, not a prefix¶
Level: 201 · for Python programmers
One line: 'Arthur: three!'.lstrip('Arthur: ') is 'ee!' — the argument is a bag of characters, not a prefix — and the three ways it differs from removeprefix can each bite on their own.
The docs say it plainly: "The chars argument is not a prefix; rather, all combinations of its values are stripped." That sentence is easy to read past, because the call site looks exactly like the thing it is not. lstrip('Arthur: ') reads as "remove 'Arthur: ' from the front", and what it means is "remove leading characters as long as they are any of A r t h u : ␠". The word after lstrip( is a string only because Python has no character-set literal.
This is the most quietly destructive method in str, because it usually works. The eight-character argument in the example above removes the eight characters you meant — and then keeps going, and takes t h r with it, and hands back a string that still looks like a plausible answer.
Verified output of strip_is_a_set_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. THE BUG, IN ONE LINE
line 'Arthur: three!'
line.lstrip('Arthur: ') 'ee!' <- not what anyone meant
line.removeprefix('Arthur: ') 'three!'
The 'thr' went with it. lstrip did not look for the prefix at
all -- it was handed a bag of characters and it ate every
leading character that was in the bag.
2. THE ARGUMENT IS A SET, SO ORDER AND REPEATS MEAN NOTHING
'Arthur: ' as a set ' ' ':' 'A' 'h' 'r' 't' 'u'
8 characters, 7 distinct
Scanning from the left, stopping at the first character not in the bag:
0 'A' in the bag, eaten
1 'r' in the bag, eaten
2 't' in the bag, eaten
3 'h' in the bag, eaten
4 'u' in the bag, eaten
5 'r' in the bag, eaten
6 ':' in the bag, eaten
7 ' ' in the bag, eaten
8 't' in the bag, eaten
9 'h' in the bag, eaten
10 'r' in the bag, eaten
11 'e' NOT in the bag -- stop
Every spelling of the same set does the same thing:
lstrip('Arthur: ' ) -> 'ee!'
lstrip(' :Arthu' ) -> 'ee!'
lstrip('rrrAAA uth:' ) -> 'ee!'
lstrip(' :Ahrtu' ) -> 'ee!'
3. THREE DIFFERENCES, NOT ONE
They are usually described as 'set vs prefix'. There are three,
and each one can bite on its own.
lstrip(x) removeprefix(x) the difference
--------------------------------------------------------------------------------
'aaab', 'a' 'b' 'aab' repeats vs once
'ababX', 'ab' 'X' 'abX' set vs whole string
'three!', 'Q: ' 'three!' 'three!' no match: both silent
(a) lstrip repeats until it stops matching; removeprefix runs once.
(b) lstrip splits its argument into characters; removeprefix does not.
(c) neither says anything when there was nothing to remove -- the
string comes back unchanged and you cannot tell the two cases
apart from the result. That is the third difference, and it is
the one both of them share.
4. THE DOCS' OWN EXAMPLE, AND ONE THAT LOOKS SAFE
'www.example.com'.strip('cmowz.') 'example'
-- six characters, both ends, and 'example' survives because
'e', 'x', 'a', 'p' and 'l' are not in the bag.
Now the one people actually write, to drop a file extension:
'report.txt' .rstrip('.txt') -> 'repor' removesuffix -> 'report'
'data.txt' .rstrip('.txt') -> 'data' removesuffix -> 'data'
'text.txt' .rstrip('.txt') -> 'te' removesuffix -> 'text'
'extract.txt' .rstrip('.txt') -> 'extrac' removesuffix -> 'extract'
One of the four is right, and it is right by accident: 'data'
happens to end in a letter that is not in the bag. That is the
worst possible outcome -- it passes whichever example you tried
first, and fails on a filename you have not seen yet.
5. NO ARGUMENT MEANS str.isspace(), WHICH IS WIDER THAN 'SPACE'
code points where str.isspace() is True: 29
Cc 10 U+0009 U+000A U+000B U+000C U+000D U+001C U+001D U+001E U+001F U+0085
Zl 1 U+2028
Zp 1 U+2029
Zs 17 U+0020 U+00A0 U+1680 U+2000 U+2001 U+2002 U+2003 U+2004 U+2005 U+2006
U+2007 U+2008 U+2009 U+200A U+202F U+205F U+3000
'\x1cshouting into the void\u2028'.strip()
-> 'shouting into the void'
A file separator and a LINE SEPARATOR, both stripped, neither of
them a space. strip() with no argument means 'everything that
str.isspace() calls whitespace': the ten line boundaries, the
tab, the unit separator and seventeen space characters -- not
the four you can type on a keyboard.
'abc'.strip('') 'abc' <- an empty bag removes nothing
'abc'.strip(None) 'abc' <- None is the same as no argument
6. bytes HAS THE SAME FIVE, AND THE SAME TRAP
raw b'Arthur: three!'
raw.lstrip(b'Arthur: ') b'ee!'
raw.removeprefix(b'Arthur: ') b'three!'
Same five methods, same semantics, one difference: on bytes the
'set' is a set of byte VALUES, so a multi-byte character in the
argument becomes several independent bytes in the bag.
'Zubr: 3' with Z-dot, encoded b'\xc5\xbbubr: 3'
.lstrip('Ż'.encode()) b'ubr: 3'
Two bytes went into the bag and both were eaten, which happens
to be right here -- and would not be if either byte turned up
on its own inside a different character.
What the run shows¶
Order and repeats in the argument mean nothing, which is the fastest way to prove that it is a set. lstrip('Arthur: '), lstrip(' :Arthu') and lstrip('rrrAAA uth:') are the same call. If the argument were a prefix, only the first of those could possibly match.
There are three differences from removeprefix, not one. They get collapsed into "set versus prefix", and then the other two surprise you separately:
- Repeat versus once.
'aaab'.lstrip('a')is'b';'aaab'.removeprefix('a')is'aab'.lstripkeeps going until it stops matching, so it can remove any number of copies — andremoveprefixremoves exactly one, always. - Set versus whole string.
'ababX'.lstrip('ab')is'X'— the bag is{a, b}and it ate four characters.removeprefix('ab')removes the two-character string'ab'once, giving'abX'. - Both are silent when nothing matched.
'three!'.removeprefix('Q: ')returns'three!', and so doeslstrip. The result of "the prefix was there and I removed it" and "the prefix was not there" is the same value, so you cannot tell them apart afterwards. If that distinction matters — and for a protocol or a filename it usually does — you have to ask separately withstartswith, or compare the result against the input.
The file-extension case is the one that ships. name.rstrip('.txt') is a natural thing to write and it is wrong in a way that hides: of report.txt, data.txt, text.txt and extract.txt, exactly one comes out right, and it comes out right by accident — data ends in a letter that is not in the bag {., t, x}. Whichever example you tried first has a 1-in-4 chance of teaching you that the code works. removesuffix('.txt') is correct on all four, and pathlib.Path(name).stem is correct on all four and stops in the right place on archive.tar.gz.
And the no-argument form is wider than "space". strip() removes every character str.isspace() accepts — 29 code points: the ten line boundaries, the tab, the unit separator and seventeen space characters. So '\x1cshouting into the void
'.strip() comes back clean, having removed a FILE SEPARATOR and a LINE SEPARATOR, neither of which is a space and neither of which you can type. That is usually what you want when cleaning a field out of a CSV — and it is worth knowing that it is what happened, because "I stripped whitespace" and "I removed four specific characters" are different claims about your data. The ten line boundaries are the same ten splitlines() splits on.
Two small ones worth keeping: strip('') removes nothing (an empty bag), and strip(None) is identical to strip() — which is why the signature can default to None rather than to ''.
bytes has the same five methods and one extra hazard. The bag is a set of byte values, so a multi-byte character in the argument becomes several independent bytes in the bag, each free to match on its own. b'\xc5\xbbubr: 3'.lstrip('Ż'.encode()) happens to give the right answer; it would not if either of those two bytes appeared alone as part of some other character. Strip bytes with a byte set you chose deliberately, or decode first.
The Rust view¶
Rust has all of this and the accident is not available. The method that looks like Python's — trim_start_matches — takes a pattern, and a &str pattern means the whole string, so the call that reads like the buggy Python line produces the answer the buggy Python line was trying to produce.
what you meant Python Rust
------------------------------------------------------------------------------------------------
remove the prefix, once 'three!' Some("three!")
s.removeprefix("Arthur: ") s.strip_prefix("Arthur: ")
remove the prefix, repeatedly -- no such method -- "three!"
s.trim_start_matches("Arthur: ")
remove a set of characters 'ee!' "ee!"
s.lstrip("Arthur: ") s.trim_start_matches(&['A','r','t','h','u',':',' '][..])
the prefix was not there 'Arthur: three!' (silent) None
s.removeprefix("Lancelot: ") s.strip_prefix("Lancelot: ")
trim whitespace 'ab' "\u{1c}ab" <- disagree
'\x1cab'.strip() "\u{1c}ab".trim()
Three things fall out of that table. Rust's &str pattern is the whole string, so you cannot reach Python's set behaviour without saying "set" out loud — &['A','r','t','h','u',':',' '][..], or a closure. The bug is still writable; it is no longer writable by accident. Second, strip_prefix returns Option<&str>, so "it was not there" is a value the compiler makes you handle, which is the third difference above turned into a type. And third, trim_start_matches is a repeating prefix removal that Python has no equivalent of — removeprefix in a while loop is the translation.
The last row is the disagreement this library has already met once: Python's isspace() counts U+001C–U+001F and Rust's White_Space property does not, so '\x1cab'.strip() and "\u{1c}ab".trim() return different strings. Is it a letter? has the same split for the predicate; here it changes the data.
And C is where the idiom comes from. strspn, strcspn and strtok all take a set of characters written as a string — strtok(line, " ,;") splits on any of three delimiters — so the "a string standing in for a character set" convention was already forty years old when str.strip inherited it. That is the historical answer to "why would anyone design it this way": nobody designed it, it was the ambient idiom.
If you are coming from ABAP¶
SHIFT lv_text LEFT DELETING LEADING lv_mask is the near neighbour, and the ABAP documentation calls the operand a mask rather than a prefix — so the same "set of characters" reading applies and the same trap is available. Check the exact semantics on your own system before relying on it, including what it does when the mask is initial: the two languages agree on the idea and there is no reason to assume they agree on the edges. For the thing you usually want — remove this exact prefix — the ABAP idiom is not SHIFT at all but REPLACE FIRST OCCURRENCE OF anchored at the front, or an offset after a CS test, and both of those tell you whether the prefix was there. Coming the other way, CONDENSE is roughly strip() plus internal whitespace collapsing, which Python does not have in one call — ' '.join(s.split()) is the idiom. (Not machine-checked — CI cannot run ABAP.)
Try it¶
- Find the extension bug in reverse: for the bag
{., t, x}, characterise every filenamerstrip('.txt')handles correctly. Then decide how you would have discovered the rule from a test suite of four filenames. - Write
removeprefixyourself withstartswithand a slice, the way everyone did before 3.9. Now write the version that tells the caller whether it matched. Which of the two is PEP 616 ↗'s, and why? strip()removes 29 different code points. Build a one-line string containing all 29 and check thatstrip()returns''. Then remove one of them fromstr.isspace()'s answer and see whether you can find a character your terminal renders as blank thatstrip()leaves alone. (There is at least one — start in theU+2000block's neighbours.)- On
bytes, construct the case the last paragraph warns about: a byte set whose members appear inside a different multi-byte character, so thatlstripcuts a UTF-8 sequence in half. What does decoding the result do?
Practice¶
Eight calls, and the one that is right by accident. Write down what each returns.
'Arthur: three!'.lstrip('Arthur: ')'Arthur: three!'.removeprefix('Arthur: ')'aaab'.lstrip('a')'aaab'.removeprefix('a')'report.txt'.rstrip('.txt')'data.txt'.rstrip('.txt')'abc'.strip('')'\x1cabc\u2028'.strip()
Then: line 6 gives the right answer. Say why that is the worst of the eight outcomes, and what you would have concluded if data.txt had been the first filename you tested.
Answers
Verified output of strip_is_a_set_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.
call result why
------------------------------------------------------------------------------------------------
'Arthur: three!'.lstrip('Arthur: ') 'ee!' the bag ate 'thr' as well
'Arthur: three!'.removeprefix('Arthur: ') 'three!' what everyone meant
'aaab'.lstrip('a') 'b' lstrip REPEATS until it stops matching
'aaab'.removeprefix('a') 'aab' removeprefix runs ONCE, always
'report.txt'.rstrip('.txt') 'repor' the bag is {'.', 't', 'x'}
'data.txt'.rstrip('.txt') 'data' right, and right by accident
'abc'.strip('') 'abc' an empty bag removes nothing
'\x1cabc\u2028'.strip() 'abc' a FILE SEPARATOR and a LINE SEPARATOR
THREE DIFFERENCES, NOT ONE
They get collapsed into 'set versus prefix'. Lines 1-4 hold all
three, and each one can bite on its own:
(a) lstrip repeats, removeprefix runs once (lines 3, 4)
(b) lstrip splits its argument into characters (lines 1, 2)
(c) neither says anything when nothing matched (below)
'three!'.lstrip('Q: ') 'three!'
'three!'.removeprefix('Q: ') 'three!'
Same value as 'the prefix was there and I removed it' would
give. You cannot tell the two cases apart from the result, so if
the distinction matters -- and for a protocol or a filename it
does -- ask separately with startswith.
WHY ORDER AND REPEATS IN THE ARGUMENT MEAN NOTHING
lstrip('Arthur: ' ) -> 'ee!'
lstrip(' :Arthu' ) -> 'ee!'
lstrip('rrrAAA uth:' ) -> 'ee!'
lstrip(' :Ahrtu' ) -> 'ee!'
Four spellings of one set, one answer. If the argument were a
prefix, only the first of those could possibly match. That is
the fastest proof available that it is a bag of characters.
THE ONE THAT SHIPS: LINE 6 IS RIGHT BY ACCIDENT
filename rstrip(.txt) removesuffix agree?
--------------------------------------------------------
report.txt 'repor' 'report' False
data.txt 'data' 'data' True
text.txt 'te' 'text' False
extract.txt 'extrac' 'extract' False
1 of 4 agree. 'data' survives because it happens to end
in a letter that is not in the bag -- nothing about the code
was right. That is the worst possible outcome: it passes whichever
example you tried first and fails on a filename you have not
seen yet. removesuffix is correct on all four, and
pathlib.Path(name).stem is correct on all four AND stops in the
right place on 'archive.tar.gz'.
WHAT LINE 8 REMOVED
'\x1cabc\u2028'.strip() -> 'abc'
code points where str.isspace() is True: 29
U+0009 U+000A U+000B U+000C U+000D U+001C U+001D U+001E U+001F U+0020
U+0085 U+00A0 U+1680 U+2000 U+2001 U+2002 U+2003 U+2004 U+2005 U+2006
U+2007 U+2008 U+2009 U+200A U+2028 U+2029 U+202F U+205F U+3000
strip() with no argument means 'everything str.isspace() calls
whitespace' -- the ten line boundaries, the tab, the unit
separator and seventeen space characters. Not the four you can
type on a keyboard. That is usually what you want when cleaning
a CSV field, and it is worth knowing that it is what happened:
'I stripped whitespace' and 'I removed four specific characters'
are different claims about your data.
See also¶
- What ends a line — the ten boundaries
strip()silently removes - Is it a letter? —
isspace()itself, and the Rust property it disagrees with stris notbytes— why thebytesversion strips byte values rather than characters- What to write next — this page closes two questions; the method tour has eight more
- The crosswalk — which idea lives in which library
str.strip()in the Python docs ↗ — including the sentence this page is about- PEP 616 ↗ —
removeprefixandremovesuffix, and the survey of real code that motivated them