split cuts characters, paste cuts delimiters, look needs a sorted file, tee does nothing¶
Level: 201 · for anyone wiring tools together
One line: None of these four reads your text, and three of them can still hand back something that is not text — split -b cuts a character in half, paste -d 'é' takes your one-character delimiter apart and uses each byte as a separate delimiter, and look answers not found on a file that plainly contains the word.
The four, and why they are on one page¶
They are the leftovers of this chapter: the commands that move text from one shape to another without ever asking what it says. split and csplit cut a file into pieces, paste puts files side by side, look finds a line in a sorted one, and tee copies a stream while passing it on. Every one of them is the obvious tool for its job, none of them has a page anywhere else, and three of the four can quietly hand you a file that is no longer text.
tee is here as the control. It is the only one that genuinely does nothing — no decode, no delimiter, no boundary, no assumption — which is exactly why it is the tool you put in the middle of a pipeline you have stopped trusting. A page that only listed the ways things break would leave out the most useful thing on it: knowing which tool has no opinion is how you find out which one had the wrong one.
The pattern in the other three is worth naming before you read the session, because it is the same shape three times. Each of them takes a number or a string that you meant as text — four bytes, one delimiter, one word — and applies it as bytes: a byte count that lands mid-character, a delimiter list read a byte at a time, a binary search that assumes the file is in byte order. None of the three warns you. Two of them exit 0.
In the terminal¶
Verified output of look_paste_tee_split_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
1. paste -d TAKES A LIST OF BYTES, NOT A DELIMITER
$ paste -d 'X' c1 c2 c3
a1Xb1Xc1
a2Xb2Xc2
Three columns, one delimiter, nothing surprising.
$ paste -d 'é' c1 c2 c3 | cat -v
a1M-Cb1M-)c1
a2M-Cb2M-)c2
One character in, two delimiters out. -d takes a LIST — paste -d ',;'
is a documented feature, alternating between them — and that list is
read a BYTE at a time, so é became the list {c3, a9}: M-C after the
first column, M-) after the second. Nothing warned, and both fields
are now separated by half a character. In a UTF-8 locale the two
pastes disagree about this; the fence on the page has that.
2. split -b CUTS WHEREVER THE COUNT LANDS
$ xxd -p text.txt
636166c3a9206e61c3af76650a
$ split -b 4 text.txt piece_
$ for f in piece_*; do printf "%s: " "$f"; xxd -p "$f"; done
piece_aa: 636166c3
piece_ab: a9206e61
piece_ac: c3af7665
piece_ad: 0a
Four bytes per piece, so the two bytes of é are now in different
FILES: c3 ends the first piece and a9 begins the second. Same for the
ï two pieces later.
$ for f in piece_*; do printf "%s is valid UTF-8? " "$f"; if iconv -f UTF-8 -t UTF-8 < "$f" >/dev/null 2>&1; then echo yes; else echo NO; fi; done
piece_aa is valid UTF-8? NO
piece_ab is valid UTF-8? NO
piece_ac is valid UTF-8? yes
piece_ad is valid UTF-8? yes
Two of the four pieces are not text at all. That is not a bug in
split: it was asked for four bytes and it gave four bytes.
$ cat piece_* | cmp - text.txt
exit=0
And the concatenation is the original, byte for byte. That is the
contract worth remembering: the PIECES are not text, the JOIN is. Any
program that reads one piece on its own — a decoder, a grep, an
uploader that validates — is looking at a broken file.
3. THE FLAG THAT DOES NOT CUT A CHARACTER IS -l, AND IT COUNTS LINES
$ split -l 1 lines.txt line_
$ for f in line_*; do printf "%s: " "$f"; xxd -p "$f"; done
line_aa: 636166c3a90a
line_ab: 6e61c3af76650a
line_ac: 7a7a7a0a
$ for f in line_*; do printf "%s is valid UTF-8? " "$f"; if iconv -f UTF-8 -t UTF-8 < "$f" >/dev/null 2>&1; then echo yes; else echo NO; fi; done
line_aa is valid UTF-8? yes
line_ab is valid UTF-8? yes
line_ac is valid UTF-8? yes
One line per file, and a line ends at an 0a, which can never be part
of a multi-byte UTF-8 character — every continuation byte has its top
bit set. So -l is safe by construction, and -b never is. The flag that
wants both (a byte budget, cut at line boundaries) is GNU-only; see
the fence.
4. csplit CUTS ON A PATTERN, WHICH IS sed's WORLD
$ cat doc.txt
café
---
naïve
$ csplit -s -f cs_ doc.txt "/---/"
$ for f in cs_*; do printf "%s: " "$f"; xxd -p "$f"; done
cs_00: 636166c3a90a
cs_01: 2d2d2d0a6e61c3af76650a
The pattern matches a LINE, so the cut is at a line boundary and the
characters survive — and the matching line starts the SECOND piece,
which is the part people get wrong. The pattern itself is a regex over
bytes here, with the same reach and the same limits sed has.
5. tee IS THE CONTROL: IT CHANGES NOTHING, AND THAT IS THE POINT
$ printf 'caf\303\251\000\377\n' | tee copy.bin | xxd -p
636166c3a900ff0a
$ xxd -p copy.bin
636166c3a900ff0a
A NUL and an ff — one byte no text tool likes and one that cannot
appear in UTF-8 at all — and both came through in both directions
untouched. tee has no text model to get wrong, which is exactly why it
is the tool to put in the middle of a pipeline you do not trust:
… | tee /tmp/raw | rest-of-pipeline, then xxd /tmp/raw to see what the
rest of the pipeline was actually handed.
look, the one that is not on your Linux box¶
look is missing from that session for a reason worth knowing before you write it into a script: it is not installed on a plain Ubuntu. apt-get install bsdextrautils brings it (util-linux 2.39.3), and macOS ships its own BSD one — so this is a tool that is on your laptop and not on your server, which is the first question to ask about any command this chapter does not measure.
What it does is a binary search, which is what makes it fast on a large word list and is also its whole problem: a binary search over a file that is not sorted in the order it compares with does not fail, it just misses.
$ cat unsorted.txt # every word is in the file; none in order
czar
apple
bee
$ look czar unsorted.txt # the FIRST line of the file
macOS (nothing) exit=1
Ubuntu (nothing) exit=1
$ look apple unsorted.txt
macOS (nothing) exit=1
Ubuntu (nothing) exit=1
$ LC_ALL=en_US.UTF-8 sort words > u.txt # resume / résumé / rz, in COLLATION order
$ LC_ALL=C look rz u.txt
macOS (nothing) exit=1
Ubuntu rz exit=0
Three words in a file, all three absent as far as look is concerned, on both platforms, with the same exit status a genuinely missing word gets. That is the same failure shape grep has on undecodable bytes and the worst one in the library: a false negative with no channel to report it on.
The second block is the encoding half, and the reason this tool is in a chapter about character data at all. sort in a UTF-8 locale puts résumé between resume and rz; sort in the C locale puts it after both, because é starts with c3. So sorted is not a property of a file — it is a property of a file and a collation, and look brings its own. Sort your word list the way a human reads it and the byte-order binary search walks straight past the answer. The two implementations do not even agree about that case: BSD look misses rz, util-linux look finds it. Neither tells you which order it assumed.
The rule that survives all of it: sort the file with the same LC_ALL you will search it with, and if you cannot promise that, use grep -x — linear, slower, and incapable of this particular lie. The order question itself is Sorting and collation.
In Python¶
Verified output of look_paste_tee_split_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. A CHUNK BOUNDARY IS A BYTE OFFSET, AND CHARACTERS DO NOT CARE
the bytes 636166c3a9206e61c3af7665
four-byte chunks ['636166c3', 'a9206e61', 'c3af7665']
chunks that will not decode alone [1, 2]
That is `split -b 4` with a different spelling, and the same result:
the pieces are not text, only their concatenation is.
2. THE FIX HAS A NAME: AN INCREMENTAL DECODER
what each chunk yielded ['caf', 'é na', 'ïve']
joined café naïve
...equals the original True
dec.decode(b'', final=True) at the end ''
Chunk 1 handed back three characters and kept a dangling c3; chunk 2
completed it and returned the é first. The decoder is holding the
partial character between calls — that is the state a stream decoder
has and a file-splitter does not, which is the whole difference.
Call it with final=True at the end: a leftover half-character there
is a truncated file, and only that call will tell you.
3. A DELIMITER IS A STRING, NOT A LIST OF BYTES
'é'.join(row) ['a1éb1éc1', 'a2éb2éc2']
the bytes of row 1 6131c3a96231c3a96331
One é between each pair, c3 a9 both times. The shell's paste -d takes
a LIST of delimiters and reads it a byte at a time, so the same
request there puts c3 after the first column and a9 after the second.
Python cannot make that mistake: you named a string, not a set.
zip_longest, fillvalue='' ['a1\tb1', 'a2\t']
And that is paste's other rule — a short column becomes empty fields,
not a short row — written down where you can see it.
4. bisect IS look, AND IT FAILS THE SAME WAY
the list ['Ada', 'bob', 'Cara', 'dan']
'Cara' in words True
bisect_left(words, 'Cara') 1
...the element actually there bob
found by binary search? False
The word is in the list and the binary search cannot find it, because
the list is ordered case-insensitively and the comparison is by code
point, where every capital sorts before every lowercase letter. Two
orders, one search, no error. `look` on a file sorted by anything but
the order it compares with is this bug with a file behind it — and a
locale is the commonest way to end up with two orders.
5. THE tee GUARANTEE IS SPELLED 'b'
the bytes on disk 636166c3a90d0a7365636f6e640d0a
read_bytes() gives them back True
read_text() gives back 'café\nsecond\n'
...same bytes? False
Two doors onto one file, and only one of them is tee. The text door
decoded and then translated the line endings, so what your code got
is not what the file holds — and it will not tell you, because for
almost every program that translation is the helpful thing to do.
When the question is what a pipeline did to the bytes, ask in binary:
that is the mode with no text model, which is what tee is.
The part that is not the same on both machines¶
$ paste -d 'é' c1 c2 c3 | cat -v # in a UTF-8 locale
macOS a1M-CM-)b1M-CM-)c1 one delimiter, both of its bytes
Ubuntu a1M-Cb1M-)c1 still two delimiters, one byte each
(in LC_ALL=C both give the Ubuntu answer — which is what the session above records)
$ split -C 8 lines.txt # byte budget, cut at line boundaries
macOS split: illegal option -- C
Ubuntu three pieces, no character cut
$ split -p PATTERN lines.txt # split where a line matches
macOS two pieces
Ubuntu split: invalid option -- 'p'
$ split -n 2 lines.txt # 17 bytes into 2 pieces
macOS 8 + 9 bytes …6e 61 | c3 af… the ï survives the cut
Ubuntu 9 + 8 bytes …6e 61 c3 | af… the ï is cut in half
paste -d is multi-byte aware on BSD in a UTF-8 locale and has never been on GNU — the same split, one tool along, as tr, whose BSD build uppercases é in a UTF-8 locale where GNU leaves the bytes alone. Under LC_ALL=C the two agree, and they agree on the wrong answer, which is the one to write your script against: use a single-byte delimiter, or build the line with something that has a string type.
The flag that would fix split -b is not portable. -C (a byte budget, but cut at a line boundary) is GNU-only; BSD answers illegal option. -p (split where a line matches a pattern) is BSD-only; GNU's answer to that job is csplit, which both ship. So the portable pair is -l and csplit, and -b is for files you have stopped calling text.
And -n is the trap in the middle, because it exists on both and is not the same: dividing 17 bytes into 2 pieces, BSD rounds the remainder into the last piece and GNU into the first, so the cut lands one byte apart — and on this file that is the difference between an ï that survives and an ï in two halves. The command is identical, the file is identical, and whether the pieces are text depends on which machine ran it.
Which one to reach for¶
| The job | Portable answer | What it costs |
|---|---|---|
Cut a file into pieces you will cat back together |
split -b, and never decode a piece alone |
pieces are not text; two of four in the session are invalid UTF-8 |
| Cut a file into pieces that are each still text | split -l N |
piece sizes vary with line length, which is what you traded |
| Cut at a marker | csplit -f pre -s file '/marker/' |
the matching line starts the next piece; the pattern is bytes, like sed's |
| Join files as columns | paste, with a single-byte delimiter |
a multi-byte delimiter becomes a delimiter list on GNU, in any locale |
| Find a word in a big sorted list | look if you can promise the collation; grep -x if you cannot |
look is a false negative when you cannot; grep is linear |
| See what a pipeline is actually handing the next stage | … \| tee /tmp/raw \| …, then xxd /tmp/raw |
nothing — this is the one with no opinion |
If you are coming from Python or ABAP¶
Python. Two of the three bugs cannot be written here, and the third has a name. A delimiter is a str, so 'é'.join(...) puts one é between fields and there is no list to read a byte at a time. A chunk boundary, though, is exactly as sharp as split -b's — data[0:4] on bytes will land inside a character — and the standard library's answer is an incremental decoder (codecs.getincrementaldecoder('utf-8')()), which holds the dangling bytes between calls and hands you the character when the rest arrives. That buffer is precisely what a file-splitter has nowhere to keep. Finish with decode(b'', final=True): a leftover half-character there means the stream was truncated, and only that call raises. And bisect is look, with look's precondition written into its own documentation — a list sorted by one key, searched by another, misses silently.
ABAP (Not machine-checked — CI cannot run ABAP.) The split -b hazard is xstring work: slicing an xstring at an arbitrary offset and calling cl_abap_conv_in_ce on the piece gives you a conversion error or a replacement character, for the same reason — the character straddles the cut. CONCATENATE ... SEPARATED BY sep on a string is paste with a real string type, so the delimiter bug cannot happen; SPLIT ... AT sep likewise. The place to be careful is any interface that chunks a payload by byte length — a fixed-width block, an IDoc segment, an HTTP body split for transport — where the receiving side must re-join before decoding, exactly as cat piece_* does above. Reassemble, then convert; never convert a chunk.
Try it¶
printf 'caf\303\251 na\303\257ve\n' > t; split -b 4 t p_; file p_*— thencat p_* | cmp - tand watch the pieces become a file again.paste -d 'é' a b c | xxd | headon any three files, and find the two delimiters. Then repeat underLC_ALL=en_US.UTF-8on a Mac.- Take a sorted word list,
looka word in it, thensortit under a differentLC_ALLandlookthe same word again. Note that nothing changed except the order. - Put
tee /tmp/rawin the middle of a pipeline you are debugging, andxxd /tmp/raw | head. It is the fastest way to find out which stage did it.
Practice¶
Three tools that take text as bytes, and one control. Predict: what split -b 4 does to a 6-byte file containing café; what paste -d 'é' uses as a delimiter (it is not é); and what look says about a word that is in an unsorted file.
Then say which of the four tools on that page has no opinion at all about your text, and why that makes it the one to insert into a pipeline you have stopped trusting.
Answers
Verified output of look_paste_tee_split_kata_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
1. split -b CUTS A CHARACTER IN HALF
piece_aa 636166c3
piece_ab a90a
piece_aa is valid UTF-8? NO piece_ab? NO
-b is a byte budget, and 4 lands inside é. Neither piece is text, and
split neither knows nor says. -l splits on lines and cannot do this.
2. paste -d TAKES A LIST OF BYTES, NOT A DELIMITER
paste -d X aX1Xx bX2Xy
paste -d é a\xc31\xa9x b\xc32\xa9y
One 'delimiter' produced TWO different separators: -d is a LIST read
one byte at a time, and é is two bytes, so c3 joins the first pair and
a9 the second. The output is not valid UTF-8 either.
3. look NEEDS A SORTED FILE, AND SAYS NOTHING WHEN IT IS NOT
look apple <unsorted> exit=1
look apple <sorted> exit=0
The word is in the file both times. look does a BINARY SEARCH, which
assumes an order -- and which order is a locale question, so even a
sorted file can be the wrong sorted file.
4. tee IS THE CONTROL
through tee: 636166c3a900ff0a
the file tee wrote: 636166c3a900ff0a
NUL, ff, an incomplete sequence -- tee passes all of it through
unchanged, because it has no text model to get wrong. That is exactly
why it is the tool to insert in a pipeline you have stopped trusting:
... | tee /tmp/raw | rest-of-pipeline then xxd /tmp/raw
THE PATTERN IN THE OTHER THREE
Each takes something you meant as TEXT -- a length, a delimiter, an
order -- and applies it as BYTES. None warns. Two of them exit 0.
See also¶
cutcounts what it is told to count — the same byte/character question on the other axistrandsortwork a byte at a time — wheresort's order comes from, and the BSD/GNU splitpaste -drepeats- Sorting and collation — why sorted is a property of a file and a locale
diffcompares lines,cmpcompares bytes — the neighbouring pair, with the same relationship to the word same- UTF-8 by hand — why a continuation byte is recognisable, which is what makes
-lsafe and-bnot