Skip to content

cat file | grep is one process too many

Level: 101 · you type cat notes.txt | grep darkfi out of habit

One line: For one file, cat f | grep x, grep x f and grep x < f print the same lines, and cat only adds a process. With several files the pipe costs information: grep cannot say which file a line came from, it counts and numbers all the files as one, it answers -l with (standard input), and a missing file makes the pipeline exit 1, the status for "not found". And cat can change the answer: a file whose last line has no newline is glued to the first line of the next.

Measured

On two files, demo/hosts.txt and demo/notes.txt:

Verified output of catgrep_sh.sh, identical on Linux and macOS — regenerated by tools/run_examples.py, never hand-typed.

$ cat hosts.txt | grep darkfi
darkfi   10.0.0.20   node
darkfi   10.0.0.21   wallet

$ grep darkfi hosts.txt
darkfi   10.0.0.20   node
darkfi   10.0.0.21   wallet

$ grep darkfi < hosts.txt
darkfi   10.0.0.20   node
darkfi   10.0.0.21   wallet

$ grep -H darkfi < hosts.txt
(standard input):darkfi   10.0.0.20   node
(standard input):darkfi   10.0.0.21   wallet

$ < hosts.txt grep darkfi
darkfi   10.0.0.20   node
darkfi   10.0.0.21   wallet

$ < notes.txt; echo "status $?"
status 0

$ cat hosts.txt notes.txt | grep darkfi
darkfi   10.0.0.20   node
darkfi   10.0.0.21   wallet
restart the darkfi node after the update

$ grep darkfi hosts.txt notes.txt
hosts.txt:darkfi   10.0.0.20   node
hosts.txt:darkfi   10.0.0.21   wallet
notes.txt:restart the darkfi node after the update

$ grep -h darkfi hosts.txt notes.txt
darkfi   10.0.0.20   node
darkfi   10.0.0.21   wallet
restart the darkfi node after the update

$ cat hosts.txt notes.txt | grep -c darkfi
3

$ grep -c darkfi hosts.txt notes.txt
hosts.txt:2
notes.txt:1

$ cat hosts.txt notes.txt | grep -l darkfi
(standard input)

$ grep -l darkfi hosts.txt notes.txt
hosts.txt
notes.txt

$ cat hosts.txt notes.txt | grep -n darkfi
2:darkfi   10.0.0.20   node
4:darkfi   10.0.0.21   wallet
5:restart the darkfi node after the update

$ grep -n darkfi hosts.txt notes.txt
hosts.txt:2:darkfi   10.0.0.20   node
hosts.txt:4:darkfi   10.0.0.21   wallet
notes.txt:1:restart the darkfi node after the update

$ cat nosuch.txt | grep darkfi; echo "status $?"
cat: nosuch.txt: No such file or directory
status 1

$ grep darkfi nosuch.txt; echo "status $?"
grep: nosuch.txt: No such file or directory
status 2

$ grep darkfi < nosuch.txt; echo "status $?"
catgrep_sh.sh: line 6: nosuch.txt: No such file or directory
status 1

$ less hosts.txt | head -n 1
alma9    10.0.0.10   build server

$ cat hosts.txt | less | head -n 1
alma9    10.0.0.10   build server

$ cat draft.txt notes.txt | grep darkfi
darkfi seed phraserestart the darkfi node after the update

$ cat draft.txt notes.txt | grep -c darkfi
1

$ grep -c darkfi draft.txt notes.txt
draft.txt:1
notes.txt:1
  • One file, and every spelling printed the same two lines. In grep darkfi < hosts.txt the shell opens the file and grep reads it on standard input, as it reads cat's output, but with no cat to run. grep is not told the name: -H, which puts the file name in front of each line, could only write (standard input). The redirection may also come first, < hosts.txt grep darkfi, which reads left to right like the cat pipeline. A redirection with no command at all, < notes.txt, printed nothing in bash and exited 0.
  • Two files through cat are one stream. grep printed three lines with no file names, -c counted 3 in all, -l could only name (standard input), and -n numbered straight through, so the first line of notes.txt came out as line 5.
  • Two files by name stay two files. grep put hosts.txt: or notes.txt: in front of each line, counted 2 and 1, listed both names, and numbered each file from 1. -h drops the names, when the cat output was what you wanted.
  • A missing file. cat nosuch.txt | grep darkfi printed cat's complaint and exited 1: grep read an empty stream, selected nothing, and a pipeline's status is its last command's. That is the same 1 as a file with no match in it. grep darkfi nosuch.txt exited 2, the status for an error. With < nosuch.txt, grep never started: bash could not open the file, said so, and set status 1. The line 6 in bash's message is the line of the script's say helper, where the command ran.
  • less is cat when its output is not a terminal. less hosts.txt and cat hosts.txt | less printed the same first line. On a terminal both open the pager; press = in each to see what less says it is reading.
  • The glued line. draft.txt ends without a newline. cat joined its last line to the first line of notes.txt, and grep printed darkfi seed phraserestart the darkfi node after the update as one line. Through cat that counts as 1; by name it is draft.txt:1 and notes.txt:1. Nothing reported an error.

On a Mac

Nothing changes: the key above is shared. BSD cat, less and grep and /bin/bash 3.2 printed the same bytes as their Linux counterparts, bash's message for the failed redirection included.

In zsh and fish

zsh takes the redirection first as bash does, and gives a redirection with no command a job:

Verified output of catgrep_zsh.zsh, identical on Linux and macOS — regenerated by tools/run_examples.py, never hand-typed.

$ cat hosts.txt notes.txt | grep -c darkfi
3

$ grep -c darkfi hosts.txt notes.txt
hosts.txt:2
notes.txt:1

$ < hosts.txt grep darkfi
darkfi   10.0.0.20   node
darkfi   10.0.0.21   wallet

$ < notes.txt
restart the darkfi node after the update
alma9 backup finished

$ grep darkfi < nosuch.txt; echo "status $?"
(eval):1: no such file or directory: nosuch.txt
status 1

$ grep darkfi nosuch.txt; echo "status $?"
grep: nosuch.txt: No such file or directory
status 2

A bare < notes.txt, which did nothing in bash, printed the file in zsh. zsh runs $READNULLCMD on it, a pager, and a pager writing to a pipe copies the file through. Which pager is not the same on the two machines: $READNULLCMD was more on macOS and pager in the Ubuntu image, so the key does not print it. zsh's message for the missing file is its own, no such file or directory: nosuch.txt, with the same status 1.

fish gets the same answers from grep, and has two differences of its own:

Verified output of catgrep_fish.fish, identical on Linux and macOS — regenerated by tools/run_examples.py, never hand-typed.

$ cat hosts.txt notes.txt | grep -c darkfi
3

$ grep -c darkfi hosts.txt notes.txt
hosts.txt:2
notes.txt:1

$ grep darkfi < hosts.txt
darkfi   10.0.0.20   node
darkfi   10.0.0.21   wallet

$ grep darkfi nosuch.txt; echo "status $status"
grep: nosuch.txt: No such file or directory
status 2

$ grep darkfi < nosuch.txt; echo "status $status"
warning: An error occurred while redirecting file 'nosuch.txt'
warning: Path 'nosuch.txt' does not exist
status 1

$ < hosts.txt grep darkfi
fish: Expected a string, but found a redirection
< hosts.txt grep darkfi
^~~~~~~~~~^
(exit status 127)

A failed redirection is two warnings from fish and status 1. fish prints them on its own stderr, not the command's, which is why the script shows them from a child fish. And fish does not take the redirection first: < hosts.txt grep darkfi is a parse error, Expected a string, but found a redirection, and the child exited 127. In fish, write grep darkfi < hosts.txt.

bash zsh fish
grep x < f works works works
< f grep x works works parse error, status 127
< f with no command nothing, status 0 shows f with $READNULLCMD not measured
grep x < missing bash's message, status 1 zsh's message, status 1 two warnings, status 1

If you are coming from another library

  • Perl. -n and -p are a loop ↗: perl -ne reads every file named on its command line in one loop, and its line counter $. keeps counting from one file into the next, as cat | grep -n did here.
  • Encodings. The trailing newline ↗ is the missing last byte behind the glued line, and what wc -l and a terminal make of a file without it.

See also