Skip to content

Redirection truncates first

Level: 101 · you have run sort names.txt > names.txt once

One line: The shell empties the file named after > before the command starts, so sort fruit.txt > fruit.txt sorts an empty file, and nosuchcommand > fruit.txt empties it without running anything. set -o noclobber makes > refuse a file that exists, and >| overrides it. zsh adds >! and also refuses >> to a missing file. fish has no option, only >? one redirection at a time, and in fish >| is a pipe.

Measured

Verified output of redir_truncate_sh.sh, identical on Linux and macOS — regenerated by tools/run_examples.py, never hand-typed.

$ printf "banana\napple\ncherry\n" > fruit.txt; cat fruit.txt
banana
apple
cherry

$ sort fruit.txt > fruit.txt; echo "fruit.txt is now $(wc -c < fruit.txt | tr -d " ") bytes"
fruit.txt is now 0 bytes

$ printf "banana\napple\ncherry\n" > fruit.txt; nosuchcommand > fruit.txt; echo "fruit.txt is now $(wc -c < fruit.txt | tr -d " ") bytes"
redir_truncate_sh.sh: line 6: nosuchcommand: command not found
fruit.txt is now 0 bytes

$ printf "banana\napple\ncherry\n" > fruit.txt; sort -o fruit.txt fruit.txt; cat fruit.txt
apple
banana
cherry

$ printf "banana\napple\ncherry\n" > fruit.txt; sort fruit.txt > sorted.tmp && mv sorted.tmp fruit.txt; cat fruit.txt
apple
banana
cherry

$ set -o noclobber

$ sort fruit.txt > fruit.txt; echo "status $?, fruit.txt is $(wc -c < fruit.txt | tr -d " ") bytes"
redir_truncate_sh.sh: line 6: fruit.txt: cannot overwrite existing file
status 1, fruit.txt is 20 bytes

$ echo date >> fruit.txt; tail -n 1 fruit.txt
date

$ echo fig > new.txt; echo plum >> new2.txt; cat new.txt new2.txt
fig
plum

$ echo grape >| fruit.txt; cat fruit.txt
grape

$ echo discard > /dev/null; echo "status $?"
status 0

$ set +o noclobber

$ echo kiwi >! fruit.txt; echo "fruit.txt: $(cat fruit.txt)"; echo "the file named !: $(cat !)"
fruit.txt: grape
the file named !: kiwi fruit.txt
  • sort fruit.txt > fruit.txt leaves 0 bytes. The shell opened fruit.txt for writing, which empties it, and only then started sort, which read the empty file.
  • The command does not even have to exist. nosuchcommand was never found, and the file is empty anyway: the redirection came first.
  • Two safe ways to sort a file in place. sort -o fruit.txt fruit.txt lets sort open the output itself, after it has read the input. Writing to a new file and mv-ing it over the old one works for any command.
  • set -o noclobber makes > refuse an existing file. The same mistake now fails with status 1, and all 20 bytes are still there.
  • What noclobber leaves alone. >> still appends to an existing file and creates a missing one, > still creates a new file, and writing to /dev/null still works. The bash manual limits the refusal to regular files, and /dev/null is not one.
  • >| is the override, for the time you meant it.
  • The last command is a zsh habit typed into bash. >! is zsh's second override and not bash syntax. bash read > and then the word !, sent the output to a file named !, and handed fruit.txt to echo as an argument. There was no error, a new file appeared, and fruit.txt was untouched.

On a Mac

Nothing changes. The key is shared: bash 3.2 and bash 5.2 truncate and refuse identically, and BSD sort -o accepts its own input file as GNU sort -o does.

In zsh and fish

zsh has the same trap and a stricter noclobber:

Verified output of redir_truncate_zsh.zsh, identical on Linux and macOS — regenerated by tools/run_examples.py, never hand-typed.

$ printf "banana\napple\ncherry\n" > fruit.txt; sort fruit.txt > fruit.txt; echo "fruit.txt is now $(wc -c < fruit.txt | tr -d " ") bytes"
fruit.txt is now 0 bytes

$ printf "banana\napple\ncherry\n" > fruit.txt; setopt noclobber

$ sort fruit.txt > fruit.txt; echo "status $?"; cat fruit.txt
(eval):1: file exists: fruit.txt
status 1
banana
apple
cherry

$ echo grape >| fruit.txt; cat fruit.txt
grape

$ echo kiwi >! fruit.txt; cat fruit.txt
kiwi

$ echo fig >> new.txt; echo "status $?"
(eval):1: no such file or directory: new.txt
status 1

$ echo fig >>| new.txt; cat new.txt
fig

$ unsetopt noclobber

setopt noclobber is zsh's spelling, and its refusal says file exists. Both >| and >! override it. The difference from bash is >>: zsh with noclobber refuses to append to a file that does not exist yet, and >>| is the override for that.

fish has no option to set, and its >| means something else entirely:

Verified output of redir_truncate_fish.fish, identical on Linux and macOS — regenerated by tools/run_examples.py, never hand-typed.

$ printf "banana\napple\ncherry\n" > fruit.txt; sort fruit.txt > fruit.txt; echo "fruit.txt is now "(wc -c < fruit.txt | tr -d " ")" bytes"
fruit.txt is now 0 bytes

$ printf "banana\napple\ncherry\n" > fruit.txt; sort fruit.txt >? fruit.txt; echo "status $status"; cat fruit.txt
warning: The file 'fruit.txt' already exists
status 1
banana
apple
cherry

$ echo fig >? new.txt; cat new.txt
fig

$ sort -o fruit.txt fruit.txt; cat fruit.txt
apple
banana
cherry

$ echo grape >| fruit.txt
fish: Unknown command: fruit.txt
fish: 
echo grape >| fruit.txt
              ^~~~~~~~^
(exit status 127)

$ cat fruit.txt
apple
banana
cherry

>? is "no clobber" for a single redirection. Against an existing file it printed a warning and returned status 1, sort did not run, and the file kept its contents. Against a new file it simply wrote. fish reports the refusal on its own stderr, where the script's eval … 2>&1 cannot catch it, so that line runs in a child fish.

>| is the bash override, and fish does not have it. In fish, >| sends stdout into a pipe, so echo grape >| fruit.txt looked for a command named fruit.txt: status 127, and the file is untouched. The script pins fish's own "Unknown command" message with -C, because Ubuntu's fish package installs a command-not-found helper that words it differently.

bash zsh fish
refuse to overwrite set -o noclobber setopt noclobber >?, one redirection at a time
overwrite anyway >\| >\|, >! plain >
>> to a missing file, with the option on creates it refused; >>\| creates it —
>\| means overwrite overwrite a pipe into the next word

If you are coming from another library

See also