Skip to content

sed writes to stdout, not to the file

Level: 101 · for anyone who has run a sed line, seen the right answer scroll past, and then found the file unchanged

One line: sed 's/a/b/' file prints a changed copy and leaves file exactly as it was — and the flag that does change the file, -i, is the one thing in this chapter each machine spells differently: sed -i on Linux, sed -i '' on a Mac, and sed -i.bak on both.

sed is a filter. It reads lines, applies an edit to each, writes the result onward, and never touches the file it read from unless you go out of your way to ask. That is why it composes: sed in the middle of a pipeline is the whole point of it, and a program that rewrote its input would be useless there.

Measured

A small log, and the four things people use sed for: substitute, substitute everywhere, pull fields out with a group, and throw lines away.

Verified output of sed_stream_sh.sh, identical on Linux and macOS — regenerated by tools/run_examples.py, never hand-typed.

$ cat log.txt
03:02:36 sshd: Disconnected from invalid user admin 10.0.0.7
03:02:37 sshd: Disconnected from invalid user root 10.0.0.9
03:04:01 sshd: Disconnected from invalid user admin 10.0.1.4
03:05:12 sshd: Accepted publickey for ci from 10.0.2.2

$ sed 's/invalid user/BAD/' log.txt
03:02:36 sshd: Disconnected from BAD admin 10.0.0.7
03:02:37 sshd: Disconnected from BAD root 10.0.0.9
03:04:01 sshd: Disconnected from BAD admin 10.0.1.4
03:05:12 sshd: Accepted publickey for ci from 10.0.2.2

$ echo 'a b a b' | sed 's/a/X/'
X b a b

$ echo 'a b a b' | sed 's/a/X/g'
X b X b

$ echo 'a b a b' | sed 's/a/X/2'
a b X b

$ sed -E 's/^([0-9:]+) sshd: Disconnected from invalid user ([a-z]+) (.*)/\2 \1 \3/' log.txt
admin 03:02:36 10.0.0.7
root 03:02:37 10.0.0.9
admin 03:04:01 10.0.1.4
03:05:12 sshd: Accepted publickey for ci from 10.0.2.2

$ sed -n -E 's/^[0-9:]+ sshd: Disconnected from invalid user ([a-z]+) .*/\1/p' log.txt
admin
root
admin

$ sed '/Disconnected/!d' log.txt
03:02:36 sshd: Disconnected from invalid user admin 10.0.0.7
03:02:37 sshd: Disconnected from invalid user root 10.0.0.9
03:04:01 sshd: Disconnected from invalid user admin 10.0.1.4

$ cat log.txt
03:02:36 sshd: Disconnected from invalid user admin 10.0.0.7
03:02:37 sshd: Disconnected from invalid user root 10.0.0.9
03:04:01 sshd: Disconnected from invalid user admin 10.0.1.4
03:05:12 sshd: Accepted publickey for ci from 10.0.2.2

Four things to read off it:

  • s/REGEX/REPLACEMENT/ changes the first match on each line, not every match. /g changes all of them, and a number changes the nth one: s/a/X/2 leaves the first a alone.
  • A line that does not match is passed through untouched. The Accepted publickey line comes out of every command above the way it went in. sed prints every line unless you tell it not to.
  • -E turns on extended expressions, where (…) groups, + repeats and | alternates without backslashes, and \1, \2, \3 in the replacement are the groups. -E is the spelling both machines take; GNU's own is -r, and writing -E costs nothing.
  • -n plus p prints only what matched, which is the difference between a report and a log with some lines rewritten. With -n, sed prints nothing on its own, and s/…/…/p prints just the lines the substitution changed. /pattern/!d — delete every line that does not match — is the other way to say it, and is what the Missing Semester ↗ lecture reaches for.

And the last command is the point of the page: cat log.txt after all of that, unchanged.

On a Mac

Everything above is a shared key — the two seds agree on all of it. Changing the file is where they part, and they part in both directions:

Verified output of sed_in_place_sh.sh on Linux — regenerated by tools/run_examples.py, never hand-typed.

$ sed -i 's/alpha/ALPHA/' one.txt; echo "status $?"
status 0

$ cat one.txt
ALPHA
beta

$ sed -i '' 's/alpha/ALPHA/' two.txt; echo "status $?"
sed: can't read s/alpha/ALPHA/: No such file or directory
status 2

$ cat two.txt
alpha
beta

$ sed -i.bak 's/alpha/ALPHA/' three.txt; echo "status $?"
status 0

$ cat three.txt three.txt.bak
ALPHA
beta
alpha
beta

$ sed 's/alpha/ALPHA/' four.txt > tmp.txt && mv tmp.txt four.txt; echo "status $?"
status 0

$ cat four.txt
ALPHA
beta

$ sed 's/beta/BETA/' four.txt > four.txt; echo "status $?"
status 0

$ cat four.txt

$ wc -c < four.txt | tr -d " "
0

Verified output of sed_in_place_sh.sh on macOS — regenerated by tools/run_examples.py, never hand-typed.

$ sed -i 's/alpha/ALPHA/' one.txt; echo "status $?"
sed: 1: "one.txt
": invalid command code o
status 1

$ cat one.txt
alpha
beta

$ sed -i '' 's/alpha/ALPHA/' two.txt; echo "status $?"
status 0

$ cat two.txt
ALPHA
beta

$ sed -i.bak 's/alpha/ALPHA/' three.txt; echo "status $?"
status 0

$ cat three.txt three.txt.bak
ALPHA
beta
alpha
beta

$ sed 's/alpha/ALPHA/' four.txt > tmp.txt && mv tmp.txt four.txt; echo "status $?"
status 0

$ cat four.txt
ALPHA
beta

$ sed 's/beta/BETA/' four.txt > four.txt; echo "status $?"
status 0

$ cat four.txt

$ wc -c < four.txt | tr -d " "
0

GNU sed takes an optional suffix, so -i on its own is in-place with no backup, and -i.bak keeps one. BSD sed takes a required argument, which may be the empty string, so in-place with no backup is -i ''. Each machine's habit is an error on the other, and the errors are unhelpful in the way only a misparsed command line can be: the Mac reads one.txt as the editing script and reports invalid command code o, while Linux reads s/alpha/ALPHA/ as a file name and cannot open it. Neither one edits anything.

sed -i.bak is the line to write. A suffix, attached to the flag, is in-place editing on both machines, and it leaves the original next to the new file. The other portable answer is the one that existed before -i did: write to a new file and mv it over the old one, which is what -i does internally anyway.

What you must not write is the shortcut in the last two commands. Redirecting into the file you are reading truncates it before sed starts, so the command reads an empty file and writes an empty file, with status 0 and not a word of complaint. That is Redirection truncates first, measured, and it is exercise 3 of the lecture this chapter follows.

In zsh and fish

Nothing changes. sed is an external program, and every sed program here is in single quotes, which mean the same thing in bash, zsh and fish: hand the bytes over untouched. The page that measures that in all three shells is the awk program goes in single quotes — its trap is sed's trap too, because sed 's/x/\1/' in double quotes is a shell substitution waiting to happen.

The one fish-shaped difference is how you check the result: echo $status, never echo $?, which fish rejects with a message naming the right variable.

If you are coming from another library

  • Perl. perl -pe 's/…/…/' is sed with a real language behind the slashes, and perl -i -pe is in-place editing that is spelled the same on every machine. The Perl library ↗ covers the one-liner switches.
  • Encodings. The tools chapter ↗ runs sed, awk, cut and sort over text that is not ASCII, where a range like [a-z] stops meaning what it looks like.
  • Python. A rewrite-this-file script in Python has the same hazard with a different name: open for reading and writing at once, and you have truncated it.

See also