Skip to content

xargs turns lines into arguments

Level: 201 · for anyone who has piped a list of file names into a command that ignored them

One line: xargs reads words from stdin and appends them to a command line, which is how a list of lines becomes a list of arguments — and it splits on whitespace, so a name with a space in it arrives as two arguments, while an empty list runs the command once with nothing on Linux and not at all on a Mac.

Most commands take their input on stdin. ls, rm, mv, chmod and git take theirs on the command line, and a pipe cannot reach the command line — that is what xargs is for.

Measured

Verified output of xargs_empty_sh.sh on Linux — regenerated by tools/run_examples.py, never hand-typed.

$ printf "a\nb\nc\n" | xargs echo
a b c

$ printf "a\nb\nc\n" | xargs -n 1 echo
a
b
c

$ printf "a\nb\n" | xargs -I {} echo "<{}>"
<a>
<b>

$ printf "" | xargs echo EMPTY; echo "status $?"
EMPTY
status 0

$ printf "" | xargs -r echo EMPTY; echo "status $?"
status 0

Verified output of xargs_empty_sh.sh on macOS — regenerated by tools/run_examples.py, never hand-typed.

$ printf "a\nb\nc\n" | xargs echo
a b c

$ printf "a\nb\nc\n" | xargs -n 1 echo
a
b
c

$ printf "a\nb\n" | xargs -I {} echo "<{}>"
<a>
<b>

$ printf "" | xargs echo EMPTY; echo "status $?"
status 0

$ printf "" | xargs -r echo EMPTY; echo "status $?"
status 0

Three shapes to know: bare xargs packs as many words as will fit onto one command line and runs it once; -n 1 runs the command once per word; and -I {} runs it once per line, substituting the line wherever {} appears — which is the form to use when the argument is not last, as in mv {} done/{}.

Then the empty list, and the first difference: GNU xargs runs the command once with no arguments at all; BSD xargs does not run it. find … | xargs rm on a day when find matches nothing is rm with no arguments on Linux — harmless for rm, which then complains, and not at all harmless for a command with a useful default. git commit, a build script, anything that does something when handed nothing: it ran, on one of your two machines.

On a Mac

-r is the fix, and it is the fix on both. GNU's --no-run-if-empty says "skip the command if the input is empty", and macOS documents -r as accepted for compatibility with GNU xargs, where it does nothing because not running is what BSD xargs already does. So -r is safe to write everywhere and makes the two machines behave alike — the second command above, on both.

The second difference is the words and the status when the command it ran fails:

Verified output of xargs_names_sh.sh on Linux — regenerated by tools/run_examples.py, never hand-typed.

$ ls d
plain.txt
two words.txt

$ printf "d/plain.txt\nd/two words.txt\n" | xargs ls; echo "status $?"
ls: cannot access 'd/two': No such file or directory
ls: cannot access 'words.txt': No such file or directory
d/plain.txt
status 123

$ printf "d/plain.txt\0d/two words.txt\0" | xargs -0 ls; echo "status $?"
d/plain.txt
d/two words.txt
status 0

$ find d -type f -print0 | xargs -0 ls; echo "status $?"
d/plain.txt
d/two words.txt
status 0

$ find d -type f -print0 | xargs -0 -n 1 basename | sort
plain.txt
two words.txt

Verified output of xargs_names_sh.sh on macOS — regenerated by tools/run_examples.py, never hand-typed.

$ ls d
plain.txt
two words.txt

$ printf "d/plain.txt\nd/two words.txt\n" | xargs ls; echo "status $?"
ls: d/two: No such file or directory
ls: words.txt: No such file or directory
d/plain.txt
status 1

$ printf "d/plain.txt\0d/two words.txt\0" | xargs -0 ls; echo "status $?"
d/plain.txt
d/two words.txt
status 0

$ find d -type f -print0 | xargs -0 ls; echo "status $?"
d/plain.txt
d/two words.txt
status 0

$ find d -type f -print0 | xargs -0 -n 1 basename | sort
plain.txt
two words.txt

d/two words.txt arrives as d/two and words.txt, because xargs splits its input on blanks and newlines alike. Both machines fail; GNU ls says cannot access where BSD ls names the file, and the exit statuses are different for a more interesting reason. BSD xargs returns 1 when the command it ran failed. GNU xargs reserves 123 for exactly that — "one or more invocations exited non-zero" — and keeps 1 for a failure of its own. A script that checks for -ne 0 is fine on both; a script that checks for -eq 1 is not.

The fix is a NUL-separated list, and it is identical on both machines: find … -print0 | xargs -0. NUL is the one byte a file name cannot contain, so it is the only separator that is always unambiguous — the last two commands above list both files correctly, spaces and all. -print0 and -0 are not POSIX, but GNU and BSD both have them, which is what matters here.

In zsh and fish

Nothing changes: xargs is an external program, and by the time it starts, the shell's part is over. In all three shells the words that reach xargs are whatever the pipeline wrote, and {} is not special to bash, zsh or fish — an empty pair of braces is not a brace expansion, so -I {} and -I{} both reach xargs as typed.

What does differ is the loop you would write instead. for f in *.txt exists in all three, and for a handful of files it is clearer than xargs; fish spells the body for f in *.txt; command; end. xargs earns its place when the list is long — it packs many arguments into each command line instead of starting one process per file — or when it comes from a program rather than a glob. To check the result afterwards, fish uses echo $status where bash and zsh use echo $?.

If you are coming from another library

  • Python. subprocess.run([cmd, *args]) passes a list straight to execve, with no shell and no splitting — the structural answer to the problem -0 works around. The Python library ↗ covers what a process receives.
  • C. argv is the array xargs is building. The C library ↗ covers how a program is handed it, and why the kernel, not the shell, has the last word on how long it may be.
  • Rust. std::process::Command::args is the same list-not-string idea; the Rust library's Unix chapter ↗ uses it.

See also