Skip to content

uniq only compares neighbours

Level: 101 · for anyone whose uniq left the duplicates in

One line: uniq collapses equal lines only when they are next to each other, which is why every counting pipeline begins with sort — and the count uniq -c prints is right-aligned in a fixed field whose width the two machines do not agree on, so read it with awk, never with cut.

sort | uniq -c | sort -rn | head is the most reused pipeline in this chapter: what are the most common values in this column. Each stage is doing something small, and the first sort is not there for tidiness — without it the answer is wrong.

Measured

Six lines, three distinct values, and no two equal lines adjacent:

Verified output of uniq_needs_sort_sh.sh, identical on Linux and macOS — regenerated by tools/run_examples.py, never hand-typed.

$ cat users.txt
admin
root
admin
ci
admin
root

$ uniq users.txt
admin
root
admin
ci
admin
root

$ sort users.txt | uniq
admin
ci
root

$ sort -u users.txt
admin
ci
root

$ sort users.txt | uniq -d
admin
root

$ sort users.txt | uniq -u
ci

$ sort users.txt | uniq | wc -l | tr -d " "
3

uniq on the unsorted file changes nothing at all: it compares each line with the line before it, and no two neighbours are equal. Sort first and the three values collapse. sort -u is the same answer in one process — it is what to reach for when you only want the distinct values. What sort -u cannot do is count, and counting is why uniq is still in the pipeline:

Verified output of uniq_counts_sh.sh on Linux — regenerated by tools/run_examples.py, never hand-typed.

$ sort users.txt | uniq -c
      3 admin
      1 ci
      2 root

$ sort users.txt | uniq -c | sed -n l
      3 admin$
      1 ci$
      2 root$

$ printf '9 nine\n10 ten\n' | sort
10 ten
9 nine

$ printf '9 nine\n10 ten\n' | sort -n
9 nine
10 ten

$ sort users.txt | uniq -c | sort -rn
      3 admin
      2 root
      1 ci

$ sort users.txt | uniq -c | sort -rn | head -n 1
      3 admin

$ sort users.txt | uniq -c | cut -d' ' -f1 | sed -n l
$
$
$

$ sort users.txt | uniq -c | awk '{print $1}'
3
1
2

$ sort users.txt | uniq -c | sed 's/^ *//' | sed -n l
3 admin$
1 ci$
2 root$

$ sort users.txt | uniq -c | sort -rn | awk '{print $2, $1}'
admin 3
root 2
ci 1

Verified output of uniq_counts_sh.sh on macOS — regenerated by tools/run_examples.py, never hand-typed.

$ sort users.txt | uniq -c
   3 admin
   1 ci
   2 root

$ sort users.txt | uniq -c | sed -n l
   3 admin$
   1 ci$
   2 root$

$ printf '9 nine\n10 ten\n' | sort
10 ten
9 nine

$ printf '9 nine\n10 ten\n' | sort -n
9 nine
10 ten

$ sort users.txt | uniq -c | sort -rn
   3 admin
   2 root
   1 ci

$ sort users.txt | uniq -c | sort -rn | head -n 1
   3 admin

$ sort users.txt | uniq -c | cut -d' ' -f1 | sed -n l
$
$
$

$ sort users.txt | uniq -c | awk '{print $1}'
3
1
2

$ sort users.txt | uniq -c | sed 's/^ *//' | sed -n l
3 admin$
1 ci$
2 root$

$ sort users.txt | uniq -c | sort -rn | awk '{print $2, $1}'
admin 3
root 2
ci 1

uniq -c prefixes each collapsed run with how many lines were in it, and sort -rn then orders by that number, largest first. -n is not decoration: a plain sort compares text, and as text 10 comes before 9 because 1 comes before 9 — the two commands in the middle of the block show it on the same two lines. -r reverses, putting the biggest at the top, and head -n 1 is then the most common value in the file.

The leading spaces in that count are real, and sed -n l above prints each line with a $ at its end so you can see exactly how many there are. They are why cut -d' ' -f1 comes back empty: cut splits on every delimiter, so the first space-separated field of 3 admin is the empty string before the first space. awk splits on runs of whitespace and ignores leading ones, so awk '{print $1}' gets 3. When the count has to be plain text, sed 's/^ *//' strips the padding.

On a Mac

The counts are padded to a different width. BSD uniq right-aligns the count in four columns, GNU uniq in seven — visible in the two fences above, and the reason this page's counting key is split while its first key is shared. Nothing else differs: -d, -u, sort -u and the ordering of sort -rn are identical.

The practical consequence is that no script may depend on the column a count lands in. cut -c over uniq -c output is a script that works on the machine it was written on. awk '{print $1}', or awk '{print $2, $1}' to put the value first, is the same everywhere — and that last form is where a counting pipeline usually wants to end up anyway.

In zsh and fish

Nothing changes. sort and uniq are external programs, the pipeline is built the same way in all three shells, and nothing here is quoted or globbed. The one thing to keep in mind is that fish has no $?: to check whether the pipeline worked, ask echo $status.

If you are coming from another library

  • Python. collections.Counter(lines).most_common() is this entire pipeline, and it does not need the sort — a hash table has no notion of neighbours. The Python library ↗ covers reading stdin into one.
  • Encodings. sort compares bytes under LC_ALL=C and something else entirely under a UTF-8 locale; tr and sort work a byte at a time ↗ measures what changes. Every example here runs under LC_ALL=C.
  • SQL. SELECT user, COUNT(*) … GROUP BY user ORDER BY 2 DESC is the same query. sort is the GROUP BY: it is doing the work that lets uniq be as simple as it is.

See also