A tr class is a set, and only two of them translate¶
Level: 101 · you have typed tr '[:lower:]' '[:upper:]' and wondered what the other ten classes hold
One line: Each of the twelve classes names a fixed set of characters, which -d deletes, -c turns inside out and -s squeezes, and in the C locale Linux and a Mac agree on every member. The second string of a translation is different: POSIX allows only [:upper:] and [:lower:] there, so GNU tr refuses tr '[:digit:]' '[:xdigit:]' with status 1, while BSD tr on a Mac runs it and exits 0.
Measured¶
A table of classes usually paraphrases the manual, with entries like "whitespace" and "punctuation". This one is computed. Every byte from \000 to \177 goes through tr -dc '[:class:]', and whatever is left is printed, with runs written the way tr itself spells a range:
Verified output of tr_classes_sh.sh, identical on Linux and macOS — regenerated by tools/run_examples.py, never hand-typed.
class bytes members, of the 128 ASCII bytes
[:alnum:] 62 [0-9A-Za-z]
[:alpha:] 52 [A-Za-z]
[:blank:] 2 [\t ]
[:cntrl:] 33 [\000-\037\177]
[:digit:] 10 [0-9]
[:graph:] 94 [!"#$%&'()*+,-./0-9:;<=>?@A-Z[\\]^_`a-z{|}~]
[:lower:] 26 [a-z]
[:print:] 95 [ !"#$%&'()*+,-./0-9:;<=>?@A-Z[\\]^_`a-z{|}~]
[:punct:] 32 [!"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~]
[:space:] 6 [\t-\r ]
[:upper:] 26 [A-Z]
[:xdigit:] 22 [0-9A-Fa-f]
[:blank:]is two bytes, space and tab. It is not whitespace in general. GNU's help says "horizontal whitespace", and that is exactly this list.[:space:]is six.\t-\ris tab, newline, vertical tab, form feed and carriage return, and then the space. So-d '[:space:]'removes line breaks and-d '[:blank:]'does not.[:print:]is[:graph:]plus the space. That is the whole difference between 95 and 94.[:punct:]is every printable character that is not a letter, a digit or a space, 32 of them._,$,`and\all count.[:cntrl:]includes tab and newline, along with the rest of\000-\037and DEL,\177. That matters for-dbelow.[:xdigit:]is 22: ten digits andA-Fin both cases.
These are the C locale's sets, which is what the runner pins. In a UTF-8 locale a Mac's tr '[:upper:]' '[:lower:]' also lowercases ŻÓŁW and Linux's does not. The Encodings library measures that on tr and sort work a byte at a time ↗.
The same classes, with one string and a switch:
Verified output of tr_d_c_s_sh.sh, identical on Linux and macOS — regenerated by tools/run_examples.py, never hand-typed.
$ cat note.txt
Meet in room 101
at 9:30. Bring two pens!
$ tr -d '[:digit:]' < note.txt
Meet in room
at :. Bring two pens!
$ tr -d '[:punct:]' < note.txt
Meet in room 101
at 930 Bring two pens
$ tr -dc '[:digit:]' < note.txt
101930
$ tr -dc '[:digit:]\n' < note.txt
101
930
$ tr -d '[:cntrl:]' < note.txt
Meet in room 101at 9:30. Bring two pens!
$ tr -s '[:blank:]' < note.txt
Meet in room 101
at 9:30. Bring two pens!
$ tr -s '[:alpha:]' < note.txt
Met in rom 101
at 9:30. Bring two pens!
$ tr -s '[:space:]' '\n' < note.txt
Meet
in
room
101
at
9:30.
Bring
two
pens!
-ddeletes every member wherever it is.room 101becameroomwith its trailing space, and9:30became:.-ccomplements the set, so-dc '[:digit:]'keeps only the digits. A newline is not a digit, so it went too: the two lines ran together into101930, with no final newline. In a terminal the next prompt would start right after the0; here the script'sechoends the line. Put\nin the set and the lines stay.-d '[:cntrl:]'joins lines, because newline is a control character. To remove only newlines, name the one you mean:tr -d '\n'.-ssqueezes a run of one repeated member.[:blank:]turned two and three spaces into one.[:alpha:]did something few people want:MeetbecameMetandroombecamerom, becauseeeandooare runs of a letter.- With two strings,
-ssqueezes after translating, using the second set.tr -s '[:space:]' '\n'turns every run of spaces and newlines into a single newline, which puts one word on each line. That is the usual first stage of a word count.
On a Mac¶
Both outputs above are shared keys: in the C locale BSD tr and GNU tr agree on every member, and on what -d, -c and -s do with them. Translating is where they part. A class on the left of a translation is fine on both. A class on the right is not:
Verified output of tr_translate_classes_sh.sh on Linux — regenerated by tools/run_examples.py, never hand-typed.
$ echo 'Room 42b, Floor 3' | tr '[:digit:]' '#'
Room ##b, Floor #
$ echo 'Room 42b, Floor 3' | tr '[:upper:]' 'a-z'
room 42b, floor 3
$ echo 'Room 42b, Floor 3' | tr '[:lower:]' '[:upper:]'
ROOM 42B, FLOOR 3
$ echo 'Room 42b, Floor 3' | tr '[:digit:]' '[:xdigit:]'; echo "status $?"
tr: when translating, the only character classes that may appear in
string2 are 'upper' and 'lower'
status 1
$ echo 'Room 42b, Floor 3' | tr '[:alpha:]' '[:digit:]'; echo "status $?"
tr: when translating, the only character classes that may appear in
string2 are 'upper' and 'lower'
status 1
$ echo 'Room 42b, Floor 3' | tr 'a-z' '[:upper:]'; echo "status $?"
tr: misaligned [:upper:] and/or [:lower:] construct
status 1
Verified output of tr_translate_classes_sh.sh on macOS — regenerated by tools/run_examples.py, never hand-typed.
$ echo 'Room 42b, Floor 3' | tr '[:digit:]' '#'
Room ##b, Floor #
$ echo 'Room 42b, Floor 3' | tr '[:upper:]' 'a-z'
room 42b, floor 3
$ echo 'Room 42b, Floor 3' | tr '[:lower:]' '[:upper:]'
ROOM 42B, FLOOR 3
$ echo 'Room 42b, Floor 3' | tr '[:digit:]' '[:xdigit:]'; echo "status $?"
Room 42b, Floor 3
status 0
$ echo 'Room 42b, Floor 3' | tr '[:alpha:]' '[:digit:]'; echo "status $?"
9999 429, 59999 3
status 0
$ echo 'Room 42b, Floor 3' | tr 'a-z' '[:upper:]'; echo "status $?"
ROOM 42B, FLOOR 3
status 0
- The first three agree. A class in string1 works with a character or a range in string2, and
[:lower:]against[:upper:]is the case conversion everyone uses. - For
[:digit:]to[:xdigit:], Linux refuses and a Mac says nothing. GNUtrprints the error, writes no output and exits 1. BSDtrpairs0-9with the first ten hex digits, which are0-9again, so the text comes out unchanged with status 0. The command did nothing on either machine, but only Linux says so. [:alpha:]to[:digit:]shows what BSD does with a short string2: it repeats string2's last character.A-Jbecame0-9and every later letter became9, soRoomis9999, andFlooris59999becauseFis the sixth capital.a-zto[:upper:]is "misaligned" to GNU. GNU takes[:upper:]in string2 only when[:lower:]is at the same position in string1. BSD maps the range onto the class and prints what you wanted.
Linux is following the standard here. Unless -d and -s are both given, POSIX says "only character class names lower or upper are valid in string2". GNU's help only mentions that the two "may be used in pairs", which is why that sentence reads like a hint and not a rule. BSD goes beyond the standard. POSIX also leaves a string2 shorter than string1 unspecified, and BSD's own manual notes that repeating the last character is allowed but not required. So a script written on a Mac can fail on Linux, while a script written on Linux runs on a Mac. For anything other than case conversion, keep the class on the left and a plain character on the right, as tr '[:digit:]' '#' does above.
In zsh and fish¶
tr is the same program whichever shell starts it, so nothing on this page changes in zsh or fish. The quotes are what keep it that way. Every command here single-quotes its sets, and single quotes pass [:digit:] to tr unchanged in all three shells. Without them, bash and zsh read [:lower:] as a filename pattern first. That is the next lesson, where zsh refuses to run the book's command at all.
If you are coming from another library¶
- Encodings.
trandsortwork a byte at a time ↗ runs these classes oncaféandŻÓŁW, wheretr -ddeletes half a character and the two machines lowercase differently. - Python.
translateis a table, keyed by ordinal ↗ is Python'str. Is it a letter? ↗ covers whatstr.isalpha()means in place of[:alpha:]. - C. Each class is a
<ctype.h>test (isalpha,isblank,ispunctand so on) asked of every byte. BSD's manual sends you toctype(3)for the members. Acharis a byte, not a character ↗ explains why they take one byte at a time. - Perl and Ruby. The class names work inside a regex bracket. Ruby's
\wis ASCII,[[:alpha:]]is not, and\bsides with Unicode ↗ and\his a hex digit in Ruby and a blank in Perl ↗ use them in patterns. Neither language's owntrtakes classes, though. Perl'str///(sees///andtr///return counts ↗) and Ruby'sString#trboth read[:lower:]as a list of the characters[,:,l,o,w,e,rand]:
$ perl -e '$_ = "hello, World"; tr/[:lower:]/X/; print "$_\n"'
hXXXX, WXXXd
$ ruby -e 'puts "hello, World".tr("[:lower:]", "X")'
hXXXX, WXXXd
h, W and d are not in that list. e, l, o and r are.
See also¶
- An unquoted
[:lower:]is a glob - GNU coreutils manual,
trinvocation ↗ and Character arrays ↗ - POSIX,
tr↗: the rule for classes in string2 - FreeBSD
tr(1)↗: the BSDtrthat macOS ships