Skip to content

Records are not always lines

Level: 201 · your data is paragraphs, one blob, fixed-size blocks, or file names from find -print0

One line: <> reads up to $/, the input record separator, which is "\n" only by default — -00 sets it to "" for paragraphs, -0777 to undef for the whole file as one record, $/ = \8 reads eight bytes at a time, and bare -0 splits on NUL — and $. counts records, not lines.

Measured

The script reads demo/notes.txt — three paragraphs, with two blank lines before the last — and demo/fixed.dat, which has no newlines at all:

Verified output of records_sh.sh — regenerated by tools/run_examples.py, never hand-typed.

$ cat notes.txt
Kraków
Main Square
Cloth Hall

Gdańsk
Long Market


Warsaw
Old Town

$ perl -ne 'END { print "$. records\n" }' notes.txt
10 records

$ perl -00 -ne 'END { print "$. records\n" }' notes.txt
3 records

$ perl -00 -ne 'print "record $.: ", (split /\n/)[0], "\n"' notes.txt
record 1: Kraków
record 2: Gdańsk
record 3: Warsaw

$ perl -MO=Deparse -00 -ne 'print' 2>/dev/null
BEGIN { $/ = ""; $\ = undef; }
LINE: while (defined($_ = readline ARGV)) {
    print $_;
}

$ perl -0777 -ne 'print length, " bytes, ", tr/\n//, " newlines, one record\n"' notes.txt
70 bytes, 10 newlines, one record

$ perl -MO=Deparse -0777 -ne 'print' 2>/dev/null
BEGIN { $/ = undef; $\ = undef; }
LINE: while (defined($_ = readline ARGV)) {
    print $_;
}

$ perl -0777 -pe 's/\n{3,}/\n\n/g' notes.txt
Kraków
Main Square
Cloth Hall

Gdańsk
Long Market

Warsaw
Old Town

$ cat fixed.dat
WAW00123KRK00045GDN00678
$ perl -ne 'BEGIN { $/ = \8 } print "[$_]\n"' fixed.dat
[WAW00123]
[KRK00045]
[GDN00678]

$ printf 'first\0second\0' | perl -0 -ne 'chomp; print "[$_]\n"'
[first]
[second]

Four separators

  • Lines — the default, $/ = "\n". notes.txt is ten records, and $. says 10.
  • Paragraphs — -00, which Deparse shows as $/ = "". The empty string is a special value: one or more blank lines end a record, so the double blank line before Warsaw still makes three records, not four. (split /\n/)[0] then takes each paragraph's first line.
  • The whole file — -0777, which Deparse shows as $/ = undef. There is no character 0777 (it is past 0377, the largest octal byte), so perl reads without a separator. With the file in one record, a substitution can see across line breaks: s/\n{3,}/\n\n/g turns the double blank line into a single one, which no line-at-a-time loop can do.
  • Fixed-size blocks — $/ = \8, a reference to a number, reads records of that size. fixed.dat is three eight-byte records with no separator between them. There is no switch for this one, so the one-liner sets $/ in a BEGIN block.

And -0 with no digits sets $/ to the NUL character, the separator find -print0 and xargs -0 use for file names that may contain spaces or newlines. chomp removes whatever $/ is, so it removes the NUL.

$. counts records

Whatever $/ is, $. goes up by one per read. After -00 it is a paragraph number; after -0777 it is 1. A message that says "error on line $." is wrong for every separator but the default.

If you are coming from another language

  • awk. RS="" is awk's paragraph mode, and it is where -00 comes from.
  • Python. A file object iterates over lines only, and splitting on anything else is yours to write. Python's What ends a line ↗ compares its own rules with $/ and Perl's \R.
  • The encodings library. The trailing newline ↗ — what tools do with a last line that has no \n.

See also