Skip to content

chomp leaves the \r

Level: 101 · your data came from Windows, Excel, or an HTTP header

One line: chomp removes $/, which is "\n", so a CRLF file's lines keep their \r — the last field of every row is city\r, five characters that print as four and are never eq to 'city'; s/\R\z// or a :crlf layer removes it from both kinds of file, while $/ = "\r\n" breaks LF files and s/\s+\z// eats data.

Measured

crlf_pl.pl writes the same three-line CSV twice, once with \r\n line endings and once with \n, and reads the first line back. A \r is shown as \r, because printed as itself it would move the cursor and hide:

Verified output of crlf_pl.pl — regenerated by tools/run_examples.py, never hand-typed.

--- chomp on a line from crlf.csv ---
  the line:        id,city\r
  the last field:  city\r   length 5
  eq 'city'?       no

--- four fixes, on both files ---
  crlf.csv
    s/\R\z//                    id,city
    :crlf layer, then chomp     id,city
    local $/ = "\r\n", chomp    id,city
    s/\s+\z//                   id,city
  lf.csv
    s/\R\z//                    id,city
    :crlf layer, then chomp     id,city
    local $/ = "\r\n", chomp    id,city\n1,Kraków\n2,Gdańsk\n
    s/\s+\z//                   id,city

--- and s/\s+\z// takes data with it ---
  s/\R\z//   [WAW   ]
  s/\s+\z//  [WAW]

--- \R is any line break, and \r\n is one of them ---
  split /\R/       4: [one] [two] [three] [four]
  split /[\r\n]/   5: [one] [] [two] [three] [four]

--- chomp removes one $/, and returns how many characters that was ---
  chomp "last\n\n"   removed 1, left "last\n"
  chomp "last"       removed 0

Why the comparison fails

A line from a Windows file ends \r\n. chomp removes $/, and $/ is "\n", so \r stays behind — on the last field, because it is last. Printed to a terminal, \r returns the cursor to the start of the line, so city\r looks like city, and the bug report says "the values are identical but eq is false".

Four fixes, two files

  • s/\R\z// — \R matches any line break, \r\n included as one. Right on both files.
  • A :crlf layer — <:encoding(UTF-8):crlf turns \r\n into \n as it reads, and then chomp works. Right on both files.
  • local $/ = "\r\n" — right on the CRLF file, and on the LF file there is no \r\n anywhere, so the first "line" is the whole file.
  • s/\s+\z// — right on both files, and wrong on data: it also strips the trailing spaces of a fixed-width field such as WAW, which s/\R\z// leaves alone.

\R against [\r\n]

\R treats \r\n as one break, so a string with three different line endings splits into four pieces. [\r\n] is one character, so \r\n is two breaks with an empty field between them.

chomp removes one, and counts

chomp removes one $/ from the end — not every trailing newline — and returns how many characters it removed: 1 from "last\n\n", leaving "last\n", and 0 from a string with no newline.

What to do instead

while (my $line = <$in>) {
    $line =~ s/\R\z//;     # instead of chomp, for data that came from anywhere
    ...
}

If you are coming from another language

See also