The trailing newline¶
Level: 101 → 201 · for anyone with a terminal
One line: A text file's last byte is normally 0a, and printf does not write one — so a perfectly good file can hold a letter, report zero lines, print as a blank line, and make your shell draw a symbol that is nowhere in the file.
The two-byte file that looked empty¶
$ wc -c one.txt
2 one.txt
$ xxd one.txt
00000000: c5bc ..
$ cat one.txt
$
The file is not empty. wc -c says two bytes, xxd shows both, and they spell ż. But cat appears to print nothing, and the reason is not cat — cat wrote the two bytes faithfully. It is that there is no third byte telling the terminal to move to the next row, so the letter is drawn in the last column the cursor was in, and the shell then draws its prompt over the top of it.
Add the byte and everything behaves:
echo with no argument writes a single 0a — it changes nothing about the file, it just finishes the line the file could not finish. That is the whole fix, and the rest of this page is why so many other tools care.
A line ends with a newline¶
POSIX defines a line as a sequence of characters terminated by a newline. Not separated by one — terminated by one. Everything on this page falls out of that one word.
So a file whose last byte is not 0a does not end with a line. It ends with what POSIX calls an incomplete last line: real content, in no line at all. wc -l counts the terminators, so it answers 0 for a file with a letter in it, and it is not being obtuse — it is answering the question it was asked.
This is the same distinction the control characters page draws between LF and CR, one level up: that page is about which byte ends a line, and this one is about whether the last line has one at all.
The character your shell draws is not in the file¶
The three shells disagree about what to show you, and none of them is showing you the file. Same two bytes, three screens:
fish x⏎ a dim U+23CE RETURN SYMBOL, then a fresh line for the prompt
zsh x% a bold reverse-video '%', then a fresh line
bash xP> nothing at all — the prompt starts in the next column
The ⏎ is not in the file and never was: it is fish telling you "the output stopped mid-line". zsh's % says the same thing in a different alphabet. bash says nothing, which is why the letter appears to have been swallowed by the prompt — and why the same command looks broken in one shell and merely odd in another.
That is this library's recurring shape, met before in od -a's question marks, which are drawn by the terminal and not by od: the screen is a rendering, and the rendering has opinions. When the two disagree, xxd is the file.
The check, and the one-byte fix¶
0a means the file ends in a newline; anything else means it does not. To add one:
In the terminal¶
Verified output of trailing_newline_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
1. THE SAME CHARACTER, TWO FILES
$ xxd no_nl.txt
00000000: c5bc ..
$ xxd with_nl.txt
00000000: c5bc 0a ...
Same letter. The second file has a third byte, 0a, and that byte is the
whole subject of this page.
2. HOW BIG, AND HOW MANY LINES
$ wc -c < no_nl.txt | tr -d ' '
2
$ wc -l < no_nl.txt | tr -d ' '
0
$ wc -l < with_nl.txt | tr -d ' '
1
A file with a letter in it has ZERO lines. wc -l counts newline bytes,
and POSIX says a line is characters ENDING WITH one — so the letter sits
in what POSIX calls an incomplete last line, and nothing counts it.
3. THE CHECK: WHAT IS THE LAST BYTE?
$ tail -c 1 no_nl.txt | xxd -p
bc
$ tail -c 1 with_nl.txt | xxd -p
0a
0a means the file ends in a newline. Anything else means it does not.
4. WHY CONCATENATION GLUES
$ cat no_nl.txt no_nl.txt | xxd
00000000: c5bc c5bc ....
$ cat with_nl.txt with_nl.txt | xxd
00000000: c5bc 0ac5 bc0a ......
Without the terminator, cat runs the two files together into one line.
That is the same fact as section 2, seen from the other side: the newline
is not a decoration at the end, it is what closes the line.
5. COMMAND SUBSTITUTION ERASES THE DIFFERENCE
$ a=$(cat no_nl.txt); b=$(cat with_nl.txt)
a is 2 bytes, b is 2 bytes
a = b -> the same string
$(...) strips EVERY trailing newline, so the difference the whole page is
about cannot survive being put in a variable. Useful when you want the
text; a trap when you were trying to measure the file.
6. A READ LOOP SILENTLY DROPS THE LAST LINE
$ printf "one\ntwo\nthree" > three.txt # no trailing newline
$ while read -r line; do echo " got: $line"; done < three.txt
got: one
got: two
Two of three. read returns false on the incomplete last line, so the loop
ends before the body runs — the classic way a data file loses its last row.
$ while read -r line || [ -n "$line" ]; do ... # the fix
got: one
got: two
got: three
7. ADDING THE BYTE
$ printf '\n' >> no_nl.txt
$ xxd no_nl.txt
00000000: c5bc 0a ...
$ wc -l < no_nl.txt | tr -d ' '
1
One byte appended, and the file now has a line in it.
In Python¶
The sharpest part of the story is here, and it is not the writing — it is that the line-splitter you reach for by default cannot see the difference at all.
Verified output of trailing_newline_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. TWO STRINGS, ONE CHARACTER, ONE BYTE APART
no_nl c5 bc 2 bytes
with_nl c5 bc 0a 3 bytes
The letter is two bytes (C5 BC). The newline is the third.
2. splitlines() CANNOT SEE THE DIFFERENCE
'ż' .splitlines() -> ['ż']
'ż\n' .splitlines() -> ['ż']
Identical. splitlines() treats a newline as a TERMINATOR: it closes the
line before it and does not open one after it. That is the right answer
for reading a file, and it is why the difference never reaches your code.
3. split('\n') CAN — BECAUSE IT ASKS A DIFFERENT QUESTION
'ż' .split(chr(10)) -> ['ż']
'ż\n' .split(chr(10)) -> ['ż', '']
split() treats it as a SEPARATOR, so a trailing one opens an empty last
field. Neither function is wrong; they answer 'what are the lines?' and
'what is between the newlines?', and only the second is a byte question.
4. THE TEST TO WRITE, AND THE ONE THAT LIES
no_nl endswith(b'\n') -> False len(splitlines()) -> 1 count('\n') -> 0
with_nl endswith(b'\n') -> True len(splitlines()) -> 1 count('\n') -> 1
Ask the bytes. Counting lines cannot tell you, because both files have
exactly one line — that is what section 2 just proved.
5. WRITING IT: WHICH CALL ADDS THE BYTE
print(x, file=f) adds it (end='\n' is the default)
print(x, file=f, end='') does not
f.write(x) does not — write() writes exactly what you gave it
f.writelines(lines) does not — the name promises a newline it never adds
The shell's pair is the same pair: echo adds it, printf does not.
6. READING A FILE THAT LACKS IT
the file: 'ż\nż'
readlines() -> ['ż\n', 'ż']
Every element ends in a newline except the last, so code that strips a
fixed number of characters off the end damages exactly one row — the
last. Use .rstrip(chr(10)), or splitlines(), and never [:-1].
7. WHICH CHARACTERS splitlines() TREATS AS A LINE ENDING
LF 000a splitlines() -> 2 lines
CRLF 000d 000a splitlines() -> 2 lines
CR 000d splitlines() -> 2 lines
VT 000b splitlines() -> 2 lines
FF 000c splitlines() -> 2 lines
FS 001c splitlines() -> 2 lines
NEL 0085 splitlines() -> 2 lines
LS 2028 splitlines() -> 2 lines
PS 2029 splitlines() -> 2 lines
All nine. split(chr(10)) breaks on exactly one of them, and Rust's
lines() on one too (plus an optional CR in front). So 'the terminator
reading' is not one reading — Python's is the widest of the three, and
the three it adds beyond ASCII (NEL, LS, PS) come from Unicode itself.
In Rust¶
Verified output of trailing_newline_rs.rs — regenerated by tools/run_examples.py, never hand-typed.
1. TWO STRINGS, ONE CHARACTER, ONE BYTE APART
no_nl c5 bc 2 bytes
with_nl c5 bc 0a 3 bytes
len() is bytes, always — the letter is two of them.
2. lines() CANNOT SEE THE DIFFERENCE
"ż" .lines() -> ["ż"]
"ż\n" .lines() -> ["ż"]
Identical, and the docs say so: the final line ending is optional.
lines() also strips a \r before the \n, so it reads CRLF files too.
3. split('\n') CAN
"ż" .split() -> ["ż"]
"ż\n" .split() -> ["ż", ""]
The trailing newline opens an empty last field. Same split as Python's.
4. THE TEST TO WRITE
no_nl ends_with('\n') -> false lines().count() -> 1
with_nl ends_with('\n') -> true lines().count() -> 1
Ask the bytes. Both strings have one line.
5. WRITING IT
println!(..) adds it
print!(..) does not — and Rust wraps stdout in a LineWriter, so
the flush happens ON the newline. Text from a bare
print! sits in the buffer looking like nothing ran.
This does NOT depend on being a terminal, which is the part worth
knowing: C and Python switch to block buffering when stdout is a
pipe, and lose a completed line if they die unflushed. Rust line-
buffers either way. The page has the measurement.
this line was printed with print! and an explicit \n
6. WHICH CHARACTERS END A LINE — AND WHY THIS IS NOT splitlines()
LF 000a lines() -> 2 line(s)
CRLF 000d000a lines() -> 2 line(s)
CR 000d lines() -> 1 line(s) (not a break)
VT 000b lines() -> 1 line(s) (not a break)
FF 000c lines() -> 1 line(s) (not a break)
FS 001c lines() -> 1 line(s) (not a break)
NEL 0085 lines() -> 1 line(s) (not a break)
LS 2028 lines() -> 1 line(s) (not a break)
PS 2029 lines() -> 1 line(s) (not a break)
Only LF ends a line, with an optional CR allowed in front of it.
Python's splitlines() breaks on ALL NINE of these. So lines() and
splitlines() are not the same reading: they agree on LF and CRLF
and part company on the other seven: four ASCII control codes
(CR, VT, FF, FS) and three that are not ASCII at all (NEL, LS, PS).
"The terminator reading" is three different readings¶
This page has been using terminator and separator as if each named one behaviour. The separator half is exact — split('\n') and Rust's split('\n') both break on one byte. The terminator half is not: the three languages disagree about which characters end a line at all, and they were measured rather than assumed.
| ends a line | agrees with | |
|---|---|---|
wc -l, read, the shell |
0a only |
— |
Rust str::lines() |
0a, with an optional 0d in front |
the shell |
Python str.splitlines() |
nine characters | neither |
Python breaks on LF, CRLF, CR, VT (000b), FF (000c), FS (001c), NEL (0085), LS (2028) and PS (2029). Rust breaks on the first two and treats the other seven as ordinary text — both generated blocks above show their own half. So a file containing a 000c between two records is two lines to Python and one to Rust, with no error either way.
Three of the seven are not ASCII at all — NEL, LINE SEPARATOR and PARAGRAPH SEPARATOR are Unicode's own line breaks, which is why this belongs on an encodings page rather than a style guide: splitlines() is making a Unicode decision on your behalf, and lines() is making a byte one. Neither is wrong; they answer different questions, and only one of them changes if your data acquires a U+2028.
The buffer flushes on the newline — and Rust does not care if you are a terminal¶
The Rust block above says print! can leave text sitting in the buffer. The part worth knowing is what that does not depend on:
stdout is a pipe stdout is a terminal
Rust line survives line survives
C LINE LOST line survives
Python LINE LOST line survives
C and Python switch to block buffering when stdout is not a terminal, so a finished line can be lost if the process dies unflushed — the classic "my logs stop just before the crash". Rust wraps stdout in a LineWriter unconditionally, so the newline flushes either way. That is the same byte doing the same job as everywhere else on this page, one layer down: the newline is what closes the line, and here it is also what sends it.
Where the missing byte actually bites¶
| Where | What you see | Why |
|---|---|---|
wc -l |
a file with content reports 0 |
it counts terminators |
cat a b |
the last line of a glued to the first of b |
nothing closed the line |
while read |
the last row of a data file never arrives | read returns false on an incomplete line |
$(cat f) |
the difference vanishes | command substitution strips all trailing newlines |
git diff |
\ No newline at end of file |
git records it, because appending later rewrites that line |
| appending | your next line starts mid-word | >> writes where the file stopped |
The git row is the one that surprises people twice. The marker itself is harmless, but the next commit that appends a line shows the previous last line as changed — one line deleted, two added — even though not one character of its text moved. Only its terminator did.
The $(cat f) row is worth knowing in the other direction too, because it is the reason a one-liner like showing a character beside its bytes works at all: substitution hands you the text without the terminator, so you can put something else after it on the same line.
If you are coming from Python or ABAP¶
Python has the same pair the shell does, under different names: print(x) is echo — it appends end='\n' unless you say otherwise — and f.write(x) is printf, writing exactly the string you gave it. The trap is writelines(), whose name promises a newline it does not add. On the reading side, splitlines() is the terminator reading and split('\n') is the separator reading; the first is almost always what you want, and it is also why a missing final newline usually never reaches your code, right up until the day you compare two files byte for byte.
ABAP (Not machine-checked — CI cannot run ABAP.) The distinction is built into the statement rather than left to a function: TRANSFER ... TO dataset appends the file's separator, while TRANSFER ... TO dataset NO END OF LINE does not — the same echo/printf choice, spelled as a clause. cl_abap_char_utilities=>newline is the byte itself, and the same cr_lf gotcha lives beside it. And READ DATASET into a string strips the terminator, so a downloaded file whose last record lacks one still reads correctly and still comes back one byte shorter than the file — which is where an ABAP-side and a Unix-side byte count disagree by exactly one on the last record and nowhere else.
Try it¶
printf 'ż' > one.txt, thenwc -l one.txt. Explain the0to someone.- Run
cat one.txtinbash,zshandfishin turn and watch three different screens for one unchanged file. - Make a two-row CSV with no final newline and read it with a
while readloop. Count the rows you get. - Commit a file with no trailing newline, append a line, and read the diff. Which line does git say you changed?
Practice¶
A file with a letter in it and zero lines. Make two files: printf 'a\n' > with.txt and printf 'a' > without.txt. Predict wc -c and wc -l for each, then explain how a file with one letter can report zero lines.
Then three consequences. How many times does while read -r line; do …; done < without.txt run its body? What does cat without.txt with.txt produce, and how many lines does it have? And what drew the % your shell showed — which byte of which file was that?
Answers
Verified output of trailing_newline_kata_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
file bytes wc -l wc -c last byte
with.txt 610a 1 2 0a
without.txt 61 0 1 61
without.txt holds a letter and reports ZERO LINES. wc -l does not
count lines; it counts newline BYTES, which is the same number only
for a file that ends in one. POSIX defines a line as ending in 0a, so
wc is right and the file is, strictly, not text.
READING IT BACK IN A LOOP
while read < without.txt iterations: 0
while read < with.txt iterations: 1
read returns false on the last piece when it hits EOF instead of a
newline, so the loop body never runs for it. The data is not lost --
the variable was set -- but the loop dropped it. This is the single
most common way a last record vanishes from a shell pipeline.
CONCATENATION, WHICH IS WHERE IT SPREADS
cat without.txt with.txt -> 61610a (one line: aa)
Two files, one line. The missing byte did not stay in its own file;
it merged two records. A CSV assembled this way has a row that is two
rows, and nothing anywhere reports an error.
WHAT DREW THE SYMBOL YOUR SHELL SHOWED
Nothing in the file. zsh prints a reverse-video % (bash prints
nothing) when the cursor is not in column 1 at the end of output. It
is your PROMPT saying the line was unfinished -- a fact about the
terminal, not a byte you can grep for.
AND THE FIX, WHICH IS ONE BYTE
printf 'a\n' writes it; printf 'a' does not.
echo appends one for you, which is why echo is the safer default for
writing a line and printf the right tool for writing exact bytes.
See also¶
- A character and its bytes on one line — the one-liner this page's section 5 explains
- Control characters — what
LFis, the CR that comes with it on Windows, and the other seven characters Python calls a line ending printfwrites bytes — the other half ofprintfversusecho- Inspecting a file — when the screen and the file disagree, which column is the file