A pipe is not a terminal¶
Level: 201 · working knowledge
One line: isatty() is the one question your program's environment answers differently depending on what is on the other end of file descriptor 1 — and measured on macOS and Linux, across Python 3.11 to 3.14, it changes exactly one thing about sys.stdout — which is enough to lose your output entirely.
Adding | cat to a command should be a no-op. It is not, and the reason is that a program can ask whether anyone is watching. isatty(1) is a single syscall with a yes/no answer, and a startling amount of behaviour hangs off it: whether grep emits colour, whether git opens a pager, whether ls uses columns, and — the one this library cares about — how the language buffers what you print.
The trap this page had to get around first¶
Every example in this library runs with its output captured. So a program that asks sys.stdout.isatty() here can only ever get False: it would be describing a fork in the road while standing on one side of it, and any claim it made about the other side would be assertion rather than measurement.
Both branches are reachable without leaving the standard library. Python's pty ↗ module opens a real pseudo-terminal — a kernel object with a terminal discipline on it, not a simulation — and subprocess runs the same child program down both roads. That comparison is the page. Everything below is one program observed twice, not one program's opinion about what would happen.
One more thing had to be arranged. tools/run_examples.py pins PYTHONUTF8=1 for every example, which is one of the very settings under discussion, so the examples build their children's environments explicitly rather than inheriting anything — {"PATH": …, "LC_ALL": "C"} and nothing else, spelled out in view, which is what CONTRIBUTING asks of an example that reads os.environ.
In Python¶
Verified output of pipe_is_not_a_terminal_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. THE SAME PROGRAM, TWO STDOUTS
LC_ALL=C, nothing else set
terminal stdout isatty=True encoding=utf-8 errors=surrogateescape line_buffering=True
terminal stderr isatty=False encoding=utf-8 errors=backslashreplace line_buffering=True
terminal euro=€
pipe stdout isatty=False encoding=utf-8 errors=surrogateescape line_buffering=False
pipe stderr isatty=False encoding=utf-8 errors=backslashreplace line_buffering=True
pipe euro=€
Four properties on stdout, and exactly one of them moved. isatty()
answered the question honestly both times; line_buffering followed
it; encoding and errors did not budge.
The stderr rows are the control. Its line_buffering is True on both
roads — stderr is line-buffered whether or not anyone is watching,
which is the whole reason a diagnostic reaches you and the stdout
line above it does not. (Its isatty answer is about the DEVNULL and
pipe this parent handed it, and is not the point.)
2. THE ENCODING FORK IS REAL — AND THE TTY IS NOT WHAT DECIDES IT
PYTHONUTF8=0 (and LC_ALL=C)
terminal stdout isatty=True encoding=ascii errors=surrogateescape line_buffering=True
terminal stderr isatty=False encoding=ascii errors=backslashreplace line_buffering=True
terminal euro -> UnicodeEncodeError (reason: ordinal not in range(128))
pipe stdout isatty=False encoding=ascii errors=surrogateescape line_buffering=False
pipe stderr isatty=False encoding=ascii errors=backslashreplace line_buffering=True
pipe euro -> UnicodeEncodeError (reason: ordinal not in range(128))
PYTHONIOENCODING=ascii
terminal stdout isatty=True encoding=ascii errors=strict line_buffering=True
terminal stderr isatty=False encoding=ascii errors=backslashreplace line_buffering=True
terminal euro -> UnicodeEncodeError (reason: ordinal not in range(128))
pipe stdout isatty=False encoding=ascii errors=strict line_buffering=False
pipe stderr isatty=False encoding=ascii errors=backslashreplace line_buffering=True
pipe euro -> UnicodeEncodeError (reason: ordinal not in range(128))
There is the UnicodeEncodeError — and it happened on BOTH roads, not
on the piped one. On this platform sys.stdout's encoding comes from
PYTHONIOENCODING, then UTF-8 Mode, then the locale, and isatty() is
not consulted at any step. Piping a program into `cat` does not
change what it can print; changing the locale does.
(Windows is the other story, and this library cannot measure it: a
console there gets a UTF-8 writer and a redirected stream gets the
ANSI code page, so on Windows the tty really is the fork. That is
where the folklore comes from.)
3. WHAT LINE BUFFERING ACTUALLY COSTS
The child alternates stdout and stderr: OUT 1, ERR 1, OUT 2, ERR 2.
Both streams are pointed at the same place, so the order you read is
the order the bytes arrived.
down a terminal: OUT 1 ERR 1 OUT 2 ERR 2
down a pipe: ERR 1 ERR 2 OUT 1 OUT 2
Through the pipe the two ERR lines came out FIRST, although the
program wrote OUT 1 before either of them. stderr is never buffered
and stdout now is, so the whole of stdout was still sitting in its
own buffer when the program ended, and went out in one piece at
exit. Down the terminal the order is the order it was written in.
How big that buffer is deliberately does not appear here: it is
8192 bytes up to Python 3.13 and 131072 in 3.14, so the size is a
fact about the interpreter and not about the program. The page has
the numbers in a dated table.
Every interleaved log you have ever read out of order is this.
4. AND WHERE THE BUFFER GOES WHEN THE PROCESS DOES NOT
down a terminal, default 'the child wrote this line'
down a pipe, default ''
down a pipe, PYTHONUNBUFFERED=1 'the child wrote this line'
The child wrote a line and then called os._exit, which does not flush.
Down the terminal the line had already left at the newline. Down a
pipe it is simply gone — no error, no exit status, no trace of it
anywhere. Same program, same byte, one of them on the floor.
This is why a crashed job's last words are missing from the log and
present on the screen, and why PYTHONUNBUFFERED is in every
Dockerfile anyone has ever debugged at 2am.
Section 1 is the finding, and it is narrower than the folklore. Four properties of sys.stdout, measured on both roads, and exactly one moved: line_buffering. The encoding did not, the error handler did not. And the stderr rows are the control — line_buffering=True on both roads, because sys.stderr is line-buffered whether or not anyone is watching. That asymmetry is the whole of section 3.
Section 2 is where the hook this page was written from turned out to be wrong, which is the ordinary outcome of pointing a program at a claim. The backlog entry said "piping the same program into cat can raise UnicodeEncodeError where running it directly does not" — and on POSIX it does not. sys.stdout's encoding comes from PYTHONIOENCODING, then UTF-8 Mode ↗, then the locale, and isatty() is not consulted at any step. The error is real and easy to produce; the thing that produces it is the locale, and it fires on both roads at once.
The folklore is not baseless, though — it is Windows, where a console gets a WindowsConsoleIO writing UTF-8 and a redirected stream gets the ANSI code page, so python x.py prints a € and python x.py > out.txt raises. That really is the tty deciding the encoding. This library runs on macOS and Ubuntu and has not measured it, so it is named here and claimed nowhere.
Sections 3 and 4 are the cost, and they are why the one property that moved is enough. A program that alternates stdout and stderr comes back in order down a terminal and inverted down a pipe — both ERR lines first, because stderr went straight out while the whole of stdout sat in its own buffer until exit. Then the sharper version: a child that writes a line and calls os._exit loses it completely down a pipe. No error, no exit status, nothing on stderr — the bytes were in a buffer that nobody flushed. Down a terminal the same line had already gone out at its newline.
That is the shape of the classic report: "it works when I run it, and the log is empty when cron runs it." Both halves are true and neither is a bug.
The Python library runs the same experiment as part of open()'s buffering argument, in section 4 of its Opening a file ↗: the same inversion against a real pty, then the same question asked of a file you opened yourself — including why buffering=0 is refused in text mode.
How much output can be in flight is a fact about your interpreter, not about your program, which is why no number for it appears in the run above:
Python 3.11 8192
Python 3.12 8192
Python 3.13 8192
Python 3.14 131072
The buffer grew sixteenfold in 3.14, which widens the window in section 4 by the same factor: an upgrade moves the amount of output a crash can swallow, without anybody changing a line of code.
In the terminal¶
Verified output of pipe_is_not_a_terminal_sh.sh — regenerated by tools/run_examples.py, never hand-typed.
1. THE TEST EVERY SHELL HAS, AND THE ANSWER IT GIVES HERE
[ -t 0 ] fd 0 is NOT a terminal
[ -t 1 ] fd 1 is NOT a terminal
[ -t 2 ] fd 2 is NOT a terminal
All three, because a test runner captured every one of them. Run this
script by hand and fd 1 and 2 change their answer while fd 0 may not.
[ -t N ] is the shell's isatty(): one syscall, three characters, and
the thing dozens of programs quietly branch on before they print.
2. WHAT A PROGRAM DOES WITH THAT ANSWER: COLOUR
--color=auto ESC bytes: 0 output: 61 6c 70 68 61 0a
--color=always ESC bytes: 4 output: 1b 5b 30 31 3b 33 31 6d 1b 5b 4b 61 6c 1b 5b 6d 1b 5b 4b 70 68 61 0a
--color=never ESC bytes: 0 output: 61 6c 70 68 61 0a
Same match, three answers. Under 'auto' grep asked [ -t 1 ], found a
pipe, and shipped plain text — which is why colour disappears the
moment you add '| less' and why nobody's log file is full of escape
codes. 'always' is the override, and it is what you want when the
thing on the other end of the pipe is a pager that understands them.
3. WHAT 'ALWAYS' ADDED, AND WHY IT IS NOT TEXT
cat -v: ^[[01;31m^[[Kal^[[m^[[Kpha
bytes : 1b 5b 30 31 3b 33 31 6d 1b 5b 4b 61 6c 1b 5b 6d 1b 5b 4b 70 68 61 0a
wc -c : 23 bytes on the wire for 6 bytes of text — 17 bytes of instruction
1b is ESC. Everything between it and the letter 'm' is an instruction
to the terminal, not content — so a coloured line is longer than it
looks, sorts differently, and will not match a regex anchored with ^.
That is the cost of letting a display decision into the byte stream,
and it is the reason 'auto' is the default rather than 'always'.
4. THE IDIOM TO WRITE IN YOUR OWN SCRIPTS
if [ -t 1 ]; then colour=always; else colour=never; fi
grep --color=$colour "$pattern" "$file"
Three lines, and it is the whole of good behaviour here: decide from
the file descriptor, never from a guess about who is running you.
The mirror of it is to always provide the override, because the one
thing [ -t 1 ] cannot see is a human on the far side of a pager.
5. WHAT THE SHELL CANNOT DO
It cannot make a terminal. There is no builtin, and the one tool that
would — script(1) — takes incompatible arguments on the two platforms
and each rejects the other's spelling. That table is on the page.
So a portable script cannot test its own both-branches behaviour,
which is exactly the gap the Python example next door fills with the
pty module. A shell script can ask the question; it cannot arrange
for the other answer.
Section 2 is isatty visible in a tool you use every day. --color=auto means ask the file descriptor, and it is the default precisely because the alternative puts display instructions into a byte stream. Section 3 counts what those instructions cost: 23 bytes on the wire for 6 bytes of text. A coloured line is not longer-looking, it is longer — wc -c says so, sort orders by it, and grep '^al' no longer matches because the line now starts with an ESC. Colour is an in-band signal, which is why it has to be switched off when nobody is watching.
Section 5 is the gap that made the Python example necessary. A shell script can ask the question and cannot arrange for the other answer. The one tool that would is script(1), and its two implementations do not share a command line:
BSD (macOS 26.6.2) script -q /dev/null CMD ARGS…
on Ubuntu: script: unexpected number of arguments
GNU (util-linux 2.39) script -q -c 'CMD ARGS…' /dev/null
on macOS: script: illegal option -- c
Each rejects the other's spelling outright, so there is no portable one-liner. stdbuf is a partial answer for the buffering half — and note it is not a substitute for a terminal, only a way to change the buffering of a child that has not already chosen its own, which Python has.
In Rust¶
Verified output of pipe_is_not_a_terminal_rs.rs — regenerated by tools/run_examples.py, never hand-typed.
1. RUST ASKS THE SAME QUESTION
stdout.is_terminal() = false
stderr.is_terminal() = false
stdin.is_terminal() = false
std::io::IsTerminal, stable since 1.70. Same syscall as the
shell's [ -t 1 ] and Python's sys.stdout.isatty(), and false
here for the same reason: a test runner captured the output.
2. AND CHANGES NOTHING ABOUT HOW IT WRITES
The child wrote OUT 1, ERR 1, OUT 2, ERR 2 with both streams
redirected to one plain FILE — no terminal anywhere. It came
back in this order:
OUT 1 ERR 1 OUT 2 ERR 2
In order. Python, given exactly this arrangement, returns
ERR 1 ERR 2 OUT 1 OUT 2, because its stdout switched to block
buffering the moment it stopped being a terminal. Rust's did
not switch, because it never asked.
3. THE TRADE, WHICH IS A REAL ONE
A LineWriter flushes on every newline, so a program printing a
million lines pays a write syscall per line. Rust's answer is
to make the fast path something you ASK for rather than
something the environment picks for you:
BufWriter::new(stdout().lock()) <- this line came
through one, and is block buffered until the flush
That is the same buffering Python gives you by default down a
pipe — with two differences that matter: you wrote it down, so
the next reader can see it; and the compiler will not let you
forget that it can fail, because flush() returns a Result.
The bug on the Python page — output lost when a process exits
without flushing — is still reachable here. It is just no
longer the default, and no longer invisible.
Rust asks the same question and uses the answer for nothing, and the demonstration needs no pty at all: with both streams pointed at one ordinary file — no terminal in sight — the output comes back in the order it was written. std::io::stdout() wraps a LineWriter unconditionally, so the ordering that Python loses the moment it stops being a terminal is simply never lost here.
That is a deliberate trade rather than a free win, and section 3 names the price: a line-flushing writer costs a write syscall per line, which for bulk output is real. Rust's answer is that the fast path is something you ask for — BufWriter::new(stdout().lock()) — and having asked for it, flush() returns a Result the compiler will not let you drop. So the bug on the Python side of this page is still reachable in Rust. It is just no longer the default, and no longer invisible.
Worth stating plainly because it is easy to get backwards, and this library once did: it is C and Python that switch buffering on isatty, not Rust. The rule they are following is written into ISO C, whose §7.21.3 makes full buffering on stdout conditional on the stream being known not to be an interactive device — so the branch is in the language standard, and Python inherits it from the C runtime it grew up beside.
What to actually do¶
| If you want | Do this | Not this |
|---|---|---|
| a log line you will still have after a crash | print(…, flush=True) on the lines that matter |
hoping |
| a whole stream flushed per line | sys.stdout.reconfigure(line_buffering=True) |
-u for a subtle case |
| a container whose logs appear live | PYTHONUNBUFFERED=1 in the image |
docker logs -f and patience |
| colour through a pager | pipe --color=always into less -R |
--color=always into a file |
| your own script to behave | if [ -t 1 ]; then …; fi, and an override flag |
assuming a human |
The last row is the one people skip. [ -t 1 ] cannot see a human on the far side of a pager, a CI runner that renders ANSI, or a colleague who piped you into tee. Detect by default, and always let the caller say otherwise.
If you are coming from Python or ABAP¶
Python. The mechanism is the Python here, so the bridge runs the other way: what transfers to the shell is that [ -t 1 ] and sys.stdout.isatty() are the same syscall, and what transfers to Rust is that is_terminal() is too. The one Python-specific thing to carry is that print(flush=True) and sys.stdout.reconfigure() are the precise tools and -u is the blunt one — and that os._exit skips every flush by design, which is what makes it the right call after fork() and the wrong call anywhere else.
ABAP. There is no terminal and no isatty, and that is the interesting part rather than a gap. An ABAP program's output goes to a list (WRITE), a spool request, or a file (TRANSFER), and which one it is was decided before your code ran — by whether the job is a dialog step or a background step, which sy-batch reports. So ABAP asks the same question with a different name, one layer up: am I being watched? is sy-batch, and the consequence is the same shape — a WRITE that a user reads on screen in a dialog step becomes a spool list nobody opens in a background job. The buffering half has an equivalent too: TRANSFER to a dataset is buffered by the application server and a CLOSE DATASET is what commits it, so a work process that dies without one loses exactly what section 4 loses. Close your datasets explicitly rather than relying on the end of the program. (Not machine-checked — CI cannot run ABAP.)
Try it¶
- Run something of yours twice — once plainly, once with
| cat— and diff the two outputs. Anything that differs is a program that asked. - Take a long-running script and watch it under
python3 script.py | tee logversuspython3 -u script.py | tee log. The first one is silent for a while and then arrives in slabs. - Find a program in your stack that has lost its last words in a log. Check whether it exits through
os._exit,os.abort, a signal, or a containerSIGKILL— all four skip the flush, and none of them reports it. git log | catandgit log. Thengit config core.pagerand work out which of the two you have been reading.
Practice¶
The same script, two commands. show.py prints sys.stdout.isatty(), sys.stdout.encoding and sys.stdout.line_buffering, then prints a euro sign. You run:
- Which of those four lines differ between the two runs?
- In some environments the euro line raises
UnicodeEncodeError. Name the environment — and say whether the pipe is what causes it. - Now
show.pyends withos._exit(0). Which run loses its output, and how much of it?
Answers
Verified output of pipe_is_not_a_terminal_kata_py.py — regenerated by tools/run_examples.py, never hand-typed.
1. WHICH LINES DIFFER
LC_ALL=C, the ordinary case
python3 show.py isatty=True encoding=utf-8 line_buffering=True
python3 show.py euro=€
python3 show.py | cat isatty=False encoding=utf-8 line_buffering=False
python3 show.py | cat euro=€
One property out of three. isatty() reports what it was asked and
line_buffering follows it; encoding is identical on both roads. The
euro printed fine down the pipe, which answers the second question
before it is asked.
2. THE UnicodeEncodeError, AND WHAT REALLY CAUSES IT
PYTHONUTF8=0 (still LC_ALL=C)
python3 show.py isatty=True encoding=ascii line_buffering=True
python3 show.py euro raised UnicodeEncodeError
python3 show.py | cat isatty=False encoding=ascii line_buffering=False
python3 show.py | cat euro raised UnicodeEncodeError
Both roads raised it, and both roads say so in the same words. The
C locale means ASCII, UTF-8 Mode was switched off, and stdout could
not encode a euro sign no matter who was on the other end. The
locale causes this, not the pipe.
Anyone who has seen it appear on adding '| cat' was on Windows,
where a console really does get a different writer from a redirected
stream — the one platform this library cannot measure.
3. WHICH RUN LOSES ITS OUTPUT
The same script again, with one line added at the end: os._exit(0),
which skips every flush on the way out.
LC_ALL=C, ending in os._exit(0)
python3 show.py isatty=True encoding=utf-8 line_buffering=True
python3 show.py euro=€
python3 show.py | cat (nothing arrived at all)
The pipe run lost BOTH lines — not truncated, not garbled, gone,
with exit status 0 and nothing on stderr. Down the terminal the same
two lines had already left at their newlines. That is the price of
the one property that changed in section 1, and it is charged only
when something goes wrong, which is when you needed the output.
The fixes, in order of how much you should like them:
flush=True on the print that matters precise
sys.stdout.reconfigure(line_buffering=True) for a whole stream
python3 -u / PYTHONUNBUFFERED=1 blunt, and fine in a
container
never call os._exit the actual bug here
See also¶
- Locale and
LC_CTYPE— the setting that really does decidesys.stdout's encoding, in full - The trailing newline — the byte a line-buffered stream flushes on
- Opening a file — the same locale default, on the other end of the program
- Opening a file, in the Python library ↗ — section 4 there is section 3 here, run against a real pty, plus
buffering=on a file you opened yourself - The byte that means something to somebody else — colour escapes as an in-band signal, and what else rides that channel
printfwrites bytes — putting exact bytes on the pipe this page is about