Skip to content

Streams — reading a big file in pieces, and backpressure

Level: 301 · for readers who have copied a file too big to read at once

Stub — an outline, not a lesson. There is no runnable example behind this page yet, so nothing on it has been through the check that backs every other claim in this library. The questions below are what the finished page has to answer.

One line: write() returning false is backpressure: the writable's buffer has reached its highWaterMark, 64 KiB by default (16 KiB on Windows). Nothing stops you writing anyway — memory just grows — unless you wait for drain or let pipeline wait for you.

Keywords: stream.pipeline, highWaterMark, drain, createReadStream, Writable

What the finished page will answer

  • After how many 1 KiB writes to a slow Writable does write() return false, and what does it return if you keep writing?
  • What is writableLength after ignoring false and writing 1 MiB more, and when does drain fire?
  • What chunk sizes does createReadStream deliver for a 200 KiB file, and how does its highWaterMark option change them?
  • How does peak rss compare when copying a large file with an unchecked write loop and with pipeline?
  • What does pipeline do with an error in the middle stage, and what happens to the other two streams?

Examples it will need

  • streams_write_returns_false_js.js — the write at which a slow Writable first returns false, its writableLength after writing on regardless, and when drain fires
  • streams_read_chunk_sizes_js.js — the chunk sizes createReadStream yields for a 200 KiB file at the default and at a 16 KiB highWaterMark
  • streams_pipeline_memory_js.js — peak rss copying a large generated file with an unchecked write loop and with pipeline

See also

Sources to start from