Skip to content

Pipeline

Category: Communication · Status: stub · Lessons: chapter 05, Message passing (planned)

One line: A chain of stages, each a task that reads from the previous stage's channel and writes to the next one's.

Also called: pipes and filters.

How it connects

flowchart LR
  n_channel["Channel"]
  n_pipeline["Pipeline"]
  n_pipeline -->|uses| n_channel
  classDef center stroke-width:3px
  class n_pipeline center
  classDef outside stroke-dasharray: 4 3
  class n_channel outside

In each language
Rust threads joined by mpsc ↗ channels; a stage's receives fail once the stage before it has hung up by dropping its sender
Go goroutines joined by channels, each stage receiving from upstream and sending downstream; the Go blog's pipelines post ↗ notes there is no formal definition of one in Go
C# TPL Dataflow ↗, in-process message passing for dataflow and pipelining tasks
JavaScript ReadableStream.pipeThrough ↗ chains transform streams
Kotlin a pipeline ↗: one coroutine produces a stream of values and others consume, process and pass it on
The operating system the shell pipeline: processes connected by pipes ↗

Where to read more