Task queue¶

Category: Communication · Status: stub · Lessons: chapter 05, Message passing (planned)

One line: A queue of jobs that workers take and run — within one program, or across processes and machines with a broker in between.

Also called: job queue, work queue, Celery.

How it connects¶

flowchart LR
  n_producer_consumer["Producer-consumer"]
  n_task_queue["Task queue"]
  n_task_queue -->|uses| n_producer_consumer
  classDef center stroke-width:3px
  class n_task_queue center
  classDef outside stroke-dasharray: 4 3
  class n_producer_consumer outside

Is built on: Producer-consumer
See also: Publish-subscribe and broadcast, Worker pool

In each language¶


Java	`BlockingQueue` ↗, which an `ExecutorService` uses for its waiting tasks
Python	`queue.Queue` ↗ between threads, `asyncio.Queue` ↗ between tasks
Elsewhere	Celery ↗ runs Python tasks on workers fed through a message broker

Where to read more¶

In the books: Programming with POSIX Threads, David R. Butenhof — ch. 7, '"Real Code"' → 'Work queue manager'
In the books: Python Concurrency with asyncio, Matthew Fowler — ch. 12, 'Asynchronous queues'
In the books: Parallel Programming with Python, Jan Palach — ch. 7, 'Distributing Tasks with Celery'
In the books: Designing Distributed Systems, Brendan Burns — ch. 10, 'Work Queue Systems'
In the books: High Performance Python, Micha Gorelick, Ian Ozsvald — ch. 10, 'Clusters and Job Queues'
In the books: Python Parallel Programming Cookbook, Giancarlo Zaccone — ch. 5, 'Distributed Python' → 'Using Celery to distribute tasks'