Skip to content

Data parallelism

Category: Parallelism · Status: stub · Lessons: chapter 07, Parallelism

One line: The same operation applied to many pieces of data at once, each piece on its own core or vector lane.

How it connects

flowchart LR
  n_data_parallelism["Data parallelism"]
  n_gpu_computing["GPU computing"]
  n_map_reduce["Map-reduce"]
  n_parallel_iterators["Parallel iterators and streams"]
  n_parallel_prefix_sum["Parallel prefix sum"]
  n_parallelism["Parallelism"]
  n_simd["SIMD"]
  n_task_parallelism["Task parallelism"]
  n_data_parallelism ---|vs| n_task_parallelism
  n_data_parallelism -->|is a| n_parallelism
  n_gpu_computing -->|is a| n_data_parallelism
  n_map_reduce -->|is a| n_data_parallelism
  n_parallel_iterators -->|is a| n_data_parallelism
  n_parallel_prefix_sum -->|is a| n_data_parallelism
  n_simd -->|is a| n_data_parallelism
  classDef center stroke-width:3px
  class n_data_parallelism center
  classDef outside stroke-dasharray: 4 3
  class n_gpu_computing,n_map_reduce,n_parallel_iterators,n_parallel_prefix_sum,n_parallelism,n_simd,n_task_parallelism outside

In each language
Rust Rayon's parallel iterators ↗; the standard library offers only thread::scope ↗
C++ std::execution::par ↗ (C++17) on a standard algorithm, or par_unseq to allow vectorization as well
Java Parallel streams ↗
Python multiprocessing.Pool.map ↗
C# Data parallelism ↗ with Parallel.For and Parallel.ForEach
Swift DispatchQueue.concurrentPerform ↗ runs a block the given number of times, balanced across cores
Haskell parMap ↗ in the parallel package

Where to read more