dataparallelism

Dataparallelism is a form of parallel computing in which the same operation is applied concurrently to multiple data items. Data items are partitioned into chunks processed independently by parallel units such as cores, vector lanes, or separate nodes, allowing large-scale tasks to be completed more quickly. The data items are typically identical in structure, and the same program is executed across elements, with occasional reductions to combine results.

Techniques include SIMD/vectorization, GPU kernels, and distributed frameworks. Vector processors can operate on multiple data points per instruction; GPUs execute the same kernel across thousands of threads; distributed systems use multiple machines coordinated by frameworks such as MPI or data-parallel libraries. Data parallelism assumes data independence or uses reductions and synchronization to manage dependencies.

Key challenges are load balancing, memory bandwidth, communication overhead, and data-layout issues that affect locality and cache performance. The effectiveness of dataparallel patterns depends on how well the data can be partitioned and how much work can be done in parallel without frequent inter-thread or inter-process communication.

Typical applications include image and signal processing, scientific simulations, and machine learning with minibatch training. Advantages include scalable performance with data size and straightforward programming when the computation is naturally data-parallel. Limitations include poor performance on irregular data, control-flow divergence in vector units, and dependence between elements that prevents parallelization.

See also: parallel computing, SIMD, GPU computing, distributed computing.