parallelization

Parallelization is the practice of performing multiple computations simultaneously by dividing a task into sub-tasks that can be executed concurrently. The goal is to reduce wall-clock time, increase throughput, or improve energy efficiency by exploiting available parallel resources such as multiple processing units or cores.

Parallel programs can follow different models. Shared-memory systems allow threads to access a common address space, while distributed-memory systems use separate memories and communicate via message passing. Common tools include OpenMP for shared memory, MPI for distributed memory, and CUDA or OpenCL for GPUs. Parallel work can be organized as data parallelism, applying the same operation to many data elements, or task parallelism, where different tasks run concurrently.

Architectures range from multi-core CPUs and GPUs to large clusters and cloud platforms. Performance is often described by speedup relative to a serial execution and by efficiency. Amdahl's law provides a bound on maximum speedup based on the program's serial fraction, while Gustafson's law suggests that larger problems can scale better with more resources.

Key challenges include the overhead of coordinating tasks, communication between compute units, load balancing, data dependencies, and limits imposed by memory bandwidth. The granularity of parallelism also influences performance; fine-grained parallelism offers more parallelism but higher overhead, whereas coarse-grained parallelism reduces overhead but may leave resources underutilized.

Common applications include scientific simulations, big data analytics, machine learning training and inference, rendering, and real-time processing. Effective parallelization requires identifying independent sub-tasks, choosing appropriate models, and balancing computation with data movement and synchronization.