rparallel

Rparallel is a library for the R programming language that provides abstractions for parallel execution of expressions. It aims to simplify parallelism by offering high-level patterns for applying a function over collections and aggregating results, while delegating the details of scheduling to pluggable backends. It supports multicore processing on a single machine and distributed execution on clusters.

Rparallel uses a modular scheduler with pluggable backends. Local backends run tasks in separate R processes, using fork on Unix-like systems or sockets on other platforms. Distributed backends dispatch work to worker nodes via a cluster manager. Tasks describe inputs, functions, and dependencies; results are returned to the master and can be recombined.

Core APIs include par_map, par_apply, and par_reduce for common patterns, as well as par_for for iterative workloads. A plan mechanism selects a backend at runtime (for example, a multicore plan or a cluster plan) and can configure chunk size, memory limits, and error handling. RNG state can be preserved across workers to ensure reproducibility.

Rparallel emphasizes compatibility with existing R data structures by serializing objects between master and workers. It incurs overhead for inter-process communication and is best for CPU-bound tasks with substantial work per unit. Developers should consider backend capabilities and platform differences when writing parallel code.

Rparallel is an open-source project that builds on established R parallel tooling and aims for reproducible results through predictable seeds and documented error semantics. See also parallel, future, and foreach.