DistributedM

DistributedM is an open-source distributed computing framework designed to orchestrate and execute workloads across clusters of machines. It aims to provide scalable processing, fault tolerance, and portability across on-premises and cloud environments.

The system consists of a scheduler and a set of workers that execute tasks arranged as directed acyclic graphs. It supports data locality, dynamic resource management, and outcome reproducibility via deterministic scheduling. A pluggable storage layer allows use of distributed filesystems or object stores.

Core features include multi-model support (batch processing, streaming, and iterative workloads), automatic scaling, task parallelism, and sandboxed execution for isolation. Security features include role-based access control, encryption in transit, and transparent access controls for data stores.

DistributedM is controlled by an open-source community-driven project. The architecture favors a modular design with separate components for job submission, scheduling, execution, and state management. It provides APIs in several languages and bindings to common data platforms.

Adoption notes: It is used for data processing pipelines, analytics, and machine learning workflows that require cross-cluster execution. It integrates with common storage backends and messaging systems. Documentation emphasizes configuration simplicity and observability through metrics and logging.

Limitations include maturity of the ecosystem and varying performance across heterogeneous hardware. The project encourages collaboration and provides a governance model to manage feature proposals and releases.