batchwerk
Batchwerk is a software framework designed to orchestrate and execute large-scale batch jobs across computing resources. It emphasizes declarative workflow descriptions, fault tolerance, and observability to enable repeatable, auditable batch runs.
Its architecture centers on the separation of concerns among a scheduler, an executor, and pluggable backends.
Key features include parallel and distributed execution, backfilling, dynamic task generation, parameterized templates, and resource-aware scheduling.
Typical use cases are ETL pipelines, batch analytics, machine learning model training workflows, and scientific computations
Development and adoption are driven by a community of contributors and users. The ecosystem features plugins