Koorest

Koorest is an open-source workflow orchestration platform designed to help researchers and developers compose, execute, and monitor complex data pipelines across diverse computing environments. The project emphasizes reproducibility, portability, and extensibility, and provides a modular architecture that can run on local clusters, cloud platforms, or hybrid systems.

Koorest was first released in 2018 by a collaborative initiative among academic labs and industry engineers. It is released under the Apache 2.0 license and is maintained by the Koorest Foundation and a community of contributors. The core engine represents workflows as directed acyclic graphs, with nodes corresponding to tasks and edges to data dependencies. It supports parallel execution, retry policies, and content-addressable data caching to speed up repeated runs.

Key components include a workflow engine, a data catalog for lineage tracking, and a set of pluggable adapters (connectors) for cloud providers, storage systems, and computational backends. Workflows can be authored using YAML or Python DSLs and can be executed via a command-line interface, a REST API, or a Kubernetes operator. Koorest integrates with existing data science tools and supports provenance capture, access control, and audit logging.

Reception and use: Koorest has been adopted by several research groups and data teams seeking a standards-based approach to pipeline management. Critics note a learning curve for new users and the overhead of running on smaller projects.