rdoWork

rdoWork is a software framework to automate and orchestrate distributed data workflows. It targets data engineering teams building pipelines, ETL processes, and remote data operations (RDO), providing a lightweight runtime, a task orchestration layer, and pluggable connectors for cloud and on‑prem environments.

Core features include DAG-based scheduling, dependency resolution, retry and timeout policies, parallel execution, and resource-aware scheduling to balance workload and performance. The system emphasizes deterministic execution and robust failure handling to support complex data workflows.

Its architecture centers on a small core engine that coordinates workers and a plugin system for executors. Workflows can be authored in YAML, JSON, or a Python DSL and are exposed via a REST API and a command-line interface, enabling automation and integration with existing tooling.

Tasks run in isolated environments with support for idempotent execution, comprehensive logging, and audit trails to ensure traceability and recoverability across runs and environments.

The project emphasizes extensibility through connectors to data stores and services such as S3, GCS, Azure Blob, Postgres, and MySQL, as well as messaging queues. SDKs for Python and Go are provided, along with a web-based dashboard for monitoring and management.

rdoWork is a community-driven open-source project released under a permissive license, with ongoing development and governance through community contributions and independent maintainers.

In practice, it is used for data pipelines, migrations, and remote processing tasks. Reviewers note ease of integration, while acknowledging a learning curve for more advanced workflow patterns.

Within the landscape of workflow orchestration tools, rdoWork is often compared with Apache Airflow, Prefect, and Dagster.