Kestra

Kestra is an open-source data workflow orchestration platform designed to help organizations author, schedule, and monitor data pipelines. It provides a centralized system for defining workflows that coordinate the execution of discrete tasks across diverse data systems, such as databases, data warehouses, messaging systems, and processing engines.

Workflows in Kestra are defined declaratively in YAML or JSON and are executed by a server that coordinates task execution. A workflow can be triggered by time-based schedules or external events, include conditional branches, support parallelism, and manage retries. Developers implement custom tasks and register them so they can be invoked as steps within workflows.

Architecture and deployment: Kestra is designed with cloud-native principles and can be deployed on Kubernetes or other environments. It typically comprises a server component that runs the orchestration logic, a web-based user interface for authoring and monitoring workflows, and an execution layer backed by a metadata store. The system emphasizes observability with logs, metrics, and tracing for each run.

Key capabilities include event-driven triggers, scheduling, retry and backoff strategies, versioning and reproducibility of workflows, and rich visibility into execution history. It provides a set of built-in tasks and connectors to common data systems and allows developers to add custom tasks to extend functionality. It also supports governance features such as audit trails and access control through its UI and API.

Kestra originated as an open-source project to offer a modern alternative to traditional workflow schedulers. It maintains an active community of users and contributors and is used by data teams to automate ETL/ELT processes, data quality checks, and data delivery pipelines in cloud-native environments.