flyte

Flyte is an open-source workflow automation platform designed to orchestrate scalable, reusable data and machine learning pipelines. Originating at Lyft, it provides end-to-end execution, management, and observability of containerized tasks running on Kubernetes. Developers author workflows and tasks using a Python SDK, and those workflows are compiled into directed acyclic graphs for execution by a Kubernetes-native engine.

Key concepts in Flyte include tasks, workflows, and launch plans. A task defines a unit of work, typically implemented as a container image. A workflow describes a DAG of tasks with strongly typed inputs and outputs. A launch plan attaches a default configuration and versioning to a workflow, enabling repeatable executions with consistent defaults. Flyte supports data artifacts, lineage, parameterization, retries, timeouts, and dependency management to ensure reproducibility.

The architecture comprises several components. FlyteKit is the Python-based SDK used to author tasks and workflows. FlytePropeller is the Kubernetes-based workflow engine that executes tasks. FlyteAdmin provides the centralized API layer for metadata, scheduling, authentication, authorization, and user management. FlyteConsole offers a web-based user interface for monitoring, governance, and collaboration. The platform stores metadata in a relational or transactional backend and utilizes Kubernetes resources such as Pods and Jobs to run tasks. It supports pluggable executors and integrations with cloud storage, data warehouses, and ML tooling.

Flyte is distributed under the Apache 2.0 license and maintained as an active open-source project with a growing ecosystem of plugins and integrations. It is used to implement data processing, ETL, and ML pipelines in organizations seeking reproducibility, versioning, and multi-user collaboration across scalable compute environments.