polyppit

Polyppit is an open-source software framework for building and executing data processing pipelines. It enables scientists and engineers to compose modular tasks, manage data lineage, and run analyses from a laptop to cloud clusters. The framework emphasizes reproducibility, portability, and collaborative development through a plug-in architecture that supports diverse data formats and compute backends.

Polyppit originated from a collaboration between computational biology and data-engineering researchers. The project began in 2019 to unify multiple workflow tools within a single extensible platform. The first public release appeared in 2020, and subsequent major updates added a declarative workflow language and a policy-driven execution engine.

Architecture: The core consists of an execution engine, a plugin registry, and a data model for polyplets, which are modular units that pair a task with its inputs and outputs. Workflows are defined as directed acyclic graphs; tasks run in containers and are scheduled for local, cloud, or HPC environments. Provenance and versioned data are built in, supporting reproducibility and auditability. The plugin system enables data connectors, compute backends, and custom task logic.

Usage: Polyppit has been applied in genomics, environmental modeling, and large-scale data analytics. It supports common data formats and integrates with cloud services and container technologies. Its emphasis on provenance makes it suitable for research workflows and regulated environments.

Reception and governance: The project is maintained by an open community with a merit-based governance model and releases under the Apache 2.0 license. Users praise modularity and reproducibility while noting a learning curve for new users and the overhead of managing large deployments.

See also: Nextflow, Apache Airflow, Snakemake.