Snakemake

Snakemake is an open-source workflow management system designed to create and execute data analysis pipelines in a reproducible and scalable way. It uses a Python-based language to define a set of rules in a Snakefile, where each rule specifies how to generate one or more output files from given input files, possibly with parameters, shell commands, Python functions, or scripts. Snakemake automatically constructs the workflow’s directed acyclic graph from the rules and determines which jobs must run to produce the requested targets.

Core concepts include wildcards, which generalize rules to many samples, and checkpoints, which enable dynamic workflows when the full set of outputs is not known in advance. Inputs, outputs, parameters, logs, and resources are declared in rules to promote reproducibility and fine-grained control over execution. Subworkflows allow reusing existing Snakefiles, and wrappers provide access to community-available steps.

Execution can occur on a local machine or scale to clusters and cloud environments. Snakemake supports various execution backends and profiles, integrates with job schedulers such as SLURM, SGE, and PBS, and can run inside container environments (Docker, Singularity) or in Conda environments for isolated dependencies. It can generate and visualize the workflow DAG and offers dry-run options to inspect planned execution.

Snakemake is widely used in genomics, transcriptomics, and other areas of computational biology but is applicable to any data-analysis workflow. It emphasizes reproducibility, portability, and auditability by recording dependencies and environment configurations, and has an active community and documentation, including a registry of wrappers and tutorials.