SourceProcessorSink

SourceProcessorSink, often written as source–processor–sink, is a simple three-stage data flow pattern used in software architecture to describe how data moves through a system. It frames a pipeline as three distinct roles: a source that emits data, one or more processors that transform or enrich the data, and a sink that consumes and stores or forwards the results. The model is widely used as a mental model for designing data integration, ETL, and streaming pipelines.

The source provides the initial data stream and can be a file, a network socket, a message broker, a sensor, or any other data provider. The processor(s) apply transformations such as filtering, mapping, aggregation, enrichment, or validation. The sink writes the resulting data to a destination, which may be a database, a file, a search index, another message bus, or a downstream service.

The pattern applies to both batch and streaming contexts. In batch processing, a complete data set is read from the source, processed, and written to the sink. In streaming, data flows continuously, with processors typically handling incremental updates and maintaining state where needed. Design considerations include stateless versus stateful processors, ordering, fault tolerance, backpressure, and idempotency.

SourceProcessorSink is a flexible, modular model that can be extended with additional stages or parallel branches. While not a formal standard, it is a common and widely understood way to describe and implement data pipelines across various technologies and ecosystems.