redFlow

redFlow is an open-source data processing framework for real-time streaming analytics and batch processing. It models computation as directed acyclic graphs of operators and emphasizes low-latency execution, backpressure-aware streaming, and fault-tolerant distributed processing.

In redFlow, programs consist of operators (such as map, filter, window, and join) connected by streams. The runtime provides a distributed scheduler, state management, and a lineage log to enable fault recovery. It supports backpressure propagation, windowed aggregations, checkpointing, and processing semantics that can be configured for exactly-once or at-least-once delivery.

Origins and development: redFlow originated at RedWave Systems as an internal streaming tool and was released as an open-source project in 2017. The project has seen several major releases, with a 2.x series in 2019 and a 3.x series in 2021–2023 introducing improved scalability, state backends, and language bindings.

As of 2023, redFlow provides connectors for Apache Kafka, AWS S3, Cassandra, and PostgreSQL, and offers client libraries for Java, Python, and Go. It is used in sectors such as finance, telecommunications, and IoT for real-time analytics, event-driven pipelines, and data enrichment tasks. The software is distributed under the Apache 2.0 license.