fs2kafka

fs2kafka is a Scala library that provides a functional, FS2-based integration with Apache Kafka. It wraps the Kafka Java client to offer resource-safe streaming interfaces for producing and consuming messages within the FS2 and Cats Effect ecosystem. The goal is to enable pure, composable streams of Kafka data with backpressure and predictable resource management.

The library exposes a producer API that uses ProducerRecords and ProducerSettings to send key-value messages. It integrates serialization logic for keys and values and presents these operations as a Stream, allowing producers to be composed with other streams and sinks in a safe, effectful context. On the consumer side, fs2kafka offers a stream-based API for consuming records, with options to emit plain records or committable records that carry offset information for explicit offset management. This enables at-least-once processing guarantees when combined with appropriate commit strategies.

Architecturally, fs2kafka sits atop the Kafka Java client and is designed to work with FS2 streams and Cats Effect. Resources such as producers and consumers are managed using the Resource abstraction, ensuring proper acquisition and release. The library emphasizes backpressure-friendly streaming semantics and seamless composition with other FS2 components, enabling pipelines that transform, enrich, and route Kafka data within a functional program.

Typical use cases include building reliable streaming data pipelines that ingest from Kafka topics and write to sinks or other systems, and conversely streaming data from external sources into Kafka. It serves as a tooling fit for Scala applications that favor functional programming, modularity, and strong type-driven interfaces within the Kafka ecosystem.