streamcuts

Stream cuts are a concept used in distributed streaming storage systems to represent a boundary in a multi-segment stream. In these systems, a stream is divided into multiple logical segments that can grow independently. A stream cut encodes a consistent position across the set of segments, allowing reads and computations to proceed from a defined boundary even as segments are split or merged during scaling.

Formally, a stream cut is a map from segment identifiers to offsets within those segments. An offset identifies a position inside a specific segment. The cut defines the boundary such that all events up to the specified offsets (across the included segments) are considered before the cut. When a segment is not part of a cut—such as during a scale operation where new segments appear—the boundary is determined by the offsets present in the cut, while other segments are treated as either not yet reached or already past their known positions.

Usage and purpose: stream cuts are used to read from a stream starting at a particular boundary, enabling time-travel or checkpointed consumption. They support resuming consumption after failures, coordinating progress across multiple segments, and preserving consistent boundaries during scaling operations. Stream cuts thus serve as a cross-segment progress marker that supplements simpler single-offset offsets found in single-segment streams.

Relationship to offsets and checkpoints: unlike a single numeric offset, a stream cut provides per-segment positions, reflecting the distributed nature of the stream. They are typically stored as metadata for durability and can be used to restart readers, perform bounded reads, or implement consistent snapshots across a stream with evolving segment topology. Stream cuts are commonly associated with modern streaming systems such as Pravega.