SegmentStore

SegmentStore is a software component that provides durable, segment-based storage for streaming data and log-like workloads. Data written to a SegmentStore is organized into segments, which are fixed-length or variable-sized contiguous chunks. Writers append events to an open segment; when no longer accepting writes, a segment is sealed and becomes immutable, allowing readers to advance through segments as part of a stream. A stream is typically composed of a sequence of segments, enabling scalable, parallelized storage and retrieval.

SegmentStore abstracts away the underlying storage medium and can be deployed as a standalone service, on each node in a cluster, or as a library integrated into an application. It commonly supports features such as: durable persistence of segment data; sequential and random read APIs by segment and offset; segment lifecycle management (create, seal, delete); compaction and retention policies; metadata management for stream-to-segment mapping; and interaction with a controller or catalog that tracks stream topology.

In practice, SegmentStore systems may implement tiered storage, where recent data resides on fast local storage and older data is migrated to object storage. They are used in event streaming platforms, log processing systems, and time-series data architectures to enable scalable ingestion, retrieval, and processing.

Origins and usage: The term is used in several architectures and projects; a notable implementation is in the Pravega project, where the Segment Store is the storage service responsible for persisting segment data and handling recovery, scaling, and durability across a cluster.