BigTable

Bigtable is a distributed storage system developed by Google for managing structured data at a very large scale. It is designed to support large applications such as web indexing, maps, and analytics, providing a scalable, flexible database-like service without conforming to a traditional relational model. Bigtable exposes a sparse, distributed, persistent multi-dimensional sorted map as its core data model.

Data in Bigtable is organized into tables. Each table contains rows identified by a row key, and each row can have multiple column families. Within a family, data is stored as cells that are timestamped, allowing multiple versions of a value to be kept. Columns that share a family are stored together on disk to improve locality and performance, while most queries specify a row key and optionally a column or time range.

The system uses tablet-based partitioning to scale. Tables are divided into tablets, which are contiguous ranges of row keys. Tablet servers manage one or more tablets and handle reads and writes. A master server coordinates metadata, balancing, and tablet assignment across tablet servers, while a highly available coordination service stores metadata about tablet locations. On-disk data is stored on a distributed file system and organized into immutable SSTable-like files, with recent writes buffered in memory as memtables and periodically flushed to disk. Writes are recorded in a commit log to ensure durability, and bloom filters are used to speed up reads.

Bigtable provides strong consistency at the row level, with atomic read and write operations for individual rows and support for read-modify-write operations. It also supports versioning through timestamps and automatic sharding and replication for fault tolerance. The design has influenced several open-source projects and inspired subsequent NoSQL column-family databases.