Sharding

Sharding is a database partitioning technique that distributes data across multiple machines, or shards, to improve scalability and performance. By dividing a large dataset, a system can store more data and handle higher query and write throughput than a single server could support. Each shard contains its own subset of data and can be managed and queried independently, while the system presents a unified view to applications.

To assign data to shards, systems use a shard key. A routing layer or query planner uses a shard map to determine which shard holds a given record or range of records, and routes requests accordingly. Some queries may need to access multiple shards and require merging results, while others can be satisfied by a single shard.

Common sharding strategies include hash-based sharding, range-based sharding, and directory-based sharding. Hash-based assigns records based on a hash of the shard key, providing even distribution but potentially unpredictable query locality. Range-based partitions data by contiguous key ranges, allowing efficient range queries but risking uneven distribution. Directory-based sharding uses an external lookup to map keys to shards, offering flexible, workload-aware placement.

Benefits include linear scale-out of capacity and isolated failures, which can improve availability. Drawbacks include added operational complexity, the need for re-sharding when shards become imbalanced, and challenges with cross-shard queries or transactions. Strong consistency across shards may require distributed transactions or careful design.

Sharding is widely used in NoSQL and NewSQL databases, such as MongoDB and Cassandra, and is also a concept in some distributed SQL systems. In blockchain and other distributed systems, sharding is explored as a means to increase throughput by processing data or transactions in parallel across shards.