dataskew

Dataskew, more commonly called data skew, is a term used in data management and distributed computing to describe an imbalance in how data are distributed across partitions, processes, or time. When data are not evenly distributed, some tasks must handle disproportionately large shares of work while others do little, leading to inefficiency and bottlenecks.

There are two related senses of the term. In statistics, skewness describes asymmetry in a data distribution, measured by a skewness coefficient. In distributed systems and data processing, data skew refers specifically to non-uniform data distribution across computing resources, which can cause hot spots and uneven workload.

Causes of data skew in practice include uneven key frequencies in partitioning schemes, long-tail or bursty time-series data, skewed join keys, and design choices that favor certain values or ranges. For example, if a partitioning function assigns many records to a small number of partitions, those partitions become bottlenecks.

The effects of data skew are increased latency and wasted resources. Some nodes or workers become overloaded while others remain underutilized, leading to longer shuffle times, slower queries, higher memory pressure, and reduced parallelism.

Detection typically involves data sampling and examination of distribution statistics. In a processing system, operators may monitor partition sizes, per-partition throughput, and skew metrics; visualizations and UI dashboards can reveal hotspots and imbalanced workloads.

Mitigation strategies include changing partitioning schemes (hash versus range), salting keys to distribute load more evenly, performing two-stage aggregations, bucketing data, broadcasting small tables for joins, and dynamically repartitioning or rebalancing data to reduce hotspots. The choice of strategy depends on the workload, data characteristics, and the processing framework.