Cachethrashing

Cachethrashing is a performance phenomenon in which a program experiences a high rate of cache misses because the data being accessed cannot remain in the cache, constantly being evicted to make room for new data. The term is often used to describe situations where the cache is unable to retain useful working data, leading to repeated fetches from lower levels of memory.

The underlying cause is typically a working set that does not fit in the cache or data access patterns that map many distinct addresses to the same cache sets. Cache misses fall into several categories, including capacity misses (the cache cannot hold the entire working set) and conflict misses (multiple addresses compete for the same cache lines due to the cache’s set-associative structure). Thrashing occurs when the likelihood of eviction is high enough that even recently used data is removed before it can be reused, resulting in a low data reuse rate and elevated latency.

Thashing can be exacerbated by multi-core or multi-threaded execution, especially when threads contend for shared caches or when memory access patterns are interleaved in ways that defeat spatial or temporal locality. NUMA (non-uniform memory access) architectures can also contribute, as remote memory accesses add latency and increase cache pressure.

Mitigation strategies focus on improving data locality and reducing contention: applying loop blocking or tiling to increase data reuse, reorganizing data structures for better spatial locality, ensuring cache-friendly access patterns, using NUMA-aware memory allocation and thread placement, and, where possible, increasing cache size or selectivity of data to fit within the cache.