GPUcacher

GPUcacher is a software layer and reference implementation intended to accelerate GPU-accelerated workloads by caching data and intermediate results near the compute units. It targets data reuse patterns common in graphics, science, and machine learning pipelines, reducing memory bandwidth pressure and latency when the same data are accessed multiple times within or across kernels.

At runtime, GPUcacher partitions memory into caches that reside on the GPU, in unified memory, and, where available, in host memory or specialized near-device storage. It tracks access patterns and stores frequently requested tiles or tensors, anticipating reuse through prefetching and eviction governed by configurable policies.

Architecturally, GPUcacher provides an API layer that intercepts memory requests from kernels and translates them to cached or uncached operations. It supports several eviction strategies (for example, LRU, LFU), coherence rules for multi-GPU and CPU-GPU scenarios, and optional hints to guide placement during kernel launches.

The project offers bindings for common GPU programming interfaces such as CUDA and OpenCL, and aims to integrate with popular frameworks used in data science and high-performance computing. It can be used as a drop-in cache layer or as a complementary optimization when larger caching subsystems are unavailable.

Performance benefits are workload-dependent. When data exhibit temporal locality or repeated access across kernels, substantial bandwidth savings and lower latency are reported, while overhead from cache management and coherence traffic can offset gains for streaming, single-pass workloads. Its practical adoption depends on hardware topology and application characteristics.

GPUcacher originated as a research idea in computer architecture and has evolved through open-source collaborations and industry experiments. It is described in technical documentation and comparative studies as a modular caching layer for GPU memory hierarchies, with ongoing work to expand portability and multi-tenant support.