perchunk

Perchunk is a concept in data processing that describes the amount of processing resources or work allocated to a single chunk of input data in streaming or batch systems. It is used to control work granularity and throughput, and can be expressed as time, byte count, or item count per chunk.

In practice, perchunk settings influence latency and fairness. A smaller perchunk budget yields lower latency and finer-grained scheduling but higher overhead from more frequent chunk boundaries; a larger budget improves throughput but may increase tail latency or imbalance if chunks vary in size. The concept helps systems manage backpressure and maintain predictable performance under varying load.

Chunk size refers to the amount of input data contained in a chunk, while perchunk refers to the maximum processing work permitted for that chunk. The two interact in dynamic systems that adapt processing rate per chunk to observed performance, allowing operators to tune responsiveness and efficiency independently from the raw input size.

Implementation typically involves measuring processing time or resource use per chunk and adjusting future budgets to target a desired latency or throughput. Perchunk can be implemented as a processing budget, a time slice limit, or a rate-limited work queue, and may be auto-tuned in adaptive pipelines to respond to changing workload characteristics.

Perchunk is not a universally standardized term; its exact meaning varies by system. It is closely related to chunking, backpressure, and workload balancing in streaming and distributed processing.