chunksize

Chunk size, often abbreviated as chunkSize, is a parameter that specifies the amount of data processed, transferred, or stored in a single unit. It is used in programming interfaces, file I/O, networking, and data processing pipelines to control the granularity of operations. Processing data in fixed-size chunks helps manage memory usage while balancing throughput and latency.

In file input/output, reading or writing data in chunks reduces peak memory requirements and can improve caching behavior. Typical chunk sizes range from a few kilobytes to several tens of kilobytes, with common defaults around 4 KiB, 8 KiB, or 64 KiB, depending on the system and library.

In streaming and networking, chunking enables progressive delivery and streaming without requiring the complete payload upfront. Protocols such as HTTP use chunked transfer formats to begin transmission before the full content is known. Data processing pipelines and streaming platforms also rely on chunking to process data as it arrives rather than in a single bulk operation.

In storage formats and databases, chunk size can determine boundaries for compression, indexing, and write amplification. Larger chunks can improve throughput for sequential writes but may increase memory usage and reduce random-access efficiency; smaller chunks improve responsiveness but add overhead from more numerous operations.

Choosing an appropriate chunk size depends on workload, hardware, and I/O characteristics. Smaller chunks lower memory use and latitude for early processing but increase overhead; larger chunks increase throughput for bulk transfers but raise memory usage and potential latency. Practical guidance includes aligning to cache lines, using power-of-two sizes when possible, and testing with representative data.