Chunked

Chunked is an adjective used in computing to describe data that has been divided into smaller pieces called chunks. Chunking is employed to enable streaming, parallel processing, caching, and error recovery. The concept is language- and protocol-agnostic and applies to memory, storage, and communication systems.

One prominent use is HTTP chunked transfer encoding in HTTP/1.1. In this scheme, the body is sent as a sequence of chunks. Each chunk begins with the size in hexadecimal, followed by CRLF, then the data, and another CRLF. The transmission ends with a final chunk of size 0, after which optional trailing headers may follow. This enables a server to start sending a response before its total length is known, allowing streaming and dynamic content generation. Clients and intermediaries reassemble the chunks to reconstruct the payload.

Beyond HTTP, chunking is used in data processing and storage systems to break large files into manageable pieces. In distributed computing and streaming pipelines, data is partitioned into chunks to enable parallel processing, fault isolation, and more efficient I/O. The choice of chunk size affects throughput, latency, and memory usage; smaller chunks reduce latency and improve cache locality but increase overhead, while larger chunks improve throughput at the expense of latency and memory pressure.

Chunking also appears in memory allocators, databases, and data formats that store or transmit large datasets as collections of blocks. Some protocols and systems implement chunking with overheads and compatibility requirements; not all intermediaries support chunked encoding, and legacy software may mishandle it. When used for streaming, chunking is particularly helpful for long-running responses such as live data feeds or media streaming.

Overall, chunked processing and transmission are fundamental techniques for handling large or variable-length data efficiently in modern networks and storage systems.