subbatch

Subbatch is a term used in data processing and machine learning to describe a subdivision of a batch of data. A batch is the set of samples processed together during a model update, while a subbatch is a smaller portion of that batch that is handled separately within the same overall cycle. Subbatching is commonly employed to manage memory constraints, enabling the processing of large batches by handling the data in smaller chunks.

In training, subbatches are often used in gradient accumulation. The model can perform forward and backward passes on each subbatch and accumulate the resulting gradients. After all subbatches belonging to the original batch have been processed, a single optimization step updates the model parameters. This approach can emulate training with a larger effective batch size while keeping memory usage within available limits. Subbatches can also facilitate distribution across multiple devices or stages in a pipeline, allowing parts of the batch to be processed in parallel or sequentially.

Trade-offs of subbatching include the balance between memory efficiency and computational overhead. Very small subbatches reduce memory usage but can increase communication or synchronization overhead and may introduce additional gradient noise, affecting training dynamics. Conversely, larger subbatches require more memory but reduce overhead.

Subbatching is sometimes referred to as micro-batching or batch splitting, and is used in various contexts from neural network training to data processing pipelines. See also: gradient accumulation, micro-batching, batch processing.