minibatch

Minibatch refers to a method in machine learning where the training data is divided into small batches used to compute gradient estimates. It sits between full-batch gradient descent, which uses the entire dataset to compute a gradient, and stochastic gradient descent, which uses a single example. Minibatch gradient descent uses a batch of examples per iteration, balancing computational efficiency with gradient variance.

Typical minibatch sizes range from 16 to 256 examples, with 32, 64, and 128 commonly used. The exact size depends on the dataset, model, and available hardware. Using minibatches enables efficient vectorized computation on modern hardware such as CPUs and GPUs and allows updates to the model parameters to be performed frequently enough to accelerate training.

Training with minibatches generally proceeds by shuffling the training data, partitioning it into batches, and iterating through these batches for one or more epochs. Compared with full-batch methods, minibatches reduce memory usage and enable parallel processing, while providing a gradient estimate that is noisier than the full dataset but more stable than a single example.

Trade-offs include the impact of batch size on convergence and generalization. Very large batches require more memory and may lead to poorer generalization, while very small batches increase gradient noise and training time. In practice, learning rate schedules and optimization algorithms (such as Adam or RMSProp) are often tuned to work well with minibatch updates. Minibatches are a standard component of modern deep learning training.