modelparallel

Modelparallel, or model parallelism, is a distributed computing approach used to train and run machine learning models that exceed the memory capacity of a single device. Rather than duplicating an entire model on each processor, its components—such as layers or subtensors—are placed on different devices, with computation and communication coordinating the forward and backward passes.

Two common strategies are inter-layer (pipeline) parallelism and intra-layer (tensor) parallelism. Inter-layer splits assign successive layers to different devices and often employ a pipeline to overlap computation. Intra-layer splits partition a layer’s weight matrices or activations across devices, requiring communication of partial results during forward and backward steps. Hybrid approaches combine both to improve utilization. Real-world systems may also integrate model parallelism with data parallelism to scale both memory and throughput.

Implementation relies on scheduling and communication optimization, including micro-batching, activation checkpointing, and careful shard placement. Performance depends on hardware bandwidth, network topology, and the model’s architecture; as a result, model parallel training can incur higher latency and more complex debugging compared with data parallelism alone.

Model parallelism is essential for very large models such as billions of parameters that do not fit on a single accelerator. It is used in large language models and other domains requiring large memory footprints. Popular software ecosystems include PyTorch with distributed backends, and domain-specific toolkits such as Megatron-LM, as well as systems that provide pipeline and tensor parallelism components and optimizations like DeepSpeed.