horovodrun

horovodrun is a command-line launcher used to start distributed training jobs with Horovod across multiple processes and hosts. It serves as a front end that uses an MPI-compatible launcher to spawn Horovod-enabled training processes on the specified machines, with each process executing a copy of the user’s training script. The launcher coordinates the ranks and initializes the distributed environment so that collective operations such as allreduce can be performed efficiently.

To use horovodrun, an MPI implementation (such as Open MPI or MPICH) must be available on the cluster and accessible to the user. A typical invocation is: horovodrun -np 4 -H host1:2,host2:2 python train.py. This runs four processes in total, with two processes on host1 and two on host2. The training script should initialize Horovod (for example, by calling hvd.init()) and proceed with the distributed components of the framework being used (TensorFlow, PyTorch, etc.). The launcher handles process startup, environment setup, and rendezvous among ranks.

horovodrun supports GPU and multi-host configurations by distributing processes across available devices according to the host list and per-host allocations. It also offers launcher options to control aspects such as process binding, oversubscription, and resource usage, depending on the underlying MPI implementation. The exact flags can vary with the MPI distribution, but the standard -np and -H options are commonly used across implementations.

In practice, horovodrun is a primary method for launching Horovod jobs in cluster environments and is compatible with major deep learning frameworks that Horovod supports, including TensorFlow, Keras, PyTorch, and MXNet. Related topics include Horovod itself, MPI-based launching, and alternative launchers provided by specific ecosystems.