ML//training//distributed training

Distributed training is a way of training one model on several GPUs or nodes at once, splitting the work among them so that the job is not limited to a single device. Data parallelism puts the same model on every GPU and splits the batch; model parallelism splits the model across GPUs; pipeline parallelism splits it by layer.

Distributed training is a way of training one model on several GPUs or nodes at once, splitting the work among them so that the job is not limited to a single device. Data parallelism puts the same model on every GPU and splits the batch; model parallelism splits the model across GPUs; pipeline parallelism splits it by layer.

Communication overhead is the bottleneck, so the NVLink and InfiniBand interconnects matter as much as the GPUs themselves.