Train Your Large Model on Multiple GPUs with Pipeline Parallelism

This article is divided into six parts; they are: • Pipeline Parallelism Overview • Model Preparation for Pipeline Parallelism • Stage and Pipeline Schedule • Training Loop • Distributed Checkpointing • Limitations of Pipeline Parallelism Pipeline parallelism means creating the model as a pipeline of stages.

source https://machinelearningmastery.com/train-your-large-model-on-multiple-gpus-with-pipeline-parallelism/