The large-scale engineering innovations underpinning the training of Llama 3 405B center on the following:
- Building a production-grade infrastructure consisting of 16,000 H100 GPUs, 240 PB of Tectonic storage, and a RoCE-based 400 Gbps three-layer Clos network;
- Designing a 4D parallelism strategy optimized in the order
[TP, CP, PP, DP] for ultra-large-scale training, with improved pipeline parallelism (PP) through flexible micro-batch scheduling and load balancing;
- Adopting all-gather-based context parallelism (CP) to efficiently support training on sequences up to 128K tokens, developing a custom NCCLX communication library for improved multi-hop network performance, and employing FP32 gradient accumulation to ensure numerical stability;
- Additionally, through tooling (such as the NCCL flight recorder) and automated operations, the team maintained over 90% effective training time over a 54-day pretraining period despite 466 job interruptions (78% caused by hardware failures), and observed engineering challenges unique to ultra-large-scale training, such as daily throughput fluctuations of 1-2% due to temperature variations and multi-megawatt power transients caused by large-scale GPU synchronization.