On the compute cluster side, the model was trained on a cluster equipped with 2,048 NVIDIA H800 GPUs, using a hierarchical interconnect architecture with high-speed NVLink / NVSwitch for intra-node communication and InfiniBand for inter-node communication.
The training framework is built on the HAI-LLM framework, employing a hybrid parallelism strategy combining 16-way pipeline parallelism, 64-way expert parallelism, and ZeRO-1 data parallelism.
To address the heavy communication overhead introduced by cross-node expert parallelism, the team designed the innovative DualPipe bidirectional pipeline parallelism algorithm. This algorithm subdivides each computation chunk into four components — attention computation, all-to-all dispatch, MLP processing, and all-to-all combine — and rearranges these components to precisely adjust the GPU streaming multiprocessor resource allocation ratio, achieving complete overlap of all-to-all communication and pipeline parallelism communication during execution.
To ensure DualPipe's computational performance, the team developed custom high-efficiency cross-node all-to-all communication kernels, fully exploiting the bandwidth differences between NVLink (approximately 160 GB/s) and InfiniBand (approximately 50 GB/s). Through a hierarchical forwarding mechanism and node-limited routing (each token routed to at most 4 nodes), IB communication traffic is reduced.
On the memory optimization front, techniques such as recomputation, asynchronous CPU updates, and parameter sharing enable DeepSeek-V3 to be trained without requiring expensive tensor parallelism.
The FP8 mixed-precision training framework validated its feasibility on an ultra-large-scale model for the first time. It uses a fine-grained mixed-precision strategy: most compute-intensive operations are executed in FP8 precision (theoretically doubling compute speed compared to BF16), while numerically sensitive critical components retain their original precision. Through fine-grained quantization (activations use 1x128 tile-wise grouping, weights use 128x128 block-wise grouping), high-precision accumulation, and a unified E4M3 format, the relative loss error of FP8 training consistently stays within 0.25% of the BF16 baseline.
Inference deployment is divided into a prefill stage (minimum deployment unit of 4 nodes, 32 GPUs) and a decode stage (minimum deployment unit of 40 nodes, 320 GPUs), employing redundant expert strategies and computation-communication overlapping optimizations.
Hardware design recommendations include designing dedicated coprocessors for communication tasks, providing unified communication primitives across IB and NVLink, and improving Tensor Core support for FP32 accumulation precision and online quantization acceleration.