The V3 architecture inherits the V2 framework while integrating several core innovations to achieve efficient inference and economical training:
- The model adopts Multi-head Latent Attention (MLA), which applies low-rank joint compression to attention key-value pairs. During inference, only the compressed latent vectors need to be cached, dramatically reducing KV cache memory usage;
- In the FFN component, the model uses the DeepSeekMoE architecture with shared experts and routed experts for efficient parameter utilization. One of its most notable innovations is the introduction of an auxiliary-loss-free load-balancing strategy, which guides routing decisions by adding dynamically adjustable bias terms for each expert. This ensures balanced expert loads while completely avoiding the negative impact that traditional auxiliary loss functions have on the model's primary task performance;
- Additionally, the architecture incorporates a multi-token prediction (MTP) training objective, which sequentially predicts multiple future tokens to enhance the model's representation learning. This objective can be repurposed for speculative decoding at inference time to accelerate generation.
Among these technical innovations, MLA ensures inference efficiency, DeepSeekMoE with its novel load-balancing strategy enables efficient and high-performance training, and multi-token prediction further enhances model capability. Working in concert, these innovations allow DeepSeek-V3, at 671B total parameters, to activate only 37B parameters per token, achieving an outstanding balance between performance and efficiency.