The pre-training phase of DeepSeek-V3 was conducted on 14.8 trillion high-quality, diverse tokens, using a vocabulary of 129,280 tokens and a maximum context length of 128K. Data preprocessing included document packing and a Fill-in-Middle strategy to enhance the model's contextual understanding.
The model architecture employs 61 Transformer layers with a hidden dimension of 7,168, featuring Multi-head Latent Attention (MLA) and a DeepSeekMoE mixture-of-experts system comprising 1 shared expert and 256 routed experts, with 8 experts activated per token.
Training uses the AdamW optimizer with a carefully scheduled learning rate: starting from an initial linear warmup to 2.2×10−4, then decaying via a cosine schedule to 2.2×10−5, and finally dropping to 7.3×10−6 in the terminal stage. The batch size gradually increases from 3,072 to 15,360.
After pre-training, the context window is extended from 4K to 32K and then to 128K using a two-stage YaRN method, with each extension stage comprising 1,000 training steps.
The entire pre-training process demonstrated remarkable stability, with no irrecoverable loss spikes or rollbacks required. It consumed only 2.664M H800 GPU hours (equivalent to 180K GPU hours per trillion tokens), ultimately producing the DeepSeek-V3-Base foundation model. This model exhibits top-tier performance across multiple benchmarks, with particularly strong results on code and math tasks.