The pretraining recipe for Llama 3 405B is a three-stage process consisting of initial pretraining, long-context pretraining, and annealing, with dynamic data mixture adjustments to fine-tune model capabilities:
- In the initial pretraining stage, the AdamW optimizer and a cosine learning rate schedule are used (peak 8×10−5, decaying after 8,000 warmup steps), with the batch size progressively doubled from 4 million to 16 million tokens to balance stability and efficiency. Meanwhile, the proportions of non-English, mathematical, and recent web data are dynamically increased to enhance multilingual and reasoning capabilities and to update the knowledge cutoff date;
- The long-context pretraining stage follows. To avoid the high computational cost early on, the context window is gradually expanded from 8K to 128K tokens across six incremental stages, with the model advancing to the next stage only after its performance on short-context tasks has fully recovered and it has perfectly passed the "needle in a haystack" test;
- Finally, in the annealing stage, the learning rate is linearly decayed to 0 over 40 million tokens while maintaining the 128K-token context length, and an extremely high proportion of high-quality data sources (especially code and math data, but excluding any benchmark test sets) is upsampled to further boost performance in key areas. The final pretrained model is produced by applying Polyak averaging to the checkpoints.