Gemma 2's pretraining was comprehensively optimized in terms of data scale, processing pipeline, and training methodology, and it innovatively applies knowledge distillation to large-scale pretraining. The core pretraining details are as follows:
- Models at different parameter scales use differentiated data volumes: the 27B model is trained on 13 trillion tokens, the 9B model on 8 trillion tokens, and the 2.6B model on 2 trillion tokens.
- These tokens are primarily drawn from English web documents, code, and scientific articles. The final data mixture ratios were determined via ablation experiments similar to those used for Gemini 1.0.
- The model uses the same SentencePiece tokenizer as Gemma 1 and Gemini, with a vocabulary size of 256k, featuring digit splitting, whitespace preservation, and byte-level encoding.
- For data safety, the same filtering techniques used in Gemma 1 are applied to reduce the risk of inappropriate or sensitive content.
- Training was conducted on Google's TPU clusters; for example, the 27B model was trained on 6144 TPUv5p chips using 768-way data replication and 8-way model sharding.