Gemma 2B was pretrained on approximately 2 trillion (2T) tokens, while Gemma 7B was trained on approximately 6 trillion (6T) tokens, all from English-only content primarily sourced from web documents (including web pages, articles, and blogs), mathematical content (formulas, proofs, and problem-solving processes), and code (code snippets and documentation across various programming languages). Gemma is not a multilingual model and is not optimized for multilingual tasks.
In terms of data filtering, Gemma employs a rigorous multi-stage filtering pipeline: first, heuristic methods and model-based classifiers are used to remove harmful or low-quality content and filter out personal information and other sensitive data; second, all evaluation sets are removed from the pretraining data mixture, and targeted contamination analysis is run to check for evaluation set leakage; finally, a series of ablation experiments on the 2B and 7B models determine the final data mixture, using a staged training approach that dynamically adjusts the corpus mixture throughout training, gradually increasing the weight of high-quality data in later stages. This dual emphasis on data quality and safety is a key reason why Gemma can outperform larger models on critical benchmarks.