Gemma 3's pretraining data saw significant improvements in scale, composition, and processing:
- In terms of data scale, the 27B model uses 14 trillion tokens, the 12B model 12 trillion tokens, the 4B model 4 trillion tokens, and the 1B model 2 trillion tokens.
- Data composition includes text-image mixed content, spanning web documents, code, mathematical text, and images, with a substantially increased proportion of multilingual data supporting over 140 languages, using a strategy similar to Chung et al. to balance language distribution.
- Data processing employs multiple filtering mechanisms, including strict CSAM (Child Sexual Abuse Material) filtering, sensitive personal information removal, and quality re-weighting steps to reduce low-quality content.
- The knowledge cutoff date for the training data is August 2024.