The core of Gemma 3's pretraining lies in the combination of efficient Knowledge Distillation and Quantization-Aware Training (QAT).
In knowledge distillation, the model uses an efficient sampling strategy: for each token in the training data, only the 256 most likely candidates are sampled from the teacher model's full logits output, weighted according to the teacher model's probabilities. The student model learns this distribution via cross-entropy loss, with the target probability for unsampled logits set to zero. The study also found that for short training cycles (<100B tokens), using a "small teacher" model of similar scale to the student yields better results, while for long training cycles (>100B tokens), using a larger teacher model brings more significant performance gains. The pretraining data scale is adjusted according to model parameters; for example, the 27B model uses 14 trillion tokens.
After pretraining is complete, to optimize deployment efficiency, the model undergoes approximately 5,000 steps of QAT fine-tuning, using the probability distribution of the original unquantized checkpoint as the target, allowing the model to adapt to low-precision computation in advance. After QAT optimization, the perplexity degradation when quantizing to Q4_0 format is reduced by 54%, and memory usage is substantially lowered. For instance, the weight memory required for the 27B model drops from 54GB in BF16 precision to 14.1GB in INT4, enabling it to run on consumer-grade GPUs.