Gemma 3's pretraining model architecture incorporates multiple innovative design choices to achieve an efficient combination of multimodal capabilities and long-context processing.
- Its core innovation lies in a 5:1 local/global attention layer interleaving design, where every 5 local layers are followed by 1 global layer. Local layers have a span of only 1024 tokens, while global layers increase the RoPE base frequency from 10k to 1M.
- The model can effectively handle contexts up to 128K tokens (32K for the 1B model), while reducing KV cache memory overhead from 60% under a "global only" configuration to below 15%.
- For multimodal capabilities, Gemma 3 uses a 400M-parameter SigLIP vision encoder (Vision Transformer) to process images, normalizing input images to 896×896 pixels and converting them into 256 fixed-size vectors.
- To handle non-square images, Gemma 3 introduces the Pan & Scan (P&S) method, which at inference time divides the image into non-overlapping equal-sized crops and resizes them to 896×896 pixels.
- All 4B, 12B, and 27B models share the same vision encoder, and image embeddings are precomputed before pretraining, adding no cost to language model training.