Gemma 2 is a family of open language models based on a Transformer decoder architecture, offered in three parameter scales — 27B, 9B, and 2.6B — with a context length of 8192 tokens, using Rotary Position Embedding (RoPE) and an approximate GeGLU activation function.
The model family introduces several key architectural improvements to enhance performance and efficiency:
-
An interleaved attention mechanism that alternates between local sliding-window attention (window size of 4096 tokens) and global attention (spanning the full 8192-token context) across the network layers, balancing computational cost against long-range dependency modeling.
-
Logit soft-capping, which constrains logit values to a specific range at each attention layer and the final output layer using the following formula:
logits←soft_cap⋅tanh(logits/soft_cap)
-
Extensive use of RMSNorm to normalize the inputs and outputs of each sub-layer, enhancing training stability.
-
Grouped-Query Attention (GQA, with 2 groups) in the 9B and 27B models, which substantially improves inference speed while largely preserving downstream task performance.