The Gemma-1 model architecture is based on the Transformer decoder but incorporates several enhancements: the 7B model uses multi-head attention, while the 2B model uses multi-query attention (num_kv_heads = 1); Rotary Position Embeddings (RoPE) replace absolute position embeddings, with embeddings shared between input and output; an approximate version of the GeGLU activation function replaces the standard ReLU; and RMSNorm is applied for normalization at each Transformer sub-layer to stabilize training.
For tokenization, Gemma uses a subset of Gemini's SentencePiece tokenizer with a vocabulary size of 256k tokens. It splits digits, does not remove extra whitespace, and uses byte-level encoding for unknown tokens. Inheriting Gemini's large vocabulary (256k entries) designed to handle multiple languages, Gemma's embedding parameter count is larger compared to models limited to one or two languages.