The Gemma-1 paper provides a comprehensive evaluation of the base (pretrained) model, covering memorization evaluation, personal information handling assessment, and academic benchmarks. For memorization evaluation, the Gemma pretrained models were tested using the same methodology as Anil et al. (2023): 10,000 documents were sampled from the corpus, with the first 50 tokens used as a prompt to evaluate exact memorization (the model's subsequent 50 generated tokens exactly match the ground-truth continuation) and approximate memorization (using a 10% edit distance threshold). The evaluation results show that Gemma's memorization rate is comparable to PaLM's, but no memorized sensitive data was found — only a small amount of data classified as "possibly personal" was memorized.
For personal information evaluation, Gemma used the Google Cloud Sensitive Data Protection tool for analysis, classifying the highest severity level as "sensitive" and the rest as "personal." The results indicated that no sensitive data was memorized. In academic benchmarks, Gemma base models were compared against similarly sized open-source models across multiple domains (physical reasoning, social reasoning, question answering, coding, mathematics, commonsense reasoning, language modeling, and reading comprehension). Gemma 7B achieved strong results on key benchmarks such as MMLU (64.3%) and MBPP (44.4%), demonstrating its competitiveness in fundamental language understanding and reasoning. These evaluations, consistent with the Gemini model family's evaluation methodology, provide a solid foundation for Gemma as a responsible open-source model.