The Gemma-1 paper's evaluation of the chat model (i.e., the instruction-tuned model) primarily consists of two core components: human preference evaluation and automated benchmark testing. In the human evaluation, the research team constructed two held-out prompt sets: one set of approximately 1,000 prompts covering diverse tasks such as creative writing, coding, and instruction following, used to evaluate the model's general instruction-following capability; and another set of approximately 400 prompts specifically designed to test the model's adherence to basic safety protocols. The evaluation employed side-by-side comparisons, pitting the responses of Gemma-7B IT and Gemma-2B IT against those of the strong competitor Mistral v0.2 7B Instruct, with human evaluators selecting the preferred response.
Additionally, the paper reports the model's performance on a broad range of automated academic benchmarks including MMLU (Massive Multitask Language Understanding), GSM8K (mathematics), and HumanEval (code). These results further validate the chat model's comprehensive capabilities across knowledge, reasoning, and code generation.