Mistral AI did not publicly disclose the full details of its pretraining data in the original paper, unlike some other models (such as LLaMA). However, based on industry practices, the model's demonstrated capabilities, and scattered official information, we can infer and describe several characteristics of its pretraining data:
- Scale and Diversity: As a high-performance 7B-parameter model, its pretraining data must be massive and diverse. This typically includes a vast collection of text from the internet, such as web pages, books, academic papers, news articles, and code repositories, ensuring the model acquires broad linguistic and world knowledge.
- Multilingual Capability: Although Mistral 7B is primarily optimized for English, its pretraining data likely includes a substantial proportion of other languages (especially European languages), giving the model a degree of multilingual understanding and generation ability.
- Code Data: The paper explicitly notes that Mistral 7B excels at code generation, even surpassing larger models. This strongly suggests that its pretraining data includes a large amount of high-quality code (e.g., from public repositories on platforms like GitHub), which is a key source of its strong programming capability.
- Data Quality and Filtering: To achieve higher performance standards, the Mistral AI team likely applied rigorous cleaning, deduplication, and quality filtering to the raw data, removing low-quality, duplicate, or harmful content to ensure a refined and efficient dataset.
Note: No module data