The model was trained on a large-scale web corpus spanning multiple European languages, with a significantly increased sampling ratio for multilingual data, enabling strong performance on multilingual benchmarks. Built on the Transformer architecture with a sparse mixture-of-experts design, the model has 46.7B total parameters but activates only 12.9B per token, achieving efficient computation through 8 experts per layer and a Top-2 routing mechanism. Training employs sliding window attention and grouped-query attention. Router analysis reveals that expert selection aligns more closely with syntactic structure than with domain specialization, and that consecutive tokens tend to be assigned to the same expert. This sparse activation mechanism allows the model to maintain large capacity while being far more efficient to train and infer than a dense model of equivalent size, ultimately matching or surpassing Llama 2 70B and GPT-3.5 across a range of tasks.