Llama 3's pretraining model architecture is based on the standard dense Transformer, and its leap in performance is primarily attributed to improvements in data quality, diversity, and training scale rather than radical architectural changes.
While adhering to this stable architecture, the team introduced four key enhancements: adopting grouped-query attention (GQA) to improve inference efficiency and reduce the key-value cache; using a mask that prevents cross-document attention to support continued pretraining on ultra-long sequences; substantially expanding the vocabulary to 128K tokens (combining tiktoken's 100K tokens with 28K newly added multilingual tokens) to improve compression ratio and multilingual capability; and raising the RoPE base frequency to 500,000 to better support long-context modeling.
Building on these, systematic scaling-law experiments were used to determine the hyperparameters for the flagship 405B model (126 layers, 16,384 dimensions, 128 attention heads), ensuring it is near compute-optimal under the given compute budget.