LLaMA-1's training details embody its core design philosophy of "achieving the best performance under a given inference budget." For efficient implementation, it uses the causal multi-head attention from the xformers library, avoiding computation on masked weights to save memory and computation. It also extensively employs activation checkpointing, trading computation for memory by saving only key activations (such as those from linear layers), enabling training of larger models or batch sizes. Its parallelism strategy combines model parallelism (primarily tensor parallelism, splitting model parameters horizontally across multiple GPUs) and sequence parallelism (splitting input sequences vertically to distribute the memory cost of intermediate activations), thereby successfully loading and efficiently training the massive 65B model without resorting to pipeline parallelism, which can introduce additional waiting time. All of this runs on powerful training infrastructure: 2,048 A100-80GB GPUs, with techniques such as model and sequence parallelism and computation-communication overlap to optimize throughput, while also attending to engineering practices like carbon footprint tracking. The result is a foundation model with outstanding performance, trained on trillions of tokens of public data.