MLA aims to solve the inference efficiency bottleneck caused by the excessively large KV cache in traditional Multi-Head Attention (MHA).
MLA uses low-rank joint key-value compression to compress high-dimensional key-value vectors into low-dimensional latent vectors. Specifically, a down-projection matrix compresses the input into a latent vector, and up-projection matrices reconstruct the keys and values. During inference, the up-projection matrices can be absorbed into the query and output projection matrices, so only the compressed vectors need to be cached rather than full key-value pairs, reducing the KV cache by 93.3%.
To address the compatibility issue between Rotary Position Embedding (RoPE) and low-rank compression, MLA adopts a decoupled RoPE strategy, using additional multi-head queries and a shared key to carry positional information.
Compared to standard MHA, MLA not only reduces the KV cache to support 128K long contexts, but also boosts maximum generation throughput by up to 5.76x, while outperforming MHA on benchmarks such as MMLU, achieving a dual breakthrough in both performance and efficiency.