Traditional self-attention has time and memory complexity that scales quadratically with sequence length, causing training and inference to become prohibitively slow and memory-intensive as sequences grow longer. While some approximate attention methods attempt to reduce computation at the expense of model quality, they often fail to deliver actual wall-clock speedups. The paper identifies a key principle that prior work overlooked: making the attention algorithm IO-aware. That is, the algorithm must carefully account for the cost of reading and writing data between different levels of GPU memory (such as high-bandwidth memory HBM and on-chip SRAM). The core innovation of FlashAttention is to use tiling and recomputation techniques to dramatically reduce the number of HBM-SRAM reads and writes of intermediate results. By optimizing memory access efficiency, FlashAttention significantly accelerates the attention mechanism and reduces memory consumption — without sacrificing computational precision — making it far more feasible to process long sequences.