Understanding why memory bandwidth dominates attention computation and how tiling saves SRAM transfers.
In standard multi-head attention, the computational bottleneck on modern GPU architectures like Hopper and Blackwell is not FLOP throughput, but memory bandwidth between High Bandwidth Memory (HBM) and on-chip SRAM.
FlashAttention-3 reorganizes the softmax normalization loop by tiling the Q, K, and V matrices into blocks that fit within SRAM, performing online softmax accumulation without writing intermediate N x N attention matrices back to HBM.
Grouped-Query Attention (GQA) takes this optimization further by sharing key and value heads across multiple query heads, slashing KV cache memory footprint by 4x to 8x during autoregressive decoding.