Giant models like GPT-3 (175B parameters), if loaded in standard 16-bit or 32-bit floating-point, require hundreds of GB of GPU memory — far exceeding the capability of ordinary consumer-grade GPUs, making large model inference costly and inaccessible.
Simple 8-bit quantization can reduce memory usage, but when applied to models with over 6 billion parameters, it causes model performance (e.g., inference accuracy) to plummet, rendering it unusable.
LLM.int8() is an innovative quantization technique whose core objective is: to halve the inference memory footprint of large language models (LLMs) without any performance degradation. First, you need to understand that the root cause of performance degradation in large models is the emergence of "outlier features," and then learn that LLM.int8()'s solution is a "mixed-precision" two-stage process: vector-wise absolute maximum quantization and mixed-precision decomposition.