The pretraining model architecture of ChatGLM4 (the GLM-4 series) is based on the Transformer, with several key optimizations for performance, efficiency, and long-context processing: it employs Grouped Query Attention (GQA) to reduce KV cache during inference and improve efficiency; uses Rotary Position Embedding (RoPE) to enhance positional modeling; adopts RMSNorm and SwiGLU activation functions to improve performance; and applies a "No Bias Except QKV" design to boost training speed and length extrapolation capability.
The model scale has increased significantly: for example, GLM-4-9B has 40 layers, a vocabulary of 151,552 tokens, and supports a standard 128K long context with a dedicated 1M ultra-long context variant. Pretraining data comprises approximately 10 trillion tokens across multiple languages. Since ChatGLM2, the series has fully transitioned from the earlier Prefix Decoder-only architecture (hybrid attention) to the standard decoder-only architecture (unidirectional attention throughout), improving training efficiency, reducing engineering complexity, and enhancing compatibility with the mainstream ecosystem.