The Prefix Decoder-only architecture originally adopted by ChatGLM was an innovative design built on top of the standard causal decoder-only architecture (as used in the GPT series). Its core characteristic is splitting the attention mechanism into two regimes: when processing the input content, bidirectional attention (Prefix Attention) is used, allowing the model to attend to both preceding and following context for more comprehensive understanding; when generating output, it switches to unidirectional attention (Causal Attention), ensuring coherence in autoregressive generation.
This hybrid attention design enhanced ChatGLM's contextual understanding while preserving its generative capability, making it particularly well-suited for dialogue tasks. In the concrete implementation, the model uses a special attention mask to distinguish the input portion (Part A) from the generation portion (Part B): the input portion uses a bidirectional mask allowing attention to all positions, while the generation portion uses a causal mask attending only to left-side historical information. However, subsequent versions ChatGLM2 and ChatGLM3 have fully transitioned to the standard decoder-only architecture, uniformly using causal attention.