The Prior model's core role is to generate a corresponding CLIP image embedding from a text description, providing the semantic foundation for the subsequent Decoder stage. Specifically, the Prior maps a text caption y to a CLIP image embedding zi, with the generation process following the probability distribution P(zi∣y).
The paper explores and compares two Prior architectures: the Autoregressive (AR) Prior reduces the CLIP image embedding via PCA dimensionality reduction and discretization, then uses a Transformer to predict the encoded sequence autoregressively; the Diffusion Prior, in contrast, directly models zi in the continuous vector space using a diffusion model, employing a decoder-only Transformer architecture that predicts the un-noised embedding and is optimized with a mean squared error loss.
Experiments show that the Diffusion Prior outperforms the Autoregressive Prior in both computational efficiency and quality. During training, text conditioning is randomly dropped (10% probability) to support Classifier-Free Guidance, and at sampling time, a strategy of generating multiple candidates and selecting the one with the highest cosine similarity to the text embedding is used to improve alignment quality. This design enables the Prior to effectively bridge the text and image semantic spaces, making it a key component for generating diverse, high-quality images.