The Decoder is a diffusion-based generator whose core task is to convert a CLIP image embedding back into the corresponding pixel image.
Specifically, the decoder architecture is built by modifying the previously released GLIDE model:
- The CLIP image embedding (learned during the Prior stage) is projected and added to the model's timestep embedding.
- The CLIP image embedding is projected into four additional context tokens, concatenated after the output sequence of GLIDE's text encoder.
Although the paper retains text captions as an optional conditioning input, experiments found that they provide limited benefit — the CLIP image embedding itself already supplies strong semantic guidance.
To improve generation quality, the model employs Classifier-Free Guidance (CFG) during training: the CLIP embedding is randomly zeroed out (10% probability) and text captions are randomly dropped (50% probability), enabling the model to balance fidelity and diversity during inference by adjusting the guidance scale.
Additionally, to efficiently generate high-resolution images (e.g., 1024x1024), the system trains two cascaded diffusion upsamplers: the first upsamples the 64x64 base image to 256x256, and the second further upsamples to 1024x1024. These upsamplers are trained with blurred or diversified degradation applied to the conditioning image to enhance robustness, and they use only convolutional layers without attention mechanisms, allowing them to generalize to higher resolutions at inference time.