Image Manipulations is a core innovative capability that fully leverages the model's two-stage generation architecture (first generating a CLIP image embedding, then decoding it into an image) to enable powerful and flexible image editing.
This capability encompasses three main operations:
- Variations: Given a single input image, generate diverse samples that preserve core semantics (e.g., objects, composition) and style (e.g., artistic style) while varying non-essential details (e.g., texture, orientation). This is achieved by fixing the CLIP image embedding (responsible for semantics/style) while perturbing the latent codes obtained via DDIM inversion (responsible for details).
- Interpolations: Enable smooth transitions between two images by performing spherical interpolation on their representations in the CLIP embedding space, naturally blending their content and style.
- Text Diffs: Leverage CLIP's ability to map images and text into the same latent space. By computing the embedding difference vector between a target text and a reference text, and shifting the original image's CLIP image embedding in that direction, this enables zero-shot, language-driven semantic editing (e.g., transforming a "daytime" scene into "nighttime") without any model retraining.
The fundamental advantage of these manipulation capabilities lies in the model's effective separation of high-level semantic information (captured by the CLIP embedding) from specific detail information (captured by the diffusion model's latent codes), enabling users to perform highly controllable, creative edits that go far beyond traditional pixel-level operations.