ColorPeel

Color prompt learning with diffusion models via color and shape disentanglement.

ECCV 2024

Muhammad Atif Butt, Kai Wang, Javier Vazquez-Corral, Joost van de Weijer

Paper arXiv Code Project page BibTeX

A color name like “red” covers a wide range of shades, so a prompt cannot pin down the exact red a designer has in mind. ColorPeel learns a new prompt token for each user-chosen color. It trains on simple 2D and 3D shapes rendered in the target colors and learns separate color and shape tokens, which peels the color away from the shape. The learned color tokens can then be used in any prompt, alone or combined.

Explore the learned colors
Each token was learned from the shapes shown on its card. Click a token to see which generations use it. Images are from the paper.

Why shapes?

When existing personalization methods learn a color from real photos, the token absorbs the shape of the objects too, and the color leaks into unrelated parts of the image. Rendering basic geometric shapes gives precise control over the color and makes shape an independent factor, and a cross-attention alignment loss keeps each token focused on its own attribute. The same idea extends to abstract attributes such as textures and materials.