GenColorBench

A color evaluation benchmark for text-to-image models. Can you beat the benchmark at judging color?

CVPR 2026

Muhammad Atif Butt, Alexandra Gomez-Villa, Tao Wu, Javier Vazquez-Corral, Joost van de Weijer, Kai Wang

Paper arXiv Code Benchmark page BibTeX

Most text-to-image benchmarks either ignore color or check it with a vision-language model, and we found that vision-language models are themselves unreliable at recognizing colors. GenColorBench evaluates color the way color science does. It is grounded in the ISCC-NBS and CSS3/X11 naming systems, contains about 44K color-focused prompts over more than 400 colors, and scores images with object segmentation and perceptual color metrics instead of a VLM.

Be the benchmark
Does each image match the color in its prompt? Make your call, then see what GenColorBench decided. Examples are from the color name understanding task in the paper.

What we found

GenColorBench covers five tasks, namely color name understanding, multi-color composition, color-object association, numeric color understanding, and implicit color association. Evaluating a wide range of recent text-to-image and unified models shows a clear drop from basic to fine-grained colors, a consistent bias toward neutral colors, and an almost complete failure to interpret numeric color codes. That last finding led directly to NumColor.