Muhammad Atif Butt

Mechanistic interpretability often relies on the Linear Representation Hypothesis, which assumes that high-level concepts are encoded as linear directions in activation space. Yet natural visual concepts often pass through genuine intermediate states, and when text-to-image models are prompted for intermediate attributes, their activations rarely fall on the straight line connecting the endpoints. KANSteer models concept traversal as a curve through these intermediate states using Kolmogorov-Arnold Networks, which provide a one-dimensional coordinate whose learned functions define the trajectory, so the steering direction varies along the concept while remaining interpretable. Across several concepts and text-to-image diffusion transformers, activation trajectories deviate substantially from straight lines, and KANSteer provides a closer fit and smoother traversal of intermediate attributes than linear steering.

Text-to-image diffusion models struggle with numerical color specifications such as hex codes and RGB values, because subword tokenization splits color codes into fragments the text encoder cannot map to consistent colors. NumColor introduces a Color Token Aggregator that detects color specifications regardless of tokenization and a ColorBook of 6,707 learnable embeddings aligned with CIE Lab space, together with auxiliary losses that keep the Lab and embedding geometries consistent. Trained on NumColor-Data, a synthetic set of 500K rendered images with exact color-to-pixel correspondence, the model is trained only on FLUX and transfers zero-shot to SD3, SD3.5, and PixArt models.

Recent text-to-image models still struggle with fine-grained color controllability and often fail to match the colors specified in prompts. GenColorBench is a benchmark of about 44K color-focused prompts covering more than 400 colors, grounded in color systems such as ISCC-NBS and CSS3/X11 and including numerical colors. It evaluates five tasks (color name understanding, multi-color composition, color-object association, numeric color understanding, and implicit color association) by combining VQA with perceptual color metrics.

Text-to-image models lack precise control over illumination. LumiCtrl learns an illuminant prompt from a single image of an object using three components: physics-based illuminant augmentation along the Planckian locus, edge-guided prompt disentanglement with a frozen ControlNet so that the prompt captures light rather than structure, and a masked reconstruction loss that focuses learning on the foreground while letting the background adapt.

Text-to-image models accept color names, but names cover wide ranges, so matching an exact color is hard, and existing personalization methods entangle color with object shape. ColorPeel generates simple 2D and 3D geometric shapes in the target color and learns a color prompt from them, explicitly disentangling color from shape so that the learned token carries color alone. The approach extends to abstract attributes such as textures and materials.