LightMover: Generative Light Movement with Color and Intensity Controls

2026-03-28

AI Summary

This work proposes a method for controllable and physically plausible lighting editing in a single image without requiring re-rendering. The approach formulates lighting manipulation as a sequence-to-sequence prediction task in visual token space, leveraging video diffusion priors to jointly regulate light source position, color, intensity, and their resulting shadows, reflections, and attenuation effects. Key innovations include a unified representation of spatial and appearance attributes of illumination, an adaptive token pruning mechanism that reduces control sequence length by 41% without compromising fidelity, and a scalable image-pair rendering pipeline. Experiments demonstrate superior performance across multiple tasks, achieving high PSNR and strong semantic consistency (as measured by DINO and CLIP), while enabling precise and independent control over lighting parameters.

Abstract

We present LightMover, a framework for controllable light manipulation in single images that leverages video diffusion priors to produce physically plausible illumination changes without re-rendering the scene. We formulate light editing as a sequence-to-sequence prediction problem in visual token space: given an image and light-control tokens, the model adjusts light position, color, and intensity together with resulting reflections, shadows, and falloff from a single view. This unified treatment of spatial (movement) and appearance (color, intensity) controls improves both manipulation and illumination understanding. We further introduce an adaptive token-pruning mechanism that preserves spatially informative tokens while compactly encoding non-spatial attributes, reducing control sequence length by 41% while maintaining editing fidelity. To train our framework, we construct a scalable rendering pipeline that generates large numbers of image pairs across varied light positions, colors, and intensities while keeping the scene content consistent with the original image. LightMover enables precise, independent control over light position, color, and intensity, and achieves high PSNR and strong semantic consistency (DINO, CLIP) across different tasks.

Problem

• light manipulation
• image editing
• illumination control
• single-image relighting
• controllable generation

Innovation

• light editing
• video diffusion priors
• visual token space
• adaptive token pruning
• controllable illumination