Talk2Move: Reinforcement Learning for Text-Instructed Object-Level Geometric Transformation in Scenes

2026-01-05

arXiv.org

AI Summary

Existing text-guided image editing methods struggle to achieve object-level geometric transformations due to the absence of paired supervision and bottlenecks in pixel-level optimization. This work proposes a reinforcement learning–based diffusion framework that enables spatial manipulation of objects driven by natural language instructions. The approach introduces a novel Group Relative Policy Optimization (GRPO) algorithm, featuring an object-centric spatial reward function, and integrates off-policy evaluation with active sampling strategies. This design allows for semantically consistent and interpretable geometric edits without requiring paired training data. Evaluated across multiple benchmarks, the method significantly outperforms existing approaches in both spatial accuracy and scene consistency, enabling precise and coherent object-level control.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We introduce Talk2Move, a reinforcement learning (RL) based diffusion framework for text-instructed spatial transformation of objects within scenes. Spatially manipulating objects in a scene through natural language poses a challenge for multimodal generation systems. While existing text-based manipulation methods can adjust appearance or style, they struggle to perform object-level geometric transformations-such as translating, rotating, or resizing objects-due to scarce paired supervision and pixel-level optimization limits. Talk2Move employs Group Relative Policy Optimization (GRPO) to explore geometric actions through diverse rollouts generated from input images and lightweight textual variations, removing the need for costly paired data. A spatial reward guided model aligns geometric transformations with linguistic description, while off-policy step evaluation and active step sampling improve learning efficiency by focusing on informative transformation stages. Furthermore, we design object-centric spatial rewards that evaluate displacement, rotation, and scaling behaviors directly, enabling interpretable and coherent transformations. Experiments on curated benchmarks demonstrate that Talk2Move achieves precise, consistent, and semantically faithful object transformations, outperforming existing text-guided editing approaches in both spatial accuracy and scene coherence.

Problem

• text-instructed manipulation
• object-level geometric transformation
• spatial transformation
• multimodal generation
• scene editing

Innovation

• Reinforcement Learning
• Text-to-3D Manipulation
• Geometric Transformation
• Spatial Reward
• Diffusion Model