Exploring Conditions for Diffusion models in Robotic Control

2025-10-17

AI Summary

This work addresses the challenge of directly applying pretrained text-to-image diffusion models to robot control. We propose ORCA, a framework that bypasses fine-tuning the diffusion model itself and instead jointly optimizes learnable task prompts and frame-level visual prompts to transform textual conditions into task-adaptive, temporally aware visual representations. By redesigning the conditioning mechanism—rather than modifying the diffusion backbone—ORCA bridges semantic and distributional gaps between linguistic priors and the robot’s perception–action space, enabling efficient guidance of control policies. Evaluated across multiple simulation and real-robot control benchmarks, ORCA consistently outperforms existing diffusion-based and vision-encoder-based approaches, achieving state-of-the-art performance. Our results empirically validate that reengineering the conditioning mechanism, rather than fine-tuning the generative model, is an effective paradigm for representation learning in embodied intelligence.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Intelligent Robots: Learning & Optimization for ROB
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

While pre-trained visual representations have significantly advanced imitation learning, they are often task-agnostic as they remain frozen during policy learning. In this work, we explore leveraging pre-trained text-to-image diffusion models to obtain task-adaptive visual representations for robotic control, without fine-tuning the model itself. However, we find that naively applying textual conditions - a successful strategy in other vision domains - yields minimal or even negative gains in control tasks. We attribute this to the domain gap between the diffusion model's training data and robotic control environments, leading us to argue for conditions that consider the specific, dynamic visual information required for control. To this end, we propose ORCA, which introduces learnable task prompts that adapt to the control environment and visual prompts that capture fine-grained, frame-specific details. Through facilitating task-adaptive representations with our newly devised conditions, our approach achieves state-of-the-art performance on various robotic control benchmarks, significantly surpassing prior methods.

Problem

• Adapting pre-trained diffusion models for robotic control tasks
• Overcoming domain gaps between image generation and control environments
• Developing task-adaptive visual representations without model fine-tuning

Innovation

• Uses pre-trained diffusion models without fine-tuning
• Introduces learnable task prompts for environment adaptation
• Employs visual prompts capturing frame-specific details