VADF: Vision-Adaptive Diffusion Policy Framework for Efficient Robotic Manipulation

2026-04-17

AI Summary

This work addresses the slow training convergence and excessive inference latency commonly observed in existing diffusion-based robotic policies, which stem from uniform sampling strategies and a lack of awareness of sample difficulty. To overcome these limitations, we propose VADF—the first model-agnostic, vision-based adaptive diffusion policy framework—featuring two key components: an Adaptive Loss Network (ALN) that employs a lightweight MLP to dynamically evaluate and reweight hard negative samples during training, and a Hierarchical Vision-based Task Segmenter (HVTS) that decomposes visual instructions to adaptively modulate noise scheduling and action sequence length. Our approach substantially accelerates training convergence, significantly reduces computational overhead at inference time, and markedly improves early-stage task success rates.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Diffusion policies are becoming mainstream in robotic manipulation but suffer from hard negative class imbalance due to uniform sampling and lack of sample difficulty awareness, leading to slow training convergence and frequent inference timeout failures. We propose VADF (Vision-Adaptive Diffusion Policy Framework), a vision-driven dual-adaptive framework that significantly reduces convergence steps and achieves early success in inference, with model-agnostic design enabling seamless integration into any diffusion policy architecture. During training, we introduce Adaptive Loss Network (ALN), a lightweight MLP-based loss predictor that quantifies per-step sample difficulty in real time. Guided by hard negative mining, it performs weighted sampling to prioritize high-loss regions, enabling adaptive weight updates and faster convergence. In inference, we design the Hierarchical Vision Task Segmenter (HVTS), which decomposes high-level task instructions into multi-stage low-level sub-instructions based on visual input. It adaptively segments action sequences into simple and complex subtasks by assigning shorter noise schedules with longer direct execution sequences to simple actions, and longer noise steps with shorter execution sequences to complex ones, thereby dramatically reducing computational overhead and significantly improving the early success rate.

Problem

• diffusion policies
• class imbalance
• robotic manipulation
• training convergence
• inference timeout

Innovation

• diffusion policy
• adaptive sampling
• vision-based manipulation
• hard negative mining
• noise schedule adaptation