Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Models

2026-06-29

AI Summary

This work proposes T²VLA, a novel framework that addresses the challenge of unsupervised policy optimization in vision-language-action (VLA) models, which typically rely on external rewards in reinforcement learning. The study reveals, for the first time, a strong correlation between the model’s confidence in generated trajectories and actual task success rates, leveraging this insight to design an intrinsic reward mechanism. This enables bootstrapped policy improvement at test time without external feedback. T²VLA introduces a confidence-driven dual-expert bootstrapping scheme, combined with trajectory-level similarity metrics, to jointly refine a local pseudo-expert and a global expert pool. The approach is compatible with diverse VLA architectures, including OpenVLA-OFT and the pi series. Experiments demonstrate that T²VLA significantly outperforms supervised baselines on the LIBERO and RoboTwin benchmarks, achieving performance approaching that of Oracle RL that uses ground-truth rewards, thereby enabling effective policy enhancement without external supervision.

Abstract

Reinforcement learning (RL) has become indispensable for pushing Vision-Language-Action Models (VLAs) beyond static imitation learning. However, existing RL methods typically require external environmental feedback, relying on predefined success signals to guide policy updates. In this work, we show that VLA models possess useful internal evaluative capabilities: in discrete-action VLAs, trajectories with higher generation confidence are significantly more likely to succeed. Based on this observation, we introduce T^2VLA (Test-time VLA), an architecture-agnostic test-time RL framework that enables VLA models to achieve self-bootstrapping policy improvement. Instead of relying on external rewards, T^2VLA leverages trajectory-level similarity to high-confidence expert demonstrations as an intrinsic reward signal. In addition, we propose a Confidence-Driven Dual Expert Bootstrapping mechanism, which dynamically balances a Local Pseudo-Expert for exploration and a Global Expert Pool for training stability. Extensive experiments on the LIBERO and RoboTwin benchmarks show that T^2VLA consistently outperforms supervised baselines and approaches oracle RL performance with ground-truth rewards, achieving effective improvement without external reward feedback. Furthermore, T^2VLA adapts to distinct VLA paradigms, including both OpenVLA-OFT and the pi series.

Problem

• Vision-Language-Action Models
• Reinforcement Learning
• Test-Time Adaptation
• Reward-Free Learning
• Policy Improvement

Innovation

• Test-Time Reinforcement Learning
• Vision-Language-Action Models
• Confidence-Driven Reward
• Intrinsic Reward
• Expert Bootstrapping