Elastic Queries Reinforcement Learning: Self-Aware Policy Execution for VLA Models

2026-06-12

AI Summary

Existing fixed inference and replanning strategies struggle to adapt to the dynamic variations in state difficulty encountered in robot control, often leading to either computational waste or insufficient control responsiveness. This work proposes an Elastic Query Reinforcement Learning framework that dynamically adjusts the input context, denoising budget, and action chunk length for each policy query—without fine-tuning the underlying Vision-Language-Action (VLA) model. The approach leverages a lightweight implicit scheduler adapter guided by a state-difficulty signal derived from ensemble critic disagreement. Integrating query-level macro-action reinforcement learning, action-chunk-aware discounting, amortized NFE budgeting, and residual learning, the method significantly reduces average inference overhead while maintaining or improving task success rates across both simulated and real-world robotic tasks.

Abstract

Vision-language-action (VLA) models are powerful action generators for robot manipulation, but they are typically executed with fixed inference and replanning schedules. This rigidity ignores the uneven difficulty of robot control: contact-rich or uncertain states may need more computation and fresher feedback, while easier states can often be handled with fewer inference steps and longer open-loop execution. We propose Elastic Queries Reinforcement Learning (EQRL), a framework that makes each VLA policy query elastic. A lightweight latent-schedule adaptor jointly selects the latent input, denoising budget, and action chunk length, without fine-tuning the underlying VLA model. To make scheduling difficulty-aware, EQRL trains a critic over the joint latent-schedule action and derives a state difficulty signal from critic ensemble disagreement. This signal guides compute toward difficult states, while a learned residual allows task-driven correction. We formulate variable chunk execution as query-level macro-action RL with chunk-dependent discounting and an amortized number-of-function-evaluations (NFE) budget. Across simulation and real-robot manipulation, EQRL reduces amortized inference cost while preserving or improving task success.

Problem

• Vision-Language-Action Models
• Adaptive Inference
• Robot Manipulation
• State Difficulty
• Computation Allocation

Innovation

• Elastic Queries
• Vision-Language-Action Models
• Difficulty-Aware Scheduling
• Macro-Action Reinforcement Learning
• Amortized Inference