COVR:Collaborative Optimization of VLMs and RL Agent for Visual-Based Control

2026-01-04

arXiv.org

AI Summary

This work addresses the low sample efficiency of visual reinforcement learning due to high-dimensional observations and the underutilization of reinforcement learning (RL) interaction data for enhancing vision-language models (VLMs). The authors propose COVR, a framework that enables the first bidirectional co-optimization between VLMs and RL agents. On one hand, RL-generated data is leveraged to fine-tune the VLM through exploration-driven dynamic filtering and reward-aware adaptive loss weighting, thereby improving its task-relevant semantic reasoning. On the other hand, the enhanced VLM provides action priors to guide policy learning, with progressive fine-tuning employed to reduce computational overhead. Experiments demonstrate that COVR significantly improves both performance and generalization across a range of complex visual control tasks.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Visual reinforcement learning (RL) suffers from poor sample efficiency due to high-dimensional observations in complex tasks. While existing works have shown that vision-language models (VLMs) can assist RL, they often focus on knowledge distillation from the VLM to RL, overlooking the potential of RL-generated interaction data to enhance the VLM. To address this, we propose COVR, a collaborative optimization framework that enables the mutual enhancement of the VLM and RL policies. Specifically, COVR fine-tunes the VLM with RL-generated data to enhance the semantic reasoning ability consistent with the target task, and uses the enhanced VLM to further guide policy learning via action priors. To improve fine-tuning efficiency, we introduce two key modules: (1) an Exploration-Driven Dynamic Filter module that preserves valuable exploration samples using adaptive thresholds based on the degree of exploration, and (2) a Return-Aware Adaptive Loss Weight module that improves the stability of training by quantifying the inconsistency of sampling actions via return signals of RL. We further design a progressive fine-tuning strategy to reduce resource consumption. Extensive experiments show that COVR achieves strong performance across various challenging visual control tasks.

Problem

• visual reinforcement learning
• vision-language models
• sample efficiency
• mutual enhancement
• high-dimensional observations

Innovation

• Collaborative Optimization
• Vision-Language Models
• Visual Reinforcement Learning
• Action Priors
• Adaptive Fine-tuning