The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewards

2024-09-24

AI Summary

This work identifies a critical reward noise problem in vision-language models (VLMs) for embodied navigation reward learning, arising from cosine similarity bias: spurious positives (incorrectly high rewards) are more detrimental than spurious negatives, severely degrading policy learning—even below autonomous exploration baselines. To address this, we propose Binary Mutual Information (BiMI), the first information-theoretic reward function that models multimodal reward confidence by replacing conventional similarity metrics with a mutual information-based discrimination mechanism between VLM embeddings and trajectory semantics. BiMI effectively suppresses spurious positives while preserving meaningful reward signals. Evaluated across multiple embodied navigation benchmarks, BiMI significantly improves sample efficiency and final task performance, consistently outperforming intrinsic-reward-only baselines. Moreover, it provides the first systematic analysis revealing the differential impact of distinct reward noise types—particularly spurious positives—on policy optimization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While Vision-Language Models (VLMs) are increasingly used to generate reward signals for training embodied agents to follow instructions, our research reveals that agents guided by VLM rewards often underperform compared to those employing only intrinsic (exploration-driven) rewards, contradicting expectations set by recent work. We hypothesize that false positive rewards -- instances where unintended trajectories are incorrectly rewarded -- are more detrimental than false negatives. Our analysis confirms this hypothesis, revealing that the widely used cosine similarity metric is prone to false positive reward estimates. To address this, we introduce BiMI ({Bi}nary {M}utual {I}nformation), a novel reward function designed to mitigate noise. BiMI significantly enhances learning efficiency across diverse and challenging embodied navigation environments. Our findings offer a nuanced understanding of how different types of reward noise impact agent learning and highlight the importance of addressing multimodal reward signal noise when training embodied agents

Problem

• Visual Language Models
• Reward Learning
• Misleading Feedback

Innovation

• BiMI
• Visual Language Models
• Reward Correction in Robot Learning