Robots That Know What to Ask: Recovering Misaligned Rewards through Targeted Explanations

2026-05-21

AI Summary

This work addresses the ambiguity in reward function learning from imperfect human demonstrations, which often arises due to overlooked or underrepresented features and leads to suboptimal behaviors. The authors propose a novel approach that automatically detects reward ambiguity by analyzing the statistical variability of features in demonstrations, generates natural language explanations to articulate the associated uncertainty, and actively solicits corrective user feedback for high-uncertainty regions. By leveraging feature variability as an intrinsic signal of ambiguity—integrating inverse reinforcement learning, explainability, and active querying—the method significantly outperforms both random querying and passive learning baselines in simulated tabletop tasks and real-world experiments with a Franka robot, effectively reducing reward ambiguity and improving human-robot alignment.

Abstract

Learning reward functions from demonstrations assumes that demonstrations provide adequate supervision over all features -- or task-relevant aspects of behavior. In practice, demonstrations are often imperfect: humans may under-emphasize certain features due to cognitive load or physical difficulty, or the training regime may fail to sufficiently cover all relevant situations. In either case, important features may be underspecified, leading to ambiguity in the learned reward function and misaligned behavior at deployment. We propose a framework that detects such underspecified features and actively solicits targeted corrective demonstrations. Our key insight is that demonstrations implicitly reveal which features are well specified: features that are consistently optimized show little variation across demonstrations, while features that are underspecified vary widely. We leverage this statistical signal to infer which features may have been insufficiently demonstrated. The robot then explains which features it is uncertain about in natural language and queries for demonstrations that explicitly address the identified gaps. We evaluate our approach in a simulated tabletop manipulation domain and in a user study with a real Franka robot. Targeted, explanation-guided queries significantly improve reward recovery compared to random querying and passive data collection, reducing ambiguity that would otherwise persist in learning from imperfect demonstrations.

Problem

• reward learning
• demonstration imperfection
• feature underspecification
• reward ambiguity
• misaligned behavior

Innovation

• reward learning from demonstrations
• active querying
• feature underspecification
• natural language explanations
• human-robot interaction