Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning

2026-07-09

AI Summary

Existing inverse reinforcement learning methods suffer from poor generalization of learned reward functions due to reliance on demonstrations from a single environment and susceptibility to environmental dynamics. This work proposes the first hierarchical machine teaching framework that leverages multi-environment settings and multimodal feedback. By actively selecting informative environments and querying low-cost feedback—such as pairwise comparisons—the framework jointly optimizes reward learning. Theoretical analysis reveals, for the first time, that comparison-based feedback imposes stronger global constraints on the reward function in the infinite-data limit. Combining greedy environment selection with a hierarchical querying mechanism, the algorithm significantly reduces regret under the same feedback budget and demonstrates superior reward generalization in unseen environments.

Abstract

As autonomous agents are increasingly deployed across diverse operational contexts, aligning their behavior with human intent demands reward functions that remain robust to such changes rather than overfitting to any single environment. Inverse reinforcement learning (IRL) provides a principled way to infer such objectives from human feedback. However, existing analyses of optimal teaching approaches for IRL focus on single-environment, demonstration-only settings, leaving underexplored how heterogeneous feedback modalities and environment dynamics jointly constrain reward functions that generalize across multiple environments. Because demonstrations in one MDP entangle reward information with that environments specific structure, the resulting rewards frequently fail to generalize when the agent is deployed in a new setting. We first analyze how different feedback modalities constrain rewards, showing that, in the unlimited-data regime, comparisons impose strictly stronger global constraints than other modalities. Beyond this theoretical analysis, we introduce a hierarchical machine teaching algorithm for reward learning that operates across multiple MDPs. The algorithm first greedily selects informative environments that expose complementary reward constraints, then strategically queries low-cost feedback within those environments. Empirically, our method achieves substantially lower regret and stronger generalization to held-out environments than uniform teaching baselines under identical feedback budgets, demonstrating the importance of multi-environment, multi-modal teaching for learning dynamics-robust reward functions.

Problem

• reward learning
• multi-environment
• multi-modal feedback
• generalization
• inverse reinforcement learning

Innovation

• multi-modal feedback
• multi-environment teaching
• reward learning
• inverse reinforcement learning
• machine teaching