Learning to Assist Humans without Inferring Rewards

2024-11-04

arXiv.org

AI Summary

In complex human-robot collaboration, conventional reward-inference-based assistance often fails due to uncertain and dynamic human intent. This paper proposes a reward-free, empowerment-driven assistance framework that enhances human controllability over environmental state evolution—rather than modeling human reward functions. Methodologically, we introduce Contrastive Successor Representations into the empowerment maximization framework, enabling scalable and robust empowerment estimation in high-dimensional state spaces. Theoretically, our formulation unifies information-theoretic controllability measures, neuroscientific predictive coding principles, and reinforcement learning–based representation learning. Empirically, our approach significantly outperforms existing reward-free baselines on synthetic benchmarks and successfully scales to the high-dimensional collaborative environment Overcooked. Results demonstrate superior effectiveness, robustness to human behavioral variability, and computational scalability—establishing empowerment maximization as a principled foundation for adaptive, human-centered assistance.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Assistive agents should make humans' lives easier. Classically, such assistance is studied through the lens of inverse reinforcement learning, where an assistive agent (e.g., a chatbot, a robot) infers a human's intention and then selects actions to help the human reach that goal. This approach requires inferring intentions, which can be difficult in high-dimensional settings. We build upon prior work that studies assistance through the lens of empowerment: an assistive agent aims to maximize the influence of the human's actions such that they exert a greater control over the environmental outcomes and can solve tasks in fewer steps. We lift the major limitation of prior work in this area--scalability to high-dimensional settings--with contrastive successor representations. We formally prove that these representations estimate a similar notion of empowerment to that studied by prior work and provide a ready-made mechanism for optimizing it. Empirically, our proposed method outperforms prior methods on synthetic benchmarks, and scales to Overcooked, a cooperative game setting. Theoretically, our work connects ideas from information theory, neuroscience, and reinforcement learning, and charts a path for representations to play a critical role in solving assistive problems.

Problem

• Complex Environment
• User Demand Prediction
• Assistive Technology

Innovation

• Empowerment-based Approach
• Enhanced Assistance Efficiency
• Overcooked Game Environment