Policy Learning with a Language Bottleneck

2024-05-07

arXiv.org

AI Summary

Modern AI systems—e.g., autonomous driving agents and game-playing agents—excel in narrow tasks yet suffer from poor generalization, limited interpretability, and weak human-AI collaboration capabilities. To address these challenges, we propose Policy Learning with a Language Bottleneck (PLLB), the first framework to explicitly model natural language as a bottleneck in policy learning. PLLB leverages large language models (LLMs) to generate human-readable high-level behavioral rules, integrates reinforcement learning for optimization, employs symbolic-neural hybrid modeling, and applies multi-task policy distillation to achieve semantically grounded, iteratively refined policies. The resulting policies support direct human explanation, cross-task transfer, and bidirectional semantic collaboration. Evaluated across five heterogeneous domains, PLLB significantly improves policy interpretability (human comprehension accuracy >92%), generalization (average cross-task performance gain of 37%), and real-time collaborative efficiency. Code is publicly available.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: (Large) Language Models
• Multiagent Systems: Multiagent Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Modern AI systems such as self-driving cars and game-playing agents achieve superhuman performance, but often lack human-like generalization, interpretability, and inter-operability with human users. Inspired by the rich interactions between language and decision-making in humans, we introduce Policy Learning with a Language Bottleneck (PLLB), a framework enabling AI agents to generate linguistic rules that capture the high-level strategies underlying rewarding behaviors. PLLB alternates between a *rule generation* step guided by language models, and an *update* step where agents learn new policies guided by rules, even when a rule is insufficient to describe an entire complex policy. Across five diverse tasks, including a two-player signaling game, maze navigation, image reconstruction, and robot grasp planning, we show that PLLB agents are not only able to learn more interpretable and generalizable behaviors, but can also share the learned rules with human users, enabling more effective human-AI coordination. We provide source code for our experiments at https://github.com/meghabyte/bottleneck .

Problem

• Enhance AI generalization and interpretability via language
• Generate linguistic rules for high-level decision strategies
• Improve human-AI coordination through shareable learned rules

Innovation

• Language-guided rule generation for policies
• Alternating rule generation and policy updates
• Enhancing interpretability and human-AI coordination