Revealing the learning process in reinforcement learning agents through attention-oriented metrics

2024-06-20

arXiv.org

AI Summary

This work addresses the “black-box” nature of reinforcement learning (RL) training by investigating the dynamic evolution of attention and its coupling with behavioral acquisition. To this end, we propose Attention-Targeted Observation Metrics (ATOMs)—a formal framework that integrates attention visualization via neural activations and gradients, temporal modeling, and controlled ablation experiments—systematically quantifying attention development across multiple Pong variants. Our key contribution is the first empirical identification of a consistent, task-invariant three-phase attentional evolution pattern, tightly synchronized with behavioral phase transitions (e.g., policy shifts and performance jumps). Moreover, ATOMs effectively discriminate attentional profiles induced by distinct training objectives (e.g., reward shaping vs. sparse rewards). By providing an interpretable, reproducible analytical lens into RL agent cognition, this work bridges explainable AI and learning dynamics, advancing principled studies of emergent intelligence in sequential decision-making systems.

Technology Category

• Machine Learning: Reinforcement Learning
• Intelligent Robots: Behavior Learning & Control
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search

Abstract

The learning process of a reinforcement learning (RL) agent remains poorly understood beyond the mathematical formulation of its learning algorithm. To address this gap, we introduce attention-oriented metrics (ATOMs) to investigate the development of an RL agent's attention during training. In a controlled experiment, we tested ATOMs on three variations of a Pong game, each designed to teach the agent distinct behaviours, complemented by a behavioural assessment. ATOMs successfully delineate the attention patterns of an agent trained on each game variation, and that these differences in attention patterns translate into differences in the agent's behaviour. Through continuous monitoring of ATOMs during training, we observed that the agent's attention developed in phases, and that these phases were consistent across game variations. Overall, we believe that ATOM could help improve our understanding of the learning processes of RL agents and better understand the relationship between attention and learning.

Problem

• Understanding RL agent learning process
• Developing attention-oriented metrics (ATOMs)
• Linking attention patterns to behaviour

Innovation

• Attention-oriented metrics reveal RL learning
• Phased attention development in RL agents
• ATOMs link attention patterns to behavior