Attention Trajectories as a Diagnostic Axis for Deep Reinforcement Learning

2025-11-25

AI Summary

Deep reinforcement learning (DRL) agents suffer from poor interpretability during training, hindering mechanistic understanding of policy acquisition. Method: We propose Attention-Oriented Metrics (ATOMs) and the diagnostic framework ATOMICs to systematically characterize the dynamic evolution of agent attention patterns throughout training. Leveraging multi-variant Pong environments, we integrate attention visualization, behavioral evaluation, and controlled experimental design to quantify the coupling between attention trajectories and policy behavior. Contribution/Results: Our study reveals, for the first time, a task-invariant, stage-wise progression in DRL attention development—comprising initial exploration, critical feature focusing, and policy stabilization—each stage aligning significantly with abrupt improvements in behavioral performance. This work establishes a reproducible, interpretable paradigm for analyzing DRL learning dynamics, advancing agent diagnostics and trustworthy AI research.

Technology Category

• Machine Learning: Reinforcement Learning
• Multiagent Systems: Multiagent Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

The learning process of a reinforcement learning (RL) agent remains poorly understood beyond the mathematical formulation of its learning algorithm. To address this gap, we introduce attention-oriented metrics (ATOMs) to investigate the development of an RL agent's attention during training. In a controlled experiment, we tested ATOMs on three variations of a Pong game, each designed to teach the agent distinct behaviours, complemented by a behavioural assessment. ATOMs successfully delineate the attention patterns of an agent trained on each game variation, and that these differences in attention patterns translate into differences in the agent's behaviour. Through continuous monitoring of ATOMs during training, we observed that the agent's attention developed in phases, and that these phases were consistent across game variations. Overall, we believe that ATOM could help improve our understanding of the learning processes of RL agents and better understand the relationship between attention and learning.

Problem

• Investigating RL agent attention development during training process
• Analyzing how attention patterns translate into behavioral differences
• Monitoring attention trajectory phases across different game variations

Innovation

• Introduces attention-oriented metrics for RL agents
• Monitors attention development phases during training
• Links attention patterns to agent behavior differences