Interpreting Reinforcement Learning Agents with Susceptibilities

2026-05-08

AI Summary

This work addresses the challenge of uncovering the internal representational dynamics of deep reinforcement learning agents, which cannot be fully inferred from policy behavior alone. For the first time, it introduces sensitivity analysis—a technique from neural network interpretability—into deep reinforcement learning, integrating it with a regret minimization framework. By examining how observable expectations respond to perturbations in the loss function, the method characterizes the agent’s phased evolution in parameter space. Evaluated in nontrivial grid-world environments, this approach successfully reveals latent stage-wise structures during training, demonstrating the efficacy of sensitivity analysis as an interpretability tool in reinforcement learning. The findings align with results from activation intervention experiments, offering a novel perspective for understanding agent learning dynamics and informing post-training optimizations such as reinforcement learning from human feedback (RLHF).

Abstract

Susceptibilities are a technique for neural network interpretability that studies the response of posterior expectation values of observables to perturbations of the loss. We generalize this construction to the setting of the regret in deep reinforcement learning and investigate the utility of susceptibilities in a simple gridworld model that nevertheless exhibits non-trivial stagewise development. We argue that susceptibilities reveal internal features of the development of the model in parameter space that one cannot detect purely by studying the development of the learned policy. We validate these results with activation-steering, and discuss the framework's extension to RLHF post-training.

Problem

• reinforcement learning
• interpretability
• susceptibilities
• parameter space
• policy development

Innovation

• susceptibilities
• reinforcement learning interpretability
• regret analysis
• activation steering
• RLHF