From Actions to Understanding: Conformal Interpretability of Temporal Concepts in LLM Agents

2026-03-27

arXiv.org

AI Summary

This work addresses the limited interpretability of large language model (LLM) agents in multi-step interactive tasks, where it is challenging to trace the dynamic origins of success or failure. The study introduces stepwise conformal prediction into the temporal interpretability analysis of LLM agents for the first time, combining reward modeling to statistically annotate internal representations at each step and training linear probes to identify temporal concept directions correlated with task outcomes. Evaluated in ScienceWorld and AlfWorld environments, the approach demonstrates that success- and failure-related concepts are linearly separable in the representation space. Furthermore, it shows preliminary evidence that steering agent behavior toward identified success directions can effectively enhance performance.

Abstract

Large Language Models (LLMs) are increasingly deployed as autonomous agents capable of reasoning, planning, and acting within interactive environments. Despite their growing capability to perform multi-step reasoning and decision-making tasks, internal mechanisms guiding their sequential behavior remain opaque. This paper presents a framework for interpreting the temporal evolution of concepts in LLM agents through a step-wise conformal lens. We introduce the conformal interpretability framework for temporal tasks, which combines step-wise reward modeling with conformal prediction to statistically label model's internal representation at each step as successful or failing. Linear probes are then trained on these representations to identify directions of temporal concepts - latent directions in the model's activation space that correspond to consistent notions of success, failure or reasoning drift. Experimental results on two simulated interactive environments, namely ScienceWorld and AlfWorld, demonstrate that these temporal concepts are linearly separable, revealing interpretable structures aligned with task success. We further show preliminary results on improving an LLM agent's performance by leveraging the proposed framework for steering the identified successful directions inside the model. The proposed approach, thus, offers a principled method for early failure detection as well as intervention in LLM-based agents, paving the path towards trustworthy autonomous language models in complex interactive settings.

Problem

• interpretability
• temporal concepts
• large language models
• autonomous agents
• conformal prediction

Innovation

• conformal interpretability
• temporal concepts
• linear probes
• LLM agents
• failure detection