A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents

2026-02-09

AI Summary

Existing methods struggle to reliably attribute goals in agent systems, limiting the interpretability and predictability of their behavior. This work proposes an integrated framework that combines behavioral evaluation with internal representation analysis to investigate the goal-directedness of language model agents navigating toward target states in 2D grid environments. Through behavioral benchmarks, comparisons with optimal policies, representation probing, and reasoning process analysis, we find that agent performance scales robustly with task difficulty, that agents coarsely encode the spatial structure of the environment, and that their reasoning processes dynamically shift representations from reliance on global cues toward supporting immediate actions. These findings reveal how language model agents nonlinearly encode spatial information and adaptively refine their internal representations to enable goal-directed decision-making.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Understanding an agent's goals helps explain and predict its behaviour, yet there is no established methodology for reliably attributing goals to agentic systems. We propose a framework for evaluating goal-directedness that integrates behavioural evaluation with interpretability-based analyses of models'internal representations. As a case study, we examine an LLM agent navigating a 2D grid world toward a goal state. Behaviourally, we evaluate the agent against an optimal policy across varying grid sizes, obstacle densities, and goal structures, finding that performance scales with task difficulty while remaining robust to difficulty-preserving transformations and complex goal structures. We then use probing methods to decode the agent's internal representations of the environment state and its multi-step action plans. We find that the LLM agent non-linearly encodes a coarse spatial map of the environment, preserving approximate task-relevant cues about its position and the goal location; that its actions are broadly consistent with these internal representations; and that reasoning reorganises them, shifting from broader environment structural cues toward information supporting immediate action selection. Our findings support the view that introspective examination is required beyond behavioural evaluations to characterise how agents represent and pursue their objectives.

Problem

• goal-directedness
• language model agents
• behavioural evaluation
• internal representations
• agent goals

Innovation

• goal-directedness
• behavioral evaluation
• representation probing
• language model agents
• internal representations