Evaluating the Goal-Directedness of Large Language Models

2025-04-16

AI Summary

This work investigates goal-directedness in large language models (LLMs)—their capacity to systematically engage in information gathering, cognitive effort, and plan execution to achieve a given objective, distinct from mere task accuracy. Method: The authors introduce and operationalize the first accuracy-agnostic metric for goal-directedness, proposing a novel evaluation framework grounded in multi-stage subtask decomposition and causal reasoning. Experiments span leading models from Google DeepMind, OpenAI, and Anthropic. Contribution/Results: Goal-directedness exhibits cross-task consistency but low sensitivity to motivational prompting; most models fall short of full goal-directed behavior, and this capability shows only weak correlation with task accuracy. The study establishes an interpretable, reproducible benchmark for designing, monitoring, and safety-evaluating LLM agents’ agentic properties.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

To what extent do LLMs use their capabilities towards their given goal? We take this as a measure of their goal-directedness. We evaluate goal-directedness on tasks that require information gathering, cognitive effort, and plan execution, where we use subtasks to infer each model's relevant capabilities. Our evaluations of LLMs from Google DeepMind, OpenAI, and Anthropic show that goal-directedness is relatively consistent across tasks, differs from task performance, and is only moderately sensitive to motivational prompts. Notably, most models are not fully goal-directed. We hope our goal-directedness evaluations will enable better monitoring of LLM progress, and enable more deliberate design choices of agentic properties in LLMs.

Problem

• Assess goal-directedness of LLMs in task execution
• Compare goal-directedness across models and tasks
• Evaluate impact of prompts on LLM goal-directedness

Innovation

• Evaluates goal-directedness in LLMs
• Uses subtasks to infer model capabilities
• Assesses models from multiple organizations