The Limits of Predicting Agents from Behaviour

2025-06-03

AI Summary

Can an AI agent’s beliefs and goals be reliably inferred solely from its behavioral data, enabling accurate prediction of its behavior in unseen environments? This foundational question bears directly on AI safety and interpretability. Method: Under the world-model assumption, we develop a formal framework integrating probabilistic inference, causal modeling, and generalization theory—leveraging behavioral observations and counterfactual environment modeling. Contribution/Results: We establish the first rigorous theoretical bounds for cross-environment behavioral prediction by intelligent agents. Crucially, we characterize the fundamental limits of purely behavior-driven intent inference, proving an insurmountable upper bound on identifiability. Our results yield the first theoretically grounded, quantitative benchmark for AI fairness evaluation, robust deployment, and safety verification—bridging formal guarantees with practical AI assurance.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

As the complexity of AI systems and their interactions with the world increases, generating explanations for their behaviour is important for safely deploying AI. For agents, the most natural abstractions for predicting behaviour attribute beliefs, intentions and goals to the system. If an agent behaves as if it has a certain goal or belief, then we can make reasonable predictions about how it will behave in novel situations, including those where comprehensive safety evaluations are untenable. How well can we infer an agent's beliefs from their behaviour, and how reliably can these inferred beliefs predict the agent's behaviour in novel situations? We provide a precise answer to this question under the assumption that the agent's behaviour is guided by a world model. Our contribution is the derivation of novel bounds on the agent's behaviour in new (unseen) deployment environments, which represent a theoretical limit for predicting intentional agents from behavioural data alone. We discuss the implications of these results for several research areas including fairness and safety.

Problem

• How to infer agent's beliefs from behavior
• Predicting agent's behavior in new situations
• Theoretical limits for intentional agent prediction

Innovation

• Predicting agent behaviour using beliefs and goals
• Deriving bounds for behaviour in unseen environments
• Theoretical limits from behavioural data alone