🤖 AI Summary
This paper addresses the foundational question: *When can a physical system be rigorously regarded as an agent possessing beliefs and goals?* We propose a POMDP-based explanatory framework, formalizing an agent as a physical system satisfying two joint constraints: (i) its state evolution must conform to Bayesian belief-updating dynamics, and (ii) its policy must be optimal with respect to a specified objective. Crucially, we introduce the *completeness of a POMDP solution*—simultaneous adherence to correct belief evolution and optimal action selection—as a necessary and empirically falsifiable criterion for agency, overcoming the limitations of prior definitions relying solely on belief mapping. This yields the first axiomatization of agency that is mathematically rigorous, computationally operational, and empirically testable. It reveals necessary constraints linking physical dynamics to agential properties, thereby establishing a theoretical foundation for AI safety verification and computational modeling of consciousness.
📝 Abstract
Under what circumstances can a system be said to have beliefs and goals, and how do such agency-related features relate to its physical state? Recent work has proposed a notion of interpretation map, a function that maps the state of a system to a probability distribution representing its beliefs about an external world. Such a map is not completely arbitrary, as the beliefs it attributes to the system must evolve over time in a manner that is consistent with Bayes' theorem, and consequently the dynamics of a system constrain its possible interpretations. Here we build on this approach, proposing a notion of interpretation not just in terms of beliefs but in terms of goals and actions. To do this we make use of the existing theory of partially observable Markov processes (POMDPs): we say that a system can be interpreted as a solution to a POMDP if it not only admits an interpretation map describing its beliefs about the hidden state of a POMDP but also takes actions that are optimal according to its belief state. An agent is then a system together with an interpretation of this system as a POMDP solution. Although POMDPs are not the only possible formulation of what it means to have a goal, this nevertheless represents a step towards a more general formal definition of what it means for a system to be an agent.