Embedded Universal Predictive Intelligence: a coherent framework for multi-agent learning

2025-11-27

AI Summary

Traditional model-free reinforcement learning assumes a static environment and treats the agent as decoupled from it, rendering it inadequate for multi-agent settings where non-stationarity arises from other agents’ learning. Method: We propose an “embedded universal predictive intelligence” framework that models the agent itself as part of the environment, enabling joint prospective prediction of both self and others’ behaviors. Our approach integrates Bayesian reinforcement learning, AIXI theory, and Solomonoff’s prior to construct an embedded agent with intrinsic self-prediction capability, supporting infinite-order theory of mind and self-consistent mutual prediction. Contribution/Results: We prove that an idealized embedded agent achieves consistent mutual predictability. Empirically, our framework yields novel game-theoretic solution concepts and emergent cooperative patterns unattainable under standard decoupled paradigms. It establishes a new benchmark for embedded multi-agent learning, bridging foundational AI theory with scalable multi-agent coordination.

Technology Category

• Multiagent Systems: Multiagent Learning
• Game Theory and Economic Paradigms: Adversarial Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The standard theory of model-free reinforcement learning assumes that the environment dynamics are stationary and that agents are decoupled from their environment, such that policies are treated as being separate from the world they inhabit. This leads to theoretical challenges in the multi-agent setting where the non-stationarity induced by the learning of other agents demands prospective learning based on prediction models. To accurately model other agents, an agent must account for the fact that those other agents are, in turn, forming beliefs about it to predict its future behavior, motivating agents to model themselves as part of the environment. Here, building upon foundational work on universal artificial intelligence (AIXI), we introduce a mathematical framework for prospective learning and embedded agency centered on self-prediction, where Bayesian RL agents predict both future perceptual inputs and their own actions, and must therefore resolve epistemic uncertainty about themselves as part of the universe they inhabit. We show that in multi-agent settings, self-prediction enables agents to reason about others running similar algorithms, leading to new game-theoretic solution concepts and novel forms of cooperation unattainable by classical decoupled agents. Moreover, we extend the theory of AIXI, and study universally intelligent embedded agents which start from a Solomonoff prior. We show that these idealized agents can form consistent mutual predictions and achieve infinite-order theory of mind, potentially setting a gold standard for embedded multi-agent learning.

Problem

• Addresses non-stationarity in multi-agent reinforcement learning via predictive models.
• Integrates self-prediction to model agents as embedded parts of their environment.
• Extends AIXI theory for universal intelligence in cooperative multi-agent settings.

Innovation

• Self-prediction framework for embedded agency
• Bayesian RL agents modeling themselves and environment
• Universal AI extension for multi-agent cooperation