Value Under Ignorance in Universal Artificial Intelligence

2025-12-18

AI Summary

This paper addresses the insufficient generalization capability of utility functions within the AIXI framework—particularly when the agent’s belief distribution includes incomplete hypotheses that predict only finite prefixes of history, leaving utility assignment theoretically unjustified. To resolve this, we model beliefs as imprecise probabilities, formalizing total ignorance via submeasures and defining generalized expected utility through the Choquet integral. This work constitutes the first integration of imprecise probability theory with universal AI value functions. We introduce the “ignorance-is-fatal” principle with a dual semantic interpretation: under the weak interpretation, the model recovers standard recursive utility; under the strong interpretation, we prove that expected utility cannot be represented as a Choquet integral, thereby rigorously establishing its computability boundary. The approach provides a theoretically grounded, robust framework for utility evaluation under deep uncertainty in universal artificial intelligence.

Technology Category

• Reasoning under Uncertainty: Decision/Utility Theory
• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Philosophy and Ethics of AI: Artificial General Intelligence

Application Category

• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We generalize the AIXI reinforcement learning agent to admit a wider class of utility functions. Assigning a utility to each possible interaction history forces us to confront the ambiguity that some hypotheses in the agent's belief distribution only predict a finite prefix of the history, which is sometimes interpreted as implying a chance of death equal to a quantity called the semimeasure loss. This death interpretation suggests one way to assign utilities to such history prefixes. We argue that it is as natural to view the belief distributions as imprecise probability distributions, with the semimeasure loss as total ignorance. This motivates us to consider the consequences of computing expected utilities with Choquet integrals from imprecise probability theory, including an investigation of their computability level. We recover the standard recursive value function as a special case. However, our most general expected utilities under the death interpretation cannot be characterized as such Choquet integrals.

Problem

• Generalizes AIXI to handle broader utility functions in reinforcement learning.
• Addresses ambiguity in utility assignment for finite history prefixes.
• Investigates expected utilities using Choquet integrals from imprecise probability theory.

Innovation

• Generalize AIXI agent for broader utility functions
• Use Choquet integrals for imprecise probability distributions
• Address semimeasure loss as total ignorance in computations