Safety from Honesty in a Disinterested AI Predictor

2026-06-28

AI Summary

This work proposes an agent-free “Scientist AI” predictor designed to mitigate unspecified implicit agentic behaviors and associated safety risks that may arise when AI prediction systems are optimized for downstream objectives. By integrating cognitively contextualized natural language representations with a Bayesian posterior-guided training objective, the approach treats goal specifications in prediction tasks as evidence to be explained rather than as intrinsic drives, thereby mechanistically precluding endogenous agency. Theoretical analysis demonstrates that, under reasonable assumptions, the resulting predictor exhibits an extremely low probability of producing residual harm exceeding a defined threshold. Empirical validation confirms that the method achieves both high predictive accuracy and strong safety guarantees without requiring access to downstream reward signals.

Abstract

As AI systems become more capable, training procedures that optimize for downstream outcomes risk introducing implicit agency: goal-directed behavior that designers never specified. We present a formal safety argument for the Scientist AI (SAI) Predictor, trained to approximate the Bayesian posterior conditioned on a dataset of "epistemically contextualized" natural-language statements. We argue that such a Predictor can honestly predict agents, actions, and their consequences without itself being an agent that selects outputs to achieve goals. This rests on data representation and on the training procedure. Epistemic contextualization of text distinguishes latent factual claims from communication acts, so expressions of goals are treated as evidence to be explained rather than drives the model adopts. With a posterior-seeking training objective, this is intended to drive the Predictor toward calibrated, cautious predictions. Training proceeds so downstream effects of deploying a prediction never serve as a reward signal; any agency the system needs is supplied by explicit scaffolding constrained by guardrails. We prove that, under assumptions on the training dynamics and on the argued sparsity of dangerous Predictors, the probability that training produces a Predictor whose guarded deployment carries residual harm above a specified threshold is small: a dangerous Predictor would have to underestimate harm in a coordinated way across many queries while such coordinated patterns are rare under the initialization distribution and receive no direct training signal. Safety and accuracy are jointly supported in this framework, since the constraints that secure accuracy are the same ones that make coordinated deception costly. These guarantees against misalignment and agency arising from within the Predictor itself do not preclude the use of the Predictor as part of an agentic system.

Problem

• AI safety
• implicit agency
• misalignment
• predictor
• goal-directed behavior

Innovation

• disinterested AI
• epistemic contextualization
• posterior-seeking training
• deception sparsity
• non-agentic predictor