Phantom Guardrails: When Self-Improving Agent Harnesses Fix Failures That Never Happened

2026-07-13

AI Summary

This work investigates the phenomenon of “phantom guardrails,” wherein self-improving agents fabricate errors and apply ineffective safeguards in the absence of actual failures. To systematically examine this behavior, the authors construct a counterfactual hallucination laboratory—a deterministic, non-interventional environment—employing a large language model proposer, byte-precise oracle verification, deterministic micro-experimental setups, and controlled variable analysis. Their experiments reveal that when rule-like patterns, open-ended rule sets, and pre-specified failure instructions coexist, agents structurally generate spurious fixes in 15 out of 60 runs. This tendency persists across both single-proposal and iterative acceptance cycles. The study introduces the first reproducible evaluation framework for this issue, offering a novel dimension for assessing the reliability of self-improving systems.

Abstract

Self-improving AI agents are designed to learn from their mistakes. We show they can also hallucinate mistakes that never happened. We study this failure mode in automated harness optimization, where an LLM-based proposer edits an agent's scaffold, including prompts, parsers, filters, validators and guardrails, to eliminate observed failures. But this process rarely asks first: was there a real failure to fix? We introduce the Counterfactual Fabrication Lab, a deterministic micro-lab where the correct action is known: do nothing. The lab plants a candidate guardrail for a failure class that provably never occurs, presents only legal episodes, and uses a byte-exact oracle to check every cited violation. The proposer behaves as expected on real violations and abstains on featureless legal input. Yet when the legal input contains a harmless pattern resembling a familiar game rule, it invents a failure: in 15/60 runs, versus 0/60 on featureless input, it enables the nonexistent-rule guardrail and cites a violation the oracle refutes. The effect is structured, not indiscriminate. In single-shot proposals it appears only when three conditions coincide: a rule-shaped pattern, an open-ended rule set and an instruction that presupposes failures. Removing any of these conditions eliminates the fabrication. Because the invented guardrail changes no true outcome and cannot improve an already-perfect suppression score, the phenomenon is neither reward hacking nor over-refusal. It is a phantom guardrail: a fix for a failure that never happened, invisible to suppression-only acceptance. Inside an add-only accept loop it re-enters even without the failure-presupposing instruction, the loop's keep-adding role supplying the demand the instruction supplied in single shot, and once in it stays. We present the Counterfactual Fabrication Lab for measuring fabricated failures in self-improving agent harnesses.

Problem

• phantom guardrails
• self-improving agents
• hallucinated failures
• counterfactual fabrication
• guardrail optimization

Innovation

• phantom guardrails
• self-improving agents
• counterfactual fabrication
• hallucinated failures
• harness optimization