Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents

2026-07-05

AI Summary

This work addresses the often-overlooked influence of evaluation frameworks on software agent performance, demonstrating that even when tasks, environments, and base models remain unchanged, the framework itself can implicitly perturb an agent’s multi-step belief state and thereby affect its decision-making. The authors propose a novel belief-unfolding diagnostic tool that, for the first time, decomposes cross-framework belief discrepancies into immediate interface shifts and temporally accumulated belief evolution. They introduce a training-free Belief-Informed Weight Matching (BIWM) protocol to align belief observations across frameworks. Leveraging structured belief trajectory collection, shadow execution, repair expansion, and validation mask logging, experiments on programming tasks and stress tests reveal that operations such as action masking and compressed repair—while preserving final success rates—significantly distort intermediate beliefs and consequently impair subsequent decisions. These findings underscore that evaluation frameworks must be treated as critical experimental variables rather than mere implementation details.

Abstract

Software-agent benchmarks usually report whether an agent solves a task, but the agent reaches that outcome through a harness that controls what it sees, which actions it can take, which failures are repaired, which states are verified, and which evidence is logged. We show that this harness can change the agent's multi-step beliefs even when the task, environment, and base LLM are fixed. We introduce a belief-rollout diagnostic that elicits structured K-step trajectories over progress, risk, recoverability, constraints, failure mode, uncertainty, future success, repair cost, and next action under alternative harnesses. We define a cross-harness belief divergence and decompose it into an arrival term for immediate interface shifts and a growth term for horizon-dependent belief changes. On controlled coding tasks and public-benchmark stress tests, blocked actions, compressed repairs, selective verification, and cost-aware evidence pruning often preserve terminal success while changing the beliefs that drive later decisions. We further introduce BIWM, a no-training protocol that canonicalizes observations, logs censored branches, expands repair traces, records verification masks, executes risky branches in shadow, and aligns belief trajectories across harness views. The results suggest that harness design is an experimental variable in agent evaluation, not an implementation detail. Our code is available at https://github.com/Hik289/Harness-induce-bias.git.

Problem

• belief divergence
• multi-step LLM agents
• evaluation harness
• agent belief
• benchmarking bias

Innovation

• belief divergence
• multi-step LLM agents
• harness-induced bias
• belief-rollout diagnostic
• BIWM protocol