Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents

2026-08-12

AI Summary

This work addresses the limitations of existing code agent evaluations, which often conflate genuine instruction following with default behavioral tendencies and overlook the impact of instruction distribution across diverse input interfaces. To this end, we introduce Harness-IF, a benchmark comprising 60 multi-turn real-world coding tasks and 642 fine-grained rules, enabling systematic assessment of agent compliance across five configurable instruction interfaces. We further propose Against-Prior Accuracy (AP-Acc), a novel metric that isolates true instruction adherence by discounting performance attributable to default behaviors. Empirical results reveal that conventional accuracy (72.1–85.9%) substantially overestimates actual instruction-following capability, with AP-Acc dropping to 66.1–78.6%. All models exhibit an average performance decline of 5.81 points on anti-prior rules, and instruction priority analysis demonstrates that system prompts, project files, and user instructions contribute more significantly to compliance than tool or skill descriptions.

Abstract

When a coding agent obeys a rule, it may simply have been going to do that anyway. Existing instruction-following benchmarks cannot tell the difference: they concentrate rules in the user turn, while coding-agent benchmarks emphasize final task success. We introduce Harness-IF, which scores operational rules one at a time from execution evidence: 60 realistic multi-turn coding items drawn from a 642-rule library, 256 rules receiving verdicts, placed on the five configurable surfaces a deployed agent reads. To separate compliance from coincidence we introduce Against-Prior Accuracy (AP-Acc), which scores only rules labeled as opposing unprompted defaults, observed by re-running tasks with the rule withheld across nine probe builds and curated otherwise. Across 12 frontier models, accuracy spans 72.1-85.9% and AP-Acc 66.1-78.6%; every model is worse on against-prior rules, by 3.6 to 7.4 points (mean 5.81), and the direction survives a common-support analysis with item-clustered intervals. Aggregate scores therefore overstate compliance by a model-specific margin: prior control leaves the top build unchanged and exchanges three adjacent rank pairs. A counterbalanced conflict pilot on nine separate builds adds a second result: pooled precedence does not follow prompt depth, with system prompts, project files, and user instructions ahead of tool and skill descriptions.

Problem

• instruction following
• coding agents
• benchmarking
• rule compliance
• against-prior accuracy

Innovation

• instruction following
• code agents
• against-prior accuracy
• instruction surfaces
• benchmarking