🤖 AI Summary
Structured output often degrades reasoning performance, yet the underlying cause remains unclear. This work disentangles the effects of output format from prompt-length confounds through carefully designed natural language controls and a four-level complexity framework, evaluated across multiple models (Sonnet, Haiku, GPT-4o-mini, Opus) and five benchmarks, including MATH-Hard and AIME. The study introduces a “capacity competition” mechanism, demonstrating that performance loss stems not from the structured format itself but from insufficient residual model capacity: high-capacity models handle JSON output without degradation, whereas capacity-constrained models suffer substantial drops (Haiku ↓36.2 pp, GPT-4o-mini ↓28.0 pp). To mitigate this, the authors propose a “reason-then-format” strategy, which recovers 80–87% of lost accuracy, effectively alleviating the issue.
📝 Abstract
Prior work treats structured output as a reasoning tax, but this framing is incomplete: the cost of formatting depends strongly on a model's spare capacity. Using information-matched prose controls and a four-level schema complexity gradient, we separate format-specific effects from prompt-length confounds across 4 models and 5 benchmarks with 0% parse failures on successfully generated responses.
We find that structured formats are capacity-dependent. Models with sufficient headroom absorb JSON constraints without degradation (Sonnet: $88.7\pm4.0$% JSON vs. $89.3\pm1.7$% CoT on MATH-Hard). In contrast, formats severely degrade models operating near their limits through two distinct mechanisms. First, under standard token budgets, Haiku drops 36.2pp ($p < 0.0001$) largely due to truncation. Second, even with extended budgets eliminating truncation, GPT-4o-mini drops 28.0pp ($p < 0.001$), revealing pure capacity competition independent of token exhaustion.
This format penalty scales with schema complexity (McNemar $p < 0.0001$) and cannot be explained by prompt length alone. Furthermore, these results qualify claims of frontier model immunity: on AIME competition math, Opus 4.7 drops from 96.2% to 91.0% under JSON ($-5.3$pp; the displayed percentages are independently rounded, exact difference is $7/133 = 5.26$pp $\approx 5.3$pp). A delayed-structure ablation -- reasoning freely before formatting -- recovers most of the lost accuracy (3-run mean: 80--87%), supporting the capacity competition mechanism. The practical implication is not to avoid structured output, but to match it to capacity: when a model is near its limits, think first, format later.