Where vs What: Decomposing Structural and Content Failures in LLM-Generated Structured Outputs

2026-08-26

AI Summary

"This study addresses the challenge of evaluating structured outputs from large language models, where positional and value errors are intertwined. It introduces the Structure-Content Decomposition (SCD) framework to independently measure structural fidelity and content accuracy. Based on SCD, the SA-RLVR method is proposed, which converts SCD metrics into verifiable rewards for reinforcement learning. This work distinguishes between positional and value errors for the first time, revealing that structural fidelity degrades more rapidly as complexity increases. The project employs SCD, GPO, and reinforcement learning, achieving a significant improvement in value placement accuracy from 26% to 63% on JSON and table tasks, demonstrating strong generalization across different domains."

Abstract

Structured outputs such as JSON and tables are central to modern LLM-based systems, yet generation failures are evaluated monolithically, conflating two distinct error modes: placement errors (correct values at wrong positions) and value errors (wrong values at intended positions). We introduce Structure-Content Decomposition (SCD), a framework that independently measures structural fidelity and content accuracy. Applying SCD to nested JSON and table tasks across six models (7B to frontier), we uncover a consistent phenomenon: structural fidelity degrades earlier and more sharply than content accuracy as complexity increases. At the highest complexity, even DeepSeek-V4-Flash (with reasoning) misplaces 35% of recalled values, while Qwen2.5-7B misplaces 74%. Controlled ablations suggest that this pattern is associated with reliance on semantic shortcuts rather than topological understanding of output structure. Based on these findings, we propose SA-RLVR, converting SCD metrics into verifiable rewards for reinforcement learning via GRPO. SA-RLVR successfully optimizes structural addressing across distinct topologies: it lifts JSON Value Placement Accuracy (VPA) from 26% to 63% while generalizing to held-out schemas; moreover, it consistently drives VPA improvements in the table domain, demonstrating that structure-aware rewards can directly enhance multi-domain structural positioning.

Problem

• Structured Outputs
• Placement Errors
• Value Errors
• Structural Fidelity
• Content Accuracy

Innovation

• Structure-Content Decomposition
• Structural Fidelity
• Content Accuracy
• SA-RLVR
• Reinforcement Learning