Parser States Already Know: Structure-Conditioned KV Persistence for Structured Generation

2026-08-28

AI Summary

"This study addresses the issue of single erroneous fields in structured data, such as JSON and SQL, generated by large language models, which can lead to downstream failures. It introduces PASK (Parser-Aware Structural KV Persistence), a method that transforms parser-exported structures into layer-specific key-value (KV) persistence decisions, thereby mitigating the mismatch between model-side KV sensitivity and task-level structural risks. PASK innovatively sets a minimum protection baseline based on task error sensitivity, allocating the remaining KV capacity through attention output distortion. This approach incorporates offline calibration for strategy formation and online lightweight structure checks, leveraging KV compression, parser state tracking, and attention mechanisms. On Qwen3-4B, PASK outperforms the strongest compression baseline by 17.39% across eight BFCL subcategories, achieving up to 2.2 times throughput and 3.3 times reduction in total processing time with only 0.53 times peak GPU memory usage compared to full KV."

Abstract

Structured generation underpins large language model (LLM) agents that produce JSON, SQL, and function calls, where a single wrong field can cause the downstream action to fail. Constrained decoding already tracks parser transitions to enforce formal validity, and these transitions expose how generated tokens participate in schema-critical decisions such as required fields, arguments, and structural boundaries under the active grammar. Existing KV compression largely leaves this task-relevant structural signal unused. We introduce PASK (Parser-Aware Structural KV Persistence), which turns parser-derived structure into layer-group-specific KV persistence decisions. PASK addresses the mismatch between model-side KV sensitivity and task-level structured risk by using task-error sensitivity to set minimum protection floors and attention-output distortion to allocate residual KV capacity. An offline calibration stage compiles these signals into a persistence policy, leaving only lightweight structure-conditioned lookup online. At a targe total KV budget of 0.33, PASK outperforms the strongest compressed baseline by 17.39 percentage points on average across eight BFCL non-live and Live subcategories on Qwen3-4B. In end-to-end serving, PASK achieves up to 2.2x higher throughput and 3.3x lower TPOT, while using 0.53x the peak GPU memory of Full KV.

Problem

• Structured Generation
• Parser Transitions
• KV Compression

Innovation

• PASK
• KV Persistence
• Structural Signal
• Task-Error Sensitivity
• Attention-Output Distortion