Attributing Structured-Output Gains in Function Calling: Interface Alignment versus Procedural Transfer

2026-07-01

AI Summary

Existing structured output benchmarks struggle to disentangle whether performance gains in function calling stem from interface alignment or genuine procedural skill transfer. This work proposes a four-tier attribution protocol—comprising standardized rescored evaluation, format-controlled baselines, repair/balancing induction strategies, and cross-task portability tests—to systematically isolate these contributions. Our analysis reveals, for the first time, that most observed improvements are attributable to mere format compliance rather than true skill transfer; on benchmarks such as BFCL, format-only prompting matches the performance of full skill prompting, and state-of-the-art advantages largely vanish after applying repair induction. To promote rigor in structured output evaluation, we concurrently release the BFCL-CANONICAL benchmark and its associated evaluation protocol.

Abstract

Structured-output benchmarks reward both task decisions and interface compliance, so prompt-induced function-calling gains require attribution before they can be interpreted as transferable skill. We introduce a four-layer gain-attribution protocol for prompt-prepended skill injection, combining canonicalized rescoring, format-only controls, repaired/balanced induction, and portability checks. Applied to the Berkeley Function Calling Leaderboard (BFCL) and scoped with API-Bank, MATH-500, and MultiHop-RAG, the protocol shows that several apparent gains are better attributed to interface alignment than to procedural transfer: format-only prompts match or exceed full skills in key BFCL cells, repaired/balanced induction removes the largest sub-frontier gains, and API-Bank target-native gains are matched within 0.5 percentage points (pp) by length-matched generic procedural prompts. These findings treat format compliance as a useful engineering capability while clarifying what a structured-output score certifies. We release BFCL-CANONICAL and recommend canonicalized metrics, balanced induction, and format-only baselines for function-calling skill-gain attribution. Code and data are available at https://github.com/couragec/skill-injection-attribution.

Problem

• structured-output
• function calling
• skill transfer
• interface alignment
• gain attribution

Innovation

• gain attribution
• interface alignment
• procedural transfer
• structured-output evaluation
• function calling