Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

2026-07-14

AI Summary

Existing code-generation agents struggle to effectively integrate the results of external tool invocations into their reasoning due to pretraining architectures that support only left-to-right autoregressive code generation. This work proposes a function-aware Fill-in-the-Middle (FIM) intermediate training strategy that, for the first time, maps the agent’s “act–observe–continue” interaction loop onto function call sites. By leveraging program dependence graphs and a dual criterion balancing complexity and inferability, the method selects appropriate functions for masking and performs self-supervised training on a large-scale, decontaminated GitHub code corpus. The approach yields performance gains of 2.8–3.2 percentage points on SWE-Bench-Verified and 3.7–5.4 points on SWE-Bench-Lite, while maintaining or improving results on general-purpose coding benchmarks such as LiveCodeBench, tau-bench, and BFCL, thereby mitigating the degradation in both tool-use proficiency and general coding ability commonly induced by post-training.

Abstract

Coding agents must integrate external tool returns into ongoing reasoning - a capability that standard left-to-right pretraining on code exposes only in its forward direction. We observe that the action-observation-continuation loop of a coding agent is structurally isomorphic to a function call site, where a caller binds arguments, a callee returns a value computed elsewhere, and downstream code consumes that value. This conditioning structure exists at internet scale in ordinary code. We exploit it through function-aware fill-in-the-middle (FIM) mid-training: a self-supervised objective that masks functions selected via program dependency graph analysis and a complexity-inferability double criterion. We mid-train Qwen2.5-Coder-Instruct (7B/14B) and Qwen3-8B on a 2.6B-token decontaminated corpus drawn from 968 GitHub repositories, then apply existing agentic post-training pipelines. Mid-training improves SWE-Bench-Verified by +2.8/+3.0 at 7B/14B and by +3.2 on Qwen3-8B; SWE-Bench-Lite gains are +3.7/+4.0/+5.4 on the same models. The improvement holds across two post-training pipelines (R2E-Gym, SWE-Smith) and on a non-Qwen2.5 base (Qwen3-8B with SWE-Lego). Beyond in-domain gains, mid-training also mitigates the capability erosion that agentic post-training otherwise inflicts on non-agent coding (e.g., LiveCodeBench) and non-coding tool-use benchmarks (tau-bench, BFCL): although the mid-training corpus contains Python code only, the function-call inductive bias survives post-training and yields consistent gains.

Problem

• coding agents
• tool integration
• reasoning continuity
• function calls
• pretraining limitation

Innovation

• function-aware FIM
• mid-training
• program dependency graph
• agentic reasoning
• tool integration