CODEBLOCK: Learning to Supervise Code at the Right Granularity

2026-06-10

AI Summary

This work addresses the limitations of conventional large code models that uniformly weight all response tokens during supervised fine-tuning, disregarding the varying learning utility of different code segments and risking syntactic or semantic integrity when applying natural language–style token-level filtering. To overcome this, the authors propose CodeBlock, a novel framework that introduces structure-aware sparse supervision into code fine-tuning for the first time. CodeBlock partitions code into syntactically coherent blocks based on program structure and evaluates their learning value using generalized cross-entropy, data-flow reachability, and bridging signals, applying supervision only to high-value blocks. This approach preserves program dependency integrity while dramatically improving training efficiency—achieving state-of-the-art pass@1 performance on six code generation benchmarks using merely 1.9% of the supervision tokens required by full-token fine-tuning and outperforming all existing baselines.

Abstract

Supervised fine-tuning of code LLMs typically applies uniform cross-entropy loss to all response tokens, implicitly assuming that every token provides equally useful learning signal. Recent token-level selection methods challenge this assumption in natural-language SFT by supervising only high-value tokens. However, directly transferring token-level masking to code can break syntactically and semantically coherent program units, because code depends on structural completeness and definition-use relations. We therefore propose CodeBlock, a structure-aware sparse supervision framework that selects structure-complete code evidence rather than isolated tokens. CodeBlock first selects high-quality instruction-response pairs, then partitions code responses into syntactically coherent coding items, estimates their utility by aggregating generalized cross-entropy over core logic tokens, and reranks them with data-flow reach and bridge signals to prioritize blocks that propagate or connect important program dependencies. During training, the full response remains available as context, while loss is applied only to selected code items and informative natural-language tokens. Experiments on six code-generation benchmarks show that CodeBlock achieves stronger average pass@1 than full-token SFT and competitive selection baselines, while using only 1.9% of supervised response tokens.

Problem

• code LLMs
• supervised fine-tuning
• token-level supervision
• code structure
• sparse supervision

Innovation

• structure-aware supervision
• code LLM fine-tuning
• sparse token selection
• syntactically coherent code blocks
• data-flow aware reranking