Hard or Just Unreached? Diagnosing the Sampling Blind Spot in Math-Reasoning Difficulty Estimation

2026-06-17

AI Summary

Current pass@k-based evaluations of mathematical reasoning struggle to disentangle intrinsic problem difficulty from limitations inherent in sampling strategies, particularly exhibiting blind spots among samples with pass@k = 0. This work proposes activation grafting—a method combining deterministic decoding with residual stream perturbations—to reveal, for the first time, that a substantial subset of these ostensibly unsolvable instances harbors structurally recoverable solutions. Through Jaccard similarity analysis and cross-model, cross-dataset validation across eight test subsets of GSM8K and MATH, the approach successfully recovers 10.3%–22.9% of previously deemed unsolvable problems, significantly outperforming greedy decoding (≤6%). These findings demonstrate that language models possess latent problem-solving capabilities often obscured by conventional sampling methods.

Abstract

Math and science reasoning benchmarks rely on pass@k, the fraction of sampled chains that reach gold, as the canonical per-example difficulty signal. The same signal drives RL with verifiable rewards, math data curation, synthetic curricula, and verifier training. We show this proxy has a persistent blind spot on its hardest stratum: on the eight free-form math cells we test (GSM8K and MATH across four open-weight models), 10.3-22.9% of the examples that no sampling seed solves in six tries are instead solved at matched compute by a six-chain deterministic regime. These are greedy decoding plus five cheap residual-stream perturbations applied via activation grafting, while greedy alone solves at most 6% on these math cells. Recovery scales with the additional budget, across perturbations whose mechanistic distinctness we verify across all twelve cells (cross-kind fix-set Jaccard <= 0.47 in every setup). Activation grafting is used as an intervention on internal representations, not a decoding method; we use it purely as a diagnostic and diversification tool, and our recovered items show that the pass@k= 0 % stratum is structurally identifiable in the residual stream rather than that the unmodified model reaches them under ordinary inference.

Problem

• sampling blind spot
• math reasoning difficulty
• pass@k
• activation grafting
• residual stream

Innovation

• activation grafting
• sampling blind spot
• math reasoning difficulty
• residual-stream perturbation
• pass@k