Learning Shrinks the Hard Tail: Training-Dependent Inference Scaling in a Solvable Linear Model

2026-01-07

arXiv.org

AI Summary

This work investigates the scaling relationship between training and inference under instance difficulty heterogeneity, particularly when difficulties follow a heavy-tailed distribution. By constructing an analytically tractable latent instance difficulty (LID) model, the study establishes the first theoretical link between training scale and the exponential decay rate of failure probability in multi-attempt inference (pass@$k$). The key insight is the “learning-induced compression of the hard tail” mechanism: as training data increases, the tail of the generalization error distribution corresponding to difficult instances is effectively compressed, thereby steepening the pass@$k$ curve. The analysis integrates linear models, heavy-tailed probabilistic modeling, and generalization error theory, and is validated on both synthetic benchmarks and real-world tasks including CIFAR-10H and mathematical reasoning distillation, offering principled guidelines for optimizing the allocation of computational resources between training and inference.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We analyze neural scaling laws in a solvable model of last-layer fine-tuning where targets have intrinsic, instance-heterogeneous difficulty. In our Latent Instance Difficulty (LID) model, each input's target variance is governed by a latent ``precision''drawn from a heavy-tailed distribution. While generalization loss recovers standard scaling laws, our main contribution connects this to inference. The pass@$k$ failure rate exhibits a power-law decay, $k^{-\beta_\text{eff}}$, but the observed exponent $\beta_\text{eff}$ is training-dependent. It grows with sample size $N$ before saturating at an intrinsic limit $\beta$ set by the difficulty distribution's tail. This coupling reveals that learning shrinks the ``hard tail''of the error distribution: improvements in the model's generalization error steepen the pass@$k$ curve until irreducible target variance dominates. The LID model yields testable, closed-form predictions for this behavior, including a compute-allocation rule that favors training before saturation and inference attempts after. We validate these predictions in simulations and in two real-data proxies: CIFAR-10H (human-label variance) and a maths teacher-student distillation task.

Problem

• scaling laws
• instance difficulty
• pass@k
• heavy-tailed distribution
• generalization error

Innovation

• scaling laws
• instance difficulty
• pass@k
• heavy-tailed distribution
• inference scaling