SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling

2026-05-27

AI Summary

Existing downstream scaling methods are hindered by benchmark-specific biases and the inability of in-distribution validation loss to reliably reflect genuine improvements in model capabilities. This work proposes SuperValid, a novel framework that establishes, for the first time, a capability-level validation mechanism. SuperValid constructs out-of-distribution (OOD) validation sets by abstracting core competencies across multiple benchmarks and synthesizing them through knowledge-intensive text generation coupled with capability-aligned data distillation. Evaluated across six capability domains and seventeen benchmarks, the framework demonstrates that its validation loss exhibits strong correlation with downstream performance. Crucially, it enables stable performance prediction across diverse architectures, model scales, and training distributions, facilitating effective model selection, early stopping, and scaling decisions—all without requiring additional training.

Abstract

Scaling laws guide large language model training by relating compute to cross-entropy loss, and recent work further extends them to predict downstream benchmark performance. However, prior approaches face generalization limitations from two aspects: focusing on benchmark-level performance introduces scenario-specific artifacts, while relying on IID validation loss fails to track capability improvements when training distributions vary. In this work, we argue that downstream scaling should be studied at the capability level, which captures shared skill factors across related tasks while abstracting away benchmark-specific noise. We propose SuperValid, a framework that synthesizes OOD (out-of-distribution), capability-aligned validation data by distilling core concepts from benchmarks within a capability domain and expanding them into diverse, knowledge-rich texts. Extensive experiments spanning 17 benchmarks grouped into 6 capability domains show that SuperValid loss exhibits strong and stable correlation with downstream performance across models of different architectures, scales, and training data distributions. As a training-free metric computable during training without benchmark evaluation, SuperValid enables effective model selection, early stopping, and scaling decisions.

Problem

• scaling laws
• out-of-distribution validation
• downstream generalization
• capability alignment
• benchmark artifacts

Innovation

• capability-aligned validation
• out-of-distribution (OOD)
• scaling laws
• SuperValid
• downstream generalization