Kinetics: Rethinking Test-Time Scaling Laws

2025-06-05

AI Summary

Existing test-time scaling laws overestimate small-model performance by ignoring memory-access bottlenecks induced by inference strategies (e.g., Best-of-N, chain-of-thought). Method: We propose Kinetics—a unified scaling law integrating both computational and memory-access costs—centered on sparse attention to establish a new test-time scaling paradigm. It incorporates cross-scale empirical validation (0.6B–32B), joint memory-computation modeling, MoE adaptation, and long-sequence generation optimization. Contribution/Results: Kinetics reveals that attention computation—not parameter count—dominates test-time cost and identifies a critical model-size threshold: beyond it, test-time scaling becomes significantly more efficient. On AIME, sparse-attention models outperform dense baselines by >60 points (low-cost regime) and +5+ points (high-cost regime), with accuracy monotonically increasing—and remaining unsaturated—under increased test-time compute.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We rethink test-time scaling laws from a practical efficiency perspective, revealing that the effectiveness of smaller models is significantly overestimated. Prior work, grounded in compute-optimality, overlooks critical memory access bottlenecks introduced by inference-time strategies (e.g., Best-of-$N$, long CoTs). Our holistic analysis, spanning models from 0.6B to 32B parameters, reveals a new Kinetics Scaling Law that better guides resource allocation by incorporating both computation and memory access costs. Kinetics Scaling Law suggests that test-time compute is more effective when used on models above a threshold than smaller ones. A key reason is that in TTS, attention, rather than parameter count, emerges as the dominant cost factor. Motivated by this, we propose a new scaling paradigm centered on sparse attention, which lowers per-token cost and enables longer generations and more parallel samples within the same resource budget. Empirically, we show that sparse attention models consistently outperform dense counterparts, achieving over 60 points gains in low-cost regimes and over 5 points gains in high-cost regimes for problem-solving accuracy on AIME, encompassing evaluations on state-of-the-art MoEs. These results suggest that sparse attention is essential and increasingly important with more computing invested, for realizing the full potential of test-time scaling where, unlike training, accuracy has yet to saturate as a function of computation, and continues to improve through increased generation. The code is available at https://github.com/Infini-AI-Lab/Kinetics.

Problem

• Rethinking test-time scaling laws for practical efficiency
• Addressing memory access bottlenecks in inference-time strategies
• Proposing sparse attention to optimize resource allocation

Innovation

• Introduces Kinetics Scaling Law for resource allocation
• Proposes sparse attention to reduce per-token cost
• Shows sparse models outperform dense ones significantly