Breaking the Overscaling Curse: Thinking Parallelism Before Parallel Thinking

2026-01-29

AI Summary

This work addresses the “curse of over-parallelization” in large model inference, where uniformly high parallelism leads to redundant per-sample computational resources and suboptimal efficiency. The study formally characterizes and quantifies this phenomenon for the first time, proposing T2—a novel method that dynamically predicts the optimal parallelism for each sample based on its latent-space representation prior to decoding, thereby enabling on-demand resource allocation. T2 introduces a lightweight, proactive adaptation mechanism combined with multi-path sampling and aggregation strategies. This approach significantly reduces computational overhead while preserving inference quality, substantially improving sample efficiency in parallelized inference pipelines.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Hardware-aware ML
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Parallel thinking enhances LLM reasoning by multi-path sampling and aggregation. In system-level evaluations, a global parallelism level N is allocated to all samples, typically set large to maximize overall dataset accuracy. However, due to sample heterogeneity, some samples can achieve comparable performance with a smaller N'<N, causing budget redundancy. This incompatibility between system-level efficacy and sample-level efficiency constitutes the overscaling curse. In this paper, we formalize and quantify the overscaling curse, showing its universality and severity in practice, and analyze its trigger mechanism. We then propose a lightweight method, T2, to break the overscaling curse, which utilizes latent representations to estimate the optimal parallelism level for each sample before decoding. Experiments show that T2 significantly reduces cost while maintaining comparable performance, enabling more efficient parallel thinking.

Problem

• overscaling curse
• parallel thinking
• sample heterogeneity
• parallelism allocation
• computational efficiency

Innovation

• overscaling curse
• parallel thinking
• sample-level parallelism
• latent representation
• cost-efficient inference