Scalable Complexity Control Facilitates Reasoning Ability of LLMs

2025-05-29

AI Summary

This study investigates the impact of model complexity control on the generalization and reasoning capabilities of large language models (LLMs). We address the instability of scaling laws arising from fixed initialization standard deviations in conventional approaches by proposing a novel complexity-control paradigm: replacing fixed initialization variance with a constant initialization rate, jointly optimized with weight decay coefficients. The method is broadly applicable, scalable, and implementation-friendly. We systematically evaluate it across model sizes up to 2.4B parameters and training data scales up to 1T tokens. Results demonstrate significant improvements in reasoning generalization performance and accelerated convergence of scaling laws along both model-size and data-scale dimensions. Crucially, this work is the first to establish the initialization rate—as a core complexity-control variable—as a decisive factor governing LLM scaling behavior.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The reasoning ability of large language models (LLMs) has been rapidly advancing in recent years, attracting interest in more fundamental approaches that can reliably enhance their generalizability. This work demonstrates that model complexity control, conveniently implementable by adjusting the initialization rate and weight decay coefficient, improves the scaling law of LLMs consistently over varying model sizes and data sizes. This gain is further illustrated by comparing the benchmark performance of 2.4B models pretrained on 1T tokens with different complexity hyperparameters. Instead of fixing the initialization std, we found that a constant initialization rate (the exponent of std) enables the scaling law to descend faster in both model and data sizes. These results indicate that complexity control is a promising direction for the continual advancement of LLMs.

Problem

• Enhancing LLMs' reasoning ability through scalable complexity control
• Improving scaling laws via initialization rate and weight decay adjustments
• Optimizing complexity hyperparameters for better model performance

Innovation

• Adjust initialization rate and weight decay
• Constant initialization rate improves scaling
• Complexity control enhances LLM generalizability