EvoSLD: Automated Neural Scaling Law Discovery With Large Language Models

2025-07-27

AI Summary

This work addresses the reliance of neural network scaling laws on manual expertise and extensive empirical experimentation. We propose the first synergistic optimization framework integrating large language models (LLMs) with evolutionary algorithms. The method leverages LLMs to guide symbolic expression generation and optimization strategy design, combined with symbolic regression, grouped-data fitting error minimization, and multivariate response surface modeling, enabling automated discovery of interpretable scaling laws. Evaluated across five real-world scenarios, our approach reproduces or surpasses manually derived scaling laws. On test sets, it achieves a normalized mean squared error one order of magnitude lower than conventional symbolic regression. The framework significantly improves the accuracy, generalizability, and interpretability of discovered scaling relationships, advancing automated scientific discovery in deep learning.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Neuro-Symbolic Learning
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Scaling laws are fundamental mathematical relationships that predict how neural network performance evolves with changes in variables such as model size, dataset size, and computational resources. Traditionally, discovering these laws requires extensive human expertise and manual experimentation. We introduce EvoSLD, an automated framework for Scaling Law Discovery (SLD) that leverages evolutionary algorithms guided by Large Language Models (LLMs) to co-evolve symbolic expressions and their optimization routines. Formulated to handle scaling variables, control variables, and response metrics across diverse experimental settings, EvoSLD searches for parsimonious, universal functional forms that minimize fitting errors on grouped data subsets. Evaluated on five real-world scenarios from recent literature, EvoSLD rediscovers exact human-derived laws in two cases and surpasses them in others, achieving up to orders-of-magnitude reductions in normalized mean squared error on held-out test sets. Compared to baselines like symbolic regression and ablated variants, EvoSLD demonstrates superior accuracy, interpretability, and efficiency, highlighting its potential to accelerate AI research. Code is available at https://github.com/linhaowei1/SLD.

Problem

• Automate discovery of neural scaling laws using LLMs
• Reduce human effort in scaling law experimentation
• Improve accuracy and efficiency in AI performance prediction

Innovation

• Uses LLMs to guide evolutionary algorithms
• Automates scaling law discovery process
• Minimizes fitting errors with universal forms