Farseer: A Refined Scaling Law in Large Language Models

2025-06-12

AI Summary

Large language model (LLM) training suffers from a “scaling gap”: insights from small-scale experiments fail to reliably guide large-scale deployments. Method: We propose Farseer, a high-accuracy, strongly generalizable scaling law that explicitly models the loss surface as ( L(N, D) ), enabling high-fidelity prediction and extrapolation across compute scales. Our approach integrates systematic surface modeling, multi-scale empirical fitting, and large-scale controlled-variable training—spanning ~1,000 LLMs trained on an H100 cluster for a total of 3 million GPU-hours. Contribution/Results: Farseer achieves the first reliable transfer of ablation conclusions from small-scale experiments to thousand-GPU training. It reduces extrapolation error by 433% relative to Chinchilla and uncovers fine-grained computational allocation principles in modern LLMs. All models, datasets, and training logs are publicly released to support reproducible scaling research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Training Large Language Models (LLMs) is prohibitively expensive, creating a critical scaling gap where insights from small-scale experiments often fail to transfer to resource-intensive production systems, thereby hindering efficient innovation. To bridge this, we introduce Farseer, a novel and refined scaling law offering enhanced predictive accuracy across scales. By systematically constructing a model loss surface $L(N,D)$, Farseer achieves a significantly better fit to empirical data than prior laws (e.g., Chinchilla's law). Our methodology yields accurate, robust, and highly generalizable predictions, demonstrating excellent extrapolation capabilities, improving upon Chinchilla's law by reducing extrapolation error by 433%. This allows for the reliable evaluation of competing training strategies across all $(N,D)$ settings, enabling conclusions from small-scale ablation studies to be confidently extrapolated to predict large-scale performance. Furthermore, Farseer provides new insights into optimal compute allocation, better reflecting the nuanced demands of modern LLM training. To validate our approach, we trained an extensive suite of approximately 1,000 LLMs across diverse scales and configurations, consuming roughly 3 million NVIDIA H100 GPU hours. We are comprehensively open-sourcing all models, data, results, and logs at https://github.com/Farseer-Scaling-Law/Farseer to foster further research.

Problem

• High cost of training large language models
• Inaccurate scaling predictions from small experiments
• Need for optimal compute allocation strategies

Innovation

• Refined scaling law for LLMs
• Better fit to empirical data
• Reduced extrapolation error significantly