The rising costs of training frontier AI models

2024-05-31

arXiv.org

AI Summary

The lack of transparent, quantified data on training costs for state-of-the-art AI models hinders sustainability assessment and policy formulation. Method: This paper introduces the first systematic, multi-dimensional cost estimation framework, incorporating hardware, energy, cloud leasing, and personnel expenses. It employs three complementary validation techniques: statistical confidence interval analysis, power-and-interconnect modeling, and cloud pricing reverse-engineering. Contribution/Results: The study reveals that AI model training costs grew at a compound annual growth rate of 2.4× (90% CI: 2.0×–2.9×) from 2016 to 2024, with chips and personnel constituting the largest cost components—each reaching tens of millions of USD. Training costs for GPT-4- and Gemini-class models have entered the hundred-million-dollar range, and single-training expenditures are projected to exceed $1 billion by 2027. These findings establish a critical quantitative benchmark for evaluating the economic sustainability of AI advancement and informing evidence-based policymaking.

Technology Category

• Humans and AI: Other Foundations of Human Computation & AI
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

The costs of training frontier AI models have grown dramatically in recent years, but there is limited public data on the magnitude and growth of these expenses. This paper develops a detailed cost model to address this gap, estimating training costs using three approaches that account for hardware, energy, cloud rental, and staff expenses. The analysis reveals that the amortized cost to train the most compute-intensive models has grown precipitously at a rate of 2.4x per year since 2016 (90% CI: 2.0x to 2.9x). For key frontier models, such as GPT-4 and Gemini, the most significant expenses are AI accelerator chips and staff costs, each costing tens of millions of dollars. Other notable costs include server components (15-22%), cluster-level interconnect (9-13%), and energy consumption (2-6%). If the trend of growing development costs continues, the largest training runs will cost more than a billion dollars by 2027, meaning that only the most well-funded organizations will be able to finance frontier AI models.

Problem

• Modeling rising AI training costs
• Estimating hardware and staff expenses
• Predicting billion-dollar training by 2027

Innovation

• Detailed cost model development
• Three estimation approaches utilized
• Focus on AI accelerator chips