Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI

2024-09-21

arXiv.org

AI Summary

This paper critically examines the prevailing “bigger-is-better” paradigm in AI, identifying its weak scientific foundations, environmental unsustainability, and exacerbation of power concentration and resource diversion from critical societal domains (e.g., health, education, climate). Methodologically, it is the first systematic deconstruction of the paradigm’s two core assumptions, introducing a multi-dimensional trade-off framework that integrates computational efficiency evaluation, resource consumption modeling, socio-technical systems analysis, and policy scenario simulation. Empirical findings reveal pronounced non-linear imbalance between compute scaling and performance gains, with diminishing marginal returns in large-scale models. The contributions include: (1) a quantifiable assessment framework jointly measuring scale, efficacy, fairness, and sustainability; (2) identification of structural risks inherent in unbounded scaling; and (3) an interdisciplinary methodology and concrete policy intervention pathways for responsible AI governance.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Humans and AI: Other Foundations of Human Computation & AI
• Machine Learning: Scalability of ML Systems

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

With the growing attention and investment in recent AI approaches such as large language models, the narrative that the larger the AI system the more valuable, powerful and interesting it is is increasingly seen as common sense. But what is this assumption based on, and how are we measuring value, power, and performance? And what are the collateral consequences of this race to ever-increasing scale? Here, we scrutinize the current scaling trends and trade-offs across multiple axes and refute two common assumptions underlying the 'bigger-is-better' AI paradigm: 1) that performance improvements are driven by increased scale, and 2) that all interesting problems addressed by AI require large-scale models. Rather, we argue that this approach is not only fragile scientifically, but comes with undesirable consequences. First, it is not sustainable, as, despite efficiency improvements, its compute demands increase faster than model performance, leading to unreasonable economic requirements and a disproportionate environmental footprint. Second, it implies focusing on certain problems at the expense of others, leaving aside important applications, e.g. health, education, or the climate. Finally, it exacerbates a concentration of power, which centralizes decision-making in the hands of a few actors while threatening to disempower others in the context of shaping both AI research and its applications throughout society.

Problem

• Challenges the assumption that larger AI systems are inherently more valuable.
• Examines sustainability and environmental impact of scaling AI models.
• Highlights power concentration and neglected applications in AI research.

Innovation

• Challenges 'bigger-is-better' AI paradigm assumptions
• Highlights unsustainable compute demands and environmental impact
• Advocates for diverse AI applications beyond large-scale models