What is the Role of Small Models in the LLM Era: A Survey

2024-09-10

arXiv.org

AI Summary

Despite the dominance of large language models (LLMs), small models (SMs) remain indispensable in resource-constrained settings due to their low computational cost and deployment flexibility—yet their structural role has been systematically undervalued. Method: This paper introduces the first “collaboration–competition” two-dimensional analytical framework to characterize the dynamic interplay between SMs and LLMs, substantiated through systematic literature review, multi-case empirical comparison, and open-source implementation (GitHub repository). Contribution/Results: The study rigorously delineates SMs’ applicability boundaries across model compression, edge deployment, and human-AI collaboration. It delivers an actionable tripartite guideline covering model selection strategies, lightweight optimization techniques, and deployment paradigms—challenging the misconception that SMs are merely degraded substitutes for LLMs and affirming their foundational complementary role within the AGI ecosystem.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Human-AI Collaboration / Human-AI Teaming

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Models (LLMs) have made significant progress in advancing artificial general intelligence (AGI), leading to the development of increasingly large models such as GPT-4 and LLaMA-405B. However, scaling up model sizes results in exponentially higher computational costs and energy consumption, making these models impractical for academic researchers and businesses with limited resources. At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated. This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior research. In this work, we systematically examine the relationship between LLMs and SMs from two key perspectives: Collaboration and Competition. We hope this survey provides valuable insights for practitioners, fostering a deeper understanding of the contribution of small models and promoting more efficient use of computational resources. The code is available at https://github.com/tigerchen52/role_of_small_models

Problem

• Examining the role of small models in the LLM era.
• Addressing high computational costs of large language models.
• Exploring collaboration and competition between LLMs and SMs.

Innovation

• Surveying collaboration between LLMs and SMs
• Analyzing competition dynamics of small models
• Promoting efficient computational resource usage