A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness

2024-11-04

arXiv.org

AI Summary

To address the ambiguity in defining small language models (SLMs), the lack of standardized evaluation criteria, and unclear deployment pathways in the era of large language models (LLMs), this work proposes a novel SLM definition framework centered on *task specialization* and *resource-constrained adaptability*. We introduce the first unified taxonomy spanning definition, acquisition, enhancement, application, and trustworthiness, and establish boundary criteria based on *minimum emergent capability* and *maximum resource tolerance*. Furthermore, we design a multi-dimensional general-purpose enhancement framework and an LLM-SLM co-processing architecture. Leveraging systematic literature review, taxonomic modeling, cross-model benchmarking, and integrated trustworthy AI assessment, we deliver the first structured, extensible research landscape for SLMs—providing a theoretical foundation, technical roadmap, and practical guidelines for lightweight AI, thereby enabling edge intelligence and domain-specific AI deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) have demonstrated emergent abilities in text generation, question answering, and reasoning, facilitating various tasks and domains. Despite their proficiency in various tasks, LLMs like PaLM 540B and Llama-3.1 405B face limitations due to large parameter sizes and computational demands, often requiring cloud API use which raises privacy concerns, limits real-time applications on edge devices, and increases fine-tuning costs. Additionally, LLMs often underperform in specialized domains such as healthcare and law due to insufficient domain-specific knowledge, necessitating specialized models. Therefore, Small Language Models (SLMs) are increasingly favored for their low inference latency, cost-effectiveness, efficient development, and easy customization and adaptability. These models are particularly well-suited for resource-limited environments and domain knowledge acquisition, addressing LLMs' challenges and proving ideal for applications that require localized data handling for privacy, minimal inference latency for efficiency, and domain knowledge acquisition through lightweight fine-tuning. The rising demand for SLMs has spurred extensive research and development. However, a comprehensive survey investigating issues related to the definition, acquisition, application, enhancement, and reliability of SLM remains lacking, prompting us to conduct a detailed survey on these topics. The definition of SLMs varies widely, thus to standardize, we propose defining SLMs by their capability to perform specialized tasks and suitability for resource-constrained settings, setting boundaries based on the minimal size for emergent abilities and the maximum size sustainable under resource constraints. For other aspects, we provide a taxonomy of relevant models/methods and develop general frameworks for each category to enhance and utilize SLMs effectively.

Problem

• Small Language Models
• Resource-constrained Environments
• Specialized Domains

Innovation

• Small Language Models
• Resource Adaptability
• Privacy Protection