Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models

2024-01-01

arXiv.org

AI Summary

Large language models (LLMs) incur prohibitive computational, memory, energy, network, and financial costs, severely limiting their deployment and sustainable adoption in resource-constrained environments. To address this, we propose the first fine-grained, resource-type–aware efficiency optimization framework for LLMs, systematically modeling multi-resource interdependencies across the full lifecycle—architecture design, pretraining, fine-tuning, and inference deployment. We introduce a “resource–technique” mapping framework and a unified evaluation benchmark, including an open-source dataset. Our approach innovatively integrates model compression, sparsification, quantization, knowledge distillation, efficient attention mechanisms, hardware-aware compilation, and green training techniques, yielding a comprehensive technology landscape spanning five resource dimensions and four development stages. The resulting framework provides standardized evaluation tools and an open research paradigm to advance efficient, sustainable LLM development and deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The burgeoning field of Large Language Models (LLMs), exemplified by sophisticated models like OpenAI's ChatGPT, represents a significant advancement in artificial intelligence. These models, however, bring forth substantial challenges in the high consumption of computational, memory, energy, and financial resources, especially in environments with limited resource capabilities. This survey aims to systematically address these challenges by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs. We categorize methods based on their optimization focus: computational, memory, energy, financial, and network resources and their applicability across various stages of an LLM's lifecycle, including architecture design, pretraining, finetuning, and system design. Additionally, the survey introduces a nuanced categorization of resource efficiency techniques by their specific resource types, which uncovers the intricate relationships and mappings between various resources and corresponding optimization techniques. A standardized set of evaluation metrics and datasets is also presented to facilitate consistent and fair comparisons across different models and techniques. By offering a comprehensive overview of the current sota and identifying open research avenues, this survey serves as a foundational reference for researchers and practitioners, aiding them in developing more sustainable and efficient LLMs in a rapidly evolving landscape.

Problem

• Resource Constraints
• Large Language Models
• Sustainability

Innovation

• Resource-efficient Models
• Standardized Evaluation
• Sustainable AI