Efficient Large Foundation Models Design: A Perspective From Model and System Co-Design

2024-09-03

AI Summary

Large language models (LLMs) suffer from prohibitively high computational costs, resource consumption, and deployment expenses in both training and inference. To address this, this work proposes the first holistic model-system co-design paradigm for efficient LLMs, unifying key optimization avenues—including sparsification, quantization, attention mechanism optimization, memory-aware scheduling, compiler-level acceleration, and hardware adaptation—into a coherent framework. Through cross-layer joint optimization, we establish a comprehensive, full-stack technical taxonomy spanning training and inference, and publicly release a structured, open-source knowledge base. Beyond systematically categorizing and evaluating state-of-the-art efficient LLM techniques, our contribution provides a reusable methodology framework and practical implementation guidelines. This significantly improves model efficiency, affordability, and accessibility, establishing a standardized research infrastructure for both academia and industry.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This paper focuses on modern efficient training and inference technologies on foundation models and illustrates them from two perspectives: model and system design. Model and System Design optimize LLM training and inference from different aspects to save computational resources, making LLMs more efficient, affordable, and more accessible. The paper list repository is available at url{https://github.com/NoakLiu/Efficient-Foundation-Models-Survey}

Problem

• Efficient Large Models
• Simplified Modeling
• Resource-saving Training

Innovation

• Efficient Method
• Large Language Model
• Resource Saving