EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models

2025-02-10

AI Summary

To address high latency, elevated operational costs, and privacy risks arising from cloud dependency in edge deployment, this work proposes a pruning-aware pretraining paradigm that breaks conventional model scaling laws and deeply integrates large-model compression with pretraining. Methodologically, we introduce the first structured-pruning-driven automated neural architecture search, jointly optimizing pretraining objectives and sparsity constraints while incorporating parameter-group minimization. We also extend the SparseGPT/LLM-Pruner frameworks to the pretraining stage—marking the first such adaptation—thereby bridging the performance gap between compressed and directly pretrained models. Evaluated across 100M–1B parameter scales, our approach consistently outperforms state-of-the-art edge-oriented models—including MobileLLM, SmolLM, and Qwen2.5-0.5B—achieving significant gains on commonsense reasoning benchmarks. It supports architecture-agnostic, data-scalable compact model generation, and all code is fully open-sourced.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Modern large language models (LLMs) driven by scaling laws, achieve intelligence emergency in large model sizes. Recently, the increasing concerns about cloud costs, latency, and privacy make it an urgent requirement to develop compact edge language models. Distinguished from direct pretraining that bounded by the scaling law, this work proposes the pruning-aware pretraining, focusing on retaining performance of much larger optimized models. It features following characteristics: 1) Data-scalable: we introduce minimal parameter groups in LLM and continuously optimize structural pruning, extending post-training pruning methods like LLM-Pruner and SparseGPT into the pretraining phase. 2) Architecture-agnostic: the LLM architecture is auto-designed using saliency-driven pruning, which is the first time to exceed SoTA human-designed LLMs in modern pretraining. We reveal that it achieves top-quality edge language models, termed EfficientLLM, by scaling up LLM compression and extending its boundary. EfficientLLM significantly outperforms SoTA baselines with $100M sim 1B$ parameters, such as MobileLLM, SmolLM, Qwen2.5-0.5B, OLMo-1B, Llama3.2-1B in common sense benchmarks. As the first attempt, EfficientLLM bridges the performance gap between traditional LLM compression and direct pretraining methods, and we will fully open source at https://github.com/Xingrun-Xing2/EfficientLLM.

Problem

• Develops compact edge language models
• Introduces pruning-aware pretraining method
• Bridges performance gap in LLM compression

Innovation

• Pruning-aware pretraining for edge models
• Architecture-agnostic auto-designed LLMs
• Scaling up LLM compression boundaries