GeLaCo: An Evolutionary Approach to Layer Compression

2025-07-14

AI Summary

Deploying large language models (LLMs) incurs substantial computational overhead, and existing structured pruning methods rely on time-consuming empirical search, often failing to identify globally optimal compression configurations. Method: This paper proposes a layer-wise compression framework based on multi-objective evolutionary optimization. It is the first to explicitly construct the Pareto frontier between compression ratio and model quality, integrating population-based evolution, layer-folding strategies, and modular similarity metrics—spanning attention mechanisms, feed-forward networks, and hidden states—to enable efficient, interpretable, and fully automated compression. Contribution/Results: The framework outperforms state-of-the-art methods on both base and instruction-tuned LLMs. Perplexity and generation capability evaluations demonstrate that it maintains high performance while achieving significant parameter reduction, validating its effectiveness and generalizability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Evolutionary Computation

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large Language Models (LLM) have achieved remarkable performance across a large number of tasks, but face critical deployment and usage barriers due to substantial computational requirements. Model compression methods, which aim to reduce model size while preserving its capacity, are an important means to mitigate these issues. Promising approaches along these lines, such as structured pruning, typically require costly empirical search for optimal variants and may run the risk of ignoring better solutions. In this work we introduce GeLaCo, an evolutionary approach to LLM compression via layer collapse. Our approach supports an efficient exploration of the compression solution space via population-based search and a module-wise similarity fitness function capturing attention, feed-forward, and hidden state representations. GeLaCo also supports both single and multi-objective evolutionary compression search, establishing the first Pareto frontier along compression and quality axes. We evaluate GeLaCo solutions via both perplexity-based and generative evaluations over foundational and instruction-tuned models, outperforming state-of-the-art alternatives.

Problem

• Reducing LLM computational requirements via compression
• Efficiently exploring optimal compression solutions evolutionarily
• Balancing model quality and compression trade-offs

Innovation

• Evolutionary approach for LLM layer compression
• Population-based search with similarity fitness
• Pareto frontier for compression and quality