FineScope : Precision Pruning for Domain-Specialized Large Language Models Using SAE-Guided Self-Data Cultivation

2025-05-01

AI Summary

Pruning domain-specific large language models (LLMs) often incurs substantial performance degradation on downstream tasks. Method: This paper proposes an SAE-guided precise pruning and self-data distillation framework. It introduces a novel sparse autoencoder (SAE)-driven domain data self-cultivation mechanism that tightly integrates interpretable feature learning, structured pruning, and post-pruning performance recovery. Domain-constrained optimization and lightweight fine-tuning are further incorporated to jointly preserve knowledge and enhance accuracy. Results: After pruning, fine-tuning the compressed model with SAE-synthesized data restores over 90% of its original performance. Moreover, fine-tuning the unpruned base model on the same synthetic data significantly improves domain-specific accuracy. The framework consistently outperforms state-of-the-art LLMs across multiple specialized tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Training large language models (LLMs) from scratch requires significant computational resources, driving interest in developing smaller, domain-specific LLMs that maintain both efficiency and strong task performance. Medium-sized models such as LLaMA, llama} have served as starting points for domain-specific adaptation, but they often suffer from accuracy degradation when tested on specialized datasets. We introduce FineScope, a framework for deriving compact, domain-optimized LLMs from larger pretrained models. FineScope leverages the Sparse Autoencoder (SAE) framework, inspired by its ability to produce interpretable feature representations, to extract domain-specific subsets from large datasets. We apply structured pruning with domain-specific constraints, ensuring that the resulting pruned models retain essential knowledge for the target domain. To further enhance performance, these pruned models undergo self-data distillation, leveraging SAE-curated datasets to restore key domain-specific information lost during pruning. Extensive experiments and ablation studies demonstrate that FineScope achieves highly competitive performance, outperforming several large-scale state-of-the-art LLMs in domain-specific tasks. Additionally, our results show that FineScope enables pruned models to regain a substantial portion of their original performance when fine-tuned with SAE-curated datasets. Furthermore, applying these datasets to fine-tune pretrained LLMs without pruning also improves their domain-specific accuracy, highlighting the robustness of our approach. The code will be released.

Problem

• Reducing computational costs by creating smaller, domain-specific LLMs
• Preventing accuracy loss in pruned models for specialized tasks
• Enhancing domain performance via SAE-guided data distillation

Innovation

• Uses SAE-guided self-data cultivation
• Applies structured pruning with constraints
• Enhances performance via self-data distillation