Target-Oriented Pretraining Data Selection via Neuron-Activated Graph

2026-04-17

AI Summary

This work addresses the challenge of efficiently selecting pretraining data highly relevant to a target task to enhance language model performance. It proposes a training-free, interpretable data selection framework that constructs cross-layer Neuron Activation Graphs (NAGs) based on sparse, high-impact neurons—capturing the functional skeleton supporting the target task—and ranks candidate data by NAG similarity. Evaluated across six benchmarks, the method achieves an average performance gain of 4.9%, surpassing the state of the art on HellaSwag by 5.3%. In multi-task settings, it outperforms baselines by 1.1% and 4.1%, respectively. Ablation studies further demonstrate its interpretability: deactivating only 0.12% of critical neurons leads to a 23.5% performance drop, confirming the centrality of the identified functional pathways.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Mixture of Experts (MoE)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Everyday tasks come with a target, and pretraining models around this target is what turns them into experts. In this paper, we study target-oriented language model (LM) pretraining by introducing Neuron-Activated Graph Ranking (NAG-based Ranking), a training-free and interpretable framework for target pretraining data selection. Rather than using black-box representations, our approach directly characterizes each target input by a sparse set of high-impact neurons in any off-the-shelf LLMs. Concretely, we quantify neuron impact and select the most influential neurons across layers into a compact Neuron-Activated Graph (NAG), and rank candidate data by NAG similarity to target examples. We conduct experiments across six benchmarks, where our NAG-based Ranking improves target-oriented pretraining by 4.9% on average over random sampling, and also outperforms state-of-the-art baselines by 5.3% accuracy on HellaSwag. It also remains effective under a more applicable multi-target setting, where our best setup surpasses two baselines by 1.1% and 4.1%, respectively. Furthermore, we provide a comprehensive analysis on why and how our NAG works, e.g., deactivating NAG-selected neurons (only 0.12% of all) causes a 23.5% performance collapse, and restricting NAG to the final layer incurs a 4.1% average drop, indicating that NAG captures a sparse "functional backbone" for learning target features. We release the code at https://github.com/asillycat/NAG.

Problem

• target-oriented pretraining
• data selection
• language model
• neuron activation
• pretraining data

Innovation

• Neuron-Activated Graph
• Target-Oriented Pretraining
• Data Selection
• Interpretable AI
• Sparse Neuron Activation