Decodable But Not Detachable: Training Data Granularity Determines Parametric Modularity in Large Language Models

2026-08-10

AI Summary

This study investigates whether domain-specific parameter modules exist in large language models and examines how the granularity of training data influences their emergence. Employing a unified causal analysis framework—integrating linear probeability, causal ablation matrices, neuron masking, and Intersection-over-Union (IoU) metrics—the authors systematically evaluate the necessity and selectivity of neurons across multiple models (1.5B–7B parameters) and eight task domains. The work establishes, for the first time, a causal link between modularity in training data and internal parameter modularity in models, demonstrating that highly selective parameter shells emerge only when training data exhibits token-level modularity. At the language and modality levels, 0.65%–1.14% of neurons show high selectivity; masking them causes significant performance drops in corresponding domains (e.g., 16–24 percentage points in mathematical reasoning), whereas no effective modules are observed at the subject level.

Abstract

Do large language models contain domain-specific parametric shells: concentrated, causally necessary neuron populations whose removal selectively degrades a target domain while sparing others? We apply a uniform causal methodology across two domain granularities, three model families (1.5B to 7B parameters), and eight domains. At the academic subject level, zero neurons exceed 60\% domain selectivity across 939,008 combined FFN neurons and causal damage matrices are flat, despite domain identity being linearly decodable above 85\% accuracy. At the language and modality level, 0.65--1.14\% of neurons exceed 60\% selectivity, damage matrices are near-perfectly diagonal (ratios up to 595:1), and shell neuron sets are essentially disjoint (IoU $< 0.003$). Masking code-selective neurons reduces mathematical reasoning accuracy by 16--24 percentage points across all models; masking Spanish or Chinese neurons leaves it at or below random. Shell strength increases monotonically with scale and shells are spatially interleaved in a pattern that precludes group-level selective quantization. Parametric shells form where and only where training data was modular at the token level.

Problem

• parametric modularity
• domain-specific neurons
• training data granularity
• causal selectivity
• large language models

Innovation

• parametric modularity
• causal neuron ablation
• training data granularity
• domain-selective neurons
• large language models