On Relation-Specific Neurons in Large Language Models

2025-02-24

AI Summary

It remains unclear whether large language models (LLMs) possess neurons dedicated exclusively to abstract semantic relation representation—decoupled from specific entity instantiations. Method: Leveraging the Llama-2 family, we propose a statistics-driven neuron localization method, integrating selective ablation with multi-relation and cross-lingual comparative experiments. Contribution/Results: We provide the first empirical evidence of “relation-specific neurons.” These neurons exhibit three key properties: (1) cumulative activation—response magnitude increases monotonically with the number of relation instances; (2) strong generalization across distinct relations and languages; and (3) targeted ablation significantly impairs generation accuracy for the target relation while unexpectedly improving performance on non-target relations (positive transfer). This demonstrates a modular, relation-centric knowledge organization in LLMs, advancing interpretability research and enabling more controllable, fact-aware reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

In large language models (LLMs), certain neurons can store distinct pieces of knowledge learned during pretraining. While knowledge typically appears as a combination of relations and entities, it remains unclear whether some neurons focus on a relation itself -- independent of any entity. We hypothesize such neurons detect a relation in the input text and guide generation involving such a relation. To investigate this, we study the Llama-2 family on a chosen set of relations with a statistics-based method. Our experiments demonstrate the existence of relation-specific neurons. We measure the effect of selectively deactivating candidate neurons specific to relation $r$ on the LLM's ability to handle (1) facts whose relation is $r$ and (2) facts whose relation is a different relation $r' eq r$. With respect to their capacity for encoding relation information, we give evidence for the following three properties of relation-specific neurons. $ extbf{(i) Neuron cumulativity.}$ The neurons for $r$ present a cumulative effect so that deactivating a larger portion of them results in the degradation of more facts in $r$. $ extbf{(ii) Neuron versatility.}$ Neurons can be shared across multiple closely related as well as less related relations. Some relation neurons transfer across languages. $ extbf{(iii) Neuron interference.}$ Deactivating neurons specific to one relation can improve LLM generation performance for facts of other relations. We will make our code publicly available at https://github.com/cisnlp/relation-specific-neurons.

Problem

• Identify relation-specific neurons in LLMs.
• Measure effects of deactivating these neurons.
• Explore properties of relation-specific neurons.

Innovation

• Identifies relation-specific neurons in LLMs
• Uses neuron deactivation for relation analysis
• Demonstrates neuron cumulativity and versatility