Where meaning lives: Layer-wise accessibility of psycholinguistic features in encoder and decoder language models

2026-01-07

arXiv.org

AI Summary

This study investigates where psycholinguistic meaning is encoded in Transformer language models and how this encoding depends on embedding extraction methods and model architectures. Through systematic layer-wise probing across ten Transformer models, we assess the accessibility of 58 psycholinguistic features under three embedding extraction approaches: linear probing, contextualized embeddings, and isolated embeddings. Our findings reveal that the location of meaningful representations is highly sensitive to the extraction method. Despite architectural differences between encoders and decoders, both exhibit a consistent depth-wise ordering of semantic dimensions: lexical properties peak in shallow layers, while experiential and affective dimensions peak in deeper layers. Contextualized embeddings substantially enhance feature selectivity. Notably, final-layer representations are often suboptimal, suggesting a universal hierarchical organization of semantic information in these models.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Representation Learning
• Cognitive Modeling & Cognitive Systems: Symbolic Representations

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• User Modeling, Personalization and Recommendation: Psychology-informed user models and recommender systems

Abstract

Understanding where transformer language models encode psychologically meaningful aspects of meaning is essential for both theory and practice. We conduct a systematic layer-wise probing study of 58 psycholinguistic features across 10 transformer models, spanning encoder-only and decoder-only architectures, and compare three embedding extraction methods. We find that apparent localization of meaning is strongly method-dependent: contextualized embeddings yield higher feature-specific selectivity and different layer-wise profiles than isolated embeddings. Across models and methods, final-layer representations are rarely optimal for recovering psycholinguistic information with linear probes. Despite these differences, models exhibit a shared depth ordering of meaning dimensions, with lexical properties peaking earlier and experiential and affective dimensions peaking later. Together, these results show that where meaning"lives"in transformer models reflects an interaction between methodological choices and architectural constraints.

Problem

• psycholinguistic features
• transformer models
• layer-wise probing
• meaning representation
• embedding extraction

Innovation

• layer-wise probing
• psycholinguistic features
• contextualized embeddings
• transformer architecture
• semantic localization