Where meaning lives: Layer-wise accessibility of psycholinguistic features in encoder and decoder language models
This study investigates where psycholinguistic meaning is encoded in Transformer language models and how this encoding depends on embedding extraction methods and model architectures. Through systematic layer-wise probing across ten Transformer models, we assess the accessibility of 58 psycholinguistic features under three embedding extraction approaches: linear probing, contextualized embeddings, and isolated embeddings. Our findings reveal that the location of meaningful representations is highly sensitive to the extraction method. Despite architectural differences between encoders and decoders, both exhibit a consistent depth-wise ordering of semantic dimensions: lexical properties peak in shallow layers, while experiential and affective dimensions peak in deeper layers. Contextualized embeddings substantially enhance feature selectivity. Notably, final-layer representations are often suboptimal, suggesting a universal hierarchical organization of semantic information in these models.