Modeling the language cortex with form-independent and enriched representations of sentence meaning reveals remarkable semantic abstractness

2025-09-27

AI Summary

This study investigates whether the language cortex encodes form-invariant, highly abstract sentence-level semantic representations—addressing the fundamental question of whether human language understanding depends on syntactic form. Method: We propose a novel neural response prediction model integrating multimodal (vision–language) embeddings, cross-image aggregation, sentence paraphrase integration, and implicit contextual enhancement. Crucially, we empirically evaluate abstraction via image generation tasks and context extension experiments. Contribution/Results: Our model achieves significantly improved fMRI response prediction accuracy—matching or surpassing large language models (LLMs) in several language-selective regions. Critically, it demonstrates stronger generalization across semantically equivalent but syntactically divergent constructions than state-of-the-art LLMs, providing direct neural evidence for abstract, syntax-agnostic semantic coding in the human language cortex. These findings reveal a distinctive advantage of human semantic representations over AI models in both richness and abstraction.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Language and Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The human language system represents both linguistic forms and meanings, but the abstractness of the meaning representations remains debated. Here, we searched for abstract representations of meaning in the language cortex by modeling neural responses to sentences using representations from vision and language models. When we generate images corresponding to sentences and extract vision model embeddings, we find that aggregating across multiple generated images yields increasingly accurate predictions of language cortex responses, sometimes rivaling large language models. Similarly, averaging embeddings across multiple paraphrases of a sentence improves prediction accuracy compared to any single paraphrase. Enriching paraphrases with contextual details that may be implicit (e.g., augmenting "I had a pancake" to include details like "maple syrup") further increases prediction accuracy, even surpassing predictions based on the embedding of the original sentence, suggesting that the language system maintains richer and broader semantic representations than language models. Together, these results demonstrate the existence of highly abstract, form-independent meaning representations within the language cortex.

Problem

• Modeling abstract meaning representations in language cortex
• Comparing vision and language models for neural prediction
• Enriching semantic representations beyond linguistic forms

Innovation

• Using vision model embeddings from generated images
• Averaging embeddings across multiple sentence paraphrases
• Enriching paraphrases with contextual implicit details