LLMs Struggle with Abstract Meaning Comprehension More Than Expected

2026-04-13

AI Summary

This work addresses the limited capacity of large language models to comprehend non-concrete, high-level abstract semantics. Focusing on SemEval-2021 Task 4 (ReCAM), the authors propose a bidirectional attention classifier inspired by human cognitive strategies, which dynamically models interactions between passages and answer options during fine-tuning of pretrained models such as BERT and RoBERTa. The proposed approach achieves accuracy improvements of 4.06% and 3.41% on Task 1 and Task 2, respectively, substantially enhancing the model’s ability to reason about abstract concepts. These results demonstrate the effectiveness of cognitively inspired mechanisms in improving abstract semantic understanding within natural language processing systems.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Understanding abstract meanings is crucial for advanced language comprehension. Despite extensive research, abstract words remain challenging due to their non-concrete, high-level semantics. SemEval-2021 Task 4 (ReCAM) evaluates models' ability to interpret abstract concepts by presenting passages with questions and five abstract options in a cloze-style format. Key findings include: (1) Most large language models (LLMs), including GPT-4o, struggle with abstract meaning comprehension under zero-shot, one-shot, and few-shot settings, while fine-tuned models like BERT and RoBERTa perform better. (2) A proposed bidirectional attention classifier, inspired by human cognitive strategies, enhances fine-tuned models by dynamically attending to passages and options. This approach improves accuracy by 4.06 percent on Task 1 and 3.41 percent on Task 2, demonstrating its potential for abstract meaning comprehension.

Problem

• abstract meaning
• language comprehension
• large language models
• semantic understanding

Innovation

• abstract meaning comprehension
• bidirectional attention classifier
• cognitive-inspired modeling
• few-shot learning
• SemEval-2021 Task 4