Comparing Human and Large Language Model Interpretation of Implicit Information

2026-04-18

AI Summary

This work addresses the limitations of current large language models (LLMs) in comprehending implicit information in human communication, which hinders their ability to fully replicate human semantic interpretation. The paper introduces the Implicit Information Extraction (IIE) task and proposes a systematic LLM-based pipeline to extract relational triples from context, validate implicit inferences, and analyze temporal relationships. For the first time, it employs a crowdsourced evaluation framework to systematically compare interpretive differences between humans and models. Experimental results show that while humans generally endorse most model-generated triples, they consistently identify substantial omissions. Moreover, models exhibit conservative behavior in socially nuanced contexts and insufficient coverage in brief factual scenarios, revealing context-dependent biases in their comprehension capabilities.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The interpretation of implicit meanings is an integral aspect of human communication. However, this framework may not transfer to interactions with Large Language Models (LLMs). To investigate this, we introduce the task of Implicit Information Extraction (IIE) and propose an LLM-based IIE pipeline that builds a structured knowledge graph from a context sentence by extracting relational triplets, validating implicit inferences, and analyzing temporal relations. We evaluate two LLMs against crowdsourced human judgments on two datasets. We find that humans agree with most model triplets yet consistently propose many additions, indicating limited coverage in current LLM-based IIE. Moreover, in our experiments, models appear to be more conservative about implicit inferences than humans in socially rich contexts, whereas humans become more conservative in shorter, fact-oriented contexts. Our code is available at https://github.com/Antonio-Dee/IIE_from_LLM.

Problem

• Implicit Information Extraction
• Large Language Models
• Human Interpretation
• Implicit Inference
• Knowledge Graph

Innovation

• Implicit Information Extraction
• Large Language Models
• Knowledge Graph
• Human-LLM Comparison
• Relational Triplet Extraction