Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Models

2026-03-13

AI Summary

Current evaluations of large language models (LLMs) in reading comprehension are largely confined to factoid questions grounded in local text spans, failing to assess their capacity to understand population-level distributional information—such as opinion proportions or topic frequencies. To address this gap, this work proposes Text2DistBench, the first benchmark specifically designed for distributional reading comprehension. Built upon YouTube comments, it features a real-world, automatically updatable dataset that requires models to infer distributional knowledge by integrating entity metadata with user-generated comments. Leveraging an automated data collection and annotation pipeline that combines NLP and statistical modeling techniques, the benchmark extracts distributional question-answer pairs from dynamic comment streams. Experimental results show that while leading LLMs substantially outperform random baselines, their performance varies markedly across different distributional tasks, revealing notable limitations in their ability to reason about collective semantic patterns.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Distributed Search

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

While most reading comprehension benchmarks for LLMs focus on factual information that can be answered by localizing specific textual evidence, many real-world tasks require understanding distributional information, such as population-level trends and preferences expressed across collections of text. We introduce Text2DistBench, a reading comprehension benchmark for evaluating LLMs'ability to infer distributional knowledge from natural language. Built from real-world YouTube comments about movie and music entities, the benchmark provides models with entity metadata and associated comments, and requires them to answer distributional questions, such as estimating the proportions of positive and negative comments, or identifying the most and second most frequent topics discussed among viewers. To support reliable and long-term evaluation, the construction pipeline of Text2DistBench is fully automated and continuously updated to incorporate newly emerging entities over time. Experiments across multiple LLMs show that while models substantially outperform random baselines, performance varies widely across different distribution types and characteristics. These findings highlight both the capabilities and limitations of current LLMs in distributional reading comprehension and demonstrate the value of Text2DistBench as a practical and scalable testbed for future research.

Problem

• distributional reading comprehension
• large language models
• benchmark
• population-level trends
• textual distribution

Innovation

• distributional reading comprehension
• Text2DistBench
• large language models
• automated benchmarking
• population-level trends