Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts

2025-08-27

AI Summary

This work addresses the lack of systematic, automated evaluation for large language models’ (LLMs) hierarchical comprehension of book-length texts. We propose HAMLET, a novel framework featuring a “root–branch–leaf” three-tiered factual hierarchy, integrated with query-guided summarization and semantic consistency–based automatic scoring to form an end-to-end evaluation pipeline. Experiments demonstrate >90% agreement between automated scores and human expert judgments, with a 25× reduction in evaluation cost. HAMLET reveals, for the first time, a pronounced “middle-level forgetting” effect—where LLMs exhibit degraded performance on branch-level abstractions—and a persistent bottleneck in leaf-level fine-grained understanding; analytical queries prove significantly more challenging than narrative ones. Furthermore, performance gaps across open- versus closed-source models and across model scales remain consistent, enabling robust comparative analysis.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We introduce HAMLET, a holistic and automated framework for evaluating the long-context comprehension of large language models (LLMs). HAMLET structures source texts into a three-level key-fact hierarchy at root-, branch-, and leaf-levels, and employs query-focused summarization to evaluate how well models recall and faithfully represent information at each level. To validate the reliability of our fully automated pipeline, we conduct a systematic human study, showing that our automatic evaluation achieves over 90% agreement with expert human judgments, while reducing the cost by up to 25 times. HAMLET reveals that LLMs struggle with fine-grained comprehension, especially at the leaf level, and are sensitive to positional effects like the lost-in-the-middle. Analytical queries pose greater challenges than narrative ones, and consistent performance gaps emerge between open-source and proprietary models, as well as across model scales. Our code and dataset are publicly available at https://github.com/DISL-Lab/HAMLET.

Problem

• Evaluating LLMs' multi-level comprehension in book-length contexts
• Assessing recall and faithful representation of hierarchical information
• Identifying performance gaps in fine-grained and analytical queries

Innovation

• Three-level key-fact hierarchy structure
• Query-focused summarization for evaluation
• Automated pipeline with human validation