Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

2026-07-09

AI Summary

This work addresses the limitation of existing long-context evaluation benchmarks, which predominantly focus on average performance and lack a multidimensional analysis of task difficulty, thereby failing to delineate model capabilities across distinct challenge dimensions. To this end, we introduce PredicateLongBench, a novel benchmark that systematically defines and explores multiple axes of difficulty—such as context length and distractor density—by identifying the longest contiguous subsequence in long texts that satisfies specified predicate constraints. The benchmark employs two data generation strategies: fully synthetic random-like token sequences and sampled natural texts preserving authentic distributional properties, both avoiding reliance on large language models for data creation or evaluation. Experimental results demonstrate that state-of-the-art models exhibit substantial performance degradation as difficulty increases, confirming the benchmark’s effectiveness in exposing the current limitations of long-context reasoning.

Abstract

Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness. Notably absent is a systematic way to probe how models perform as we scale up the difficulty of tasks along various axes. We address this gap by proposing PredicateLongBench, a benchmark that stress-tests long-context reasoning by asking models to identify the longest contiguous subsequence of words in a long input that satisfies given predicates/constraints (e.g., lexicographic ordering), drawn from a broader predicate class. The central innovation of our benchmark is the identification and systematic exploration of multiple different axes of difficulty which test multiple aspects of long context understanding. We provide two complementary generation pipelines - a fully synthetic setup using random word-like strings, and a real-world setup that samples words from natural documents while preserving their distributional properties. We find that frontier models struggle to perform well as we scale up the difficulty of tasks along our axes, demonstrating the utility of our benchmark in understanding the limitations of current long-context capabilities. Furthermore, the tasks in PredicateLongBench, though challenging, are conceptually simple and do not require LLM-based generations or judges.

Problem

• long-context evaluation
• difficulty axes
• benchmark
• large language models
• systematic probing

Innovation

• long-context reasoning
• benchmarking
• difficulty axes
• predicate-based tasks
• contiguous subsequence