Lost-in-Distance: Impact of Contextual Proximity on LLM Performance in Graph Tasks

2024-10-02

arXiv.org

AI Summary

This paper identifies a “distance disorientation” phenomenon in large language models (LLMs) for graph reasoning: model accuracy degrades sharply—by up to 6×—as the physical positional distance between critical edges increases within the input context. Method: We formally define this phenomenon and prove its orthogonality to the known “lost-in-the-middle” effect. Through systematic evaluation across adjacency-list, path-sequence, and subgraph-unfolding encodings, we assess three open-source LLMs on two graph reasoning tasks—two-node connectivity identification and three-node similarity judgment. Contribution/Results: Distance disorientation exhibits robustness across encoding schemes and model scales. Our findings reveal that LLMs’ positional sensitivity in context constitutes a fundamental bottleneck for graph-structured reasoning, offering a novel perspective on their limitations in structured inference.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Despite significant advancements, Large Language Models (LLMs) exhibit blind spots that impair their ability to retrieve and process relevant contextual data effectively. We demonstrate that LLM performance in graph tasks with complexities beyond the"needle-in-a-haystack"scenario-where solving the problem requires cross-referencing and reasoning across multiple subproblems jointly-is influenced by the proximity of relevant information within the context, a phenomenon we term"lost-in-distance". We examine two fundamental graph tasks: identifying common connections between two nodes and assessing similarity among three nodes, and show that the model's performance in these tasks significantly depends on the relative positioning of common edges. We evaluate three publicly available LLMs using various graph encoding techniques that represent graph structures for LLM input. We propose a formulation for the lost-in-distance phenomenon and demonstrate that lost-in-distance and lost-in-the middle phenomenas occur independently. Results indicate that model accuracy can decline by up to 6x as the distance between node connections increases, independent of graph encoding and model size.

Problem

• Large Language Models
• Graph Tasks
• Distance Bias

Innovation

• Large Language Models
• Distance Bias Phenomenon
• Graph Tasks Performance