ENTLORE: A Graph-Grounded Benchmark for Latent Organizational Reasoning in Enterprise Question Answering

2026-08-11

AI Summary

Existing enterprise question-answering benchmarks focus solely on evaluating explicit factual composition, making them inadequate for assessing models’ ability to reason about implicit organizational relationships absent from documents. To address this gap, this work introduces the ENTLORE benchmark, which formally defines the novel task of “implicit organizational reasoning.” ENTLORE integrates everyday corporate documents, authoritative organizational charts, and operational logs, constructing an auditable enterprise knowledge graph grounded in versioned organizational policies and generating gold-standard answers with verifiable reasoning paths. The benchmark comprises 2,341 anonymized documents and 907 questions spanning three task types: explicit queries, cross-source composition, and implicit reasoning. Experiments reveal that even when provided with gold documents, state-of-the-art models fail on 30.4% of implicit reasoning questions, underscoring the benchmark’s challenge and its value for evaluating advanced reasoning capabilities.

Abstract

Enterprise question answering is framed as retrieving internal documents and generating grounded answers. Routine enterprise records, however, are work by-products in which required organizational relations remain implicit across heterogeneous sources. Existing benchmarks provide realistic multi-source evidence, but often materialize a predefined answer path and therefore test the composition of stated facts rather than recovery of a target relation absent from the corpus. We call the latter capability latent organizational reasoning. We introduce ENTLORE, a graph-grounded benchmark construction framework that reconstructs an audited enterprise world from routine documents, authoritative organizational tables, and operational records. Versioned organizational conventions certify derived relations in a truth graph, enabling complete golden answers and proof certificates. The aligned anonymized release exposes only the document corpus while withholding private structure and target relations. ENTLORE contains 2,341 documents from three source types and 907 questions spanning explicit lookup, cross-source composition, and latent organizational reasoning, evaluated across 56 model and access configurations. Structuring the released world as an induced entity graph or navigable knowledge base gives the strongest deployable results. Yet supplying gold documents still leaves 30.4% of latent questions unanswered, versus 12.6% and 6.2% for explicit and compositional questions. Enterprise QA therefore depends not only on document recall, but also on whether implicit organizational relations become usable. The benchmark, data, and code are publicly available at ENTLORE.

Problem

• latent organizational reasoning
• enterprise question answering
• implicit relations
• heterogeneous sources
• organizational knowledge

Innovation

• latent organizational reasoning
• graph-grounded benchmark
• enterprise question answering
• truth graph
• implicit relation recovery