EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval

2026-08-11

AI Summary

Enterprise-scale Retrieval-Augmented Generation (RAG) systems often exhibit insufficient instruction-following capabilities in real-world scenarios due to retrieval noise, missing knowledge, and factual inconsistencies—failure modes inadequately captured by existing benchmarks. To address this gap, this work introduces a novel benchmark comprising 983 expert-validated samples across six domains, featuring the first evaluation paradigm that jointly incorporates multidimensional constraints and non-ideal retrieval conditions to systematically simulate three representative failure types. The study evaluates 13 state-of-the-art large language models and finds that only 26.8% of responses satisfy all specified constraints, revealing a systemic breakdown in holistic instruction adherence masked by high individual constraint satisfaction rates. These findings underscore the necessity of explicit context-aware protocols and calibrated judgment mechanisms, establishing a reproducible, fine-grained foundation for evaluating industrial-grade RAG systems.

Abstract

Enterprise RAG deployments face a critical reliability gap: while LLMs satisfy 80% of individual constraints, only 26.8% of responses meet all requirements simultaneously, revealing a 57-point orchestration gap. Existing benchmarks assume clean retrieval with simple queries, failing to capture production conditions where noisy documents and multi-dimensional constraints coexist. We introduce EnterpriseRAG, a benchmark of 983 expert-validated samples across six domains that systematically simulates three failure modes absent from prior work: retrieval noise, knowledge gaps, and factual conflicts, coupled with complex instructions. Evaluation of 13 state-of-the-art LLMs reveals a severe instruction adherence collapse, where high per-constraint satisfaction masks low holistic compliance. Critical findings expose deep barriers under knowledge gaps and factual conflicts, even with reasoning-enhanced inference, indicating production RAG requires explicit context-aware protocols and calibrated judgment. EnterpriseRAG provides a reproducible foundation for measuring and closing these gaps, directly informing deployment decisions for enterprise-scale RAG systems. We will release the benchmark and evaluation framework upon publication.

Problem

• Enterprise RAG
• instruction adherence
• retrieval noise
• knowledge gaps
• factual conflicts

Innovation

• EnterpriseRAG
• instruction adherence
• retrieval noise
• knowledge gaps
• factual conflicts