Counterfactual Benchmarking and Training for Factuality Consistency and Order-Robust Grounded Reasoning in LLMs over Heterogeneous Knowledge

2026-08-07

AI Summary

This work addresses the challenges large language models (LLMs) face in multi-hop reasoning over heterogeneous knowledge sources—such as text, tables, and knowledge graphs—including factual inconsistency and sensitivity to input order. To this end, the authors introduce TKFQA, a novel benchmark that incorporates counterfactual reasoning chains and evaluates model performance along three dimensions: answer correctness, reasoning chain accuracy, and order robustness. They further propose ORLF, a model-agnostic training framework that explicitly models cross-context topological relationships through knowledge-specific latent vectors, contextual positional encoding, latent bridging attention masks, and topological knowledge bias. Experiments demonstrate that ORLF significantly outperforms both untrained and LoRA-finetuned baselines across four mainstream LLMs, improving Exact Match and reasoning chain accuracy by 2.15% and 4.29%, respectively, while reducing performance variance under input order perturbations by 0.04%–3.01%.

Abstract

Large language models (LLMs) have increasingly supported response generation grounded in user-provided knowledge spanning heterogeneous structures. However, existing benchmarks provide limited assessment of whether LLMs can faithfully perform multi-hop reasoning chains across such knowledge contexts while remaining robust to variations in their input order. We introduce TKFQA, a factuality consistency benchmark comprising 10,130 question-answering (QA) pairs grounded in tables, texts, and knowledge graphs (KGs). Each example is constructed from an explicit counterfactual reasoning chain, enabling the joint evaluation of answer correctness, reasoning-chain accuracy, and robustness to different input-order. An extensive evaluation of 14 open- and closed-source LLMs reveals that state-of-the-art models exhibit limited reasoning-chain accuracy and remain sensitive to variations in the input order of heterogeneous knowledge contexts. To address these limitations, we propose ORLF, an LLM-agnostic training framework that models cross-context topological relations through knowledge-specific latent vectors. ORLF integrates context-wise position encoding, a latent-bridge attention mask, and topological knowledge bias to preserve knowledge-specific bias and encode topological semantics. Experiments across four LLM backbones show that ORLF outperforms competitive training-free and LoRA-based baselines, improving average Exact Match and Reasoning-Chain Accuracy by 2.15% and 4.29%, respectively, while reducing order-induced performance standard deviation by 0.04% to 3.01%.

Problem

• factuality consistency
• order-robustness
• grounded reasoning
• heterogeneous knowledge
• counterfactual benchmarking

Innovation

• counterfactual reasoning
• heterogeneous knowledge grounding
• order-robustness
• topological relation modeling
• factuality consistency