A Two-Stage LLM Framework for Accessible and Verified XAI Explanations

2026-04-14

AI Summary

This work addresses the limitations of current large language model (LLM)-based approaches for generating natural language explanations from explainable AI (XAI) outputs, which often lack guarantees of accuracy, faithfulness, and completeness and rely on subjective or post-hoc evaluations that fail to prevent the propagation of erroneous explanations. To overcome these issues, the authors propose a two-stage LLM meta-verification framework: an explainer LLM first translates XAI outputs into natural language explanations, which are then evaluated by a verifier LLM for faithfulness, coherence, completeness, and hallucination risk, with iterative feedback used to refine the explanations. Experimental results across three open-source LLMs, five XAI techniques, and entropy production rate (EPR) analysis demonstrate that the framework significantly enhances explanation reliability and consistency while preserving or even improving readability, effectively filtering out unreliable content.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) are increasingly used to translate the technical outputs of eXplainable Artificial Intelligence (XAI) methods into accessible natural-language explanations. However, existing approaches often lack guarantees of accuracy, faithfulness, and completeness. At the same time, current efforts to evaluate such narratives remain largely subjective or confined to post-hoc scoring, offering no safeguards to prevent flawed explanations from reaching end-users. To address these limitations, this paper proposes a Two-Stage LLM Meta-Verification Framework that consists of (i) an Explainer LLM that converts raw XAI outputs into natural-language narratives, (ii) a Verifier LLM that assesses them in terms of faithfulness, coherence, completeness, and hallucination risk, and (iii) an iterative refeed mechanism that uses the Verifier's feedback to refine and improve them. Experiments across five XAI techniques and datasets, using three families of open-weight LLMs, show that verification is crucial for filtering unreliable explanations while improving linguistic accessibility compared with raw XAI outputs. In addition, the analysis of the Entropy Production Rate (EPR) during the refinement process indicates that the Verifier's feedback progressively guides the Explainer toward more stable and coherent reasoning. Overall, the proposed framework provides an efficient pathway toward more trustworthy and democratized XAI systems.

Problem

• Explainable AI
• Large Language Models
• Faithfulness
• Explanation Verification
• Hallucination

Innovation

• Two-Stage LLM Framework
• XAI Verification
• Faithfulness
• Iterative Refinement
• Entropy Production Rate