LLMs for Explainable AI: A Comprehensive Survey

2025-03-31

AI Summary

The “black-box” nature of AI models undermines interpretability and erodes user trust. Method: This paper systematically reviews advances in leveraging large language models (LLMs) to advance eXplainable Artificial Intelligence (XAI), proposing the first comprehensive LLM-XAI taxonomy comprising six core paradigms: prompt-driven explanation generation, instruction-tuned explainability, human-in-the-loop explanation evaluation, multimodal explanation synthesis, model self-explanation enhancement, and verifiable explanation construction—based on analysis of 120+ scholarly works. Contribution/Results: It identifies critical bottlenecks in explanation fidelity, standardized evaluation, and real-world deployment. Innovatively, it advocates a user-centered, interdisciplinary XAI paradigm and outlines a next-generation XAI roadmap emphasizing scalability, automation, and formal verifiability—providing both theoretical foundations and practical frameworks to enhance model transparency and human-AI collaborative decision-making.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large Language Models (LLMs) offer a promising approach to enhancing Explainable AI (XAI) by transforming complex machine learning outputs into easy-to-understand narratives, making model predictions more accessible to users, and helping bridge the gap between sophisticated model behavior and human interpretability. AI models, such as state-of-the-art neural networks and deep learning models, are often seen as"black boxes"due to a lack of transparency. As users cannot fully understand how the models reach conclusions, users have difficulty trusting decisions from AI models, which leads to less effective decision-making processes, reduced accountabilities, and unclear potential biases. A challenge arises in developing explainable AI (XAI) models to gain users' trust and provide insights into how models generate their outputs. With the development of Large Language Models, we want to explore the possibilities of using human language-based models, LLMs, for model explainabilities. This survey provides a comprehensive overview of existing approaches regarding LLMs for XAI, and evaluation techniques for LLM-generated explanation, discusses the corresponding challenges and limitations, and examines real-world applications. Finally, we discuss future directions by emphasizing the need for more interpretable, automated, user-centric, and multidisciplinary approaches for XAI via LLMs.

Problem

• Enhancing AI transparency using LLMs for human-readable explanations
• Addressing black-box AI issues to improve user trust and understanding
• Exploring LLM-based approaches for explainable AI (XAI) challenges

Innovation

• LLMs transform complex outputs into narratives
• LLMs bridge model behavior and human interpretability
• LLMs enhance trust via automated explanations