A Taxonomy for Design and Evaluation of Prompt-Based Natural Language Explanations

2025-07-11

AI Summary

This paper addresses the lack of systematic design and evaluation frameworks for prompt-based natural language explanations (NLEs) in transparent AI governance. We propose the first taxonomy specifically tailored to prompt-driven NLEs, structured along three dimensions: context, generation & presentation, and evaluation. The taxonomy integrates task objectives, data characteristics, stakeholder needs, generation mechanisms, interaction modalities, output formats, and user-centered evaluation criteria into a multidimensional, structured classification. Its key innovation lies in adapting eXplainable AI (XAI) taxonomic principles to the prompt-based NLE paradigm—moving beyond conventional model-internal explanations. The resulting framework provides researchers, auditors, and policymakers with actionable design guidelines and standardized evaluation benchmarks, thereby enhancing the practical utility of NLEs in transparency, auditability, and regulatory compliance. (149 words)

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Humans and AI: Explainable AI (XAI) for Human Understanding
• Machine Learning: Transparent, Interpretable, Explainable ML

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

Effective AI governance requires structured approaches for stakeholders to access and verify AI system behavior. With the rise of large language models, Natural Language Explanations (NLEs) are now key to articulating model behavior, which necessitates a focused examination of their characteristics and governance implications. We draw on Explainable AI (XAI) literature to create an updated XAI taxonomy, adapted to prompt-based NLEs, across three dimensions: (1) Context, including task, data, audience, and goals; (2) Generation and Presentation, covering generation methods, inputs, interactivity, outputs, and forms; and (3) Evaluation, focusing on content, presentation, and user-centered properties, as well as the setting of the evaluation. This taxonomy provides a framework for researchers, auditors, and policymakers to characterize, design, and enhance NLEs for transparent AI systems.

Problem

• Develop taxonomy for prompt-based natural language explanations
• Examine characteristics and governance of AI system behavior
• Provide framework to design and evaluate transparent AI systems

Innovation

• Updated XAI taxonomy for prompt-based NLEs
• Three-dimensional framework for NLE design
• Structured evaluation of NLE characteristics