Toward Explaining Large Language Models in Software Engineering Tasks

2025-12-23

AI Summary

The black-box nature of large language models (LLMs) hinders their adoption in safety-critical software engineering (SE) contexts, primarily due to the lack of domain-adapted interpretability methods aligned with SE practitioners’ reasoning patterns. Method: We propose FeatureSHAP—the first fully automated, model-agnostic explanation framework tailored to SE’s dual-modality tasks: code generation and code summarization. It introduces a SE-specific, higher-order semantic feature attribution mechanism grounded in Shapley values, integrating task-aware similarity metrics and dual-modality feature abstraction. Contribution/Results: Experiments demonstrate that FeatureSHAP significantly reduces spurious attributions to irrelevant inputs and achieves superior explanation fidelity over existing baselines. An empirical study with 37 SE practitioners confirms that FeatureSHAP meaningfully enhances model understanding depth and decision trustworthiness—addressing a critical gap in practice-oriented, explainable AI for software engineering.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Recent progress in Large Language Models (LLMs) has substantially advanced the automation of software engineering (SE) tasks, enabling complex activities such as code generation and code summarization. However, the black-box nature of LLMs remains a major barrier to their adoption in high-stakes and safety-critical domains, where explainability and transparency are vital for trust, accountability, and effective human supervision. Despite increasing interest in explainable AI for software engineering, existing methods lack domain-specific explanations aligned with how practitioners reason about SE artifacts. To address this gap, we introduce FeatureSHAP, the first fully automated, model-agnostic explainability framework tailored to software engineering tasks. Based on Shapley values, FeatureSHAP attributes model outputs to high-level input features through systematic input perturbation and task-specific similarity comparisons, while remaining compatible with both open-source and proprietary LLMs. We evaluate FeatureSHAP on two bi-modal SE tasks: code generation and code summarization. The results show that FeatureSHAP assigns less importance to irrelevant input features and produces explanations with higher fidelity than baseline methods. A practitioner survey involving 37 participants shows that FeatureSHAP helps practitioners better interpret model outputs and make more informed decisions. Collectively, FeatureSHAP represents a meaningful step toward practical explainable AI in software engineering. FeatureSHAP is available at https://github.com/deviserlab/FeatureSHAP.

Problem

• Explaining black-box LLMs in software engineering tasks
• Addressing lack of domain-specific explanations for SE artifacts
• Enhancing trust and accountability in high-stakes SE domains

Innovation

• Automated model-agnostic framework for SE tasks
• Uses Shapley values with input perturbation and similarity
• Tailored explanations for code generation and summarization