Utilizing Large Language Models for Machine Learning Explainability

2025-10-08

AI Summary

This work investigates the capability of large language models (LLMs) to autonomously construct interpretable machine learning (ML) pipelines. Addressing driver alertness prediction and yeast multilabel classification, we systematically prompt GPT, Claude, and DeepSeek to generate end-to-end ML workflows—including Random Forest, XGBoost, MLP, and LSTM—alongside explanation components. We introduce SHAP fidelity and feature sparsity as novel quantitative metrics for evaluating interpretability, the first such application in LLM-driven ML automation. Results demonstrate that LLM-generated models match human-designed baselines in predictive performance (e.g., accuracy, F1-score), while achieving high SHAP fidelity and stable feature sparsity—indicating reliable attribution and concise explanations. These findings establish the feasibility and robustness of LLMs in automating interpretable ML. The study provides both a new paradigm and empirical foundation for trustworthy, automated AI systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

This study explores the explainability capabilities of large language models (LLMs), when employed to autonomously generate machine learning (ML) solutions. We examine two classification tasks: (i) a binary classification problem focused on predicting driver alertness states, and (ii) a multilabel classification problem based on the yeast dataset. Three state-of-the-art LLMs (i.e. OpenAI GPT, Anthropic Claude, and DeepSeek) are prompted to design training pipelines for four common classifiers: Random Forest, XGBoost, Multilayer Perceptron, and Long Short-Term Memory networks. The generated models are evaluated in terms of predictive performance (recall, precision, and F1-score) and explainability using SHAP (SHapley Additive exPlanations). Specifically, we measure Average SHAP Fidelity (Mean Squared Error between SHAP approximations and model outputs) and Average SHAP Sparsity (number of features deemed influential). The results reveal that LLMs are capable of producing effective and interpretable models, achieving high fidelity and consistent sparsity, highlighting their potential as automated tools for interpretable ML pipeline generation. The results show that LLMs can produce effective, interpretable pipelines with high fidelity and consistent sparsity, closely matching manually engineered baselines.

Problem

• Explores LLMs' ability to generate interpretable machine learning solutions
• Evaluates automated ML pipelines on classification tasks using SHAP metrics
• Assesses fidelity and sparsity of LLM-generated models versus manual baselines

Innovation

• LLMs autonomously generate machine learning training pipelines
• LLMs design classifiers like Random Forest and XGBoost
• LLMs produce interpretable models using SHAP explainability metrics