Explainable Collaborative Problem Solving Diagnosis with BERT using SHAP and its Implications for Teacher Adoption

2025-07-19

AI Summary

BERT-based models for automated diagnosis of Collaborative Problem Solving (CPS) suffer from poor interpretability, undermining teacher trust and educational deployment. Method: This study pioneers the application of SHAP (Shapley Additive Explanations) in CPS to enable fine-grained, token-level attribution analysis, decomposing BERT’s classification decisions. Contribution/Results: We uncover a critical anomaly—semantically irrelevant tokens (e.g., punctuation, stopwords) frequently dominate predictions—demonstrating that high accuracy does not guarantee valid explanations. Moreover, model interpretability degrades significantly as the number of diagnostic categories increases. To address this, we propose a dual-dimension evaluation framework—“explanation quality” and “prediction performance”—providing both methodological guidance and practical warnings for trustworthy AI deployment in education.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

The use of Bidirectional Encoder Representations from Transformers (BERT) model and its variants for classifying collaborative problem solving (CPS) has been extensively explored within the AI in Education community. However, limited attention has been given to understanding how individual tokenised words in the dataset contribute to the model's classification decisions. Enhancing the explainability of BERT-based CPS diagnostics is essential to better inform end users such as teachers, thereby fostering greater trust and facilitating wider adoption in education. This study undertook a preliminary step towards model transparency and explainability by using SHapley Additive exPlanations (SHAP) to examine how different tokenised words in transcription data contributed to a BERT model's classification of CPS processes. The findings suggested that well-performing classifications did not necessarily equate to a reasonable explanation for the classification decisions. Particular tokenised words were used frequently to affect classifications. The analysis also identified a spurious word, which contributed positively to the classification but was not semantically meaningful to the class. While such model transparency is unlikely to be useful to an end user to improve their practice, it can help them not to overrely on LLM diagnostics and ignore their human expertise. We conclude the workshop paper by noting that the extent to which the model appropriately uses the tokens for its classification is associated with the number of classes involved. It calls for an investigation into the exploration of ensemble model architectures and the involvement of human-AI complementarity for CPS diagnosis, since considerable human reasoning is still required for fine-grained discrimination of CPS subskills.

Problem

• Enhancing explainability of BERT-based CPS diagnostics for teachers
• Identifying token contributions to BERT's CPS classification decisions
• Reducing overreliance on LLM diagnostics by improving model transparency

Innovation

• BERT model for CPS classification
• SHAP for explainable token contributions
• Ensemble models with human-AI complementarity