AIriskEval-edu: New Dataset for Risk Assessment in AI-mediated K-12 Educational Explanations

2026-07-02

AI Summary

This study addresses the critical gap in high-quality datasets and interpretable auditing tools for evaluating educational risks in AI-generated K–12 instructional explanations. The authors introduce AIriskEval-edu-db2, a novel dataset comprising 1,639 human- and LLM-generated teaching explanations, annotated with structured interpretability labels that include precise risk localization and descriptive justifications. Annotations are grounded in a five-dimensional risk framework and produced via a semi-automated pipeline validated by expert educators. Experimental results demonstrate that a lightweight, locally deployable model (Llama 3.1 8B), fine-tuned under supervised learning and strict privacy constraints, achieves risk assessment performance comparable to or exceeding that of state-of-the-art closed-source models. This work thus offers an effective, interpretable, and practical solution for safeguarding AI-generated educational content.

Abstract

This work introduces AIriskEval-edu-db2, a new dataset designed to train and evaluate auditors based on LLMs for an explainable pedagogical risk assessment in instructional content for grades K-12. The dataset comprises 1,639 explanations from 170 curated ScienceQA questions, covering science, language arts, and social sciences. For each question, the dataset includes an explanation written by a human teacher alongside 11 explanations generated by LLM-simulated teacher profiles associated with distinct pedagogical risks. We propose a comprehensive risk rubric aligned with established educational standards that covers five complementary dimensions: factual precision, depth and completeness, focus and relevance, student-level appropriateness, and ideological bias. A key contribution is the addition of 785 explanations with structured explainability annotations, including risk localization and risk description. The annotations are produced through a semi-automatic process with expert teacher validation. Finally, we present validation experiments comparing state-of-the-art proprietary models with a lightweight local Llama 3.1 8B model in both the pedagogical risk detection and the explainability assessment. These experiments evaluate whether supervised fine-tuning on AIriskEval-edu-db2 enables a locally deployable model to approach or outperform stronger frontier models while preserving privacy in educational auditing and assessment tasks.

Problem

• AI-mediated education
• pedagogical risk assessment
• explainable AI
• K-12 educational explanations
• instructional content safety

Innovation

• pedagogical risk assessment
• explainable AI
• educational auditing
• structured explainability annotations
• local LLM deployment