Child Safety in Generative AI: An Expert-Guided and Incident-Grounded Evaluation Framework

2026-06-30

AI Summary

This study addresses a critical gap in generative AI safety evaluation by proposing the first child-centered assessment framework tailored for educational contexts. Recognizing that existing safety benchmarks largely overlook the unique vulnerabilities of minors, the work integrates developmental guidelines from child experts with real-world AI incident data to systematically evaluate model responses to child-unsafe content. The framework employs expert-derived risk factor analysis, incident data mining, and synthetically generated test cases to probe model robustness. Experimental results reveal that prevailing safety models, including Llama Guard, exhibit significant shortcomings when handling education-related prompts involving children, thereby exposing the inadequacy of current safeguards for minors and establishing a foundational benchmark for future research in pediatric AI safety.

Abstract

As generative AI is increasingly used by children and adolescents, there is a growing need for risk evaluation frameworks that account for child-specific harms. However, most existing safety evaluation frameworks focus on general user populations, often overlooking risks unique to younger users. To address this gap, we propose an evaluation framework that integrates expert-guided risk factors with real-world AI incident data for child safety. The framework identifies hazard categories from expert guidelines and AI incident databases and uses this information to construct a synthetic test set for model evaluation. Particularly, we apply the framework to the education domain and evaluate three Llama Guard models on their ability to detect unsafe user prompts. Our results show that current Llama Guard models struggle to identify education-related unsafe user prompts. We conclude by discussing how future work can extend the evaluation to additional risk categories and incorporate domain experts throughout the evaluation pipeline.

Problem

• child safety
• generative AI
• risk evaluation
• harm prevention
• AI safety

Innovation

• child safety
• generative AI
• expert-guided evaluation
• incident-grounded framework
• synthetic test set