Learning About Algorithm Auditing in Five Steps: Scaffolding How High School Youth Can Systematically and Critically Evaluate Machine Learning Applications

2024-12-09

AI Summary

Current AI literacy education for high school students lacks systematic instruction in algorithmic auditing—a critical competency for critically engaging with AI systems. Method: This study designs and empirically validates a five-step, cognitively appropriate algorithmic auditing pedagogical scaffold grounded in human-centered computing, participatory design, and critical technical literacy. Using TikTok’s generative AI filters as a real-world “black-box” audit target, the scaffold is implemented in extracurricular workshops to guide students through critical analysis of system logic, technical limitations, and sociotechnical implications. Contribution/Results: This work introduces the first structured, learner-centered algorithmic auditing scaffold for non-specialist adolescents and provides empirical evidence that high school students—when appropriately supported—can successfully conduct end-to-end audits of authentic AI systems. The study yields reusable teaching cases and classroom integration pathways, advancing both theoretical frameworks and practical pedagogical models for K–12 AI literacy education.

Technology Category

• Humans and AI: AI for Accessibility
• Machine Learning: Causal Learning
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Responsible Web: Algorithmic accountability and transparency on the web
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

While there is widespread interest in supporting young people to critically evaluate machine learning-powered systems, there is little research on how we can support them in inquiring about how these systems work and what their limitations and implications may be. Outside of K-12 education, an effective strategy in evaluating black-boxed systems is algorithm auditing-a method for understanding algorithmic systems' opaque inner workings and external impacts from the outside in. In this paper, we review how expert researchers conduct algorithm audits and how end users engage in auditing practices to propose five steps that, when incorporated into learning activities, can support young people in auditing algorithms. We present a case study of a team of teenagers engaging with each step during an out-of-school workshop in which they audited peer-designed generative AI TikTok filters. We discuss the kind of scaffolds we provided to support youth in algorithm auditing and directions and challenges for integrating algorithm auditing into classroom activities. This paper contributes: (a) a conceptualization of five steps to scaffold algorithm auditing learning activities, and (b) examples of how youth engaged with each step during our pilot study.

Problem

• AI Education
• Machine Learning
• Algorithm Auditing

Innovation

• AI Education
• Algorithmic Literacy
• High School Students