Practitioner Voices Summit: How Teachers Evaluate AI Tools through Deliberative Sensemaking

2026-03-23

AI Summary

This study addresses the limited agentic authority teachers experience when integrating AI tools into classrooms by convening a two-day national summit with 61 K–12 mathematics educators. Grounded in the TPACK framework and teacher agency, the intervention facilitated “deliberative meaning-making,” enabling participants to collaboratively develop their own AI evaluation criteria. Through qualitative thematic analysis of over 200 criteria alongside interview, survey, and discussion data, four core themes—practicality, equity, flexibility, and rigor—and their inherent tensions emerged, supported by five underlying mechanisms. The research innovatively integrates TPACK with deliberative agency, revealing teachers’ distinct conceptualization of AI as a pedagogical assistant rather than a professional coach. It further demonstrates a reciprocal cycle between knowledge construction and agency, offering theoretical and practical guidance for the design and adoption of educational technologies.

Technology Category

• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Multiagent Systems: Agent/AI Theories and Architectures
• Humans and AI: Human-AI Collaboration / Human-AI Teaming

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Teachers face growing pressure to integrate AI tools into their classrooms, yet are rarely positioned as agentic decision-makers in this process. Understanding the criteria teachers use to evaluate AI tools, and the conditions that support such reasoning, is essential for responsible AI integration. We address this gap through a two-day national summit in which 61 U.S. K-12 mathematics educators developed personal rubrics for evaluating AI classroom tools. The summit was designed to support deliberative sensemaking, a process we conceptualize by integrating Technological Pedagogical Content Knowledge (TPACK) with deliberative agency. Teachers generated over 200 criteria - initial articulations spanning four higher-order themes (Practical, Equitable, Flexible, and Rigorous) - that addressed both AI outputs and the process of using AI. Criteria contained productive tensions (e.g., personalization versus fairness, adaptability versus efficiency), and the vast majority framed AI as an assistant rather than a coaching tool for professional learning. Analysis of surveys, interviews, and summit discussions revealed five mechanisms supporting deliberative sensemaking: time and space for deliberation, artifact-centered sensemaking, collaborative reflection through diverse viewpoints, knowledge-building, and psychological safety. Across these mechanisms, TPACK and agency operated in a mutually reinforcing cycle - knowledge-building enabled more grounded evaluative judgment, while the act of constructing criteria deepened teachers' understanding of tools. We discuss implications for edtech developers seeking practitioner input, school leaders making adoption decisions, educators and professional learning designers, and researchers working to elicit teachers' evaluative reasoning about rapidly evolving technologies.

Problem

• teacher agency
• AI evaluation
• deliberative sensemaking
• responsible AI integration
• educational technology

Innovation

• deliberative sensemaking
• TPACK
• teacher agency
• AI evaluation rubrics
• collaborative reflection