Connecting Feedback to Choice: Understanding Educator Preferences in GenAI vs. Human-Created Lesson Plans in K-12 Education -- A Comparative Analysis

2025-04-07

AI Summary

This study investigates K–12 mathematics teachers’ comparative preferences for lesson plans authored by humans versus those generated by fine-tuned LLaMA-2-13B and customized GPT-4 models, across warm-up, main instruction, and closure phases—as well as holistic quality. A mixed-methods approach was employed: a large-scale pairwise preference experiment (N = 217) complemented by LDA topic modeling and qualitative analysis via manual coding. Results reveal—novelly and systematically—that teacher preferences vary dynamically by educational level (elementary teachers strongly favor human-authored plans; AI excels in high school closures and structured activities) and instructional phase. The study proposes a “human–AI collaborative lesson design paradigm,” positioning generative AI as a structured scaffolding tool—not a replacement—for educators. Key drivers of preference include differentiated instruction, authentic contextualization, and facilitation of student discourse. These findings provide empirically grounded guidance for the pedagogically sound integration of generative AI in mathematics education.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

As generative AI (GenAI) models are increasingly explored for educational applications, understanding educator preferences for AI-generated lesson plans is critical for their effective integration into K-12 instruction. This exploratory study compares lesson plans authored by human curriculum designers, a fine-tuned LLaMA-2-13b model trained on K-12 content, and a customized GPT-4 model to evaluate their pedagogical quality across multiple instructional measures: warm-up activities, main tasks, cool-down activities, and overall quality. Using a large-scale preference study with K-12 math educators, we examine how preferences vary across grade levels and instructional components. We employ both qualitative and quantitative analyses. The raw preference results indicate that human-authored lesson plans are generally favored, particularly for elementary education, where educators emphasize student engagement, scaffolding, and collaborative learning. However, AI-generated models demonstrate increasing competitiveness in cool-down tasks and structured learning activities, particularly in high school settings. Beyond quantitative results, we conduct thematic analysis using LDA and manual coding to identify key factors influencing educator preferences. Educators value human-authored plans for their nuanced differentiation, real-world contextualization, and student discourse facilitation. Meanwhile, AI-generated lesson plans are often praised for their structure and adaptability for specific instructional tasks. Findings suggest a human-AI collaborative approach to lesson planning, where GenAI can serve as an assistive tool rather than a replacement for educator expertise in lesson planning. This study contributes to the growing discourse on responsible AI integration in education, highlighting both opportunities and challenges in leveraging GenAI for curriculum development.

Problem

• Compare educator preferences for AI vs human-created K-12 lesson plans
• Evaluate pedagogical quality of AI-generated vs human-authored lesson content
• Identify key factors influencing educator choices in human-AI lesson planning

Innovation

• Fine-tuned LLaMA-2-13b model for K-12 content
• Customized GPT-4 model for lesson plans
• Human-AI collaborative approach in education