Enabling Multi-Agent Systems as Learning Designers: Applying Learning Sciences to AI Instructional Design

2025-08-20

AI Summary

K–12 educators widely adopt commercial large language models (LLMs) for instructional material generation, yet these tools lack grounding in learning science and demand advanced prompt engineering skills, compromising pedagogical effectiveness. Method: We propose MAS-CMD, a multi-agent system that uniquely embeds the Knowledge–Learning–Instruction (KLI) framework within a role-based agent architecture. It employs a “divide-and-integrate” collaboration mechanism to autonomously generate educationally principled teaching activities. MAS-CMD integrates Quality Matters (QM) K–12 standards and an LLM-as-a-judge evaluation paradigm. Contribution/Results: Evaluated by 20 in-service teachers via dual-track assessment, MAS-CMD showed no statistically significant difference in quantitative scores versus baselines; however, teachers consistently rated its outputs as superior in creativity, contextual appropriateness, and classroom usability—demonstrating the feasibility and pedagogical advantage of deeply integrating learning theory into AI-driven instructional design.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

K-12 educators are increasingly using Large Language Models (LLMs) to create instructional materials. These systems excel at producing fluent, coherent content, but often lack support for high-quality teaching. The reason is twofold: first, commercial LLMs, such as ChatGPT and Gemini which are among the most widely accessible to teachers, do not come preloaded with the depth of pedagogical theory needed to design truly effective activities; second, although sophisticated prompt engineering can bridge this gap, most teachers lack the time or expertise and find it difficult to encode such pedagogical nuance into their requests. This study shifts pedagogical expertise from the user's prompt to the LLM's internal architecture. We embed the well-established Knowledge-Learning-Instruction (KLI) framework into a Multi-Agent System (MAS) to act as a sophisticated instructional designer. We tested three systems for generating secondary Math and Science learning activities: a Single-Agent baseline simulating typical teacher prompts; a role-based MAS where agents work sequentially; and a collaborative MAS-CMD where agents co-construct activities through conquer and merge discussion. The generated materials were evaluated by 20 practicing teachers and a complementary LLM-as-a-judge system using the Quality Matters (QM) K-12 standards. While the rubric scores showed only small, often statistically insignificant differences between the systems, the qualitative feedback from educators painted a clear and compelling picture. Teachers strongly preferred the activities from the collaborative MAS-CMD, describing them as significantly more creative, contextually relevant, and classroom-ready. Our findings show that embedding pedagogical principles into LLM systems offers a scalable path for creating high-quality educational content.

Problem

• Addressing lack of pedagogical theory in LLM-generated instructional materials
• Shifting pedagogical expertise from user prompts to LLM architecture
• Creating scalable high-quality educational content through embedded learning frameworks

Innovation

• Multi-Agent System with embedded pedagogical framework
• Collaborative agent design through conquer-merge discussion
• LLM architecture integrating Knowledge-Learning-Instruction principles