A Meta-Analysis of LLM Effects on Students across Qualification, Socialisation, and Subjectification

2025-09-25

AI Summary

Existing research predominantly examines narrow performance metrics of large language models (LLMs) in education, neglecting their impact on education’s fundamental aims. This study addresses this gap by applying Biesta’s tripartite educational framework—qualification, socialization, and subjectification—to conduct the first systematic meta-analysis integrating 133 experimental and quasi-experimental studies. Results reveal that LLMs yield moderate-to-large positive effects on qualification acquisition, produce inconsistent and context-dependent outcomes for socialization, and support subjectification only under sustained, small-scale, and pedagogically rigorous interventions. Methodologically, the work innovatively bridges educational philosophy with empirical meta-analytic techniques. Its key contribution lies in demonstrating that technological integration must be purpose-driven—subordinated to intrinsic educational goals rather than instrumental efficiency gains—and underscores instructional design as decisive for realizing higher-order educational outcomes.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) are increasingly positioned as solutions for education, yet evaluations often reduce their impact to narrow performance metrics. This paper reframes the question by asking "what kind of impact should LLMs have in education?" Drawing on Biesta's tripartite account of good education: qualification, socialisation, and subjectification, we present a meta-analysis of 133 experimental and quasi-experimental studies (k = 188). Overall, the impact of LLMs on student learning is positive but uneven. Strong effects emerge in qualification, particularly when LLMs function as tutors in sustained interventions. Socialisation outcomes appear more variable, concentrated in sustained, reflective interventions. Subjectification, linked to autonomy and learner development, remains fragile, with improvements confined to small-scale, long-term studies. This purpose-level view highlights design as the decisive factor: without scaffolds for participation and agency, LLMs privilege what is easiest to measure while neglecting broader aims of education. For HCI and education, the issue is not just whether LLMs work, but what futures they enable or foreclose.

Problem

• Analyzing LLM educational impact beyond narrow performance metrics
• Evaluating LLM effects on qualification, socialisation, and subjectification domains
• Identifying how LLM design influences educational participation and agency

Innovation

• LLMs function as tutors in sustained interventions
• Socialisation outcomes improved in reflective interventions
• Design scaffolds for participation and learner agency