Challenges for AI in Multimodal STEM Assessments: a Human-AI Comparison

2025-07-02

AI Summary

This study investigates performance bottlenecks of generative AI in multimodal STEM education assessment, specifically why large language models (LLMs) significantly underperform humans on visually grounded questions. Method: We construct a novel, fine-grained dataset of 201 university-level multimodal STEM items—annotated for image type, role, visual complexity, and format—and conduct human–AI comparative experiments (n = 546 students) across four LLM families and five prompting strategies. Contribution/Results: We establish that human performance is primarily modulated by disciplinary domain, whereas AI accuracy is strongly contingent on visual features and item structural design. The best-performing ensemble method (majority voting) achieves only 58.5% accuracy—significantly below human performance across all image-containing item types. Based on these findings, we propose the “Feature Discriminability” framework: a principled, actionable paradigm for item design that enhances detectability of AI-generated responses and strengthens academic integrity in multimodal assessment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Generative AI systems have rapidly advanced, with multimodal input capabilities enabling reasoning beyond text-based tasks. In education, these advancements could influence assessment design and question answering, presenting both opportunities and challenges. To investigate these effects, we introduce a high-quality dataset of 201 university-level STEM questions, manually annotated with features such as image type, role, problem complexity, and question format. Our study analyzes how these features affect generative AI performance compared to students. We evaluate four model families with five prompting strategies, comparing results to the average of 546 student responses per question. Although the best model correctly answers on average 58.5 % of the questions using majority vote aggregation, human participants consistently outperform AI on questions involving visual components. Interestingly, human performance remains stable across question features but varies by subject, whereas AI performance is susceptible to both subject matter and question features. Finally, we provide actionable insights for educators, demonstrating how question design can enhance academic integrity by leveraging features that challenge current AI systems without increasing the cognitive burden for students.

Problem

• Evaluates AI performance on multimodal STEM questions compared to students
• Analyzes impact of question features on AI and human response accuracy
• Provides insights for designing AI-resistant assessments without burdening students

Innovation

• Multimodal dataset for STEM assessment analysis
• Comparative study of AI and student performance
• Question design insights to challenge AI systems