DrawEduMath: Evaluating Vision Language Models with Expert-Annotated Students' Hand-Drawn Math Images

2025-01-24

AI Summary

Existing vision-language models (VLMs) exhibit insufficient capability in parsing cluttered, real-world handwritten mathematics assignments within authentic K–12 educational settings. Method: We introduce the first K–12 handwritten mathematics image benchmark—comprising 2,030 real student assignment images, 11,661 expert teacher-annotated question-answer (QA) pairs, and 44,362 high-fidelity synthetic QA pairs generated by large language models (LLMs). We propose an education-driven evaluation paradigm featuring teacher-expert free-form descriptions and strategic annotation protocols. Contribution/Results: Our analysis confirms that synthetic QA pairs reliably substitute human annotations (Spearman’s ρ = 0.92 for model ranking correlation). Multi-granularity evaluation reveals a substantial performance gap between state-of-the-art VLMs and human teachers. The benchmark is publicly released to advance robustness and trustworthiness research in educational AI.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Question Answering

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In real-world settings, vision language models (VLMs) should robustly handle naturalistic, noisy visual content as well as domain-specific language and concepts. For example, K-12 educators using digital learning platforms may need to examine and provide feedback across many images of students' math work. To assess the potential of VLMs to support educators in settings like this one, we introduce DrawEduMath, an English-language dataset of 2,030 images of students' handwritten responses to K-12 math problems. Teachers provided detailed annotations, including free-form descriptions of each image and 11,661 question-answer (QA) pairs. These annotations capture a wealth of pedagogical insights, ranging from students' problem-solving strategies to the composition of their drawings, diagrams, and writing. We evaluate VLMs on teachers' QA pairs, as well as 44,362 synthetic QA pairs derived from teachers' descriptions using language models (LMs). We show that even state-of-the-art VLMs leave much room for improvement on DrawEduMath questions. We also find that synthetic QAs, though imperfect, can yield similar model rankings as teacher-written QAs. We release DrawEduMath to support the evaluation of VLMs' abilities to reason mathematically over images gathered with educational contexts in mind.

Problem

• Visual Language Models
• Educational Scene Understanding
• Handwritten Math Evaluation

Innovation

• DrawEduMath Dataset
• Visual Language Models
• Math Problem Solving