Tailored Teaching with Balanced Difficulty: Elevating Reasoning in Multimodal Chain-of-Thought via Prompt Curriculum

2025-08-26

AI Summary

Existing multimodal chain-of-thought (MCoT) prompting methods rely on random or manual example selection, ignoring both the model’s knowledge distribution and the intrinsic complexity of tasks—leading to unstable performance. To address this, we propose the first prompting curriculum framework for multimodal reasoning, inspired by the pedagogical principles of “teaching according to individual aptitude” and “difficulty balancing.” Our method constructs an ordered, model-capability-aligned sequence of demonstration examples. We introduce a novel dual-dimensional difficulty assessment: (i) model-perceived difficulty, quantified via prediction disagreement measured through active learning; and (ii) intrinsic difficulty, modeled from semantic and visual complexity of question–image pairs. Difficulty-balanced sampling then generates a progressive prompting curriculum. Evaluated across five mainstream multimodal benchmarks and multiple large multimodal models, our approach significantly improves reasoning accuracy and reduces performance variance, demonstrating strong robustness and generalizability.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The effectiveness of Multimodal Chain-of-Thought (MCoT) prompting is often limited by the use of randomly or manually selected examples. These examples fail to account for both model-specific knowledge distributions and the intrinsic complexity of the tasks, resulting in suboptimal and unstable model performance. To address this, we propose a novel framework inspired by the pedagogical principle of "tailored teaching with balanced difficulty". We reframe prompt selection as a prompt curriculum design problem: constructing a well ordered set of training examples that align with the model's current capabilities. Our approach integrates two complementary signals: (1) model-perceived difficulty, quantified through prediction disagreement in an active learning setup, capturing what the model itself finds challenging; and (2) intrinsic sample complexity, which measures the inherent difficulty of each question-image pair independently of any model. By jointly analyzing these signals, we develop a difficulty-balanced sampling strategy that ensures the selected prompt examples are diverse across both dimensions. Extensive experiments conducted on five challenging benchmarks and multiple popular Multimodal Large Language Models (MLLMs) demonstrate that our method yields substantial and consistent improvements and greatly reduces performance discrepancies caused by random sampling, providing a principled and robust approach for enhancing multimodal reasoning.

Problem

• Optimizing multimodal reasoning by selecting tailored examples
• Addressing suboptimal performance from random prompt selection
• Balancing model-perceived and intrinsic difficulty in examples

Innovation

• Integrates model-perceived and intrinsic difficulty metrics
• Develops difficulty-balanced sampling strategy for prompts
• Constructs ordered prompt curriculum aligned with model capabilities