When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning

2026-03-22

AI Summary

This work proposes an unsupervised self-evolution training framework for multimodal large language models, enabling continuous improvement in reasoning capabilities without reliance on costly human annotations or external teacher models. By sampling multiple reasoning trajectories, the method dynamically reweights trajectory quality through intra-group relative advantage modeling and bounded critic modulation, integrated with a self-consistency prior and group distribution modeling. Evaluated across five mathematical reasoning benchmarks, the approach demonstrates significant and consistent gains in both performance and generalization, establishing its effectiveness and scalability for annotation-free multimodal reasoning optimization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent progress in multimodal large language models has led to strong performance on reasoning tasks, but these improvements largely rely on high-quality annotated data or teacher-model distillation, both of which are costly and difficult to scale.To address this, we propose an unsupervised self-evolution training framework for multimodal reasoning that achieves stable performance improvements without using human-annotated answers or external reward models. For each input, we sample multiple reasoning trajectories and jointly model their within group structure.We use the Actor's self-consistency signal as a training prior, and introduce a bounded Judge based modulation to continuously reweight trajectories of different quality.We further model the modulated scores as a group level distribution and convert absolute scores into relative advantages within each group, enabling more robust policy updates. Trained with Group Relative Policy Optimization (GRPO) on unlabeled data, our method consistently improves reasoning performance and generalization on five mathematical reasoning benchmarks, offering a scalable path toward self-evolving multimodal models.The code are available at https://dingwu1021.github.io/SelfJudge/.

Problem

• multimodal reasoning
• annotated data
• teacher-model distillation
• scalability
• unsupervised learning

Innovation

• unsupervised self-evolution
• multimodal reasoning
• self-consistency
• Group Relative Policy Optimization
• trajectory reweighting