Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoning

2025-09-26

AI Summary

Existing visual reasoning methods are typically designed for specific reasoning patterns, limiting their generalizability across diverse tasks and domains. To address this, we propose Mixture-of-Visual-Thoughts (MoVT), the first unified visual reasoning framework supporting context-aware, multi-modal adaptive switching. Our approach introduces two key innovations: (1) a two-stage learning mechanism that jointly models heterogeneous reasoning patterns and dynamically selects the most suitable one per input; and (2) AdaGRPO—a novel algorithm integrating supervised cold-start initialization with reinforcement learning to optimize pattern selection policies. Evaluated across multiple visual reasoning benchmarks—including compositional VQA, spatial reasoning, and counterfactual QA—MoVT achieves significant improvements in both performance stability and cross-scenario generalization. Empirical results demonstrate MoVT’s ability to effectively identify, learn, and contextually invoke appropriate reasoning paths, establishing a new paradigm for adaptive, interpretable visual reasoning.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Current visual reasoning methods mainly focus on exploring specific reasoning modes. Although improvements can be achieved in particular domains, they struggle to develop general reasoning capabilities. Inspired by this, we propose a novel adaptive reasoning paradigm, Mixture-of-Visual-Thoughts (MoVT), which unifies different reasoning modes within a single model and guides it to select the appropriate mode based on context. To achieve this, we introduce AdaVaR, a two-stage Adaptive Visual Reasoning learning framework: different modes are unified and learned during the supervised cold-start stage, and the mode selection capability is induced via an RL process with a carefully designed AdaGRPO algorithm. Extensive experiments show that AdaVaR effectively guides the model to learn and differentiate multiple modes and perform context-adaptive mode selection, achieving consistent improvement across various scenarios, highlighting MoVT as an effective solution for building general visual reasoning models.

Problem

• Unifying multiple reasoning modes in visual reasoning
• Enabling context-adaptive selection of reasoning modes
• Developing general visual reasoning capabilities across scenarios

Innovation

• Unifies multiple reasoning modes in single model
• Uses two-stage learning framework for adaptation
• Performs context-adaptive mode selection via RL