Vision Matters: Simple Visual Perturbations Can Boost Multimodal Math Reasoning

2025-06-11

AI Summary

Current multimodal large language models (MLLMs) suffer from a critical decoupling between visual representation and reasoning processes, leading to insufficient integration of image information in mathematical reasoning. To address this, we propose a lightweight, plug-and-play visual perturbation framework that requires no retraining. Our method introduces three novel, annotation-free perturbation strategies: distractor concatenation, dominance-preserving mixup, and random rotation—designed to enhance perceptual robustness and reasoning consistency over visual content. The framework is fully compatible with open-source MLLMs (e.g., Qwen2.5-VL-7B) and integrates seamlessly via standard post-training paradigms such as SFT, DPO, or GRPO, without architectural modifications. Extensive experiments demonstrate consistent performance gains across multiple mathematical reasoning benchmarks, matching the efficacy of algorithm-level improvements. Crucially, our work provides the first empirical evidence that visual representation quality fundamentally constrains the upper bound of multimodal mathematical reasoning capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Visual Reasoning & Symbolic Representations
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Despite the rapid progress of multimodal large language models (MLLMs), they have largely overlooked the importance of visual processing. In a simple yet revealing experiment, we interestingly find that language-only models, when provided with image captions, can achieve comparable or even better performance than MLLMs that consume raw visual inputs. This suggests that current MLLMs may generate accurate visual descriptions but fail to effectively integrate them during reasoning. Motivated by this, we propose a simple visual perturbation framework that enhances perceptual robustness without requiring algorithmic modifications or additional training data. Our approach introduces three targeted perturbations: distractor concatenation, dominance-preserving mixup, and random rotation, that can be easily integrated into existing post-training pipelines including SFT, DPO, and GRPO. Through extensive experiments across multiple datasets, we demonstrate consistent improvements in mathematical reasoning performance, with gains comparable to those achieved through algorithmic changes. Additionally, we achieve competitive performance among open-source 7B RL-tuned models by training Qwen2.5-VL-7B with visual perturbation. Through comprehensive ablation studies, we analyze the effectiveness of different perturbation strategies, revealing that each perturbation type contributes uniquely to different aspects of visual reasoning. Our findings highlight the critical role of visual perturbation in multimodal mathematical reasoning: better reasoning begins with better seeing. Our code is available at https://github.com/YutingLi0606/Vision-Matters.

Problem

• MLLMs overlook visual processing importance in reasoning
• Current MLLMs fail to integrate visual descriptions effectively
• Visual perturbation enhances multimodal math reasoning performance

Innovation

• Simple visual perturbation framework enhances robustness
• Three targeted perturbations improve visual reasoning
• No algorithm changes or extra data needed