Can Textual Reasoning Improve the Performance of MLLMs on Fine-grained Visual Classification?

2026-01-11

arXiv.org

AI Summary

This work addresses the challenge that multimodal large language models (MLLMs) struggle to capture fine-grained visual distinctions in fine-grained visual classification (FGVC) tasks, and that over-elaborated chain-of-thought (CoT) reasoning often degrades accuracy—a phenomenon revealing the "cost of thinking" in perceptual tasks. To mitigate this, the authors propose the ReFine-RFT framework, which constrains reasoning length and optimizes model behavior through a fusion of multiple reward signals. The framework further introduces a general-purpose multi-reward normalization algorithm and an integrated reward mechanism. Extensive experiments demonstrate that ReFine-RFT achieves state-of-the-art performance across multiple FGVC benchmarks under diverse training paradigms and zero-shot settings, confirming its effectiveness and broad applicability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multi-modal large language models (MLLMs) exhibit strong general-purpose capabilities, yet still struggle on Fine-Grained Visual Classification (FGVC), a core perception task that requires subtle visual discrimination and is crucial for many real-world applications. A widely adopted strategy for boosting performance on challenging tasks such as math and coding is Chain-of-Thought (CoT) reasoning. However, several prior works have reported that CoT can actually harm performance on visual perception tasks. These studies, though, examine the issue from relatively narrow angles and leave open why CoT degrades perception-heavy performance. We systematically re-examine the role of CoT in FGVC through the lenses of zero-shot evaluation and multiple training paradigms. Across these settings, we uncover a central paradox: the degradation induced by CoT is largely driven by the reasoning length, in which longer textual reasoning consistently lowers classification accuracy. We term this phenomenon the ``Cost of Thinking''. Building on this finding, we make two key contributions: (1) \alg, a simple and general plug-and-play normalization method for multi-reward optimization that balances heterogeneous reward signals, and (2) ReFine-RFT, a framework that combines ensemble rewards with \alg to constrain reasoning length while providing dense accuracy-oriented feedback. Extensive experiments demonstrate the effectiveness of our findings and the proposed ReFine-RFT, achieving state-of-the-art performance across FGVC benchmarks. Code and models are available at \href{https://github.com/jiezhu23/ReFine-RFT}{Project Link}.

Problem

• Fine-Grained Visual Classification
• Multi-modal Large Language Models
• Chain-of-Thought Reasoning
• Visual Perception
• Reasoning Length

Innovation

• Chain-of-Thought
• Fine-Grained Visual Classification
• Multi-modal LLMs
• Reward Normalization
• Reasoning Length