Specificity-aware reinforcement learning for fine-grained open-world classification

2026-03-03

AI Summary

This work addresses the challenge in open-world fine-grained image classification where existing models often produce overly generalized predictions, failing to balance accuracy and specificity. To tackle this issue, the authors propose SpeciaRL, a novel framework that introduces a specificity-aware mechanism into reinforcement learning for the first time. By leveraging a verifier-driven dynamic reward signal, SpeciaRL guides large multimodal language models (LMMs) to adaptively balance correctness and concreteness during online rollouts. This approach effectively activates the model’s intrinsic fine-grained knowledge, mitigating both over-generalization and erroneous predictions. Extensive experiments demonstrate that SpeciaRL achieves state-of-the-art performance across multiple benchmarks, establishing a new Pareto-optimal trade-off between classification accuracy and prediction specificity.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Classifying fine-grained visual concepts under open-world settings, i.e., without a predefined label set, demands models to be both accurate and specific. Recent reasoning Large Multimodal Models (LMMs) exhibit strong visual understanding capability but tend to produce overly generic predictions when performing fine-grained image classification. Our preliminary analysis reveals that models do possess the intrinsic fine-grained domain knowledge. However, promoting more specific predictions (specificity) without compromising correct ones (correctness) remains a non-trivial and understudied challenge. In this work, we investigate how to steer reasoning LMMs toward predictions that are both correct and specific. We propose a novel specificity-aware reinforcement learning framework, SpeciaRL, to fine-tune reasoning LMMs on fine-grained image classification under the open-world setting. SpeciaRL introduces a dynamic, verifier-based reward signal anchored to the best predictions within online rollouts, promoting specificity while respecting the model's capabilities to prevent incorrect predictions. Our out-of-domain experiments show that SpeciaRL delivers the best trade-off between correctness and specificity across extensive fine-grained benchmarks, surpassing existing methods and advancing open-world fine-grained image classification. Code and model are publicly available at https://github.com/s-angheben/SpeciaRL.

Problem

• fine-grained classification
• open-world setting
• specificity
• correctness
• large multimodal models

Innovation

• specificity-aware reinforcement learning
• fine-grained classification
• open-world recognition
• Large Multimodal Models
• dynamic reward