VR-RAG: Open-vocabulary Species Recognition with RAG-Assisted Large Multi-Modal Models

2025-05-08

AI Summary

Open-vocabulary bird species identification—zero-shot recognition of newly discovered species from textual descriptions without predefined class constraints—remains challenging due to semantic gaps between natural language and visual representations. Method: We propose Visual Re-ranking Retrieval-Augmented Generation (VR-RAG), a novel framework that leverages visual similarity to dynamically retrieve and re-rank knowledge from structured Wikipedia text descriptions of 11,202 bird species, jointly with multimodal vision-language embeddings from Qwen2.5-VL. We further enhance semantic fidelity via GPT-4o-distilled descriptive summaries. Contribution/Results: VR-RAG achieves an average 15.4% performance gain over Qwen2.5-VL across five benchmarks, substantially outperforming existing vision-language models. It establishes a scalable, generalizable open-set recognition paradigm for biodiversity monitoring and continuous species discovery.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Open-vocabulary recognition remains a challenging problem in computer vision, as it requires identifying objects from an unbounded set of categories. This is particularly relevant in nature, where new species are discovered every year. In this work, we focus on open-vocabulary bird species recognition, where the goal is to classify species based on their descriptions without being constrained to a predefined set of taxonomic categories. Traditional benchmarks like CUB-200-2011 and Birdsnap have been evaluated in a closed-vocabulary paradigm, limiting their applicability to real-world scenarios where novel species continually emerge. We show that the performance of current systems when evaluated under settings closely aligned with open-vocabulary drops by a huge margin. To address this gap, we propose a scalable framework integrating structured textual knowledge from Wikipedia articles of 11,202 bird species distilled via GPT-4o into concise, discriminative summaries. We propose Visual Re-ranking Retrieval-Augmented Generation(VR-RAG), a novel, retrieval-augmented generation framework that uses visual similarities to rerank the top m candidates retrieved by a set of multimodal vision language encoders. This allows for the recognition of unseen taxa. Extensive experiments across five established classification benchmarks show that our approach is highly effective. By integrating VR-RAG, we improve the average performance of state-of-the-art Large Multi-Modal Model QWEN2.5-VL by 15.4% across five benchmarks. Our approach outperforms conventional VLM-based approaches, which struggle with unseen species. By bridging the gap between encyclopedic knowledge and visual recognition, our work advances open-vocabulary recognition, offering a flexible, scalable solution for biodiversity monitoring and ecological research.

Problem

• Open-vocabulary bird species recognition from descriptions
• Overcoming performance drop in open-vocabulary settings
• Integrating textual knowledge for unseen taxa recognition

Innovation

• Integrates GPT-4 distilled Wikipedia bird summaries
• Uses visual re-ranking in retrieval-augmented generation
• Combines multimodal encoders for unseen taxa recognition