PestVL-Net: Enabling Multimodal Pest Learning via Fine-grained Vision-Language Interaction

2026-04-19

AI Summary

Existing methods struggle to achieve fine-grained recognition and semantic understanding of agricultural pests due to their diverse morphologies and high species variability in real-world settings. To address this challenge, this work proposes PestVL-Net, the first framework to enable vision–language collaborative learning in the domain of agricultural pest identification. It employs an efficient vision backbone based on the RWKV architecture and introduces a saliency-guided adaptive window partitioning mechanism to capture fine-grained visual features. Furthermore, it integrates a multimodal large language model with structured expert knowledge to generate semantically rich pest descriptions through a multimodal chain-of-thought (CoT) reasoning process. Experimental results demonstrate that the proposed method significantly improves recognition accuracy across multiple pest datasets, highlighting its strong potential for practical deployment in real agricultural scenarios.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Effective pest recognition and management are crucial for sustainable agricultural development. However, collecting pest data in real scenarios is often challenging. Compared to other domains, pests exhibit a wide variety of species with complex and diverse morphological characteristics. Existing techniques struggle to effectively model the key visual and high-level semantic features of pests in a fine-grained manner. These limitations hinder the practical application of such methods in real agricultural scenarios. To address these critical challenges, we present a synergistic approach that integrates PestVL-Net, a novel vision-language framework, with two multi-species pest datasets to facilitate fine-grained pest learning. The visual pathway of PestVL-Net utilizes the Recurrent Weighted Key Value (RWKV) architecture, incorporating a saliency-guided adaptive window partitioning scheme to effectively model the fine-grained visual characteristics of pests. Concurrently, the linguistic component generates precise pest semantic descriptions by leveraging Multimodal Large Language Models (MLLMs) priors, critically informed by agricultural expert knowledge and structured via multimodal Chain-of-Thought (CoT) reasoning. The deep fusion of these complementary visual and textual representations enables fine-grained multimodal pest learning. Extensive experimental evaluations on multiple pest datasets validate the superior performance of PestVL-Net, highlighting its potential for effective real-world pest management.

Problem

• pest recognition
• fine-grained learning
• multimodal learning
• vision-language interaction
• agricultural pest management

Innovation

• fine-grained pest recognition
• vision-language interaction
• RWKV architecture
• multimodal large language models
• Chain-of-Thought reasoning