UVLM: Benchmarking Video Language Model for Underwater World Understanding

2025-07-03

AI Summary

The absence of a dedicated benchmark for underwater video-language understanding hinders progress in this domain. Method: This paper introduces UVLM—the first video-language model evaluation benchmark specifically designed for underwater environments. UVLM systematically models underwater-specific challenges, including illumination distortion, water turbidity, and multi-view variability, covering 419 marine species and diverse static scenes. It defines 20 structured observational tasks across two categories: biological identification and environmental analysis. The dataset integrates expert knowledge with AI-assisted annotation, employs fine-grained prompt engineering, and adopts rigorous multi-dimensional evaluation metrics to support fine-grained semantic understanding and dynamic change reasoning. Contribution/Results: Fine-tuning mainstream vision-language models (VLMs) on UVLM significantly improves their underwater scene comprehension. Moreover, such fine-tuned models demonstrate transferable performance gains on general-purpose benchmarks like VideoMME, validating UVLM’s generalizability and its potential to advance VLM development for extreme visual environments.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Recently, the remarkable success of large language models (LLMs) has achieved a profound impact on the field of artificial intelligence. Numerous advanced works based on LLMs have been proposed and applied in various scenarios. Among them, video language models (VidLMs) are particularly widely used. However, existing works primarily focus on terrestrial scenarios, overlooking the highly demanding application needs of underwater observation. To overcome this gap, we introduce UVLM, an under water observation benchmark which is build through a collaborative approach combining human expertise and AI models. To ensure data quality, we have conducted in-depth considerations from multiple perspectives. First, to address the unique challenges of underwater environments, we selected videos that represent typical underwater challenges including light variations, water turbidity, and diverse viewing angles to construct the dataset. Second, to ensure data diversity, the dataset covers a wide range of frame rates, resolutions, 419 classes of marine animals, and various static plants and terrains. Next, for task diversity, we adopted a structured design where observation targets are categorized into two major classes: biological and environmental. Each category includes content observation and change/action observation, totaling 20 distinct task types. Finally, we designed several challenging evaluation metrics to enable quantitative comparison and analysis of different methods. Experiments on two representative VidLMs demonstrate that fine-tuning VidLMs on UVLM significantly improves underwater world understanding while also showing potential for slight improvements on existing in-air VidLM benchmarks, such as VideoMME and Perception text. The dataset and prompt engineering will be released publicly.

Problem

• Addressing lack of underwater-focused video language models
• Creating diverse dataset for underwater observation challenges
• Enhancing VidLM performance in underwater environments

Innovation

• Collaborative human-AI dataset construction for underwater videos
• Diverse underwater challenges and marine life coverage
• Structured task design with 20 distinct types