MERLION: Marine ExploRation with Language guIded Online iNformative Visual Sampling and Enhancement

2025-03-10

AI Summary

To address bandwidth constraints, limited onboard storage, and inefficient manual frame selection in autonomous underwater vehicle (AUV) visual monitoring under turbid conditions, this paper proposes a language-guided end-to-end keyframe sampling framework. Methodologically, it introduces the first integration of language-driven semantic alignment, physics-inspired turbid image enhancement, and online informativeness evaluation—implemented via a multimodal image–text alignment model, a physically grounded enhancement network, and a dynamic sampler jointly optimizing entropy and semantic relevance. The framework directly maps user-specified natural language intents to compact, high-informativeness underwater visual summaries. Evaluated on a real-world AUV dataset, our approach achieves a 32% improvement in summary semantic relevance and a 27% increase in keyframe recall over state-of-the-art methods, significantly enhancing targeted perception efficiency and interpretability in low-visibility underwater environments.

Technology Category

• Computer Vision: Language and Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Summarization

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Autonomous and targeted underwater visual monitoring and exploration using Autonomous Underwater Vehicles (AUVs) can be a challenging task due to both online and offline constraints. The online constraints comprise limited onboard storage capacity and communication bandwidth to the surface, whereas the offline constraints entail the time and effort required for the selection of desired key frames from the video data. An example use case of targeted underwater visual monitoring is finding the most interesting visual frames of fish in a long sequence of an AUV's visual experience. This challenge of targeted informative sampling is further aggravated in murky waters with poor visibility. In this paper, we present MERLION, a novel framework that provides semantically aligned and visually enhanced summaries for murky underwater marine environment monitoring and exploration. Specifically, our framework integrates (a) an image-text model for semantically aligning the visual samples to the users' needs, (b) an image enhancement model for murky water visual data and (c) an informative sampler for summarizing the monitoring experience. We validate our proposed MERLION framework on real-world data with user studies and present qualitative and quantitative results using our evaluation metric and show improved results compared to the state-of-the-art approaches. We have open-sourced the code for MERLION at the following link https://github.com/MARVL-Lab/MERLION.git.

Problem

• Autonomous underwater visual monitoring with limited resources.
• Semantically aligning visual samples to user needs.
• Enhancing and summarizing murky underwater visual data.

Innovation

• Image-text model for semantic alignment
• Image enhancement for murky water visuals
• Informative sampler for summarization