AUTV: Creating Underwater Video Datasets with Pixel-wise Annotations

2025-03-17

AI Summary

Underwater video analysis faces severe challenges due to dynamic illumination, camera motion, and scarce annotated data; existing training-free inter-frame modeling approaches often yield motion discontinuities and misalignment. This paper introduces the first end-to-end underwater video synthesis framework, integrating physics-driven imaging modeling, motion-consistent generative networks, and cross-modal video–text alignment. Concurrently, we construct dual-track benchmark datasets: the real-world Underwater Text-Video (UTV) dataset comprising 2,000 high-quality video–text pairs, and the synthetic Underwater Text-Video (SUTV) dataset containing 10,000 videos with fine-grained semantic segmentation masks and multi-dimensional physical attribute annotations (e.g., scattering coefficients, depth maps). Experiments demonstrate substantial performance gains on underwater video inpainting and video object segmentation tasks. Our work establishes a new benchmark and technical paradigm for underwater visual understanding.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Web Mining and Content Analysis: Web data generation and simulation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT

Abstract

Underwater video analysis, hampered by the dynamic marine environment and camera motion, remains a challenging task in computer vision. Existing training-free video generation techniques, learning motion dynamics on the frame-by-frame basis, often produce poor results with noticeable motion interruptions and misaligments. To address these issues, we propose AUTV, a framework for synthesizing marine video data with pixel-wise annotations. We demonstrate the effectiveness of this framework by constructing two video datasets, namely UTV, a real-world dataset comprising 2,000 video-text pairs, and SUTV, a synthetic video dataset including 10,000 videos with segmentation masks for marine objects. UTV provides diverse underwater videos with comprehensive annotations including appearance, texture, camera intrinsics, lighting, and animal behavior. SUTV can be used to improve underwater downstream tasks, which are demonstrated in video inpainting and video object segmentation.

Problem

• Challenges in underwater video analysis due to dynamic environments and camera motion.
• Poor results from existing training-free video generation techniques.
• Need for pixel-wise annotated marine video datasets for improved analysis.

Innovation

• AUTV synthesizes marine video with pixel-wise annotations
• UTV dataset includes 2000 annotated video-text pairs
• SUTV dataset offers 10000 synthetic videos with segmentation masks