Understanding Degradation with Vision Language Model

2026-02-04

AI Summary

This work addresses the challenge that vision-language models struggle to comprehend the physical mechanisms underlying image degradation. To this end, it formalizes degradation understanding as a hierarchical structured prediction task that jointly predicts degradation types, associated parameter keys, and their continuous physical values. The authors propose DU-VLM, a multimodal chain-of-thought architecture trained via supervised fine-tuning and reinforcement learning with structured rewards, augmented by quantized grid constraints to mitigate value-space errors. A new physically annotated dataset, DU-110k, comprising 110,000 samples, is introduced to support this framework. Experiments demonstrate that DU-VLM significantly outperforms general-purpose baselines in both accuracy and robustness, exhibits strong out-of-distribution generalization, and functions effectively as a zero-shot controller to guide diffusion models for high-fidelity image restoration.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Understanding visual degradations is a critical yet challenging problem in computer vision. While recent Vision-Language Models (VLMs) excel at qualitative description, they often fall short in understanding the parametric physics underlying image degradations. In this work, we redefine degradation understanding as a hierarchical structured prediction task, necessitating the concurrent estimation of degradation types, parameter keys, and their continuous physical values. Although these sub-tasks operate in disparate spaces, we prove that they can be unified under one autoregressive next-token prediction paradigm, whose error is bounded by the value-space quantization grid. Building on this insight, we introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning using structured rewards. Furthermore, we show that DU-VLM can serve as a zero-shot controller for pre-trained diffusion models, enabling high-fidelity image restoration without fine-tuning the generative backbone. We also introduce \textbf{DU-110k}, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations. Extensive experiments demonstrate that our approach significantly outperforms generalist baselines in both accuracy and robustness, exhibiting generalization to unseen distributions.

Problem

• image degradation
• vision-language models
• physical parameters
• structured prediction
• degradation understanding

Innovation

• Vision-Language Model
• Structured Prediction
• Autoregressive Modeling
• Zero-shot Image Restoration
• Degradation Understanding