AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimization

2025-08-06

AI Summary

Multimodal large language models (MLLMs) exhibit weak domain adaptability, low training data efficiency, and unsupervised inference in industrial anomaly detection. Method: We propose a novel approach based on multi-stage progressive reasoning and fine-grained reward optimization. It employs a structured, three-stage reasoning pipeline—region localization → defect analysis → binary discrimination—to enhance response diversity and analytical depth. A continuous reward signal, jointly measuring classification accuracy and localization precision, is integrated with Group Relative Policy Optimization (GRPO) to enable precise, stepwise control over the reasoning chain. Contribution/Results: Evaluated across multiple industrial benchmarks, our method significantly outperforms state-of-the-art approaches, achieving superior detection accuracy under limited annotation budgets. It effectively bridges the gap between general-purpose MLLMs and domain-specific visual discrimination capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

While Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities across diverse domains, their application to specialized anomaly detection (AD) remains constrained by domain adaptation challenges. Existing Group Relative Policy Optimization (GRPO) based approaches suffer from two critical limitations: inadequate training data utilization when models produce uniform responses, and insufficient supervision over reasoning processes that encourage immediate binary decisions without deliberative analysis. We propose a comprehensive framework addressing these limitations through two synergistic innovations. First, we introduce a multi-stage deliberative reasoning process that guides models from region identification to focused examination, generating diverse response patterns essential for GRPO optimization while enabling structured supervision over analytical workflows. Second, we develop a fine-grained reward mechanism incorporating classification accuracy and localization supervision, transforming binary feedback into continuous signals that distinguish genuine analytical insight from spurious correctness. Comprehensive evaluation across multiple industrial datasets demonstrates substantial performance improvements in adapting general vision-language models to specialized anomaly detection. Our method achieves superior accuracy with efficient adaptation of existing annotations, effectively bridging the gap between general-purpose MLLM capabilities and the fine-grained visual discrimination required for detecting subtle manufacturing defects and structural irregularities.

Problem

• Adapting MLLMs to specialized anomaly detection with domain challenges
• Overcoming uniform responses and lack of reasoning supervision in GRPO
• Improving accuracy in detecting subtle defects with structured reasoning

Innovation

• Multi-stage reasoning for structured anomaly analysis
• Fine-grained reward mechanism for continuous feedback
• Efficient adaptation of general vision-language models