ADSeeker: A Knowledge-Infused Framework for Anomaly Detection and Reasoning

2025-08-05

AI Summary

In industrial visual anomaly detection, multimodal large language models (MLLMs) underperform human experts due to insufficient anomaly knowledge in pretraining and imprecise, context-agnostic generative reasoning. To address this, we propose ADSeeker: a framework that constructs SEEK-M&V—a semantically rich visual-document knowledge base—integrates Q2K, a retrieval-augmented generation mechanism with hierarchical sparse prompting for deep synergy between structured domain knowledge and MLLMs, and leverages MulA, a large-scale, multi-type anomaly dataset enabling robust zero-shot generalization. Experiments demonstrate that ADSeeker achieves state-of-the-art zero-shot anomaly detection performance across multiple benchmarks, delivering high accuracy, strong interpretability, and plug-and-play deployment capability without task-specific fine-tuning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Automatic vision inspection holds significant importance in industry inspection. While multimodal large language models (MLLMs) exhibit strong language understanding capabilities and hold promise for this task, their performance remains significantly inferior to that of human experts. In this context, we identify two key challenges: (i) insufficient integration of anomaly detection (AD) knowledge during pre-training, and (ii) the lack of technically precise and conte-aware language generation for anomaly reasoning. To address these issues, we propose ADSeeker, an anomaly task assistant designed to enhance inspection performance through knowledge-grounded reasoning. ADSeeker leverages a curated visual document knowledge base, SEEK-MVTec&VisA (SEEK-M&V), which we construct to address the limitations of existing resources that rely solely on unstructured text. SEEK-M&V includes semantic-rich descriptions and image-document pairs, enabling more comprehensive anomaly understanding. To effectively retrieve and utilize this knowledge, we introduce the Query Image-Knowledge Retrieval-Augmented Generation (Q2K RAG) framework. To further enhance the performance in zero-shot anomaly detection (ZSAD), ADSeeker leverages the Hierarchical Sparse Prompt mechanism and type-level features to efficiently extract anomaly patterns. Furthermore, to tackle the challenge of limited in industry anomaly detection (IAD) data, we introduce the largest-scale AD dataset, Multi-type Anomaly (MulA), encompassing 72 multi-scale defect types across 26 Categories. Extensive experiments show that our plug-and-play framework, ADSeeker, achieves state-of-the-art zero-shot performance on several benchmark datasets.

Problem

• Insufficient integration of anomaly detection knowledge in pre-training
• Lack of precise context-aware language for anomaly reasoning
• Limited data availability for industrial anomaly detection tasks

Innovation

• Knowledge-infused framework for anomaly detection
• Query Image-Knowledge Retrieval-Augmented Generation
• Hierarchical Sparse Prompt for zero-shot detection