Agent-based Condition Monitoring Assistance with Multimodal Industrial Database Retrieval Augmented Generation

2025-06-10

AI Summary

To address critical challenges in process industry condition monitoring—namely, heavy reliance on manual expertise for fault severity assessment and maintenance decision-making, high false-alarm rates, and poor interpretability—this paper proposes a large language model (LLM)-driven multimodal reasoning agent system. Methodologically, we introduce the first semi-structured multimodal vector database construction technique tailored for industrial condition monitoring (CM); design a multimodal retrieval-augmented generation (RAG) framework specifically adapted to heterogeneous time-series and textual data; integrate weak supervision with operating-condition-aware mechanisms to build an interpretable and responsive reasoning agent; and establish the first dedicated evaluation framework for industrial CM agents. Experimental results, validated by senior domain analysts, demonstrate significant reductions in false alarms, improved accuracy in fault severity estimation, and enhanced interpretability of maintenance recommendations—enabling efficient, traceable, and closed-loop alarm resolution.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent Communication

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Condition monitoring (CM) plays a crucial role in ensuring reliability and efficiency in the process industry. Although computerised maintenance systems effectively detect and classify faults, tasks like fault severity estimation, and maintenance decisions still largely depend on human expert analysis. The analysis and decision making automatically performed by current systems typically exhibit considerable uncertainty and high false alarm rates, leading to increased workload and reduced efficiency. This work integrates large language model (LLM)-based reasoning agents with CM workflows to address analyst and industry needs, namely reducing false alarms, enhancing fault severity estimation, improving decision support, and offering explainable interfaces. We propose MindRAG, a modular framework combining multimodal retrieval-augmented generation (RAG) with novel vector store structures designed specifically for CM data. The framework leverages existing annotations and maintenance work orders as surrogates for labels in a supervised learning protocol, addressing the common challenge of training predictive models on unlabelled and noisy real-world datasets. The primary contributions include: (1) an approach for structuring industry CM data into a semi-structured multimodal vector store compatible with LLM-driven workflows; (2) developing multimodal RAG techniques tailored for CM data; (3) developing practical reasoning agents capable of addressing real-world CM queries; and (4) presenting an experimental framework for integrating and evaluating such agents in realistic industrial scenarios. Preliminary results, evaluated with the help of an experienced analyst, indicate that MindRAG provide meaningful decision support for more efficient management of alarms, thereby improving the interpretability of CM systems.

Problem

• Reducing false alarms in condition monitoring systems
• Improving fault severity estimation accuracy
• Enhancing decision support with explainable interfaces

Innovation

• LLM-based reasoning agents for CM workflows
• Multimodal RAG with novel vector stores
• Structuring CM data for LLM-driven workflows