JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees

2026-03-24

AI Summary

This work addresses the challenge that large language models (LLMs) cannot directly process fault trees represented as images by proposing a novel textual representation method that converts fault trees into structured, LLM-parsable text for the first time. Leveraging this representation, the authors construct a high-quality benchmark dataset comprising 3,130 multi-turn dialogues with an average of 40.75 turns per conversation, incorporating a long-range error rollback and recovery mechanism to simulate real-world user misoperations during interactive troubleshooting. Experimental results demonstrate that the proposed approach effectively enables LLMs to perform fault localization and tracing in complex systems. Among evaluated models, Gemini 2.5 Pro achieves the best performance on this benchmark, validating the effectiveness of the framework in enhancing the robustness of interactive diagnostic capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

In the maintenance of complex systems, fault trees are used to locate problems and provide targeted solutions. To enable fault trees stored as images to be directly processed by large language models, which can assist in tracking and analyzing malfunctions, we propose a novel textual representation of fault trees. Building on it, we construct a benchmark for multi-turn dialogue systems that emphasizes robust interaction in complex environments, evaluating a model's ability to assist in malfunction localization, which contains $3130$ entries and $40.75$ turns per entry on average. We train an end-to-end model to generate vague information to reflect user behavior and introduce long-range rollback and recovery procedures to simulate user error scenarios, enabling assessment of a model's integrated capabilities in task tracking and error recovery, and Gemini 2.5 pro archives the best performance.

Problem

• fault tree
• large language model
• malfunction analysis
• multi-turn dialogue
• error recovery

Innovation

• fault tree representation
• multi-turn dialogue benchmark
• LLM-based malfunction analysis
• error recovery simulation
• vague user behavior modeling