Beyond Final Answers: Auditing Trajectory-Level Hallucinations in Multi-Agent Industrial Workflows

2026-05-22

AI Summary

This work addresses the limitation of existing hallucination evaluation methods, which predominantly focus on final outputs while overlooking hallucinations in intermediate reasoning trajectories within multi-agent industrial workflows. To bridge this gap, the authors introduce the Trajel dataset and an accompanying evaluation framework, leveraging expert-annotated trajectories from AssetOpsBench to establish a five-dimensional taxonomy of hallucinations—encompassing factuality, referentiality, logicality, procedural coherence, and scope adherence—enabling, for the first time, fine-grained identification of concurrent, multi-type trajectory-level hallucinations. By deploying supervised detection models across subtask, trajectory, and long-context levels, the study reveals that nearly half of hallucinated trajectories exhibit multiple hallucination types, a complexity largely missed by current benchmarks; notably, trajectory-aware detection substantially outperforms conventional post-hoc verification approaches.

Abstract

Large Language Models (LLMs) are increasingly deployed as autonomous agents that reason, use tools, and act over multiple steps. Yet most hallucination benchmarks still evaluate only the final output, missing failures that originate in intermediate Thought-Action-Observation steps. We present Trajel, a dataset and evaluation framework for auditing trajectory-level hallucinations in multi-agent industrial workflows. Trajel introduces a five-type hallucination taxonomy (factual, referential, logical, procedural, and scope-based) over expert-annotated agent traces from AssetOpsBench. We benchmark supervised detection models at the subtask, trajectory, and long-context levels. Our results show that the most common failure modes are missed by existing benchmarks, that nearly half of hallucinated trajectories involve multiple types at once, and that automated detectors with high binary accuracy still misclassify the subtlest types. Trajectory-aware detection significantly outperforms standard post-hoc verification, making taxonomy-grounded evaluation necessary for safer agentic deployment.

Problem

• hallucination
• multi-agent
• trajectory-level
• industrial workflows
• LLMs

Innovation

• trajectory-level hallucination
• multi-agent workflows
• hallucination taxonomy
• LLM agent auditing
• Trajel