Neither Replacement nor Panacea: Comparing LLM-Based Conversational and Graphical Decision Support in Industrial Tasks

2026-05-29

AI Summary

This study addresses the growing challenge of extracting critical decision-making insights from increasingly complex industrial operational data. Through a 2×3 mixed-factor experiment, it compares a large language model–driven conversational user interface (CUI) against traditional graphical dashboards in manufacturing decision support, evaluating how task complexity influences cognitive load, decision accuracy, and completion time. Results indicate that the CUI significantly reduces perceived cognitive load and improves efficiency for low-complexity tasks, though these advantages diminish as task complexity increases. No significant differences in decision accuracy were observed between the two interface types, and users consistently preferred integrating visual analytics tools for subsequent decision stages. The findings reveal that conversational interaction offers conditional benefits in industrial contexts rather than serving as a universal replacement for conventional visualization-based interfaces.

Abstract

Managers in manufacturing settings rely on digital interfaces to interpret operational data for decision-making, but growing data volume and complexity can make relevant insights difficult to identify efficiently. While dashboards remain dominant in industrial contexts, Large Language Model (LLM)-based conversational agents (CAs), accessed through conversational user interfaces (CUIs), may provide more direct access to such data. However, their effectiveness may depend on the information-processing demands of the task. This study compares an LLM-based CA delivered through a CUI with a dashboard in a manufacturing decision-support scenario. In a mixed factorial experiment with a 2x3 design, 134 industrial decision-makers were assigned to one interface condition and completed three tasks of increasing complexity. We examined perceived Mental Workload (MWL), decision accuracy, completion time, and intended reliance, and tested self-reported data literacy as a moderator. Results showed that the CUI reduced perceived MWL overall and supported faster completion in less demanding tasks, but both advantages diminished as task complexity increased. Neither interface produced a consistent overall advantage in decision accuracy, and the CUI was not preferred as a sole basis for subsequent decisions. Furthermore, data literacy did not reliably moderate interface effects. These findings indicate that conversational interaction offers conditional rather than universal benefits for industrial decision support. LLM-based CAs may reduce information-access effort, whereas complex decisions continue to benefit from persistent, inspectable visual representations.

Problem

• decision support
• conversational agents
• dashboards
• task complexity
• manufacturing

Innovation

• LLM-based conversational agent
• conversational user interface (CUI)
• industrial decision support
• task complexity
• mental workload