From Clicks to Conversations: Evaluating the Effectiveness of Conversational Agents in Statistical Analysis

2025-02-12

AI Summary

This paper addresses the high usability barrier and unnatural interaction paradigms of traditional GUI-based statistical software (e.g., SPSS, SAS) for non-expert users. To this end, we propose and implement StatZ—a conversational intelligent agent tailored for statistical analysis. Methodologically, StatZ integrates large language models with domain-specific statistical knowledge and employs a task-oriented dialogue management framework that supports natural language understanding, analytical intent recognition, and generation of interpretable, actionable results. Its key contribution is the first systematic empirical validation demonstrating statistically significant advantages of conversational interaction over mainstream GUI tools in statistical analysis tasks. A user study with 51 participants from diverse backgrounds shows that StatZ significantly improves task completion time (−37%), analysis accuracy (+29%), and user experience and satisfaction (+44%), all at *p* < 0.01. These results confirm StatZ’s effectiveness and broad applicability across user populations.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search

Abstract

The rapid proliferation of data science forced different groups of individuals with different backgrounds to adapt to statistical analysis. We hypothesize that conversational agents are better suited for statistical analysis than traditional graphical user interfaces (GUI). In this work, we propose a novel conversational agent, StatZ, for statistical analysis. We evaluate the efficacy of StatZ relative to established statistical software:SPSS, SAS, Stata, and JMP in terms of accuracy, task completion time, user experience, and user satisfaction. We combined the proposed analysis question from state-of-the-art language models with suggestions from statistical analysis experts and tested with 51 participants from diverse backgrounds. Our experimental design assessed each participant's ability to perform statistical analysis tasks using traditional statistical analysis tools with GUI and our conversational agent. Results indicate that the proposed conversational agents significantly outperform GUI statistical software in all assessed metrics, including quantitative (task completion time, accuracy, and user experience), and qualitative (user satisfaction) metrics. Our findings underscore the potential of using conversational agents to enhance statistical analysis processes, reducing cognitive load and learning curves and thereby proliferating data analysis capabilities, to individuals with limited knowledge of statistics.

Problem

• Evaluate conversational agents' effectiveness in statistical analysis
• Compare StatZ with traditional GUI statistical software
• Assess task accuracy, time, user experience, and satisfaction

Innovation

• Conversational agent for statistics
• Combines AI with expert input
• Outperforms traditional GUI tools