Towards Preventing Overreliance on Task-Oriented Conversational AI Through Accountability Modeling

2025-01-17

AI Summary

Task-oriented dialogues suffer from user overreliance on AI and model hallucination, leading to erroneous slot value acceptance and reduced system trustworthiness. Method: We propose an accountability modeling mechanism that actively triggers friction-based interactions—such as confirmation requests and explanations—when dialogue state tracking (DST) exhibits uncertainty or error. A lightweight “accountability head”, trained as a binary classifier, operates alongside the LLM decoder to detect slot-level errors in real time; crucially, it explicitly maps model uncertainty to executable interaction policies. We implement a multi-task joint fine-tuning framework atop open-source LLMs (Llama, Mistral, Gemma), jointly optimizing DST supervision and accountability classification objectives. Contribution/Results: On MultiWOZ, our method improves joint goal accuracy by 3 percentage points; integrating self-correction yields an additional 3-point gain. Empirical evaluation demonstrates a significant reduction in user error acceptance rate, enhancing human-AI collaboration reliability and system credibility.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent LLMs have enabled significant advancements for conversational agents. However, they are also well-known to hallucinate, i.e., they often produce responses that seem plausible but are not factually correct. On the other hand, users tend to over-rely on LLM-based AI agents; they accept the AI's suggestion even when it is wrong. Adding good friction, such as explanations or getting user confirmations, has been proposed as a mitigation in AI-supported decision-making systems. In this paper, we propose an accountability model for LLM-based task-oriented dialogue agents to address user overreliance via friction turns in cases of model uncertainty and errors associated with dialogue state tracking (DST). The accountability model is an augmented LLM with an additional accountability head, which functions as a binary classifier to predict the slots of the dialogue states. We perform our experiments with three backbone LLMs (Llama, Mistral, Gemma) on two established task-oriented datasets (MultiWOZ and Snips). Our empirical findings demonstrate that this approach not only enables reliable estimation of AI agent errors but also guides the LLM decoder in generating more accurate actions. We observe around 3% absolute improvement in joint goal accuracy by incorporating accountability heads in modern LLMs for the MultiWOZ dataset. We also show that this method enables the agent to self-correct its actions, further boosting its performance by 3%. Finally, we discuss the application of accountability modeling to prevent user overreliance by introducing friction.

Problem

• Over-reliance
• Trust
• Error Handling

Innovation

• Responsibility Modeling
• AI Accuracy Improvement
• User Over-reliance Reduction