TRACER: Early Failure Detection for Task-Oriented Dialogue

2026-07-04

AI Summary

This work addresses the limitation of existing task-oriented dialogue system evaluation methods, which typically intervene only after clear failures have occurred, thereby hindering early mitigation. To enable timely intervention, the authors propose a multi-granularity early failure prediction approach that jointly models dialogue state utterances and belief state transition trajectories for the first time. Leveraging a dual-stream architecture to fuse these complementary signals, the method effectively identifies failure risk during the 25%–75% progress window of a dialogue—well before completion. The approach demonstrates consistent effectiveness under both real and generated belief states, significantly outperforming heuristic, classical, and single-stream baselines across multiple dialogue progress points. By providing actionable early warnings, it opens a critical window for system self-repair and proactive recovery.

Abstract

Task-oriented dialogue systems often fail before the final breakdown is obvious, but most evaluation only measures failure after the conversation has already gone wrong. We present TRACER, a method for early failure detection in task-oriented dialogue. TRACER predicts from a partial dialogue whether the full conversation will eventually fail by combining simple trajectory signals from belief-state changes with text representations of the evolving dialogue state. We evaluate the method in both oracle and generated belief-state settings, and test how well it works when only 25%, 50%, 75%, or 100% of the dialogue is visible. Across these settings, TRACER detects useful failure signals well before the end of the conversation and outperforms heuristic, classical, and single-stream baselines. These results suggest that early failure detection can provide a practical warning signal for dialogue systems before the interaction fully breaks down.

Problem

• task-oriented dialogue
• early failure detection
• dialogue system failure
• belief-state changes
• dialogue evaluation

Innovation

• early failure detection
• task-oriented dialogue
• belief-state trajectory
• dialogue state representation
• TRACER