OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models

2025-08-28

AI Summary

To address the challenge of assessing goal progress in long-horizon, multi-turn LLM dialogues, this paper proposes a real-time goal alignment tracking framework. Methodologically, it pioneers the use of large language models for dynamic dialogue goal evaluation and interpretable feedback generation, integrated with an interactive visualization interface that enables users to monitor goal completion status, deviation causes, and optimization pathways in real time. Key contributions include: (1) a fine-grained goal alignment quantification mechanism; (2) chain-of-reasoning–based attribution of goal progress and actionable feedback; and (3) a lightweight, low-intrusion human-AI collaborative tracking paradigm. User studies demonstrate that the system significantly reduces goal attainment time (−32.7%) and cognitive load (−28.4%), while enhancing prompt strategy exploration diversity and dialogue resilience.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Conversational AI/Dialog Systems

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

As multi-turn dialogues with large language models (LLMs) grow longer and more complex, how can users better evaluate and review progress on their conversational goals? We present OnGoal, an LLM chat interface that helps users better manage goal progress. OnGoal provides real-time feedback on goal alignment through LLM-assisted evaluation, explanations for evaluation results with examples, and overviews of goal progression over time, enabling users to navigate complex dialogues more effectively. Through a study with 20 participants on a writing task, we evaluate OnGoal against a baseline chat interface without goal tracking. Using OnGoal, participants spent less time and effort to achieve their goals while exploring new prompting strategies to overcome miscommunication, suggesting tracking and visualizing goals can enhance engagement and resilience in LLM dialogues. Our findings inspired design implications for future LLM chat interfaces that improve goal communication, reduce cognitive load, enhance interactivity, and enable feedback to improve LLM performance.

Problem

• Tracking conversational goals in multi-turn LLM dialogues
• Evaluating progress on user goals during complex conversations
• Visualizing goal progression to reduce cognitive load

Innovation

• Real-time goal alignment feedback via LLM evaluation
• Explanation of evaluation results with examples
• Goal progression overviews for effective dialogue navigation