Highlight All the Phrases: Enhancing LLM Transparency through Visual Factuality Indicators

2025-08-09

AI Summary

Large language models (LLMs) frequently generate factually incorrect outputs (“hallucinations”), yet existing research lacks effective interactive mechanisms to communicate response veracity to users. To address this, we propose a phrase-level factuality visualization method: leveraging fine-grained factual verification results, each phrase in the model’s output is color-coded to enable end-to-end, transparent factuality assessment. Through two controlled user studies involving 208 participants, we demonstrate that our design significantly improves user verification efficiency (+31.2%), trust calibration, and interface preference—outperforming both unannotated and coarse-grained annotation baselines. This work constitutes the first systematic investigation into the alignment between visualization granularity of factual feedback and human cognitive processing, thereby bridging a critical gap in explainable AI interaction design for LLMs. It provides empirically validated, reproducible, and deployable guidelines for building trustworthy AI interfaces.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Humans and AI: Intelligent User Interfaces
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Large language models (LLMs) are susceptible to generating inaccurate or false information, often referred to as "hallucinations" or "confabulations." While several technical advancements have been made to detect hallucinated content by assessing the factuality of the model's responses, there is still limited research on how to effectively communicate this information to users. To address this gap, we conducted two scenario-based experiments with a total of 208 participants to systematically compare the effects of various design strategies for communicating factuality scores by assessing participants' ratings of trust, ease in validating response accuracy, and preference. Our findings reveal that participants preferred and trusted a design in which all phrases within a response were color-coded based on factuality scores. Participants also found it easier to validate accuracy of the response in this style compared to a baseline with no style applied. Our study offers practical design guidelines for LLM application developers and designers, aimed at calibrating user trust, aligning with user preferences, and enhancing users' ability to scrutinize LLM outputs.

Problem

• Detecting and communicating LLM hallucinated content effectively
• Comparing design strategies for factuality score visualization
• Enhancing user trust and accuracy validation in LLM outputs

Innovation

• Color-coding phrases by factuality scores
• Enhancing user trust through visual indicators
• Improving accuracy validation with design strategies