TRACE: Trustworthy Retrieval-Augmented Conversational Engine

2026-08-10

AI Summary

This work addresses the challenge faced by public service chatbots in delivering reliable, constraint-compliant recommendations from noisy and inconsistently structured service directories. The authors propose a retrieval-augmented, constraint-aware dialogue framework that parses user queries into structured and semantic constraints and employs dual data representations—knowledge graphs and non-graph variants—for downstream retrieval. Notably, this approach significantly enhances recommendation reliability without relying on large language model capabilities. Evaluated on both synthetic query benchmarks and real-world food pantry directories using open- and closed-source large language models, the method demonstrates improved constraint satisfaction rates and reduced hallucinated recommendations. Moreover, as retrieval quality increases, performance gaps between models of varying scales diminish, yielding a more robust overall system.

Abstract

Public service chatbots are expected to deliver recommendations from an underlying public service directory, while also making sure that the recommendations respect explicit user constraints. In practice, public service directories are noisy and inconsistent, and general-purpose large language model (LLM) or AI-based chatbots frequently generate unreliable recommendations, citing unverified sources from the web. We investigate the impact of retrieval quality on constraint-aware recommendation in public service conversational systems built over noisy and heterogeneous service directories. We propose TRACE (Trustworthy Retrieval-Augmented Conversational Engine), a retrieval-based, constraint-aware framework that parses input user queries into structural and semantic constraints for downstream retrieval, with the help of a dual data representation schema. Using a curated statewide pantry directory and a synthetic query benchmark, we evaluate multiple knowledge-representation variants with and without knowledge graphs (KGs). We experiment with several open-source LLMs and a proprietary model, showing that strengthening retrieval substantially improves user constraint satisfaction while reducing hallucinated recommendations. Performance differences across LLMs narrowed in our experiments as retrieval quality improved, making results less sensitive to model size. These findings suggest that the quality of retrieval is key for robust public service conversational systems.

Problem

• public service chatbots
• noisy service directories
• constraint-aware recommendation
• retrieval quality
• hallucinated recommendations

Innovation

• retrieval-augmented generation
• constraint-aware recommendation
• public service chatbots
• knowledge graph
• hallucination mitigation