LLM-Mediated Domain-Specific Voice Agents: The Case of TextileBot

2024-06-15

arXiv.org

AI Summary

Developing conversational agents for the textile circular economy is hindered by severe scarcity of domain-specific data. Method: We propose a prompt-driven paradigm to build TextileBot, a voice-based dialogue agent that enhances large language models’ (LLMs) zero-shot domain reasoning via structured domain knowledge embedded directly into prompts—eliminating the need for domain fine-tuning or training data. We design an interpretable, reusable structured prompt template tailored to this niche vertical domain. Contribution/Results: To our knowledge, this is the first empirical validation of a purely prompt-engineering–driven voice agent in a specialized domain. A mixed-method field evaluation with 30 participants—combining quantitative task performance metrics with qualitative analysis of usability and trustworthiness—demonstrates TextileBot’s capability for multi-turn expert dialogue. Significant performance differences across three prompt variants in information-seeking tasks confirm that structured prompt design critically determines interaction quality.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Application Domains: Humanities & Computational Social Science

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Developing domain-specific conversational agents (CAs) has been challenged by the need for extensive domain-focused data. Recent advancements in Large Language Models (LLMs) make them a viable option as a knowledge backbone. LLMs behaviour can be enhanced through prompting, instructing them to perform downstream tasks in a zero-shot fashion (i.e. without training). To this end, we incorporated structural knowledge into prompts and used prompted LLMs to prototyping domain-specific CAs. We demonstrate a case study in a specific domain-textile circularity - TextileBot, we present the design, development, and evaluation of the TextileBot. Specially, we conducted an in-person user study (N=30) with Free Chat and Information-Gathering tasks with TextileBots to gather insights from the interaction. We analyse the human-agent interactions, combining quantitative and qualitative methods. Our results suggest that participants engaged in multi-turn conversations, and their perceptions of the three variation agents and respective interactions varied demonstrating the effectiveness of our prompt-based LLM approach. We discuss the dynamics of these interactions and their implications for designing future voice-based CAs.

Problem

• Developing domain-specific conversational agents
• Enhancing LLMs with structural knowledge
• Evaluating TextileBot through user interactions

Innovation

• LLM as knowledge backbone
• Zero-shot task execution
• Prompt-based domain-specific CAs