Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support

2025-10-08

AI Summary

Large language model (LLM)-based customer service systems suffer from heavy reliance on offline annotation and prohibitively long iteration cycles (monthly), hindering rapid adaptation to evolving user needs. Method: This paper proposes a closed-loop, agent-cooperative framework driven by real-time human feedback. It integrates four online feedback signals—response preference, agent adoption rationale, knowledge relevance judgment, and missing-knowledge identification—to establish a “data flywheel” enabling weekly model updates. The approach synergistically combines retrieval-augmented generation (RAG), online feedback integration, continual learning, and a multi-dimensional annotation schema to support dynamic model evolution in live operations. Contribution/Results: Empirical evaluation demonstrates significant improvements: retrieval recall@75 increases by 11.7% and precision@8 by 14.8%; response usefulness rises by 8.4%; and agent adoption rate improves by 4.5%. These results validate the feasibility and effectiveness of human-feedback-driven autonomous evolution for LLM-based service systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Agent Communication
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We introduce an Agent-in-the-Loop (AITL) framework that implements a continuous data flywheel for iteratively improving an LLM-based customer support system. Unlike standard offline approaches that rely on batch annotations, AITL integrates four key types of annotations directly into live customer operations: (1) pairwise response preferences, (2) agent adoption and rationales, (3) knowledge relevance checks, and (4) identification of missing knowledge. These feedback signals seamlessly feed back into models' updates, reducing retraining cycles from months to weeks. Our production pilot involving US-based customer support agents demonstrated significant improvements in retrieval accuracy (+11.7% recall@75, +14.8% precision@8), generation quality (+8.4% helpfulness) and agent adoption rates (+4.5%). These results underscore the effectiveness of embedding human feedback loops directly into operational workflows to continuously refine LLM-based customer support system.

Problem

• Implementing continuous data flywheel for LLM customer support
• Integrating live annotations to reduce model retraining cycles
• Improving retrieval accuracy and generation quality via feedback

Innovation

• Agent-in-the-Loop framework enables continuous data flywheel
• Integrates four annotation types into live customer operations
• Reduces model retraining cycles from months to weeks