Enhancing Customer Service Chatbots with Context-Aware NLU through Selective Attention and Multi-task Learning

2025-06-02

AI Summary

In e-commerce customer service, intent classification accuracy is low when relying solely on customer queries, particularly for ambiguous expressions (e.g., “I haven’t received my package”), due to the lack of contextual grounding. Method: This paper proposes a context-aware natural language understanding (NLU) model that explicitly and scalably integrates structured contextual features—such as order status—into the NLU pipeline. We design a selective attention module to dynamically weight contextual signals and develop a Transformer-based multi-task learning framework that jointly optimizes coarse-grained intents and fine-grained operational labels. Contribution/Results: Deployed in Walmart’s production environment, our model achieves a 4.8% absolute improvement in top-2 accuracy over query-only baselines and outperforms the current state-of-the-art by 3.5%. It reduces annual human escalation costs by over one million RMB, demonstrating both technical efficacy and operational impact.

Technology Category

• Natural Language Processing: Safety and Robustness
• Data Mining & Knowledge Management: Intelligent Query Processing
• Machine Learning: Structured Learning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Customer service chatbots are conversational systems aimed at addressing customer queries, often by directing them to automated workflows. A crucial aspect of this process is the classification of the customer's intent. Presently, most intent classification models for customer care utilise only customer query for intent prediction. This may result in low-accuracy models, which cannot handle ambiguous queries. An ambiguous query like"I didn't receive my package"could indicate a delayed order, or an order that was delivered but the customer failed to receive it. Resolution of each of these scenarios requires the execution of very different sequence of steps. Utilizing additional information, such as the customer's order delivery status, in the right manner can help identify the intent for such ambiguous queries. In this paper, we have introduced a context-aware NLU model that incorporates both, the customer query and contextual information from the customer's order status for predicting customer intent. A novel selective attention module is used to extract relevant context features. We have also proposed a multi-task learning paradigm for the effective utilization of different label types available in our training data. Our suggested method, Multi-Task Learning Contextual NLU with Selective Attention Weighted Context (MTL-CNLU-SAWC), yields a 4.8% increase in top 2 accuracy score over the baseline model which only uses user queries, and a 3.5% improvement over existing state-of-the-art models that combine query and context. We have deployed our model to production for Walmart's customer care domain. Accurate intent prediction through MTL-CNLU-SAWC helps to better direct customers to automated workflows, thereby significantly reducing escalations to human agents, leading to almost a million dollars in yearly savings for the company.

Problem

• Improving chatbot intent classification accuracy using context-aware NLU
• Resolving ambiguous customer queries with selective attention and contextual data
• Enhancing automated workflows by reducing escalations to human agents

Innovation

• Context-aware NLU with selective attention
• Multi-task learning for diverse label utilization
• Combining query and order status for intent prediction