Learning Selective LLM Autonomy from Copilot Feedback in Enterprise Customer Support Workflows

2026-04-26

AI Summary

This work addresses the safety and reliability challenges of deploying large language models for enterprise customer support automation by proposing a selective automation framework grounded in real operator feedback. Leveraging user acceptance or correction of Copilot suggestions, the approach jointly trains a UI action policy and a confidence estimation module. It integrates pattern-driven BPM-based interface modeling, runtime safety fallback mechanisms, and multi-session concurrency monitoring to enable high-confidence steps to execute automatically while seamlessly deferring low-confidence cases to human agents. Requiring only lightweight supervision, the framework supports rapid deployment of new workflows within two weeks. In production, it achieves fully automated handling in 45% of sessions, reduces average handling time by 39%, and maintains service quality without degradation.

Technology Category

• Natural Language Processing: Safety and Robustness
• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We present a deployed system that automates end-to-end customer support workflows inside an enterprise Business Process Management (BPM) platform. The approach is scalable in production and reaches selective automation within two weeks for a new process, leveraging supervision already generated at scale: structured per-case UI interaction traces and low-overhead copilot feedback, where operators either accept a suggestion or provide a correction. A staged deployment pipeline trains a next UI action policy, learns a critic from copilot feedback to calibrate abstention, and executes only high-confidence steps in the background while deferring uncertain decisions to operators and resuming from the updated UI state. This setup lets one operator supervise multiple concurrent sessions and be interrupted only when the system is uncertain. The system operates on a schema-driven view of the BPM interface and includes monitoring and safe fallbacks for production. In production, it automated 45% of sessions and reduced average handling time by 39% without degrading support quality level.

Problem

• Selective Autonomy
• LLM Automation
• Customer Support
• Copilot Feedback
• Enterprise Workflows

Innovation

• selective automation
• copilot feedback
• LLM autonomy
• UI action policy
• abstention calibration