SHERLOCK: Towards Dynamic Knowledge Adaptation in LLM-enhanced E-commerce Risk Management

2025-10-09

AI Summary

E-commerce fraud risk management suffers from low fraud-pattern identification efficiency, excessive reliance on manual expertise, and inconsistent analytical standards. To address these challenges, this paper proposes an LLM-based intelligent risk investigation framework. Its core contributions are threefold: (1) a Reflect & Refine collaborative reasoning mechanism enabling iterative reflection and refinement over suspicious cases; (2) a domain-knowledge-base-driven dynamic adaptation architecture supporting cross-dimensional semantic understanding and knowledge transfer across heterogeneous risk types; and (3) a closed-loop data flywheel integrating expert annotations and model feedback to enable continuous model optimization within live business workflows. Evaluated on real transaction data from JD.com, the framework achieves substantial improvements: +28.6% in factual alignment rate, +32.1% in risk localization accuracy, and a 3.4× increase in case processing throughput—thereby advancing standardization and intelligence in risk control operations.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The growth of the e-commerce industry has intensified the adversarial dynamics between shadow economy actors and risk management teams. Companies often conduct risk investigations into suspicious cases to identify emerging fraud patterns, thereby enhancing both preemptive risk prevention and post-hoc governance. However, the sheer volume of case analyses imposes a substantial workload on risk management analysts, as each case requires the integration of long-term expert experience and meticulous scrutiny across multiple risk dimensions. Additionally, individual disparities among analysts hinder the establishment of uniform and high-standard workflows. To address these challenges, we propose the SHERLOCK framework, which leverages the reasoning capabilities of large language models (LLMs) to assist analysts in risk investigations. Our approach consists of three primary components: (1) extracting risk management knowledge from multi-modal data and constructing a domain knowledge base (KB), (2) building an intelligent platform guided by the data flywheel paradigm that integrates daily operations, expert annotations, and model evaluations, with iteratively fine-tuning for preference alignment, and (3) introducing a Reflect & Refine (R&R) module that collaborates with the domain KB to establish a rapid response mechanism for evolving risk patterns. Experiments conducted on the real-world transaction dataset from JD.com demonstrate that our method significantly improves the precision of both factual alignment and risk localization within the LLM analysis results. Deployment of the SHERLOCK-based LLM system on JD.com has substantially enhanced the efficiency of case investigation workflows for risk managers.

Problem

• Addresses high workload from manual e-commerce fraud case analysis
• Mitigates analyst skill disparities through standardized risk workflows
• Adapts to evolving fraud patterns with dynamic knowledge systems

Innovation

• Extracts knowledge from multimodal data for domain KB
• Builds intelligent platform with data flywheel paradigm
• Introduces Reflect & Refine module for rapid response