ORCHID: Orchestrated Retrieval-Augmented Classification with Human-in-the-Loop Intelligent Decision-Making for High-Risk Property

2025-11-07

AI Summary

To address stringent compliance requirements, manual process bottlenecks, and challenges in dynamically updating policies under the U.S. Department of Energy’s High-Risk Property (HRP) classification framework, this paper proposes an auditable intelligent decision-support system. Methodologically, we design a modular multi-agent architecture integrating Retrieval-Augmented Generation (RAG), Model Context Protocol (MCP), and localized reasoning to realize a closed-loop workflow from “item → evidence → decision.” The system supports inline policy citations, automatic escalation of uncertain items to human experts, comprehensive audit logging, and retention of expert feedback. Our key contributions are: (1) the first application of human-in-the-loop agent paradigms to HRP export control classification, significantly improving classification accuracy and traceability; and (2) establishing a trustworthy LLM-augmented compliance paradigm featuring single-item submission, evidence anchoring, and exportable audit packages.

Technology Category

• Humans and AI: Human-in-the-loop Machine Learning
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Adversarial Agents

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

High-Risk Property (HRP) classification is critical at U.S. Department of Energy (DOE) sites, where inventories include sensitive and often dual-use equipment. Compliance must track evolving rules designated by various export control policies to make transparent and auditable decisions. Traditional expert-only workflows are time-consuming, backlog-prone, and struggle to keep pace with shifting regulatory boundaries. We demo ORCHID, a modular agentic system for HRP classification that pairs retrieval-augmented generation (RAG) with human oversight to produce policy-based outputs that can be audited. Small cooperating agents, retrieval, description refiner, classifier, validator, and feedback logger, coordinate via agent-to-agent messaging and invoke tools through the Model Context Protocol (MCP) for model-agnostic on-premise operation. The interface follows an Item to Evidence to Decision loop with step-by-step reasoning, on-policy citations, and append-only audit bundles (run-cards, prompts, evidence). In preliminary tests on real HRP cases, ORCHID improves accuracy and traceability over a non-agentic baseline while deferring uncertain items to Subject Matter Experts (SMEs). The demonstration shows single item submission, grounded citations, SME feedback capture, and exportable audit artifacts, illustrating a practical path to trustworthy LLM assistance in sensitive DOE compliance workflows.

Problem

• Automating high-risk property classification for DOE compliance workflows
• Reducing time-consuming expert-only processes prone to backlogs
• Improving accuracy and traceability in sensitive equipment classification

Innovation

• Retrieval-augmented generation with human oversight
• Modular agentic system using Model Context Protocol
• Step-by-step reasoning with append-only audit bundles