TLDR: ORCHID is a new AI system designed to classify High-Risk Property (HRP) at U.S. Department of Energy sites. It uses a modular, agent-based approach combining retrieval-augmented generation (RAG) with human oversight to make transparent, auditable, and policy-compliant decisions. The system improves accuracy and traceability over traditional methods by providing step-by-step reasoning, on-policy citations, and capturing expert feedback for uncertain items, ensuring secure on-premise operation and reproducibility.
The U.S. Department of Energy (DOE) sites manage inventories that include sensitive and often dual-use equipment, making the classification of High-Risk Property (HRP) a critical task. Ensuring compliance with evolving export control policies and maintaining transparent, auditable decisions has traditionally been a time-consuming process, prone to backlogs and struggling to adapt to shifting regulations. To address these challenges, researchers have developed ORCHID, a new modular AI system designed to streamline HRP classification.
ORCHID, which stands for Orchestrated Retrieval-Augmented Classification with Human-in-the-Loop Intelligent Decision-Making for High-Risk Property, combines advanced AI techniques with essential human oversight. This system aims to produce policy-based outputs that are not only accurate but also fully auditable, offering a practical path to trustworthy AI assistance in sensitive compliance workflows.
How ORCHID Works
At its core, ORCHID operates through a network of small, cooperating AI agents. These agents include a retrieval agent, a description refiner, a classifier, a validator, and a feedback logger. They communicate with each other using a specialized messaging system and interact with tools through a Model Context Protocol (MCP), allowing for flexible, model-agnostic operation within secure on-premise environments.
The system follows an intuitive “Item to Evidence to Decision” loop, providing step-by-step reasoning for its classifications. Crucially, it includes on-policy citations for every decision and generates append-only audit bundles, which contain run-cards, prompts, and evidence. This ensures that every decision is traceable and verifiable.
Key Principles Guiding ORCHID
ORCHID’s design is built on several fundamental principles:
- Evidence-first: Every classification must be supported by at least one citation from authoritative sources like the USML, NRC, or CCL texts. Conflicting information is also made visible.
- Human-in-the-loop: Initial model outputs are treated as proposals. If the system has low confidence, a human reviewer (Subject Matter Expert or SME) must accept or override the decision, providing feedback that the system learns from.
- Grounded retrieval only: The search for information is strictly limited to a whitelisted and versioned policy corpus, using a hybrid retrieval method that combines lexical and semantic matching.
- Reproducible by default: Each session generates an audit bundle, logging all inputs, outputs, and decision patterns, making it possible to replay and verify any decision.
- Secure on-premise operation: The system is designed for on-premise use, meaning no external data sharing occurs, and logs are used solely for governance, not for training or fine-tuning.
- Configurable and fail-safe: Stakeholders can set confidence thresholds. If inputs are ambiguous or confidence is low, ORCHID defers the item to a human expert for review.
The Agentic Workflow
The Orchestrator acts as the central controller, managing the workflow of the agents. When an item is submitted, the Information Retrieval (IR) agent converts the item’s details (manufacturer, equipment, model number, description) into queries to fetch relevant policy snippets from the Vector Store tool. The Description Refiner (DR) can ask for clearer descriptions or rewrite existing ones. The HRP Classifier then uses this grounded information to propose a decision, including a label, confidence score, and cited snippets.
The Validator (VR) is a critical gatekeeper, checking if there’s enough on-policy support and if any contradictory snippets exist. It issues a verdict – “AGREE,” “REVIEW,” or “CONFLICT” – either leading to a final response or routing the case to a human. Finally, the Feedback Logger (FL) records reviewer decisions and rationales into an immutable audit store, ensuring continuous improvement and accountability.
User Experience
ORCHID’s user interface is designed for transparency and ease of use. Users can submit items individually or in batches, providing vendor names, item names, model numbers, and optional descriptions. The interface then displays the system’s prediction (HRP or Not HRP), the control category (USML, NRC, CCL, or EAR99), and a confidence score. Each result includes a concise, step-by-step reasoning trace with clickable citations, allowing users to inspect the underlying evidence. Users can also provide structured feedback, which is stored for audit and future model refinement. For more details, you can refer to the original research paper.
Also Read:
- Advancing Urban Planning with AI: Beyond Prediction to Transparent Reasoning
- LLM Agents Enhance Predictive Maintenance by Cleaning Noisy Logs
Preliminary Findings and Future Outlook
Preliminary tests on real HRP cases have shown that ORCHID significantly improves accuracy and traceability compared to non-agentic baselines, particularly by deferring uncertain items to human experts. While the system demonstrates strong capabilities, it does have limitations. Its performance relies on curated policy corpora, making it sensitive to outdated information. Challenges remain in distinguishing fine-grained classifications, especially between CCL and EAR99 items, and the system currently only supports English text without multimodal inputs like images. Despite these areas for future development, ORCHID represents a significant step forward in enhancing compliance workflows with intelligent, auditable AI assistance.
