TLDR: A new framework uses Large Language Model (LLM) agents to automate the complex and time-consuming process of managing CT scan protocols. This system interprets natural language or structured requests to modify acquisition, reconstruction, and postprocessing settings in patient-specific ways, aiming to reduce radiologist workload and improve efficiency. It employs a hierarchical understanding of protocols, a router for request decomposition, a planner for action proposals, and a human-in-the-loop for verification. Evaluations show promising results in generating syntactically correct and user-aligned protocols, though challenges remain with device-specific knowledge and complex requests.
Managing Computed Tomography (CT) scan protocols is a highly intricate and time-consuming task, demanding extensive clinical and technical expertise. This complexity is further exacerbated by a growing shortage of skilled professionals in radiology. A recent research paper, “Scan-do Attitude: Towards Autonomous CT Protocol Management using a Large Language Model Agent”, introduces an innovative solution to address these challenges: an AI-powered agent framework based on Large Language Models (LLMs).
The proposed framework aims to streamline the interpretation and execution of CT protocol configuration requests, whether they are given in natural language or a structured, device-independent format. By doing so, it seeks to enhance workflow efficiency and significantly reduce the workload on technologists.
How the LLM Agent Works
The core of this system is an LLM-based agent that leverages several advanced capabilities, including in-context learning, instruction-following, and structured tool-calling. This allows the agent to accurately identify relevant protocol elements and apply precise modifications.
CT scan protocols themselves are complex, often structured as hierarchical XML documents. These documents define technical parameters and procedures for examinations, with entities representing different imaging workflow phases (e.g., data acquisition, reconstruction operations) and essential parameters controlling specific settings like slice thickness or kernel type.
The agent’s workflow begins when it receives a technologist’s request. It then uses a suite of tools and a memory component to interact with the underlying scan protocol. A key component is the Router, which decomposes complex user requests into smaller, manageable sub-requests, classifying them into categories like Adding, Modification, Deleting, or Others. These sub-requests are then dispatched to specialized downstream agents.
The agent also incorporates a Memory module that stores static and protocol-specific knowledge, including descriptions of various protocol entities and a simplified representation of the protocol’s basic structure. A Planner component retrieves relevant entities and parameters, then uses the LLM’s reasoning abilities to formulate a concrete proposal. This proposal includes a structured set of actions detailing modifications and a human-readable plan explaining these steps.
To ensure accuracy and mitigate potential errors from LLM limitations, the framework includes a Human-in-the-Loop verification step. After the planner generates a proposal, it is submitted to a human user for review and confirmation, creating a closed-loop system for enhanced reliability.
Evaluating Performance and Identifying Limitations
To systematically assess the agent’s reliability, the researchers designed a multi-level evaluation pipeline. This included:
- Syntax Evaluation: Measuring the Syntax Correctness Rate (SCR) to ensure that modified XML protocols remain readable and interpretable by the scanner’s protocol browser.
- Semantic Evaluation: Assessing Plan Accuracy (how well the agent’s modifications match ground truth) and Plan Faithfulness (how accurately an LLM can reconstruct the original user request from the agent’s actions).
- Retrieval Evaluation: Checking the agent’s ability to fetch the correct context from the structured scan protocol at both entity and essential levels.
Experiments were conducted using an adult thorax CT protocol, processing both natural language and structured JSON-formatted requests. The study compared the performance of different LLMs, including GPT-4o, GPT-4o-mini, and a locally deployed open-source LLM, Gemma3:27B.
The results indicated that GPT-4o generally achieved the highest overall SCR and demonstrated superior performance in semantic evaluations, particularly for structure-sensitive tasks like adding and deleting protocol elements. While other LLMs showed selective strengths, GPT-4o consistently performed better.
Despite these promising outcomes, the study also highlighted several limitations. Common errors included misclassification of requests by the router, structural errors leading to invalid XML hierarchies, and a lack of comprehensive device-specific knowledge. This knowledge gap can cause the agent to misunderstand inter-dependencies between attributes or use unsupported values, leading to semantically incorrect modifications.
Also Read:
- SignalLLM: A New Era for Automated Signal Processing Tasks
- Boosting Vision System Reliability with Agentic AI: The Visual Reasoning Agent
Future Directions
The research demonstrates the feasibility of using LLM agents to retrieve information from real protocol definition files and generate valid, device-compatible modifications based on various clinical requests. Future work will focus on expanding the agent’s access to relevant knowledge sources, such as semantically meaningful device APIs and structured radiology knowledge bases, to overcome the current limitations in device-specific understanding. Developing more robust evaluation methodologies that minimize model-dependent biases is also crucial for strengthening the assessment of agent performance in real-world clinical settings.
