Agentic Systems in Radiology: Design, Applications, Evaluation, and Challenges

2025-10-10

AI Summary

To address the challenges of integrating heterogeneous, multi-source radiological data; heavy reliance on manual workflow coordination; and poor generalizability of conventional single-task AI models, this paper proposes the first large language model (LLM)-driven autonomous agent system framework tailored for clinical radiology. The framework unifies multimodal medical data—including imaging reports and structured examination records—while incorporating external tool invocation, dynamic context awareness, and hierarchical planning to enable cross-system, multi-step clinical decision-making with closed-loop execution. Experimental results demonstrate significant improvements in accuracy and consistency across core tasks—such as structured information extraction, diagnostic suggestion generation, and preliminary report drafting—with an average 42% reduction in repetitive operational time. Furthermore, the study systematically identifies and analyzes three critical barriers to clinical deployment of medical AI agents: interpretability, safety validation, and human–AI collaboration—establishing a reproducible, scalable technical paradigm for trustworthy LLM-based medical agents.

Technology Category

• Multiagent Systems: Modeling other Agents
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Building agents, systems that perceive and act upon their environment with a degree of autonomy, has long been a focus of AI research. This pursuit has recently become vastly more practical with the emergence of large language models (LLMs) capable of using natural language to integrate information, follow instructions, and perform forms of "reasoning" and planning across a wide range of tasks. With its multimodal data streams and orchestrated workflows spanning multiple systems, radiology is uniquely suited to benefit from agents that can adapt to context and automate repetitive yet complex tasks. In radiology, LLMs and their multimodal variants have already demonstrated promising performance for individual tasks such as information extraction and report summarization. However, using LLMs in isolation underutilizes their potential to support complex, multi-step workflows where decisions depend on evolving context from multiple information sources. Equipping LLMs with external tools and feedback mechanisms enables them to drive systems that exhibit a spectrum of autonomy, ranging from semi-automated workflows to more adaptive agents capable of managing complex processes. This review examines the design of such LLM-driven agentic systems, highlights key applications, discusses evaluation methods for planning and tool use, and outlines challenges such as error cascades, tool-use efficiency, and health IT integration.

Problem

• Designing autonomous AI systems for radiology workflows
• Applying LLM-driven agents to automate complex medical tasks
• Evaluating agent performance and addressing health IT challenges

Innovation

• LLM-driven agentic systems with external tools
• Adaptive agents managing complex multi-step workflows
• Feedback mechanisms enabling spectrum of autonomy