Can Agents Distinguish Visually Hard-to-Separate Diseases in a Zero-Shot Setting? A Pilot Study

2026-02-26

AI Summary

This work addresses the challenge of distinguishing visually similar yet clinically distinct diseases—such as melanoma versus atypical nevi or pulmonary edema versus pneumonia—in zero-shot medical image diagnosis, where existing methods often fail. The authors propose the first multi-agent framework tailored for zero-shot medical imaging, incorporating a contrastive adjudication mechanism and leveraging multimodal large language models to enhance discriminative capability without labeled training data. Experimental results demonstrate an 11-percentage-point improvement in accuracy on dermoscopic images, alongside a significant reduction in unsupported assertions, thereby increasing the reliability of agent-based diagnostic decisions. This approach establishes a novel paradigm for zero-shot medical diagnosis by effectively mitigating confusion among visually ambiguous conditions.

Technology Category

• Computer Vision: Multi-modal Vision
• Multiagent Systems: Adversarial Agents
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The rapid progress of multimodal large language models (MLLMs) has led to increasing interest in agent-based systems. While most prior work in medical imaging concentrates on automating routine clinical workflows, we study an underexplored yet clinically significant setting: distinguishing visually hard-to-separate diseases in a zero-shot setting. We benchmark representative agents on two imaging-only proxy diagnostic tasks, (1) melanoma vs. atypical nevus and (2) pulmonary edema vs. pneumonia, where visual features are highly confounded despite substantial differences in clinical management. We introduce a multi-agent framework based on contrastive adjudication. Experimental results show improved diagnostic performance (an 11-percentage-point gain in accuracy on dermoscopy data) and reduced unsupported claims on qualitative samples, although overall performance remains insufficient for clinical deployment. We acknowledge the inherent uncertainty in human annotations and the absence of clinical context, which further limit the translation to real-world settings. Within this controlled setting, this pilot study provides preliminary insights into zero-shot agent performance in visually confounded scenarios.

Problem

• zero-shot
• visually hard-to-separate diseases
• medical imaging
• diagnostic ambiguity
• multimodal agents

Innovation

• zero-shot
• multimodal large language models
• multi-agent framework
• contrastive adjudication
• visually confounded diseases