Rethinking Test-Time Scaling for Medical AI: Model and Task-Aware Strategies for LLMs and VLMs

2025-06-16

AI Summary

This study addresses test-time scaling optimization for medical large language models (LLMs) and vision-language models (VLMs) during inference. We propose the first model- and task-aware test-time scaling framework, integrating adaptive sampling, inference path reweighting, uncertainty-guided multi-step reasoning, and adversarial prompt robustness analysis. Our methodology systematically investigates how model scale, task complexity, and prompt robustness jointly influence performance. Experiments across multiple medical QA and radiology report generation benchmarks demonstrate accuracy improvements of 8.2–14.7%, alongside significantly enhanced interpretability and clinical consistency. The work reveals scenario-dependent optimal scaling trajectories across diverse clinical settings and delivers the first reusable, medical-grade test-time optimization guideline—establishing foundational principles for reliable, context-sensitive deployment of multimodal AI in healthcare.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Test-time scaling has recently emerged as a promising approach for enhancing the reasoning capabilities of large language models or vision-language models during inference. Although a variety of test-time scaling strategies have been proposed, and interest in their application to the medical domain is growing, many critical aspects remain underexplored, including their effectiveness for vision-language models and the identification of optimal strategies for different settings. In this paper, we conduct a comprehensive investigation of test-time scaling in the medical domain. We evaluate its impact on both large language models and vision-language models, considering factors such as model size, inherent model characteristics, and task complexity. Finally, we assess the robustness of these strategies under user-driven factors, such as misleading information embedded in prompts. Our findings offer practical guidelines for the effective use of test-time scaling in medical applications and provide insights into how these strategies can be further refined to meet the reliability and interpretability demands of the medical domain.

Problem

• Evaluating test-time scaling for medical LLMs and VLMs
• Identifying optimal scaling strategies for diverse medical tasks
• Assessing robustness against misleading prompts in medical AI

Innovation

• Evaluates test-time scaling for medical LLMs and VLMs
• Assesses impact of model size and task complexity
• Tests robustness against misleading prompt information