Aloe-Vision: Robust Vision-Language Models for Healthcare

2026-06-25

AI Summary

This work addresses key challenges in medical vision-language models—namely, the scarcity of high-quality multimodal data, insufficient robustness in safety-critical settings, and evaluation benchmarks vulnerable to contamination. To this end, we introduce Aloe-Vision-Data, a large-scale hybrid dataset combining medical and general multimodal data, and release the fully reproducible Aloe-Vision model series (7B/72B) with complete training recipes and weights. We also propose CareQA-Vision, a low-contamination-risk medical visual benchmark derived from the Spanish medical residency entrance examination. Through multimodal and text-based quality filtering, targeted fine-tuning, and adversarial robustness analysis, our models achieve state-of-the-art performance across multiple medical vision tasks while preserving strong general capabilities, and further expose the vulnerability of existing models to misleading inputs.

Abstract

Large Vision-Language Models (LVLMs) specialized in healthcare are emerging as a promising research direction due to their potential impact in clinical and biomedical applications. However, progress is constrained by the scarcity of high-quality medical multimodal data, concerns about robustness in safety-critical settings, and the narrow and potentially contaminated evaluation benchmarks that limit reliable assessment. To address these issues, the field requires state-of-the-art solutions to be fully open and reproducible systems in which all components can be inspected, evaluated, and improved. This work introduces Aloe-Vision-Data, a large-scale, quality-filtered mixture which integrates both medical and general domains across multimodal and text-only sources, designed for direct use in model fine-tuning. Building on this dataset, we train the Aloe-Vision family of medical LVLMs, openly released with full weights, training recipes and data, in two scales (7B and 72B). Through comprehensive benchmarking, we demonstrate that high quality training mixtures produce balanced LVLMs which yield significant gains over the baseline models without compromising general capabilities, achieving competitive performance with respect to state-of-the-art alternatives. To support reliable evaluation, we introduce CareQA-Vision, a carefully curated vision benchmark derived from MIR and EIR exams, the residency entrance exams for medical and nursing specialists in Spain, offering novel vision questions with low likelihood of contamination. Finally, we show that current LVLMs remain vulnerable to adversarial and misleading inputs, underscoring reliability challenges in clinical contexts.

Problem

• medical vision-language models
• data scarcity
• robustness
• evaluation benchmark contamination
• clinical reliability

Innovation

• Vision-Language Models
• Medical Multimodal Data
• Open Reproducible AI
• CareQA-Vision Benchmark
• Adversarial Robustness