Learning To Focus: Anatomy-Guided Attention Regularization for Medical Image Classification

2026-07-12

AI Summary

Current medical image classification models often lack spatial supervision tied to diagnostically relevant anatomical structures, leading them to attend to irrelevant regions. This work proposes Locus, a novel framework that leverages pretrained segmentation foundation models as sources of anatomical priors to guide classifiers toward clinically meaningful regions through adaptive attention regularization—without requiring pixel-level annotations or rigid feature alignment. Evaluated across eight diverse datasets spanning dermoscopy, chest X-rays, histopathology, and cardiac MRI, the method consistently improves classification performance while significantly enhancing the anatomical plausibility of model attention maps.

Abstract

Medical image classification models are ideally expected to identify diagnostically relevant regions while making predictions, yet standard classification losses rarely provide spatial supervision. Explicit supervision via anatomical shape information, such as segmentation masks of task-relevant anatomy, has been shown to guide the network toward regions relevant to the target prediction. However, obtaining such masks incurs substantial manual annotation effort and computational overhead. With the advent of segmentation foundation models that exhibit strong localization of anatomical structures across diverse imaging modalities, we leverage this capability to extract anatomical shape priors without the burden of training a dedicated segmentation model. In this paper, we propose a new framework, Locus, an anatomical attention regularization framework that leverages pretrained segmentation foundation models to guide a classifier's attention toward diagnostically meaningful anatomical structures across diverse imaging modalities. Instead of enforcing pixel-wise alignment with the foundation-model-derived mask, we introduce a regularization term that adaptively balances attention between anatomical (foreground) and background regions, penalizing the classifier when background attention dominates. We validate Locus on eight diverse medical imaging datasets spanning dermoscopy, X-ray, histopathology, and cardiac MRI, showing consistent gains in classification performance alongside improved anatomically grounded attention.

Problem

• medical image classification
• spatial supervision
• anatomical attention
• segmentation masks
• attention regularization

Innovation

• anatomy-guided attention
• segmentation foundation model
• attention regularization
• medical image classification
• self-supervised localization