BackSplit: The Importance of Sub-dividing the Background in Biomedical Lesion Segmentation

2025-11-24

AI Summary

Medical image segmentation of tiny lesions is hindered by conventional modeling that lumps all non-lesion regions into a single “background” class, ignoring the high anatomical heterogeneity of background tissues. To address this, we propose BackSplit—a paradigm that decomposes the background into multiple anatomically defined classes (e.g., organs, tissues). Fine-grained background labels are automatically generated using pre-trained models and integrated with interactive segmentation outputs for multi-scenario validation. From an information-theoretic perspective, we formally prove—first in this domain—that background decomposition increases the Fisher information of model parameters, thereby improving optimization stability and tightening convergence bounds. Crucially, BackSplit incurs no additional inference overhead and consistently boosts small-lesion segmentation accuracy across diverse datasets and network architectures. Notably, it retains robust performance gains even when background labels are auto-generated, demonstrating broad generalizability and practical utility.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Segmenting small lesions in medical images remains notoriously difficult. Most prior work tackles this challenge by either designing better architectures, loss functions, or data augmentation schemes; and collecting more labeled data. We take a different view, arguing that part of the problem lies in how the background is modeled. Common lesion segmentation collapses all non-lesion pixels into a single "background" class, ignoring the rich anatomical context in which lesions appear. In reality, the background is highly heterogeneous-composed of tissues, organs, and other structures that can now be labeled manually or inferred automatically using existing segmentation models. In this paper, we argue that training with fine-grained labels that sub-divide the background class, which we call BackSplit, is a simple yet powerful paradigm that can offer a significant performance boost without increasing inference costs. From an information theoretic standpoint, we prove that BackSplit increases the expected Fisher Information relative to conventional binary training, leading to tighter asymptotic bounds and more stable optimization. With extensive experiments across multiple datasets and architectures, we empirically show that BackSplit consistently boosts small-lesion segmentation performance, even when auxiliary labels are generated automatically using pretrained segmentation models. Additionally, we demonstrate that auxiliary labels derived from interactive segmentation frameworks exhibit the same beneficial effect, demonstrating its robustness, simplicity, and broad applicability.

Problem

• Segmenting small lesions in medical images remains notoriously difficult
• Common lesion segmentation collapses all non-lesion pixels into single background class
• Background modeling ignores rich anatomical context where lesions appear

Innovation

• Subdividing background into fine-grained anatomical labels
• Increasing Fisher Information for stable optimization
• Using pretrained models to generate auxiliary labels