Sparse Task Vector Mixup with Hypernetworks for Efficient Knowledge Transfer in Whole-Slide Image Prognosis

2026-03-11

AI Summary

This work addresses the limitations of cancer type–specific learning in whole-slide image–based prognosis, which suffers from scarce samples and tumor heterogeneity, leading to poor generalization. The authors propose STEPH, a novel approach that, for the first time, integrates task vector interpolation with hypernetwork-driven sparse aggregation to enable efficient cross-cancer knowledge transfer. By interpolating task vectors from source–target cancer pairs and sparsely aggregating them via a hypernetwork to refine the target model, STEPH avoids the need for large-scale joint training or multi-model inference. Evaluated across 13 cancer datasets, STEPH improves prognostic performance by 5.14% over cancer-specific models and by 2.01% over existing transfer learning baselines, while substantially enhancing computational efficiency.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Multi-modal Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Whole-Slide Images (WSIs) are widely used for estimating the prognosis of cancer patients. Current studies generally follow a cancer-specific learning paradigm. However, the available training samples for one cancer type are usually scarce in pathology. Consequently, the model often struggles to learn generalizable knowledge, thus performing worse on the tumor samples with inherent high heterogeneity. Although multi-cancer joint learning and knowledge transfer approaches have been explored recently to address it, they either rely on large-scale joint training or extensive inference across multiple models, posing new challenges in computational efficiency. To this end, this paper proposes a new scheme, Sparse Task Vector Mixup with Hypernetworks (STEPH). Unlike previous ones, it efficiently absorbs generalizable knowledge from other cancers for the target via model merging: i) applying task vector mixup to each source-target pair and then ii) sparsely aggregating task vector mixtures to obtain an improved target model, driven by hypernetworks. Extensive experiments on 13 cancer datasets show that STEPH improves over cancer-specific learning and an existing knowledge transfer baseline by 5.14% and 2.01%, respectively. Moreover, it is a more efficient solution for learning prognostic knowledge from other cancers, without requiring large-scale joint training or extensive multi-model inference. Code is publicly available at https://github.com/liupei101/STEPH.

Problem

• Whole-Slide Image
• Prognosis
• Knowledge Transfer
• Data Scarcity
• Computational Efficiency

Innovation

• Task Vector Mixup
• Hypernetworks
• Sparse Aggregation
• Knowledge Transfer
• Whole-Slide Image Prognosis