PatchGen: Learning Soft Intra-Image Predictive Subsets for Visual Generalization

2026-08-12

AI Summary

This work addresses the limited generalization of existing visual classification methods under data and target distribution shifts, which often stems from neglecting predictive sufficiency within individual images. The authors posit that each image contains a sample-adaptive predictive subset sufficient for accurate classification, while other regions may introduce bias. To exploit this insight, they formalize predictive sufficiency as a learnable soft subset selection problem and introduce PatchGen—a text-free, task-driven module that generates sample-dependent soft masks to highlight informative regions. The training is guided by a theoretical analysis balancing empirical risk and model complexity. Experiments demonstrate that the proposed approach significantly improves average accuracy and out-of-distribution generalization across diverse natural and pathological image benchmarks under various distribution shifts, achieving performance on par with vision–language models.

Abstract

Visual classifiers are expected to generalize under data shifts, target shifts, and their combinations, yet most existing methods focus on domain invariance while failing to address intra-image predictive sufficiency. We investigate the structural hypothesis that each image contains a sample-adaptive oracle intra-image predictive subset sufficient for label prediction, while the remaining patches form non-essential complementary context that may correlate with the label. The theoretical analysis shows that restricting prediction to this oracle subset preserves the Bayes risk achievable by the full-patch representation while admitting a complexity bound that tightens with the oracle-subset size. Based on this view, we propose PatchGen, a text-free module that learns a sample-dependent soft predictive-subset mask as a task-driven proxy for the unobserved oracle subset mask. Specifically, histopathology visualizations suggest that PatchGen assigns higher scores to tumor-consistent regions than to some frequently co-occurring inflammatory context. Extensive experiments on natural and histopathological image benchmarks spanning all three shift settings show that PatchGen improves average performance over matched-backbone baselines in most evaluated configurations, enhances generalization to unknown classes, and remains competitive with vision-language methods without text supervision.

Problem

• visual generalization
• intra-image predictive subset
• data shift
• target shift
• predictive sufficiency

Innovation

• intra-image predictive subset
• sample-adaptive masking
• visual generalization
• domain shift
• text-free learning