SAGE: Saliency-Guided Contrastive Embeddings

2025-11-16

AI Summary

To address the challenge of reliably integrating human visual saliency priors into neural network training—leading to poor generalization and limited interpretability—this paper proposes SAGE. SAGE is the first method to shift saliency guidance from the image space to the model’s latent embedding space. It constructs a contrastive triplet loss grounded in saliency preservation and degradation enhancement, and introduces a logit-distribution verification mechanism to align model decisions with human perception. Crucially, SAGE avoids reliance on unreliable internal gradients or saliency heatmaps used by conventional approaches, simultaneously delivering regularization and knowledge distillation effects. Extensive experiments demonstrate that SAGE consistently outperforms existing saliency-guided methods across both open-set and closed-set classification tasks. Moreover, it exhibits strong generalizability and robustness across diverse backbone architectures and multi-task settings.

Technology Category

• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Integrating human perceptual priors into the training of neural networks has been shown to raise model generalization, serve as an effective regularizer, and align models with human expertise for applications in high-risk domains. Existing approaches to integrate saliency into model training often rely on internal model mechanisms, which recent research suggests may be unreliable. Our insight is that many challenges associated with saliency-guided training stem from the placement of the guidance approaches solely within the image space. Instead, we move away from the image space, use the model's latent space embeddings to steer human guidance during training, and we propose SAGE (Saliency-Guided Contrastive Embeddings): a loss function that integrates human saliency into network training using contrastive embeddings. We apply salient-preserving and saliency-degrading signal augmentations to the input and capture the changes in embeddings and model logits. We guide the model towards salient features and away from non-salient features using a contrastive triplet loss. Additionally, we perform a sanity check on the logit distributions to ensure that the model outputs match the saliency-based augmentations. We demonstrate a boost in classification performance across both open- and closed-set scenarios against SOTA saliency-based methods, showing SAGE's effectiveness across various backbones, and include experiments to suggest its wide generalization across tasks.

Problem

• Integrating human saliency into neural network training using contrastive embeddings
• Moving saliency guidance from image space to latent embedding space
• Improving classification performance across open- and closed-set scenarios

Innovation

• Uses contrastive embeddings in latent space
• Applies saliency-guided triplet loss function
• Performs sanity checks on logit distributions