Deciding When to Rely on Visual Information: Gated Multimodal Fusion in Sequential Recommendation

2026-08-11

AI Summary

This work addresses the limitation of conventional multimodal sequential recommendation methods, which uniformly fuse visual and collaborative signals while overlooking the dynamic nature of visual information utility across user interaction histories and item contexts. To resolve this, we propose VisGate, a novel framework that models visual utility as a context-dependent latent variable and employs a gating mechanism to adaptively determine—on an item-by-item basis—whether to incorporate visual features, conditioned jointly on the item embedding and the user’s current sequential context. Additionally, we introduce a contrastive learning objective to encourage visual representations that complement, rather than merely align with, collaborative signals. Experiments demonstrate that VisGate maintains strong recommendation performance while effectively identifying scenarios where visual information is beneficial, revealing its correlation with interaction sparsity and visual distinctiveness, and achieving significant gains when collaborative signals are weak.

Abstract

Multimodal sequential recommender systems commonly fuse visual and collaborative signals uniformly, treating visual features as generically informative regardless of item or user context. We argue that visual utility, defined as the contribution of visual signals to recommendation quality, is a latent contextual variable that depends on both the item and the user's interaction history rather than a fixed item property. To model this variability, we introduce VisGate, a framework that makes adaptive item-level fusion decisions conditioned on item embeddings and the user's current sequence context. Visual representations are learned through a contrastive objective over sequential co-occurrence patterns, preserving complementarity with collaborative embeddings rather than aligning them into a shared space. Beyond achieving competitive recommendation performance, VisGate's learned gate serves as a measurement tool for understanding when and why visual information is beneficial. Our analyses show that visual utility varies across items, increases under interaction sparsity when collaborative signals are weak, and correlates with visual distinctiveness in semantically meaningful ways. Together, these findings highlight the importance of both fine-grained fusion and modality complementarity, while demonstrating that item-level visual utility can be estimated and interpreted through learned gating behaviour.

Problem

• multimodal fusion
• sequential recommendation
• visual utility
• contextual variability
• item-level fusion

Innovation

• gated multimodal fusion
• visual utility
• sequential recommendation
• contrastive learning
• modality complementarity