AFMRL: Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning in E-commerce

2026-04-21

AI Summary

This work addresses the challenge of fine-grained differentiation among highly similar products in e-commerce scenarios, which existing large models struggle to resolve. The authors propose a multimodal understanding framework that formulates fine-grained product recognition as a key attribute generation task. Their approach employs a two-stage training strategy: the first stage leverages Attribute-Guided Contrastive Learning (AGCL) to identify hard examples and filter noisy negative samples, while the second stage introduces a Retrieval-Aware Reinforcement (RAR) mechanism that optimizes the attribute generation capability of multimodal large language models (MLLMs) using retrieval performance as a reward signal. By uniquely integrating attribute generation with contrastive learning, this work achieves bidirectional enhancement between generative modeling and representation learning, significantly outperforming state-of-the-art methods on large-scale e-commerce datasets and achieving leading performance across multiple downstream retrieval tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Multimodal representation is crucial for E-commerce tasks such as identical product retrieval. Large representation models (e.g., VLM2Vec) demonstrate strong multimodal understanding capabilities, yet they struggle with fine-grained semantic comprehension, which is essential for distinguishing highly similar items. To address this, we propose Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning (AFMRL), which defines product fine-grained understanding as an attribute generation task. It leverages the generative power of Multimodal Large Language Models (MLLMs) to extract key attributes from product images and text, and enhances representation learning through a two-stage training framework: 1) Attribute-Guided Contrastive Learning (AGCL), where the key attributes generated by the MLLM are used in the image-text contrastive learning training process to identify hard samples and filter out noisy false negatives. 2) Retrieval-aware Attribute Reinforcement (RAR), where the improved retrieval performance of the representation model post-attribute integration serves as a reward signal to enhance MLLM's attribute generation during multimodal fine-tuning. Extensive experiments on large-scale E-commerce datasets demonstrate that our method achieves state-of-the-art performance on multiple downstream retrieval tasks, validating the effectiveness of harnessing generative models to advance fine-grained representation learning.

Problem

• fine-grained representation learning
• multimodal understanding
• e-commerce retrieval
• attribute comprehension
• similar product discrimination

Innovation

• attribute-enhanced representation
• fine-grained multimodal learning
• multimodal large language models
• contrastive learning with attributes
• retrieval-aware reinforcement