Attribute-Conditioned Multimodal Slot Factorization for Controllable Fashion Retrieval

2026-08-12

AI Summary

This work addresses the limitation of conventional fashion retrieval methods, which conflate multi-attribute information into a single embedding, thereby hindering fine-grained control over specific attributes such as color or category. To overcome this, the authors propose the MM-slotgate model, which introduces— for the first time—a named and intervenable multimodal attribute-slot architecture. This framework decomposes Fashion-CLIP’s joint image-text embeddings into four semantically distinct attribute slots, each equipped with an independent image-text gating mechanism that automatically learns modality weights without explicit supervision. The approach achieves both semantic interpretability and controllable retrieval performance, attaining a ConstraintSatisfied@10 score of 0.7566 on the H&M dataset, improving color retrieval success by 0.568, and yielding a 15.3× enhancement in color intervention efficacy through quantized slot representations.

Abstract

Fashion retrieval often requires satisfying multiple attributes at once, such as category, color, pattern, and demographic. Monolithic embeddings mix these signals into a single vector, making attribute-specific control difficult at retrieval time. Many existing semantic-ID methods provide discrete item codes, but these codes are typically optimized as item-level or residual addresses and do not expose named, independently controllable attribute slots. We introduce MM-slotgate, a multimodal slot encoder that factorizes Fashion-CLIP text and image embeddings into four named attribute slots. Each slot learns its own text-image gate, so visually grounded attributes such as color and pattern can rely more on image evidence, while taxonomy-oriented attributes such as category and demographic can remain more text-driven. On H&M, using a combined slot-similarity and slot-logit retrieval score, MM-slotgate achieves 0.7566 macro ConstraintSatisfied@10, outperforming equal-weight multimodal fusion (0.7142) and fCLIP text-only retrieval (0.4755). The largest gain is on color, which improves from 0.321 to 0.889 (+0.568 absolute), as the learned color gate assigns 57.4% weight to image evidence. The learned gates are interpretable without modality supervision: color is image-leaning, category is text-leaning, and pattern and demographic lie near the middle. The resulting slots also remain controllable: linear probes show no measured excess leakage beyond the label-correlation baseline, and quantized slot codes support targeted intervention, including a 15.3x lift for color. These results suggest that controllable fashion retrieval benefits from typed, attribute-conditioned multimodal slots rather than either a single global embedding or opaque item-level semantic IDs.

Problem

• fashion retrieval
• attribute control
• multimodal embedding
• slot factorization
• controllable representation

Innovation

• multimodal slot factorization
• attribute-conditioned retrieval
• controllable fashion search
• text-image gating
• semantic slot encoding