ZooClaw-FashionSigLIP2: Distilled Fine-tuning for Robust Fashion Retrieval

2026-06-26

AI Summary

This work addresses the challenge of balancing domain-specific performance and generalization capability in general-purpose vision-language models for fashion retrieval. We propose an efficient adaptation strategy that integrates full-parameter fine-tuning, knowledge distillation, and WISEFT weight interpolation. Starting from the SigLIP2-base model, we optimize on curated fashion data and interpolate the resulting weights with those of the original model to obtain robust representations. Our approach significantly outperforms LoRA, larger-scale models (up to 1B parameters), and methods relying on external data, all while preserving strong generalization. Additionally, we introduce ZooClaw-Fashion, a new high-quality benchmark for fashion retrieval, provide a thorough analysis of systematic biases in existing benchmarks, and release our model weights and evaluation toolkit.

Abstract

Adapting a foundation vision-language encoder to a specialized retrieval task creates a fundamental tradeoff: gains on the target distribution come at the cost of the foundation model's broad generalization, and fashion retrieval is a stringent instance of this problem. We present ZooClaw-FashionSigLIP2, a fashion-specialized SigLIP2-base model that resolves this tradeoff with a simple recipe -- full fine-tuning with knowledge distillation on curated in-domain data, followed by \wiseft~\citep{wortsman2022wiseft} weight interpolation with the base model -- and outperforms LoRA, larger backbones (up to 1B parameters), and external training data. Under fair evaluation, ZooClaw-FashionSigLIP2 outperforms all baselines on every benchmark in our suite. In addition, we release ZooClaw-Fashion, a new high-quality fashion retrieval benchmark, and a systematic quality analysis of widely-used benchmarks that exposes and mitigates structural biases in their public ground truth. We open-source the model weights and all evaluation artifacts to facilitate future research.

Problem

• fashion retrieval
• vision-language model
• generalization
• domain adaptation
• foundation model

Innovation

• knowledge distillation
• weight interpolation
• fashion retrieval
• foundation model adaptation
• domain-specific fine-tuning