PUMA: Post-Hoc Sparsification of Universal Multimodal Embeddings for Efficient Retrieval

2026-08-26

AI Summary

为解决多模态嵌入的高内存和推理成本问题，提出PUMA方法，通过后处理稀疏化技术在不重新训练主干模型的情况下生成紧凑稀疏编码，有效降低存储需求并加快检索速度。

Abstract

Universal multimodal embedders enable retrieval across text, image, and combined queries, but their dense representations incur high memory and inference costs. Post-hoc sparsification could reduce these costs but remains underexplored for multimodal retrieval. We introduce PUMA, a sparse autoencoder recipe that maps universal multimodal embeddings to compact sparse codes without retraining the backbone: a pretraining stage preserves dense dot-product geometry, after which the sparse encoder is fine-tuned for retrieval. We evaluate on five benchmarks covering text-to-image and composed image retrieval. On Qwen3-VL-Embedding-2B, PUMA is statistically indistinguishable from or improves over dense retrieval on four of five datasets. We further identify two failure modes of post-hoc sparsification: insufficient pre-TopK support and retrieval-misaligned active support. PUMA reduces vector storage by 8-16x (FP32) and is up to 25x faster than exact dense scoring on larger candidate pools, enabling efficient multimodal retrieval.

Problem

• universal multimodal embeddings
• post-hoc sparsification
• retrieval
• memory costs
• inference costs

Innovation

• sparse autoencoder
• multimodal embeddings
• post-hoc sparsification
• efficient retrieval