From Tokens to Concepts: Leveraging SAE for SPLADE

2026-04-23

AI Summary

This work addresses the limitations of the traditional SPLADE model, which is constrained by its reliance on a fixed vocabulary and struggles with polysemy, synonymy, and lacks extensibility to multilingual and multimodal settings. To overcome these issues, the authors propose SAE-SPLADE, the first integration of sparse autoencoders (SAEs) into the SPLADE framework, replacing original tokens with semantic concepts learned by the SAE. They further introduce an end-to-end training strategy to effectively couple the two components. Experimental results demonstrate that SAE-SPLADE achieves retrieval performance on par with SPLADE across both in-domain and cross-domain tasks, while significantly improving computational efficiency and enhancing model generalization and cross-modal adaptability.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Learned Sparse IR models, such as SPLADE, offer an excellent efficiency-effectiveness tradeoff. However, they rely on the underlying backbone vocabulary, which might hinder performance (polysemicity and synonymy) and pose a challenge for multi-lingual and multi-modal usages. To solve this limitation, we propose to replace the backbone vocabulary with a latent space of semantic concepts learned using Sparse Auto-Encoders (SAE). Throughout this paper, we study the compatibility of these 2 concepts, explore training approaches, and analyze the differences between our SAE-SPLADE model and traditional SPLADE models. Our experiments demonstrate that SAE-SPLADE achieves retrieval performance comparable to SPLADE on both in-domain and out-of-domain tasks while offering improved efficiency.

Problem

• sparse retrieval
• SPLADE
• vocabulary limitation
• polysemy
• synonymy

Innovation

• Sparse Auto-Encoders
• SPLADE
• Learned Sparse Retrieval
• Semantic Concepts
• Efficiency-Effectiveness Tradeoff