CSRv2: Unlocking Ultra-Sparse Embeddings

2026-02-05

AI Summary

This work addresses the severe performance degradation of existing contrastive sparse representations (CSR) under ultra-sparse conditions, where excessive neuron deactivation compromises both efficiency and representational quality. To overcome this limitation, the authors propose CSRv2, which integrates progressive k-value annealing, supervised contrastive learning, and end-to-end backbone fine-tuning. CSRv2 achieves, for the first time, competitive performance with dense or moderately sparse methods using embeddings that activate only two features. The approach reduces the proportion of inactive neurons from 80% to 20%, yielding a 14% accuracy gain at k=2. Moreover, it offers up to 300× improvements in computational and memory efficiency over dense embeddings and attains a 7× faster inference speed than MRL, substantially expanding the design space for efficient representations in edge AI systems.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Representation Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

In the era of large foundation models, the quality of embeddings has become a central determinant of downstream task performance and overall system capability. Yet widely used dense embeddings are often extremely high-dimensional, incurring substantial costs in storage, memory, and inference latency. To address these, Contrastive Sparse Representation (CSR) is recently proposed as a promising direction, mapping dense embeddings into high-dimensional but k-sparse vectors, in contrast to compact dense embeddings such as Matryoshka Representation Learning (MRL). Despite its promise, CSR suffers severe degradation in the ultra-sparse regime, where over 80% of neurons remain inactive, leaving much of its efficiency potential unrealized. In this paper, we introduce CSRv2, a principled training approach designed to make ultra-sparse embeddings viable. CSRv2 stabilizes sparsity learning through progressive k-annealing, enhances representational quality via supervised contrastive objectives, and ensures end-to-end adaptability with full backbone finetuning. CSRv2 reduces dead neurons from 80% to 20% and delivers a 14% accuracy gain at k=2, bringing ultra-sparse embeddings on par with CSR at k=8 and MRL at 32 dimensions, all with only two active features. While maintaining comparable performance, CSRv2 delivers a 7x speedup over MRL, and yields up to 300x improvements in compute and memory efficiency relative to dense embeddings in text representation. Extensive experiments across text and vision demonstrate that CSRv2 makes ultra-sparse embeddings practical without compromising performance, where CSRv2 achieves 7%/4% improvement over CSR when k=4 and further increases this gap to 14%/6% when k=2 in text/vision representation. By making extreme sparsity viable, CSRv2 broadens the design space for real-time and edge-deployable AI systems where both embedding quality and efficiency are critical.

Problem

• ultra-sparse embeddings
• dead neurons
• sparsity degradation
• embedding efficiency
• contrastive sparse representation

Innovation

• ultra-sparse embeddings
• contrastive sparse representation
• progressive k-annealing
• supervised contrastive learning
• efficient AI