Non-negative Elastic Net Decoding for Information Retrieval

2026-06-16

AI Summary

This work addresses the limitation of conventional dense retrieval methods, which often neglect global corpus context and consequently yield semantically redundant and insufficiently diverse results. The authors formulate retrieval as a joint decoding problem and propose a sparse non-negative linear combination approach based on the Non-Negative Elastic Net (NNN) to jointly reconstruct the query vector, enabling global-aware document set selection. They establish, for the first time, a theoretical separation bound between dense retrieval and NNN decoding, demonstrating that NNN effectively handles scenarios where dense retrieval fails to retrieve relevant documents. An end-to-end optimization framework is introduced to jointly train embeddings and the decoder. Experiments show that even with frozen embeddings, NNN decoding outperforms existing dense retrieval methods; after end-to-end training, it achieves significant gains across multiple benchmarks, surpassing state-of-the-art approaches on all evaluated metrics.

Abstract

Dense retrieval has become the dominant paradigm in information retrieval, in which each document is scored against a query by the inner product of their vector embeddings, and the top-$k$ documents by score are retrieved for this query. However, since each document's score depends solely on the embedding of the query and itself, the retrieval process is oblivious to the content of the entire corpus. Therefore, dense retrieval cannot avoid selecting semantically similar documents from the corpus, which may result in a non-diverse, redundant set of retrieved documents. To this end, we approach retrieval as a joint decoding problem, in which documents are selected as a set with regard to the context of the rest of the corpus. To achieve this, we propose Non-Negative elastic Net (NNN) decoding, which selects documents whose embeddings jointly reconstruct the query embedding as a sparse non-negative linear combination. Our main theoretical result establishes a strict separation between dense retrieval and NNN decoding. For any corpus, every query correctly handled by dense retrieval is also handled by NNN decoding, while on corpora containing correlated documents, NNN decoding additionally handles queries that dense retrieval cannot. Experimental results indicate that applying NNN decoding to frozen embeddings trained for inner-product scoring yields consistent improvements across several benchmarks. Moreover, we introduce an end-to-end training procedure which optimizes the embeddings for NNN decoding, producing significant performance gains surpassing in all metrics and benchmarks compared to dense retrieval. Our work establishes a new paradigm for leveraging dense embeddings in information retrieval, beyond the standard practice of inner-product scoring.

Problem

• dense retrieval
• document redundancy
• retrieval diversity
• corpus context
• information retrieval

Innovation

• Non-Negative Elastic Net
• joint decoding
• dense retrieval
• sparse reconstruction
• end-to-end training