Differentiable Geometric Indexing for End-to-End Generative Retrieval

2026-03-11

AI Summary

This work addresses the limitations of generative retrieval, where discrete, non-differentiable indices and inner-product-based objectives suppress long-tail items in favor of popular ones. To overcome this, the authors propose the first end-to-end differentiable framework that unifies indexing and retrieval. By modeling semantic relevance on the unit hypersphere and replacing inner products with normalized cosine similarity, the method enables isotropic geometric optimization. It further incorporates Gumbel-Softmax-based soft teacher forcing and symmetric weight sharing to jointly achieve popularity debiasing and representation learning. This approach uniquely integrates fully differentiable indexing with geometric debiasing mechanisms, significantly outperforming state-of-the-art sparse, dense, and generative retrieval baselines on large-scale industrial search and e-commerce datasets, with particularly pronounced gains in retrieving long-tail items.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Generative Retrieval (GR) has emerged as a promising paradigm to unify indexing and search within a single probabilistic framework. However, existing approaches suffer from two intrinsic conflicts: (1) an Optimization Blockage, where the non-differentiable nature of discrete indexing creates a gradient blockage, decoupling index construction from the downstream retrieval objective; and (2) a Geometric Conflict, where standard unnormalized inner-product objectives induce norm-inflation instability, causing popular"hub"items to geometrically overshadow relevant long-tail items. To systematically resolve these misalignments, we propose Differentiable Geometric Indexing (DGI). First, to bridge the optimization gap, DGI enforces Operational Unification. It employs Soft Teacher Forcing via Gumbel-Softmax to establish a fully differentiable pathway, combined with Symmetric Weight Sharing to effectively align the quantizer's indexing space with the retriever's decoding space. Second, to restore geometric fidelity, DGI introduces Isotropic Geometric Optimization. We replace inner-product logits with scaled cosine similarity on the unit hypersphere to effectively decouple popularity bias from semantic relevance. Extensive experiments on large-scale industry search datasets and online e-commerce platform demonstrate that DGI outperforms competitive sparse, dense, and generative baselines. Notably, DGI exhibits superior robustness in long-tail scenarios, validating the necessity of harmonizing structural differentiability with geometric isotropy.

Problem

• Generative Retrieval
• Optimization Blockage
• Geometric Conflict
• Differentiable Indexing
• Long-tail Retrieval

Innovation

• Differentiable Geometric Indexing
• Generative Retrieval
• Soft Teacher Forcing
• Isotropic Geometric Optimization
• Long-tail Retrieval