Set-Encoder: Permutation-Invariant Inter-Passage Attention for Listwise Passage Re-Ranking with Cross-Encoders

2024-04-10

arXiv.org

AI Summary

Existing cross-encoders for paragraph re-ranking effectively model inter-paragraph interactions but suffer from low computational efficiency and sensitivity to input ordering—compromising robustness. This paper proposes Set-Encoder, a novel cross-encoder architecture designed specifically for list-level re-ranking. Its core innovation is a permutation-invariant cross-paragraph attention mechanism, integrated with a set encoding paradigm, list-level ranking loss, and lightweight parameter sharing—achieving strict input-order invariance without sacrificing inference efficiency. Evaluated on TREC Deep Learning and TIREx benchmarks, Set-Encoder achieves state-of-the-art effectiveness while accelerating inference by over 35%. Moreover, it significantly outperforms pointwise models on highly interactive tasks such as novelty detection, demonstrating superior capability in modeling complex inter-document dependencies.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Existing cross-encoder models can be categorized as pointwise, pairwise, or listwise. Pairwise and listwise models allow passage interactions, which typically makes them more effective than pointwise models but less efficient and less robust to input passage order permutations. To enable efficient permutation-invariant passage interactions during re-ranking, we propose a new cross-encoder architecture with inter-passage attention: the Set-Encoder. In experiments on TREC Deep Learning and TIREx, the Set-Encoder is as effective as state-of-the-art listwise models while being more efficient and invariant to input passage order permutations. Compared to pointwise models, the Set-Encoder is particularly more effective when considering inter-passage information, such as novelty, and retains its advantageous properties compared to other listwise models. Our code is publicly available at https://github.com/webis-de/ECIR-25.

Problem

• Enables efficient permutation-invariant passage interactions in re-ranking
• Improves effectiveness over pointwise models using inter-passage information
• Maintains efficiency and order invariance compared to listwise models

Innovation

• Proposes Set-Encoder for passage re-ranking
• Uses inter-passage attention for permutation-invariance
• Balances efficiency and effectiveness in ranking