On the Scaling of Robustness and Effectiveness in Dense Retrieval

2025-05-30

AI Summary

This work investigates whether robustness (out-of-distribution and adversarial) and effectiveness in dense retrieval models obey scaling laws—and, if so, what their fundamental relationship is. We establish, for the first time, that robustness indeed follows a scaling law, but with a distinct functional form from effectiveness; crucially, the two are not inherently antagonistic but admit joint optimization. Building on this insight, we propose Pareto Training: a multi-objective optimization framework grounded in Pareto frontier modeling that simultaneously enhances both robustness and effectiveness—without increasing model parameters or training data volume. Extensive evaluation across DPR and BGE architectures demonstrates that our method achieves joint performance comparable to conventional scaling with 2–3× larger models, all at zero additional resource cost. This yields substantial reductions in industrial deployment overhead and establishes a new paradigm for efficient, robust retrieval.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Robustness and Effectiveness are critical aspects of developing dense retrieval models for real-world applications. It is known that there is a trade-off between the two. Recent work has addressed scaling laws of effectiveness in dense retrieval, revealing a power-law relationship between effectiveness and the size of models and data. Does robustness follow scaling laws too? If so, can scaling improve both robustness and effectiveness together, or do they remain locked in a trade-off? To answer these questions, we conduct a comprehensive experimental study. We find that:(i) Robustness, including out-of-distribution and adversarial robustness, also follows a scaling law.(ii) Robustness and effectiveness exhibit different scaling patterns, leading to significant resource costs when jointly improving both. Given these findings, we shift to the third factor that affects model performance, namely the optimization strategy, beyond the model size and data size. We find that: (i) By fitting different optimization strategies, the joint performance of robustness and effectiveness traces out a Pareto frontier. (ii) When the optimization strategy strays from Pareto efficiency, the joint performance scales in a sub-optimal direction. (iii) By adjusting the optimization weights to fit the Pareto efficiency, we can achieve Pareto training, where the scaling of joint performance becomes most efficient. Even without requiring additional resources, Pareto training is comparable to the performance of scaling resources several times under optimization strategies that overly prioritize either robustness or effectiveness. Finally, we demonstrate that our findings can help deploy dense retrieval models in real-world applications that scale efficiently and are balanced for robustness and effectiveness.

Problem

• Investigates scaling laws for robustness in dense retrieval models
• Explores trade-offs between robustness and effectiveness during scaling
• Proposes Pareto training to optimize joint performance efficiently

Innovation

• Scaling laws apply to robustness and effectiveness
• Pareto frontier optimizes joint performance efficiently
• Pareto training balances robustness and effectiveness