Investigating Task Arithmetic for Zero-Shot Information Retrieval

2025-05-01

AI Summary

To address the degraded cross-task and cross-domain generalization of large language models (LLMs) in zero-shot information retrieval—caused by lexical and distributional shifts—this paper introduces Task Arithmetic to zero-shot retrieval for the first time. We propose a fine-tuning-free, weight-arithmetic adaptation method that combines instruction-tuned LLM weights via addition and subtraction to enable multi-task and multi-domain knowledge fusion, yielding plug-and-play zero-shot re-ranking. Evaluated on scientific, biomedical, and multilingual retrieval benchmarks, our approach achieves up to +18% improvement in NDCG@10 and +15% in P@10 over state-of-the-art zero-shot re-rankers. This work establishes a new paradigm for lightweight, transferable, LLM-driven retrieval—requiring no parameter updates or task-specific training—while demonstrating strong zero-shot adaptability across heterogeneous domains and tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Language Models (LLMs) have shown impressive zero-shot performance across a variety of Natural Language Processing tasks, including document re-ranking. However, their effectiveness degrades on unseen tasks and domains, largely due to shifts in vocabulary and word distributions. In this paper, we investigate Task Arithmetic, a technique that combines the weights of LLMs pre-trained on different tasks or domains via simple mathematical operations, such as addition or subtraction, to adapt retrieval models without requiring additional fine-tuning. Our method is able to synthesize diverse tasks and domain knowledge into a single model, enabling effective zero-shot adaptation in different retrieval contexts. Extensive experiments on publicly available scientific, biomedical, and multilingual datasets show that our method improves state-of-the-art re-ranking performance by up to 18% in NDCG@10 and 15% in P@10. In addition to these empirical gains, our analysis provides insights into the strengths and limitations of Task Arithmetic as a practical strategy for zero-shot learning and model adaptation. We make our code publicly available at https://github.com/DetectiveMB/Task-Arithmetic-for-ZS-IR.

Problem

• Adapting LLMs for unseen tasks without fine-tuning
• Combining diverse task knowledge via weight arithmetic
• Improving zero-shot retrieval performance across domains

Innovation

• Combines LLM weights via arithmetic operations
• Synthesizes diverse tasks without fine-tuning
• Improves zero-shot retrieval performance significantly