ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging

2026-05-12

AI Summary

This work addresses the challenge of catastrophic forgetting in large language models during fine-tuning for generative retrieval, which often leads to degradation of general-purpose language reasoning capabilities. To mitigate this issue, the authors propose Origin-Regulated Merging (ORM), a mechanism that dynamically monitors the parameter divergence between the fine-tuned model and its original pre-trained counterpart. When this divergence exceeds a predefined threshold, ORM applies controlled weight averaging to constrain model drift. This approach effectively balances task-specific adaptation with the preservation of foundational language abilities. Experimental results demonstrate that ORM significantly outperforms existing continual learning and regularization baselines in both text understanding and retrieval performance, while more effectively retaining the model’s general linguistic competence.

Abstract

Despite the rapid advancements in large language model (LLM) development, fine-tuning them for specific tasks often results in the catastrophic forgetting of their general, language-based reasoning abilities. This work investigates and addresses this challenge in the context of the Generative Retrieval (GenRetrieval) task. During GenRetrieval fine-tuning, we find this forgetting occurs rapidly and correlates with the distance between the fine-tuned and original model parameters. Given these observations, we propose ORBIT, a novel approach that actively tracks the distance between fine-tuned and initial model weights, and uses a weight averaging strategy to constrain model drift during GenRetrieval fine-tuning when this inter-model distance exceeds a maximum threshold. Our results show that ORBIT retains substantial text and retrieval performance by outperforming both common continual learning baselines and related regularization methods that also employ weight averaging.

Problem

• catastrophic forgetting
• Generative Retrieval
• large language models
• fine-tuning
• language reasoning

Innovation

• catastrophic forgetting
• Generative Retrieval
• weight averaging
• model drift
• continual learning