Neuro-RIT: Neuron-Guided Instruction Tuning for Robust Retrieval-Augmented Language Model

2026-04-02

AI Summary

This work addresses the significant performance degradation of current retrieval-augmented language models when exposed to irrelevant or noisy retrieved contexts, a challenge exacerbated by the neglect of neuron-level sparsity inherent in large language models. To this end, the authors propose Neuro-RIT, a novel framework that pioneers the integration of neuron-level sparsity into robustness optimization for retrieval-augmented systems. By leveraging attribution analysis to identify neurons critical for distinguishing relevant from irrelevant contexts, Neuro-RIT employs a two-stage instruction tuning strategy to functionally deactivate noise-sensitive neurons and distill evidence into key layers. This approach marks a paradigm shift from dense parameter updates to precise neuron-level alignment, consistently outperforming strong baselines across multiple question-answering benchmarks while demonstrating enhanced robustness to retrieval noise and superior task performance.

Abstract

Retrieval-Augmented Language Models (RALMs) have demonstrated significant potential in knowledge-intensive tasks; however, they remain vulnerable to performance degradation when presented with irrelevant or noisy retrieved contexts. Existing approaches to enhance robustness typically operate via coarse-grained parameter updates at the layer or module level, often overlooking the inherent neuron-level sparsity of Large Language Models (LLMs). To address this limitation, we propose Neuro-RIT (Neuron-guided Robust Instruction Tuning), a novel framework that shifts the paradigm from dense adaptation to precision-driven neuron alignment. Our method explicitly disentangles neurons that are responsible for processing relevant versus irrelevant contexts using attribution-based neuron mining. Subsequently, we introduce a two-stage instruction tuning strategy that enforces a dual capability for noise robustness: achieving direct noise suppression by functionally deactivating neurons exclusive to irrelevant contexts, while simultaneously optimizing targeted layers for evidence distillation. Extensive experiments across diverse QA benchmarks demonstrate that Neuro-RIT consistently outperforms strong baselines and robustness-enhancing methods.

Problem

• Retrieval-Augmented Language Models
• robustness
• neuron-level sparsity
• noisy contexts
• irrelevant contexts

Innovation

• Neuron-level Sparsity
• Retrieval-Augmented Language Models
• Instruction Tuning
• Noise Robustness
• Attribution-based Neuron Mining