The Powerless Noise: How Experimental Settings Shape the Reported Power of Noise

2026-07-03

AI Summary

This study investigates the "power of noise" phenomenon in retrieval-augmented generation (RAG)—specifically, whether incorporating irrelevant documents genuinely enhances question-answering performance. Through systematic replication and extension of the original experiments, we conduct ablation analyses across diverse large language models, prompting strategies, and decoding constraints, while also examining the impact of truncation and formatting errors via error attribution. Our findings reveal that the "power of noise" manifests only under highly constrained inference settings; once output length limits are relaxed or prompting strategies adjusted, the effect markedly diminishes or vanishes entirely. These results suggest that the phenomenon does not stem from any intrinsic benefit of noise itself, but rather arises as an artifact of specific experimental configurations, thereby underscoring the critical role of inference design in RAG systems.

Abstract

Recent work has suggested that adding irrelevant documents to the input of retrieval-augmented generation (RAG) systems can improve question-answering performance, a phenomenon referred to as the ``\textit{Power of Noise}.'' This motivated investigations into the role of noise in information retrieval. In this paper, we reproduce the main findings of Cuconasu et al. \cite{cuconasu2024power} and evaluate the robustness of the effect under extended experimental settings. We first confirm that the phenomenon holds under the original setup, which uses earlier-generation LLMs, restrictive prompting and constrained decoding settings. We subsequently introduce a series of extensions to investigate the underlying causes of the noise effect, examining the authors' original design choices including the use of different models, instruction prompting, and relaxed output length constraints. Across these ablations, the Power-of-Noise pattern proves highly sensitive to inference configuration: it can appear, weaken, or disappear under small changes to prompt formulation and decoding limits. Combined with our error analysis, which shows substantial contributions from truncation and malformed generations, this variance indicates that the original effect cannot be robustly confirmed as a general benefit of noisy retrieval under these experimental conditions. More broadly, our work highlights the importance of carefully scrutinizing inference design in retrieval-augmented generation systems. Our code is available at https://github.com/ina0105/The-Power-of-Noise-Reproduction.

Problem

• retrieval-augmented generation
• noise
• robustness
• experimental settings
• question answering

Innovation

• Power of Noise
• retrieval-augmented generation
• inference configuration
• prompt design
• robustness evaluation