Search-Induced Issues in Web-Augmented LLM Code Generation: Detecting and Repairing Error-Inducing Pages

2026-03-27

AI Summary

This work addresses a novel failure mode in web-augmented large language models (LLMs), termed Search-Induced Inaccuracy (SII), wherein unreliable or malicious web content misleads LLMs into generating erroneous code. The study presents the first systematic characterization of SII and introduces Sherlock, an end-to-end automated defense pipeline that integrates LLMs, web search, static and dynamic code analysis, trusted source verification, and repair strategies to accurately detect and effectively mitigate SII. Experimental evaluation demonstrates that Sherlock achieves a 95% F1 score in detecting erroneous induced pages (EIPs) across multiple mainstream LLMs and search APIs, and successfully repairs between 71% and 100% of the generated errors, all with manageable computational overhead.

Abstract

Web-augmented large language models (LLMs) offer promising capabilities for automatic code generation. However, integrating live web search exposes models to unreliable or malicious content, leading to Search-Induced Issues (SII), a novel failure mode in which external pages mislead LLMs into producing incorrect code. This paper presents a comprehensive empirical study of the prevalence and impact of SII across three commercial search APIs and six advanced LLMs. Our analysis reveals that all evaluated web-augmented LLMs are vulnerable to SII, with root causes arising from either misaligned specifications or flawed code implementations in the searched Error-Inducing Pages (EIPs). To address this challenge, we propose Sherlock, an automated framework that enables LLM service providers to proactively safeguard web-augmented generation systems at scale. Sherlock operates as a continuous pipeline that first detects potential SII instances, then debugs them to identify the responsible EIPs and pinpoint their root causes, and finally repairs them by either annotating misaligned content or replacing erroneous code snippets with evaluated solutions from trusted sources. Experiments show that Sherlock identifies EIPs with an F1 score of up to 95% and repairs 71% to 100% of affected generations across the evaluated models, with modest computational overhead. Our findings and framework provide practical guidance for improving the reliability of web-augmented LLM-based code generation systems in real-world software engineering scenarios.

Problem

• Search-Induced Issues
• Web-Augmented LLMs
• Error-Inducing Pages
• Code Generation
• LLM Reliability

Innovation

• Search-Induced Issues
• Web-Augmented LLMs
• Error-Inducing Pages
• Automated Repair
• Code Generation Reliability