That's Deprecated! Understanding, Detecting, and Steering Knowledge Conflicts in Language Models for Code Generation

2025-10-21

AI Summary

This study investigates how large language models (LLMs) resolve implicit knowledge conflicts between parametric knowledge and prompt-specified information in code generation. To address this, we propose the first knowledge conflict modeling and resolution framework tailored to code generation: (1) a domain-agnostic conflict construction methodology; (2) a layer-wise activation-based steering intervention mechanism; and (3) the first dedicated code-specific conflict evaluation dataset and automated detection method. Experimental results demonstrate that LLMs possess implicit conflict recognition capability; targeted layer-wise steering significantly improves guided generation success—outperforming random baselines by 12.6%; and our conflict detector achieves up to 80.65% accuracy. This work pioneers the systematic analysis of knowledge conflicts in code generation, revealing fundamental trade-offs among model scale, task difficulty, and intervention directionality.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

This paper investigates how large language models (LLMs) behave when faced with discrepancies between their parametric knowledge and conflicting information contained in a prompt. Building on prior question-answering (QA) research, we extend the investigation of knowledge conflicts to the realm of code generation. We propose a domain-agnostic framework for constructing and interpreting such conflicts, along with a novel evaluation method and dataset tailored to code conflict scenarios. Our experiments indicate that sufficiently large LLMs encode the notion of a knowledge conflict in their parameters, enabling us to detect knowledge conflicts with up to extbf{80.65%} accuracy. Building on these insights, we show that activation-level steering can achieve up to a extbf{12.6%} improvement in steering success over a random baseline. However, effectiveness depends critically on balancing model size, task domain, and steering direction. The experiment code and data will be made publicly available after acceptance.

Problem

• Investigating LLM behavior with parametric knowledge versus prompt conflicts
• Detecting knowledge conflicts in code generation with high accuracy
• Improving conflict steering through activation-level model interventions

Innovation

• Detecting knowledge conflicts in code generation models
• Activation-level steering improves conflict resolution success
• Domain-agnostic framework for constructing code conflicts