ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models

2026-01-07

arXiv.org

AI Summary

This work addresses the tendency of large language models to rely on internal memorized knowledge over conflicting but relevant external retrieval context, often resulting in unfaithful outputs. To mitigate this issue, the authors propose a lightweight activation steering method that intervenes in the activation signals of key transformer layers. Without requiring fine-tuning or incurring significant inference overhead, this approach substantially enhances contextual faithfulness while preserving generation fluency. The method is compatible with existing prompting strategies and scales effectively to large models. Evaluated on the ConFiQA benchmark, it significantly outperforms strong baselines such as ContextDPO and COIECD, demonstrating its effectiveness, robustness, and computational efficiency.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Reasoning with Beliefs

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Large Language Models (LLMs) encode vast amounts of parametric knowledge during pre-training. As world knowledge evolves, effective deployment increasingly depends on their ability to faithfully follow externally retrieved context. When such evidence conflicts with the model's internal knowledge, LLMs often default to memorized facts, producing unfaithful outputs. In this work, we introduce ContextFocus, a lightweight activation steering approach that improves context faithfulness in such knowledge-conflict settings while preserving fluency and efficiency. Unlike prior approaches, our solution requires no model finetuning and incurs minimal inference-time overhead, making it highly efficient. We evaluate ContextFocus on the ConFiQA benchmark, comparing it against strong baselines including ContextDPO, COIECD, and prompting-based methods. Furthermore, we show that our method is complementary to prompting strategies and remains effective on larger models. Extensive experiments show that ContextFocus significantly improves contextual-faithfulness. Our results highlight the effectiveness, robustness, and efficiency of ContextFocus in improving contextual-faithfulness of LLM outputs.

Problem

• contextual faithfulness
• knowledge conflict
• large language models
• retrieval-augmented generation

Innovation

• activation steering
• contextual faithfulness
• knowledge conflict
• parameter-free intervention
• retrieval-augmented generation