Context Engineering for Trustworthiness: Rescorla Wagner Steering Under Mixed and Inappropriate Contexts

2025-09-01

AI Summary

This work addresses the degradation in response reliability of large language models (LLMs) when processing realistic mixed-context inputs containing irrelevant or harmful information. To tackle this, we propose a solution grounded in associative learning theory—specifically, we introduce the Rescorla–Wagner (RW) model to formalize LLMs’ contextual signal attribution, uncovering their anomalous preference for low-frequency harmful tokens. Building on this insight, we design RW-Steering: a lightweight, annotation-free mechanism for adaptive context filtering. We further construct the Poisoned Context Testbed and employ a two-stage fine-tuning strategy to separately identify and suppress competing contextual signals. Experiments demonstrate that our approach improves response quality by 39.8%, effectively reverses behavioral biases induced by harmful content, and generalizes robustly across varying contamination ratios.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Incorporating external context can significantly enhance the response quality of Large Language Models (LLMs). However, real-world contexts often mix relevant information with disproportionate inappropriate content, posing reliability risks. How do LLMs process and prioritize mixed context? To study this, we introduce the Poisoned Context Testbed, pairing queries with real-world contexts containing relevant and inappropriate content. Inspired by associative learning in animals, we adapt the Rescorla-Wagner (RW) model from neuroscience to quantify how competing contextual signals influence LLM outputs. Our adapted model reveals a consistent behavioral pattern: LLMs exhibit a strong tendency to incorporate information that is less prevalent in the context. This susceptibility is harmful in real-world settings, where small amounts of inappropriate content can substantially degrade response quality. Empirical evaluations on our testbed further confirm this vulnerability. To tackle this, we introduce RW-Steering, a two-stage finetuning-based approach that enables the model to internally identify and ignore inappropriate signals. Unlike prior methods that rely on extensive supervision across diverse context mixtures, RW-Steering generalizes robustly across varying proportions of inappropriate content. Experiments show that our best fine-tuned model improves response quality by 39.8% and reverses the undesirable behavior curve, establishing RW-Steering as a robust, generalizable context engineering solution for improving LLM safety in real-world use.

Problem

• LLMs struggle with mixed relevant and inappropriate context
• Small inappropriate content degrades LLM response quality
• Need robust method to ignore harmful contextual signals

Innovation

• Rescorla-Wagner model adaptation for context quantification
• Two-stage fine-tuning approach for inappropriate signal suppression
• Poisoned Context Testbed for empirical vulnerability evaluation