In-Context Examples Suppress Scientific Knowledge Recall in LLMs

2026-04-30

AI Summary

This study investigates how in-context examples in large language models often suppress the activation of pretrained scientific knowledge during scientific reasoning, steering models toward empirical pattern matching rather than structured, knowledge-driven inference. Through 6,000 controlled experiments across four mainstream models and 60 interdisciplinary latent-structure recovery tasks, the work systematically demonstrates that in-context learning consistently shifts reasoning mechanisms—manifesting as decreased, unchanged, or deceptively improved accuracy—while fundamentally deviating from reliance on intrinsic scientific principles. The findings reveal, for the first time, that contextual examples can undermine models’ dependence on embedded scientific formulas, highlighting the dual-edged nature of in-context learning in scientific reasoning and its consistent effects across diverse architectures.

Abstract

Scientific reasoning rarely stops at what is directly observable; it often requires uncovering hidden structure from data. From estimating reaction constants in chemistry to inferring demand elasticities in economics, this latent structure recovery is what distinguishes scientific reasoning from curve fitting. Large language models (LLMs) can often recall and apply relevant scientific formulas, but we show that this ability is surprisingly easy to suppress. We show that adding in-context examples makes models rely less on pretrained domain knowledge, even when those examples are generated by the very same formula. Rather than reinforcing knowledge-driven derivation, examples shift computation toward empirical pattern fitting. We document this knowledge displacement on 60 latent structure recovery tasks across five scientific domains, 6,000 trials, and four models. This displacement is consistent across domains, but its accuracy consequences depend on how the displaced strategy compares to the one that replaces it: the same shift can lower accuracy, leave it unchanged, or appear to improve it. In all cases, however, the model shifts away from knowledge-driven reasoning. For practitioners deploying LLMs on scientific tasks, the message is cautionary: in-context examples may displace, rather than reinforce, the knowledge they are intended to support.

Problem

• in-context learning
• scientific reasoning
• latent structure recovery
• knowledge recall
• large language models

Innovation

• in-context learning
• scientific reasoning
• knowledge displacement
• latent structure recovery
• large language models