Causal Interventions on Continuous Variables: A Case Study on Verb Bias in Steering Vectors for In-Context Learning

2026-05-28

AI Summary

Existing causal intervention methods are largely confined to discrete features and struggle to handle continuous variables—such as verb bias—in language models. This work proposes the first causal intervention framework tailored for continuous variables, pairing activation vectors with continuous target attributes to identify low-dimensional intervention directions and perform counterfactual edits. Applied to verb bias in large language models, the method reveals a causal representation of verb bias within steering vectors. Although this representation encodes error signals, it is not causally leveraged by downstream generation processes. Experiments demonstrate that verb bias can be systematically manipulated, significantly altering syntactic preferences, thereby offering a novel perspective on the causal mechanisms underlying in-context learning.

Abstract

Causal interventions in language model representations have largely targeted discrete features, like grammatical number. However, language models must also make use of features that are graded. We introduce a method for causal intervention on continuous variables: given activation vectors paired with a graded target variable, we localize a low-dimensional direction for that variable and use this direction to edit a vectors toward counterfactual target values. We apply this method to a continuous feature that is well-studied in psycholinguistics, namely verb bias (which reflects which syntactic structures tend to follow a given verb). We show that verb bias is causally represented in steering vectors extracted from large language models: counterfactual edits to verb bias systematically shift downstream structural preferences. Verb bias has also previously been linked to in-context learning; in further analyses, we find that steering vectors encode error signals that could drive the error-driven update behavior seen in in-context learning but that these aspects of the steering vectors are not causally used in downstream production. Overall, these results show causal interventions can be applied to continuous variables, though connecting continuous variables to in-context learning remains a challenge.

Problem

• causal interventions
• continuous variables
• verb bias
• in-context learning
• steering vectors

Innovation

• causal intervention
• continuous variables
• verb bias
• steering vectors
• in-context learning