Where Does Authorship Signal Emerge in Encoder-Based Language Models?

2026-05-19

AI Summary

Under identical pre-trained encoders, training data, and loss functions, different scoring mechanisms yield up to a fourfold difference in authorship attribution performance—a discrepancy whose origin has long remained unclear. This work integrates causal intervention, mechanistic interpretability analysis, gradient structure decomposition, and training dynamics tracking to reveal, for the first time, that scoring mechanisms shape gradient structures to guide where authorial style signals are integrated across network layers. Specifically, mean pooling concentrates signal utilization in early-to-mid layers, whereas late interaction defers it to deeper layers. Moving beyond the conventional focus on representation quality alone, this study demonstrates that the choice of scoring mechanism fundamentally determines the layer-wise locus of signal exploitation, thereby inducing divergent learning trajectories that account for the observed performance gap.

Abstract

Authorship attribution models fine-tuned with the same pretrained encoder, data, and loss can differ four-fold in performance depending only on their scoring mechanism. We use mechanistic interpretability tools to explain this gap. Stylistic features such as word length, punctuation density, and function-word frequency are equally available at every layer in every model, including in an off-the-shelf control encoder, hence the gap not coming from representation quality. Instead, causal intervention shows that the scorer determines where the encoder consolidates authorship signal. Mean pooling forces consolidation by early to mid layers, while late interaction defers it to later layers. We further derive this difference from the gradient structure of each scorer, and training dynamics reveal distinct learning trajectories that follow from that difference.

Problem

• authorship attribution
• encoder-based language models
• scoring mechanism
• mechanistic interpretability
• stylistic features

Innovation

• authorship attribution
• mechanistic interpretability
• scoring mechanism
• gradient structure
• training dynamics