ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions

2026-05-22

AI Summary

This work addresses the susceptibility of state-of-the-art language models to persona drift during extended, multi-turn programming sessions involving tool use, which undermines their intended role as preference-agnostic coding assistants—a phenomenon inadequately captured by existing evaluation methods. To this end, the authors introduce ContextEcho, a novel benchmark that establishes the first framework for assessing persona stability over thousands of interaction rounds in realistic deployment scenarios. ContextEcho features a snapshot-probe protocol, 25 identity probes, dialogue state forking, and a combined discriminative and non-discriminative evaluation mechanism. Experiments reveal that persona drift is prevalent across 23 mainstream models and consistent across institutions; a single anchoring intervention can restore original linguistic style, while dialogue compression fails to reliably reset drift. Notably, drift may be beneficial during tool-augmented interactions but degrades output formatting and increases verbosity in pure chat settings.

Abstract

A frontier language model's acknowledged "helpful programming assistant" persona does not survive long agentic-coding sessions in the deployment regime that production products actually run. After hours of tool-using debugging, a model that initially hedges preferences ("I don't have preferences") may begin asserting them ("Python - the feedback loop is instant..."), revealing user-visible drift that deployer evaluations may miss. Existing persona-stability studies focus on short dialogues and report little shift, leaving real-world code-generation regimes - thousands of tool-using turns, compaction, and hours-long sessions - largely uncharacterized. We introduce ContextEcho, a benchmark and reusable harness for measuring persona drift at deployment scale. It combines a 25-probe identity suite, a snapshot-then-probe protocol that forks conversation state without perturbing the main session, complementary judged and judge-free measurement surfaces, and three anonymized Claude Code sessions spanning 3,746-9,716 turns. Across 23 frontier models, ContextEcho shows that persona drift is general across organizations rather than family-specific, that in-session compaction does not reliably reset it, and that a single-shot anchor restores the trained register across measured targets. It also reveals mode-dependent downstream effects: while drift can facilitate tool-using continuation, in tool-free chat it breaks formatting contracts and inflates output length. Overall, ContextEcho provides researchers and deployers an open-source framework to audit whether the persona a model ships with is the persona users encounter at session end, across chat-completions API targets and without retraining.

Problem

• persona drift
• long-context coding
• agentic sessions
• language model deployment
• identity stability

Innovation

• persona drift
• long-context agentic coding
• ContextEcho benchmark
• snapshot-then-probe protocol
• deployment-scale evaluation