Old Habits Die Hard: How Conversational History Geometrically Traps LLMs

2026-02-08

AI Summary

This study investigates behavioral biases in large language models induced by dialogue history, with a particular focus on how hallucinations from prior interactions persistently distort subsequent responses. To address this, the authors propose the History-Echoes framework, which for the first time models history-induced generation bias as “geometric traps” in latent space, revealing a strong correlation between probabilistic state consistency and geometric consistency of hidden representations. Methodologically, the approach combines Markov chain modeling of dialogue states with a geometric consistency metric derived from continuous hidden representations to quantify historical bias. Extensive experiments across three prominent model families and six datasets validate the high correlation between these dual perspectives, demonstrating that dialogue history can indeed form geometric traps in latent space that constrain the model’s generation trajectory.

Abstract

How does the conversational past of large language models (LLMs) influence their future performance? Recent work suggests that LLMs are affected by their conversational history in unexpected ways. For instance, hallucinations in prior interactions may influence subsequent model responses. In this work, we introduce History-Echoes, a framework that investigates how conversational history biases subsequent generations. The framework explores this bias from two perspectives: probabilistically, we model conversations as Markov chains to quantify state consistency; geometrically, we measure the consistency of consecutive hidden representations. Across three model families and six datasets spanning diverse phenomena, our analysis reveals a strong correlation between the two perspectives. By bridging these perspectives, we demonstrate that behavioral persistence manifests as a geometric trap, where gaps in the latent space confine the model's trajectory. Code available at https://github.com/technion-cs-nlp/OldHabitsDieHard.

Problem

• conversational history
• large language models
• behavioral persistence
• geometric trap
• hidden representations

Innovation

• conversational history
• geometric trap
• hidden representations
• Markov chains
• behavioral persistence