Pigeonholing: Bad prompts hurt models to collapse and make mistakes

2026-06-23

AI Summary

This study identifies and quantifies a previously uncharacterized phenomenon termed the “pigeonhole effect,” wherein large language models, when exposed to adverse context—such as user suggestions or their own prior erroneous responses—tend to repeat mistakes, lose response diversity, and shift stance, even when the contextual cues are non-adversarial or contain correct examples. The authors demonstrate the prevalence of this effect across 10 diverse tasks and 10 prominent models, showing that error repetition degrades performance by 38–40%, with multi-turn interactions exacerbating the decline by over 14%. To mitigate this issue, they propose RLVR, a reinforcement learning-based fine-tuning approach augmented with synthetically generated erroneous data, which substantially improves model robustness in adverse contexts, yielding performance gains of 43–60%.

Abstract

While in-context learning is generally shown to be effective in Large Language Models (LLMs), bad contexts can cause performance degradation and mode collapse, a phenomenon we call "pigeonholing." **Unintentionally bad** contexts can happen without malicious jailbreaking intents: For example, a user asks the model to justify an incorrect math theorem or fails to correct the model's buggy code. Specifically, we investigate ``pigeonholing" in two scenarios: (1) when the user suggests a solution, and (2) when the conversation context includes the assistant's previous (incorrect) responses. Our experiments across 10 verifiable and open-ended tasks with 10 different models show that pigeonholing manifests in several ways: (1) repeating the incorrect answers from context (leading to 38-40% performance drop), (2) converging on a narrow set of answers in coding and text generation without exploring alternatives, and (3) flipping stance on controversial topics to align with the user or the assistant's previous claims. We find that pigeonholing worsens almost monotonically with the number of conversation turns (performance drops by additional 14+% as repeated mistakes increase from 1 to 5), and pigeonholing-induced mode collapse can happen even when the provided example is correct. As a step toward mitigation, we propose RLVR with synthetic errors which improves models by 43-60% under bad contexts compared to vanilla RLVR baselines.

Problem

• pigeonholing
• in-context learning
• mode collapse
• large language models
• bad prompts

Innovation

• pigeonholing
• in-context learning
• mode collapse
• RLVR
• synthetic errors