One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness

2026-04-14

AI Summary

This study reveals that instruction-tuned large language models exhibit systematic response collapse under minimal lexical constraints—such as the prohibition of a single punctuation mark or common word—manifesting as a significant drop (14–48%) in answer comprehensiveness, a phenomenon absent in base models. Through two-stage generation, linear representation probing, and LLM-as-judge preference evaluation, the authors demonstrate that this collapse stems not from inherent model limitations but from surface-form template dependencies introduced during instruction tuning, which disrupt planning processes. Standard automated metrics substantially underestimate the resulting quality degradation. The proposed linear probes effectively predict collapse severity (R² = 0.51–0.93), while two-stage generation recovers 59–96% of response length and demonstrates strong generalization across multi-task settings in MT-Bench evaluations.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Instruction-tuned large language models produce helpful, structured responses, but how robust is this helpfulness when trivially constrained? We show that simple lexical constraints (banning a single punctuation character or common word) cause instruction-tuned LLMs to collapse their responses, losing 14--48% of comprehensiveness in pairwise evaluation across three open-weight model families and one closed-weight model (GPT-4o-mini). The baseline response is preferred in 77--100% of 1,920 pairwise comparisons judged by GPT-4o-mini and GPT-4o. Notably, GPT-4o-mini suffers 31% comprehensiveness loss (99% baseline win rate), demonstrating that the fragility extends to commercially deployed closed-weight models, contrary to prior findings on format-level constraints. Through mechanistic analysis, we identify this as a planning failure: two-pass generation (free generation followed by constrained rewriting) recovers 59--96% of response length, and linear probes on prompt representations predict response length with $R^2 = 0.51$--$0.93$ before generation begins, with $R^2$ tracking collapse severity across models. The same probes yield negative $R^2$ on base models, confirming that instruction tuning creates the representational structure encoding the collapse decision. Crucially, base models show no systematic collapse under identical constraints, with effects that are small, noisy, and bidirectional, demonstrating that instruction tuning creates this fragility by coupling task competence to narrow surface-form templates. The effect replicates on MT-Bench across all eight task categories. We further show that standard independent LLM-as-judge evaluation detects only a 3.5% average quality drop where pairwise evaluation reveals 23%, exposing a methodological blind spot in how constrained generation is assessed.

Problem

• instruction tuning
• lexical constraints
• helpfulness fragility
• large language models
• constrained generation

Innovation

• instruction tuning
• lexical constraints
• response collapse
• mechanistic analysis
• LLM robustness