Chunky Post-Training: Data Driven Failures of Generalization

2026-02-05

AI Summary

This work identifies and formalizes a previously uncharacterized issue in large language models termed “Chunky Post-Training,” wherein the segmentation of post-training data into discrete chunks introduces spurious correlations among formatting, phrasing, and content. These artificial associations lead to degraded generalization and inconsistent model behavior. To address this, the study proposes SURF, a black-box runtime diagnostic tool for behavioral analysis, and TURF, a data provenance tracer that enables retrospective identification of problematic training chunks. Empirical evaluations across leading models—including Claude 4.5, GPT-5.1, Grok 4.1, Gemini 3, and Tülü 3—demonstrate the widespread prevalence of this phenomenon, which stems primarily from imbalanced or semantically ambiguous data chunks in the post-training corpus.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

LLM post-training involves many diverse datasets, each targeting a specific behavior. But these datasets encode incidental patterns alongside intended ones: correlations between formatting and content, narrow phrasings across diverse problems, and implicit associations arising from the discrete data curation process. These patterns are often invisible to developers yet salient to models, producing behaviors that surprise their creators, such as rejecting true facts presented in a particular question format. We call this chunky post-training: the model learns spurious correlations as a result of distinct chunks of post-training data. We introduce SURF, a black-box pipeline which surfaces these unintended behaviors at run time, and TURF, a tool that traces these failures back to specific post-training data. Applying these tools to frontier models (Claude 4.5, GPT-5.1, Grok 4.1, Gemini 3) and open models (T\"ulu 3), we show that chunky post-training produces miscalibrated behaviors, which often result from imbalanced or underspecified chunks of post-training data.

Problem

• chunky post-training
• spurious correlations
• unintended behaviors
• data curation
• generalization failures

Innovation

• Chunky Post-Training
• spurious correlations
• SURF
• TURF
• post-training data