Prototype Language Models

2026-07-01

AI Summary

This work addresses the challenge of efficiently tracing the influence of training samples on outputs in large language models. The authors propose PRISM, an architecture that performs prediction via sparse non-negative prototype mixtures, where each prototype is anchored by clustering objectives to coherent neighborhoods in the training data, thereby explicitly linking predictions to specific training instances. PRISM enables highly efficient attribution—approximately 500× faster than baseline methods—and supports behavior modification without fine-tuning. The approach further incorporates Hessian curvature localization and a linear prototype controller for calibration. Evaluated across models ranging from 130M to 1.6B parameters, PRISM incurs at most a 2.5-point accuracy drop relative to dense baselines on downstream tasks; prototype calibration recovers about 3 points of accuracy, and selective suppression of prototypes can eliminate targeted behaviors without degrading generation quality.

Abstract

Knowing which training examples drive outputs is fundamental to auditing, correcting, and understanding language models, yet for modern LLMs this remains expensive, approximate, and largely post-hoc. Standard language models generate tokens through a dense network pathway, causing training data's influence to be distributed across parameters rather than organized along explicit, traceable components. We introduce a prototype language model architecture, Prototypes for Interpretable Sequence Modeling (PRISM), that forms each prediction via a sparse, non-negative mixture of learned prototypes, trained with clustering objectives that anchor each prototype to coherent neighborhoods of training examples. Across architectures from 130M to 1.6B parameters trained on up to 50B tokens, prototype language models either surpass or remain within 2.5 percentage points on average downstream accuracy of matched dense baselines. We show that sparse prototype structure localizes curvature in the loss landscape, yielding a more tractable Hessian and enabling training data attribution that is ~500x faster than post hoc baselines when consuming equivalent memory. Calibrating linear prototype controllers can improve downstream accuracy by roughly 3 points while tracing those corrections back to training neighborhoods, and targeted prototype suppression can remove model behaviors without finetuning or measurable loss in generation quality.

Problem

• training data attribution
• interpretability
• language models
• prototype-based modeling
• model auditing

Innovation

• prototype language models
• interpretable sequence modeling
• training data attribution
• sparse mixture of prototypes
• loss landscape curvature