At the Edge of Understanding: Sparse Autoencoders Trace The Limits of Transformer Generalization

2026-06-24

AI Summary

This work addresses the vulnerability of pretrained Transformers to out-of-distribution (OOD) inputs—such as misspellings or jailbreak prompts—which perturb internal representations and compromise model reliability and safety. Moving beyond input-space analyses, the study innovatively reframes the OOD problem within the model’s internal computation, employing sparse autoencoders to dissect activation patterns under OOD conditions. This reveals a marked proliferation of spurious or erroneous concepts in hidden representations. Leveraging this mechanistic insight, the authors propose an inference-time diagnostic method and a targeted fine-tuning strategy that operate directly on internal representations. Their approach effectively quantifies distributional shift in prompts and substantially enhances the robustness and safety of large language models against adversarial and anomalous inputs, establishing a novel paradigm for secure deployment.

Abstract

Pre-trained transformers have demonstrated remarkable generalization abilities, at times extending beyond the scope of their training data. Yet, real-world deployments often face unexpected or adversarial data that diverges from training data distributions. Without explicit mechanisms for handling such shifts, model reliability and safety degrade, urging more disciplined study of out-of-distribution (OOD) settings for transformers. By systematic experiments, we present a mechanistic framework for delineating the precise contours of transformer model robustness. We find that OOD inputs, including subtle typos and jailbreak prompts, drive language models to operate on an increased number of fallacious concepts in their internals. We leverage this device to quantify and understand the degree of distributional shift in prompts, enabling a mechanistically grounded fine-tuning strategy to robustify LLMs. Expanding the very notion of OOD from input data to a model's private computational processes, a new transformer diagnostic at inference time is a critical step toward making AI systems safe for deployment across science, business, and government.

Problem

• out-of-distribution
• transformer robustness
• distributional shift
• language model safety
• adversarial inputs

Innovation

• sparse autoencoders
• out-of-distribution generalization
• mechanistic interpretability
• transformer robustness
• LLM safety