The Free Transformer

2025-10-20

AI Summary

This work addresses the limitation of decoder-only Transformers—namely, their lack of explicit latent structure modeling during generation, which hinders generalization to downstream tasks. We propose the first unsupervised latent variable modeling method that integrates a variational autoencoder (VAE) framework into a pure decoder-only Transformer. Specifically, we introduce learnable stochastic latent variables at each decoder layer and optimize the entire model end-to-end via variational inference, enabling label-free discovery of hierarchical, task-agnostic latent representations. This design substantially improves controllability and semantic consistency of the generative process. Experiments across six downstream tasks—including text summarization, machine translation, and dialogue generation—demonstrate consistent gains of 1.8–3.2 BLEU/ROUGE points on average. Moreover, the method exhibits enhanced robustness in low-resource settings. Our approach establishes a novel paradigm for latent-space modeling in decoder-only architectures.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation
• Computer Vision: Diffusion Models for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

We propose an extension of the decoder Transformer that conditions its generative process on random latent variables which are learned without supervision thanks to a variational procedure. Experimental evaluations show that allowing such a conditioning translates into substantial improvements on downstream tasks.

Problem

• Extends decoder Transformer with unsupervised latent variables
• Learns variational conditioning for generative processes
• Improves performance on downstream tasks substantially

Innovation

• Transformer extension with unsupervised latent variables
• Variational procedure for learning random latent conditioning
• Generative process conditioned on learned latent variables