Simplifying the Modeling of Arbitrary Conditionals in Natural Language

2026-06-12

AI Summary

Standard causal Transformers struggle to efficiently model and sample arbitrary conditional distributions, such as text that depends simultaneously on both past and future context. This work proposes a lightweight architectural modification that enables flexible conditioning—including mixed temporal contexts—within a single forward pass, while preserving the standard left-to-right training objective and autoregressive decoding mechanism. The approach seamlessly integrates into existing large language model training pipelines, allowing direct fine-tuning without disrupting conventional autoregressive generation. Empirical results demonstrate that the method significantly outperforms current baselines on tasks requiring arbitrary conditional modeling, all while maintaining competitive performance in standard autoregressive text generation.

Abstract

Causal Transformers model sequences through an autoregressive factorization of the joint distribution, which enables efficient left-to-right decoding and conditional likelihood computation. However, they cannot tractably sample from or evaluate arbitrary conditionals -- e.g., a block of text conditioned on past and future tokens. Recent work aims to solve this problem through novel architectures, but they often lead to sub-optimal modeling of such conditionals and degraded generations. We propose Arbitrary Conditionals GPT (AC-GPT) which introduces a simple modification to standard causal Transformers to enable evaluating and sampling from arbitrary conditionals -- including past, future, and mixed contexts -- within a single forward pass. Unlike prior approaches, our method preserves the standard left-to-right ordering and next-token prediction objective essential for both strong performance and efficient training on natural language. Crucially, this compatibility allows existing LLMs to be fine-tuned for arbitrary conditioning. Our empirical results indicate that our method outperforms baselines on modeling arbitrary conditionals, without degrading standard left-to-right performance.

Problem

• arbitrary conditionals
• causal Transformers
• natural language modeling
• conditional sampling
• language generation

Innovation

• Arbitrary Conditionals
• Causal Transformers
• AC-GPT
• Conditional Generation
• Language Modeling