Conditional Attribute Estimation with Autoregressive Sequence Models

2026-05-13

AI Summary

This work addresses the limitations of traditional autoregressive models, which rely solely on next-token prediction and struggle to capture sequence-level properties, often resulting in local overfitting and poor global structure modeling. Moreover, controllable generation typically demands costly sampling or architectural modifications. To overcome these challenges, the authors propose the Conditional Attribute Transformer, which jointly optimizes next-token prediction and sequence attribute estimation conditioned on the current token within a single forward pass. This approach uniquely enables token-level attribute attribution, counterfactual attribute evaluation, and attribute-guided generation—all within a unified model without additional sampling or structural changes. Experiments demonstrate state-of-the-art performance on sparse-reward tasks, improved language modeling at large scales, attribute estimation orders of magnitude faster than sampling-based methods, and effective guidance across diverse language generation tasks.

Abstract

Generative models are often trained with a next-token prediction objective, yet many downstream applications require the ability to estimate or control sequence-level properties. Next-token prediction can lead to overfitting of local patterns during training, underfitting of global structure, and requires significant downstream modifications or expensive sampling to guide or predict the global attributes of generated samples at inference time. Here, we introduce Conditional Attribute Transformers, a novel method for jointly estimating the next-token probability and the value of an attribute conditional on each potential next token selection. This framework enables three critical capabilities within a single forward pass, without modification of the input sequence: (1) per-token credit assignment across an entire sequence, by identifying how each token in a sequence is associated with an attribute's value; (2) counterfactual analysis, by quantifying attribute differences conditional on alternative next token choices; (3) steerable generation, by decoding sequences based on a combination of next-token and attribute likelihoods. Our approach achieves state of the art performance on sparse reward tasks, improves next-token prediction at sufficient model sizes, estimates attribute probabilities orders of magnitude faster than sampling, and can guide decoding of autoregressive sequence models on a range of language tasks.

Problem

• attribute estimation
• autoregressive sequence models
• sequence-level properties
• next-token prediction
• global structure

Innovation

• Conditional Attribute Estimation
• Autoregressive Sequence Models
• Steerable Generation
• Counterfactual Analysis
• Per-token Credit Assignment