Architectural and Inferential Inductive Biases For Exchangeable Sequence Modeling

2025-03-03

AI Summary

This paper addresses two critical limitations in exchangeable sequence modeling: (i) the inability to disentangle epistemic from aleatoric uncertainty, and (ii) the lack of theoretical guarantees—particularly strict exchangeability—in existing architectures. We systematically analyze how inference mechanisms and structural inductive biases affect posterior uncertainty quantification. We show that standard single-step autoregressive modeling conflates uncertainty types, and current exchangeable Transformers violate strict permutation invariance. To resolve these issues, we propose a multi-step autoregressive generative framework grounded in Bayesian posterior inference and causal masking analysis, along with a novel architectural design principle ensuring provable exchangeability. Through rigorous theoretical analysis and controlled synthetic experiments, we demonstrate that our architecture achieves significantly improved uncertainty calibration. It consistently outperforms baselines on downstream decision-making tasks—including active learning and contextual bandits—thereby exposing structural inefficiencies and redundant computation inherent in prevailing models.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Autoregressive models have emerged as a powerful framework for modeling exchangeable sequences - i.i.d. observations when conditioned on some latent factor - enabling direct modeling of uncertainty from missing data (rather than a latent). Motivated by the critical role posterior inference plays as a subroutine in decision-making (e.g., active learning, bandits), we study the inferential and architectural inductive biases that are most effective for exchangeable sequence modeling. For the inference stage, we highlight a fundamental limitation of the prevalent single-step generation approach: inability to distinguish between epistemic and aleatoric uncertainty. Instead, a long line of works in Bayesian statistics advocates for multi-step autoregressive generation; we demonstrate this"correct approach"enables superior uncertainty quantification that translates into better performance on downstream decision-making tasks. This naturally leads to the next question: which architectures are best suited for multi-step inference? We identify a subtle yet important gap between recently proposed Transformer architectures for exchangeable sequences (Muller et al., 2022; Nguyen&Grover, 2022; Ye&Namkoong, 2024), and prove that they in fact cannot guarantee exchangeability despite introducing significant computational overhead. We illustrate our findings using controlled synthetic settings, demonstrating how custom architectures can significantly underperform standard causal masks, underscoring the need for new architectural innovations.

Problem

• Addresses limitations in modeling exchangeable sequences with autoregressive models.
• Explores effective inferential and architectural biases for uncertainty quantification.
• Identifies gaps in Transformer architectures for ensuring sequence exchangeability.

Innovation

• Multi-step autoregressive generation for uncertainty quantification
• Custom architectures underperform standard causal masks
• Transformers fail to guarantee exchangeability despite overhead