Order-Agnostic Autoregressive Modelling with Missing Data

2026-05-07

AI Summary

This work addresses the challenges of training and inference in permutation-invariant autoregressive models under missing-not-at-random data by reframing the problem as a general missing data modeling task. It introduces, for the first time, an end-to-end training framework that is compatible with arbitrary missingness mechanisms. The proposed approach integrates amortized conditional density estimation with an active variable selection strategy, enabling both efficient imputation and targeted information acquisition. The resulting model, termed MO-ARM, demonstrates substantial performance gains over existing methods across multiple real-world benchmarks, exhibiting exceptional robustness and generalization capability under diverse missing data scenarios.

Abstract

Order-Agnostic autoregressive models have demonstrated strong performance in deep generative modeling, yet their use in settings with incomplete data remains largely unexplored. In this work, we reinterpret them through the lens of missing data. First, we show that their standard training procedure on fully observed data implicitly performs imputation under a missing completely at random mechanism, resulting in robust out-of-sample imputation performance in settings with high missingness. Second, we introduce the first principled framework for training them directly on incomplete datasets under general missingness mechanisms. Third, we leverage their amortized conditional density estimation to perform active information acquisition, i.e., sequentially selecting the most informative missing variables for downstream prediction or inference. Across a suite of real-world benchmarks, our Missingness-Aware Order-Agnostic Autoregressive Model (MO-ARM) consistently outperforms established imputation baselines.

Problem

• missing data
• order-agnostic autoregressive models
• imputation
• missingness mechanisms
• incomplete datasets

Innovation

• order-agnostic autoregressive modeling
• missing data imputation
• amortized conditional density estimation
• active information acquisition
• missingness-aware learning