Invariant Subspace Decomposition

2024-04-15

arXiv.org

AI Summary

This paper addresses response prediction under time-varying covariates. We propose the Invariant Subspace Decomposition (ISD) framework, which decomposes the linear conditional distribution $Y mid X$ into a time-invariant subspace and a residual time-varying component. ISD is the first method to achieve both statistical identifiability and theoretical tractability for this decomposition, enabling zero-shot forecasting and few-shot temporal adaptation. The approach leverages unsupervised approximate joint matrix diagonalization, complemented by finite-sample statistical analysis and time-varying causal modeling. We establish consistency of the estimator under mild regularity conditions. Empirical results demonstrate that ISD significantly improves cross-temporal generalization and cold-start prediction accuracy compared to baseline methods that ignore invariant structure. (124 words)

Technology Category

• Machine Learning: Time-Series/Data Streams
• Intelligent Robots: State Estimation
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We consider the task of predicting a response Y from a set of covariates X in settings where the conditional distribution of Y given X changes over time. For this to be feasible, assumptions on how the conditional distribution changes over time are required. Existing approaches assume, for example, that changes occur smoothly over time so that short-term prediction using only the recent past becomes feasible. To additionally exploit observations further in the past, we propose a novel invariance-based framework for linear conditionals, called Invariant Subspace Decomposition (ISD), that splits the conditional distribution into a time-invariant and a residual time-dependent component. As we show, this decomposition can be utilized both for zero-shot and time-adaptation prediction tasks, that is, settings where either no or a small amount of training data is available at the time points we want to predict Y at, respectively. We propose a practical estimation procedure, which automatically infers the decomposition using tools from approximate joint matrix diagonalization. Furthermore, we provide finite sample guarantees for the proposed estimator and demonstrate empirically that it indeed improves on approaches that do not use the additional invariant structure.

Problem

• Predict Y from X with changing conditional distribution
• Decompose conditional distribution into time-invariant and dependent components
• Enable zero-shot and time-adaptation prediction tasks

Innovation

• Invariant Subspace Decomposition method
• Time-invariant and residual components
• Approximate joint matrix diagonalization