Unsupervised Invariant Risk Minimization

2025-05-18

AI Summary

This paper addresses the challenge of applying Invariant Risk Minimization (IRM) in fully unsupervised settings, where no labeled data is available. To this end, we propose the first unsupervised IRM framework. Our method comprises two key components: (1) an unsupervised structural causal model that formally characterizes invariance under environmental shifts; and (2) a dual-path algorithm—PICA (Principal Invariant Component Analysis) and VIAE (Variational Interventional Autoencoder)—that achieves identifiable disentanglement of environment-invariant (causal) factors from environment-dependent (spurious) factors. PICA leverages PCA and invariant direction estimation, while VIAE employs a variational autoencoder with generative intervention modeling. Experiments on synthetic benchmarks and environment-shifted MNIST demonstrate that the learned representations capture true causal structure without supervision, significantly improving zero-shot cross-environment generalization and preserving semantic interpretability.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

We propose a novel unsupervised framework for emph{Invariant Risk Minimization} (IRM), extending the concept of invariance to settings where labels are unavailable. Traditional IRM methods rely on labeled data to learn representations that are robust to distributional shifts across environments. In contrast, our approach redefines invariance through feature distribution alignment, enabling robust representation learning from unlabeled data. We introduce two methods within this framework: Principal Invariant Component Analysis (PICA), a linear method that extracts invariant directions under Gaussian assumptions, and Variational Invariant Autoencoder (VIAE), a deep generative model that disentangles environment-invariant and environment-dependent latent factors. Our approach is based on a novel ``unsupervised'' structural causal model and supports environment-conditioned sample-generation and intervention. Empirical evaluations on synthetic dataset and modified versions of MNIST demonstrate the effectiveness of our methods in capturing invariant structure, preserving relevant information, and generalizing across environments without access to labels.

Problem

• Extends invariance learning to unlabeled data settings
• Proposes methods for robust representation without labels
• Demonstrates generalization across environments without labels

Innovation

• Unsupervised framework for Invariant Risk Minimization
• Feature distribution alignment for robust representation
• Principal Invariant Component Analysis and Variational Autoencoder