Compositional Risk Minimization

2024-10-08

arXiv.org

AI Summary

This paper addresses the challenge of generalization under compositional shift—where test samples exhibit attribute combinations entirely absent during training. To tackle this, we propose Compositional Risk Minimization (CRM), a framework that models the joint distribution of multiple attributes via an additive energy function. CRM unifies multi-attribute prediction and classifier calibration, and theoretically guarantees extrapolation to the affine hull spanned by seen attribute combinations—enabling robust generalization to novel compositions. Unlike existing subgroup shift methods, CRM achieves significant improvements in compositional generalization across multiple benchmark datasets. Crucially, it is the first approach to systematically resolve discriminative generalization under compositional distribution shift, providing both theoretical foundations and empirical validation.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Compositional generalization is a crucial step towards developing data-efficient intelligent machines that generalize in human-like ways. In this work, we tackle a challenging form of distribution shift, termed compositional shift, where some attribute combinations are completely absent at training but present in the test distribution. This shift tests the model's ability to generalize compositionally to novel attribute combinations in discriminative tasks. We model the data with flexible additive energy distributions, where each energy term represents an attribute, and derive a simple alternative to empirical risk minimization termed compositional risk minimization (CRM). We first train an additive energy classifier to predict the multiple attributes and then adjust this classifier to tackle compositional shifts. We provide an extensive theoretical analysis of CRM, where we show that our proposal extrapolates to special affine hulls of seen attribute combinations. Empirical evaluations on benchmark datasets confirms the improved robustness of CRM compared to other methods from the literature designed to tackle various forms of subpopulation shifts.

Problem

• Addresses distribution shift in machine learning
• Focuses on compositional generalization in AI models
• Introduces compositional risk minimization for robustness

Innovation

• Compositional Risk Minimization (CRM)
• Flexible additive energy distributions
• Extrapolates to affine hulls