One-Step Generalization Ratio Guided Optimization for Domain Generalization

2026-06-15

AI Summary

This work addresses the challenge in domain generalization where models tend to overfit to domain-specific features and struggle to generalize to unseen target domains. To mitigate this, the authors propose GENIE, a novel optimizer that introduces the One-Step Generalization Ratio (OSGR) to dynamically assess each parameter’s contribution to loss reduction and the alignment of its gradient with respect to generalization. Leveraging OSGR, GENIE employs a preconditioning mechanism that balances parameter updates, preventing a few dominant parameters from steering optimization and thereby promoting the learning of domain-invariant features. Notably, GENIE is the first to utilize OSGR for guiding optimization, achieving enhanced generalization while preserving the convergence rate of standard SGD. It seamlessly integrates into various domain generalization frameworks, and extensive experiments demonstrate its consistent superiority over existing optimizers and its effectiveness in boosting the performance of diverse DG methods across multiple benchmarks.

Abstract

Domain Generalization (DG) aims to train models that generalize to unseen target domains but often overfit to domain-specific features, known as undesired correlations. Gradient-based DG methods typically guide gradients in a dominant direction but often inadvertently reinforce spurious correlations. Recent work has employed dropout to regularize overconfident parameters, but has not explicitly adjusted gradient alignment or ensured balanced parameter updates. We propose GENIE (Generalization-ENhancing Iterative Equalizer), a novel optimizer that leverages the One-Step Generalization Ratio (OSGR) to quantify each parameter's contribution to loss reduction and assess gradient alignment. By dynamically equalizing OSGR via a preconditioning factor, GENIE prevents a small subset of parameters from dominating optimization, thereby promoting domain-invariant feature learning. Theoretically, GENIE balances convergence contribution and gradient alignment among parameters, achieving higher OSGR while retaining SGD's convergence rate. Empirically, it outperforms existing optimizers and enhances performance when integrated with various DG and single-DG methods.

Problem

• Domain Generalization
• spurious correlations
• gradient alignment
• parameter update balance
• generalization

Innovation

• Domain Generalization
• One-Step Generalization Ratio
• Gradient Alignment
• Parameter Equalization
• Optimization