From Objectives to What Models Learn: A Landau Theory of Invariant Learning

2026-08-10

AI Summary

This work addresses the behavioral gap between objective functions and the representations actually learned in invariant learning by introducing, for the first time, the Landau free energy framework from statistical physics, drawing an analogy between representation learning and multimodal magnetization processes. By deriving a Landau-type effective free energy corresponding to invariant learning objectives and constructing a low-order coefficient–based “signature” of the objective function, the study quantitatively characterizes the evolution of representations along regularization paths. Combining theoretical analysis—supported by closed-form solutions in bilinear models—with experiments on ReLU networks and spectral methods, the paper uncovers connections among phase transition boundaries, steady-state load distributions, and regularization phenotypes. It further demonstrates the signature’s cross-layer predictive power in deep networks and introduces novel concepts such as the “selective retention window.”

Abstract

Invariant learning seeks representations that remain predictive across environments, yet the behavior of its objectives along the regularization path is often opaque. We address this objective-behavior gap by viewing representation learning as multimode magnetization and deriving, from concrete invariant-learning objectives, a Landau-type effective free energy whose low-order coefficients form objective signatures and induce distinct regularization phenotypes. Effective quadratic corrections move the phase boundary and enable finite-strength mode elimination; quartic corrections regulate post-onset amplitude and typically leave residual loading at finite strength; higher-order structure governs non-monotone tails, instability, and collapse at large regularization. In a canonical bilinear model, the theory yields closed-form phase boundaries and steady-state loadings, as well as distinct critical strengths for shortcut and stable modes that define a selective-retention window. Controlled experiments confirm the predicted phase boundaries, loadings, and regularization phenotypes. In one- and two-hidden-layer ReLU networks, the same signatures remain predictive of qualitative regularization-path behavior despite depth-dependent shifts in scale. A matrix extension generalizes the framework to coupled collective modes and yields a spectral phase-boundary criterion. Together, the framework turns low-order objective signatures into predictions of regularization phenotypes and, ultimately, of what models learn as regularization varies.

Problem

• invariant learning
• regularization path
• objective-behavior gap
• representation learning
• phase boundary

Innovation

• Invariant Learning
• Landau Theory
• Regularization Phenotypes
• Phase Boundary
• Effective Free Energy