The SIGReg Objective as Variational Free Energy: A Theoretical Active-Inference Account of JEPA World Models

2026-07-15

AI Summary

This work addresses the lack of theoretical grounding in regularizer selection for current Joint Embedding Predictive Architecture (JEPA) world models, particularly regarding their alignment with the variational free energy principle from active inference. By constructing a hierarchical entropy estimator and analyzing how non-contrastive regularizers influence latent space entropy, the study establishes a formal connection between the JEPA objective and variational free energy. Key contributions include proving that SIGReg eliminates the prior misalignment gap, thereby rendering the JEPA objective an exact information bottleneck; identifying the absence of an epistemic value term in existing JEPA formulations; demonstrating SIGReg’s safety in bounding surprise; and deriving a correspondence between multi-step expected free energy and ensemble epistemic value. These theoretical findings are supported by information bottleneck analysis, active inference principles, and formal verification in Lean 4, yielding testable predictions for future empirical work.

Abstract

Joint-Embedding Predictive Architectures (JEPAs) are the dominant design for latent world models, yet they are usually justified by empirical performance rather than a normative principle. We show that the choice of anti-collapse regulariser determines whether a JEPA's training objective, a prediction loss plus a weighted embedding regulariser, is a valid Active Inference (AIF) variational free energy. We organise four non-contrastive regularisers (VICReg, LogDet, PairDist, and SIGReg) into an entropy-estimator hierarchy indexed by a prior-miscalibration gap, and show that the gap's sign, whether the estimator bounds the latent entropy from above or below, decides whether the AIF surprise bound survives: VICReg and LogDet are unsafe upper bounds, PairDist a safe lower bound, and SIGReg eliminates the gap. We then prove a correspondence theorem: under the standard constant-noise encoder model and successful SIGReg enforcement (isotropic-Gaussian embeddings), the gap vanishes, the objective becomes an exact information bottleneck, the surprise bound is preserved, and the latent goal cost becomes an exact proxy for AIF pragmatic value, whereas VICReg leaves an irreducible second-order anisotropy term. We extend the correspondence to multi-step expected free energy, ensemble epistemic value, and a learned-policy regime, and we identify the one AIF term no current JEPA world model computes: the state-epistemic value, a future-state coverage signal. The predictions differ in kind, not degree, and are stated here as theoretical consequences left for empirical test in separate work; full proofs are in Appendix A, and the algebraic core of every result is machine-verified in Lean 4 (Appendix D).

Problem

• JEPA
• Active Inference
• variational free energy
• anti-collapse regulariser
• world models

Innovation

• Active Inference
• JEPA
• SIGReg
• variational free energy
• information bottleneck